You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/10/29 20:58:47 UTC
[GitHub] [beam] pabloem opened a new pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
pabloem opened a new pull request #15848:
URL: https://github.com/apache/beam/pull/15848
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r798110611
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -1071,74 +1112,136 @@ public void populateDisplayData(DisplayData.Builder builder) {
* strides for generated WHERE clause expressions used to split the column withPartitionColumn
* evenly. When the input is less than 1, the number is set to 1.
*/
- public ReadWithPartitions<T> withNumPartitions(int numPartitions) {
+ public ReadWithPartitions<T, PartitionColumnT> withNumPartitions(int numPartitions) {
checkArgument(numPartitions > 0, "numPartitions can not be less than 1");
return toBuilder().setNumPartitions(numPartitions).build();
}
/** The name of a column of numeric type that will be used for partitioning. */
- public ReadWithPartitions<T> withPartitionColumn(String partitionColumn) {
+ public ReadWithPartitions<T, PartitionColumnT> withPartitionColumn(String partitionColumn) {
checkNotNull(partitionColumn, "partitionColumn can not be null");
return toBuilder().setPartitionColumn(partitionColumn).build();
}
- public ReadWithPartitions<T> withLowerBound(Long lowerBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withLowerBound(PartitionColumnT lowerBound) {
return toBuilder().setLowerBound(lowerBound).build();
}
- public ReadWithPartitions<T> withUpperBound(Long upperBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withUpperBound(PartitionColumnT upperBound) {
return toBuilder().setUpperBound(upperBound).build();
}
/** Name of the table in the external database. Can be used to pass a user-defined subqery. */
- public ReadWithPartitions<T> withTable(String tableName) {
+ public ReadWithPartitions<T, PartitionColumnT> withTable(String tableName) {
checkNotNull(tableName, "table can not be null");
return toBuilder().setTable(tableName).build();
}
+ private static final int EQUAL = 0;
+
@Override
public PCollection<T> expand(PBegin input) {
checkNotNull(getRowMapper(), "withRowMapper() is required");
checkNotNull(
getDataSourceProviderFn(),
"withDataSourceConfiguration() or withDataSourceProviderFn() is required");
checkNotNull(getPartitionColumn(), "withPartitionColumn() is required");
- checkArgument(
- getUpperBound() != null && getLowerBound() != null,
- "Upper and lower bounds are mandatory parameters for JdbcIO.readWithPartitions");
checkNotNull(getTable(), "withTable() is required");
+ if (getLowerBound() != null && getLowerBound() instanceof Comparable<?>) {
+ // Not all partition types are comparable. For example, LocalDateTime, which is a valid
+ // partitioning type, is not Comparable, so we can't enforce this for all sorts of
+ // partitioning.
+ checkArgument(
+ ((Comparable<PartitionColumnT>) getLowerBound()).compareTo(getUpperBound()) < EQUAL,
+ "The lower bound of partitioning column is larger or equal than the upper bound");
+ }
checkArgument(
- getLowerBound() < getUpperBound(),
- "The lower bound of partitioning column is larger or equal than the upper bound");
- checkArgument(
- getUpperBound() - getLowerBound() >= getNumPartitions(),
- "The specified number of partitions is more than the difference between upper bound and lower bound");
+ JdbcUtil.PRESET_HELPERS.containsKey(getPartitionColumnType().getRawType()),
+ "readWithPartitions only supports the following types: %s",
+ JdbcUtil.PRESET_HELPERS.keySet());
+
+ PCollection<KV<Integer, KV<PartitionColumnT, PartitionColumnT>>> params;
+
+ if (getLowerBound() == null && getUpperBound() == null) {
+ String query =
+ String.format(
+ "SELECT min(%s), max(%s) FROM %s",
+ getPartitionColumn(), getPartitionColumn(), getTable());
+ if (getNumPartitions() == null) {
+ query =
+ String.format(
+ "SELECT min(%s), max(%s), count(*) FROM %s",
+ getPartitionColumn(), getPartitionColumn(), getTable());
+ }
+ params =
+ input
+ .apply(
+ JdbcIO.<KV<Integer, KV<PartitionColumnT, PartitionColumnT>>>read()
+ .withQuery(query)
+ .withDataSourceProviderFn(getDataSourceProviderFn())
+ .withRowMapper(
+ (JdbcReadWithPartitionsHelper<PartitionColumnT>)
+ JdbcUtil.PRESET_HELPERS.get(getPartitionColumnType().getRawType())))
+ .apply(
+ MapElements.via(
+ new SimpleFunction<
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>>,
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>>>() {
+ @Override
+ public KV<Integer, KV<PartitionColumnT, PartitionColumnT>> apply(
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>> input) {
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>> result;
+ if (getNumPartitions() == null) {
+ // In this case, we use the table row count to infer a number of
+ // partitions.
+ // We take the square root of the number of rows, and divide it by 5
+ // to keep a relatively low number of partitions, given that an RDBMS
+ // cannot usually accept a very large number of connections.
+ Integer numPartitions =
Review comment:
hmmm we could increase the factor by which we divide, to keep a smaller number of partitions in general. WDYT? It all may also depend on the downstream transform, after all...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8412e56) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51984 +42795
- Misses 9612 10162 +550
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/coderx/varint.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L3ZhcmludC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/window.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy93aW5kb3cuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/max.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21heC5nbw==) | | |
| [sdks/go/pkg/beam/transforms/filter/filter.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL2ZpbHRlci9maWx0ZXIuc2hpbXMuZ28=) | | |
| [sdks/go/pkg/beam/util/gcsx/gcs.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL2djc3gvZ2NzLmdv) | | |
| [sdks/go/pkg/beam/core/typex/fulltype.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3R5cGV4L2Z1bGx0eXBlLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/input.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9pbnB1dC5nbw==) | | |
| [sdks/go/pkg/beam/core/util/hooks/hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvaG9va3MvaG9va3MuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/hash.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9oYXNoLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/windows.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3dpbmRvd3MuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...8412e56](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a change in pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r802489521
##########
File path: sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOAutoPartitioningIT.java
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.jdbc;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.Objects;
+import java.util.Random;
+import javax.sql.DataSource;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.io.GenerateSequence;
+import org.apache.beam.sdk.io.common.DatabaseTestHelper;
+import org.apache.beam.sdk.io.jdbc.JdbcIO.RowMapper;
+import org.apache.beam.sdk.metrics.Distribution;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.options.PipelineOptions.CheckEnabled;
+import org.apache.beam.sdk.schemas.JavaFieldSchema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.annotations.SchemaCreate;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.joda.time.DateTime;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TestRule;
+import org.junit.runner.Description;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.model.Statement;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.JdbcDatabaseContainer;
+import org.testcontainers.containers.MySQLContainer;
+import org.testcontainers.containers.PostgreSQLContainer;
+
+/** A test of {@link org.apache.beam.sdk.io.jdbc.JdbcIO} on test containers. */
+@RunWith(Parameterized.class)
+public class JdbcIOAutoPartitioningIT {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcIOAutoPartitioningIT.class);
+
+ public static final Integer NUM_ROWS = 1_000;
+ public static final String TABLE_NAME = "baseTable";
+
+ @ClassRule public static TestPipeline pipelineWrite = TestPipeline.create();
+ @Rule public TestPipeline pipelineRead = TestPipeline.create();
+
+ @Parameterized.Parameters(name = "{0}")
+ public static Iterable<String> params() {
+ return Lists.newArrayList("mysql", "postgres");
+ }
+
+ @Parameterized.Parameter(0)
+ public String dbms;
+
+ public static JdbcDatabaseContainer<?> getDb(String dbName) {
+ if (dbName.equals("mysql")) {
+ return mysql;
+ } else {
+ return postgres;
+ }
+ }
+
+ // We need to implement this retrying rule because we're running ~18 pipelines that connect to
+ // two databases in a few seconds. This may cause the databases to be overwhelmed, and reject
+ // connections or error out. By using this rule, we ensure that each pipeline is trued twice so
+ // that flakiness from databases being overwhelmed can be managed.
+ @Rule
+ public TestRule retryRule =
+ new TestRule() {
+ // We establish max number of retries at 2.
+ public final int maxRetries = 2;
+
+ @Override
+ public Statement apply(Statement base, Description description) {
+ return new Statement() {
+ @Override
+ public void evaluate() throws Throwable {
+ Throwable caughtThrowable = null;
+ // implement retry logic here
+ for (int i = 0; i < maxRetries; i++) {
+ try {
+ pipelineRead.apply(base, description);
+ base.evaluate();
+ return;
+ } catch (Throwable t) {
+ caughtThrowable = t;
+ System.err.println(
+ description.getDisplayName() + ": run " + (i + 1) + " failed.");
+ }
+ }
+ System.err.println(
+ description.getDisplayName() + ": Giving up after " + maxRetries + " failures.");
+ throw Objects.requireNonNull(caughtThrowable);
+ }
+ };
+ }
+ };
+
+ @ClassRule public static JdbcDatabaseContainer<?> mysql = new MySQLContainer<>("mysql");
+
+ @ClassRule
+ public static JdbcDatabaseContainer<?> postgres = new PostgreSQLContainer<>("postgres");
+
+ @Before
+ public void prepareDatabase() throws SQLException {
+ pipelineRead.getOptions().setStableUniqueNames(CheckEnabled.OFF);
+ DataSource dbDs = DatabaseTestHelper.getDataSourceForContainer(getDb(dbms));
+ try {
+ DatabaseTestHelper.createTable(
+ dbDs,
+ TABLE_NAME,
+ Lists.newArrayList(
+ KV.of("id", "INTEGER"),
+ KV.of("name", "VARCHAR(50)"),
+ KV.of("specialDate", "TIMESTAMP")));
+ } catch (SQLException e) {
+ LOG.warn("Exception occurred when preparing database {}", dbms, e);
Review comment:
Logging a warning here might be confusion, maybe just comment that this is expected as the db already exists when running this before the 2nd test...? Though, nothing critical ....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a change in pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r801332204
##########
File path: sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java
##########
@@ -41,4 +53,126 @@ public void testGetPreparedStatementSetCaller() throws Exception {
String expectedStmt = "INSERT INTO test_table(col1, col2, col3) VALUES(?, ?, ?)";
assertEquals(expectedStmt, generatedStmt);
}
+
+ @Test
+ public void testStringPartitioningWithSingleKeyFn() {
+ JdbcReadWithPartitionsHelper<String> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(TypeDescriptors.strings());
+ List<KV<String, String>> expectedRanges =
Review comment:
Sounds like the best way forward 👍 Such a range class would also make the interfaces a lot more comprehensive (to eventually expose these to users)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (014888e) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.19%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19%
===========================================
Files 201 452 +251
Lines 19787 62175 +42388
===========================================
+ Hits 9189 52002 +42813
- Misses 9612 10173 +561
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/sideinput.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zaWRlaW5wdXQuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/mean.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21lYW4uZ28=) | | |
| [sdks/go/pkg/beam/io/textio/textio.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby90ZXh0aW8vdGV4dGlvLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/harness/harness.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9oYXJuZXNzLmdv) | | |
| [...ks/go/pkg/beam/runners/dataflow/dataflowlib/job.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL2pvYi5nbw==) | | |
| [sdks/go/pkg/beam/transforms/stats/count.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL2NvdW50Lmdv) | | |
| [sdks/go/pkg/beam/core/runtime/harness/logging.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9sb2dnaW5nLmdv) | | |
| [sdks/go/pkg/beam/io/databaseio/mapper.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhYmFzZWlvL21hcHBlci5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/datasink.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kYXRhc2luay5nbw==) | | |
| [.../go/pkg/beam/runners/dataflow/dataflowlib/fixup.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL2ZpeHVwLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...014888e](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (014888e) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.19%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 014888e differs from pull request most recent head 47aa02a. Consider uploading reports for the commit 47aa02a to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19%
===========================================
Files 201 452 +251
Lines 19787 62175 +42388
===========================================
+ Hits 9189 52002 +42813
- Misses 9612 10173 +561
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/graph/coder/time.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3RpbWUuZ28=) | | |
| [sdks/go/pkg/beam/core/util/reflectx/call.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvY2FsbC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/flatten.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbGF0dGVuLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/data.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kYXRhLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/payload.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L3BheWxvYWQuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/genx/genx.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ2VueC9nZW54Lmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/bytes.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2J5dGVzLmdv) | | |
| [sdks/go/pkg/beam/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsLmdv) | | |
| [...pkg/beam/runners/dataflow/dataflowlib/translate.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL3RyYW5zbGF0ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/coderx/varint.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L3ZhcmludC5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...47aa02a](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1034138825
thanks @mosche ! your code review made this PR significantly better : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #15848:
URL: https://github.com/apache/beam/pull/15848
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r801982984
##########
File path: sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java
##########
@@ -28,6 +44,66 @@
@RunWith(JUnit4.class)
public class JdbcUtilTest {
+ static final JdbcReadWithPartitionsHelper<String> PROTOTYPE_STRING_PARTITIONER =
Review comment:
I've added a comment referencing the issue. LMK if it looks good. Thanks so much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f155b81) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.61% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62252 +42465
===========================================
+ Hits 9189 52050 +42861
- Misses 9612 10202 +590
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/decode.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kZWNvZGUuZ28=) | | |
| [sdks/go/pkg/beam/runners/direct/impulse.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9pbXB1bHNlLmdv) | | |
| [sdks/go/pkg/beam/combine.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb21iaW5lLmdv) | | |
| [sdks/go/pkg/beam/artifact/materialize.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9hcnRpZmFjdC9tYXRlcmlhbGl6ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/serialize.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NlcmlhbGl6ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/coder/row\_encoder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3Jvd19lbmNvZGVyLmdv) | | |
| [sdks/go/pkg/beam/io/datastoreio/datastore.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhc3RvcmVpby9kYXRhc3RvcmUuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/coderx/int.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L2ludC5nbw==) | | |
| [sdks/go/pkg/beam/transforms/stats/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL3V0aWwuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...f155b81](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8412e56) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51984 +42795
- Misses 9612 10162 +550
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/metricsx/urns.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvdXJucy5nbw==) | | |
| [sdks/go/pkg/beam/transforms/stats/quantiles.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL3F1YW50aWxlcy5nbw==) | | |
| [...s/go/pkg/beam/core/runtime/graphx/schema/schema.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NjaGVtYS9zY2hlbWEuZ28=) | | |
| [sdks/go/pkg/beam/runners/dataflow/dataflow.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93Lmdv) | | |
| [sdks/go/pkg/beam/testing/passert/equals.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvZXF1YWxzLmdv) | | |
| [sdks/go/pkg/beam/schema.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9zY2hlbWEuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/flatten.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbGF0dGVuLmdv) | | |
| [sdks/go/pkg/beam/artifact/materialize.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9hcnRpZmFjdC9tYXRlcmlhbGl6ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/namespace.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L25hbWVzcGFjZS5nbw==) | | |
| [sdks/go/pkg/beam/transforms/stats/max.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21heC5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...8412e56](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (19b9a69) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51979 +42790
- Misses 9612 10167 +555
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/discard.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kaXNjYXJkLmdv) | | |
| [sdks/go/pkg/beam/core/util/protox/any.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcHJvdG94L2FueS5nbw==) | | |
| [...ks/go/pkg/beam/core/runtime/coderx/coderx.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L2NvZGVyeC5zaGltcy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/multiplex.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9tdWx0aXBsZXguZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/decode.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kZWNvZGUuZ28=) | | |
| [sdks/go/pkg/beam/io/databaseio/writer.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhYmFzZWlvL3dyaXRlci5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/types.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvdHlwZXMuZ28=) | | |
| [sdks/go/pkg/beam/testing/passert/hash.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvaGFzaC5nbw==) | | |
| [sdks/go/pkg/beam/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsLmdv) | | |
| [sdks/go/pkg/beam/core/util/hooks/hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvaG9va3MvaG9va3MuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...19b9a69](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e87c4bb) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62161 +42374
===========================================
+ Hits 9189 51998 +42809
- Misses 9612 10163 +551
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/util/shimx/generate.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL3NoaW14L2dlbmVyYXRlLmdv) | | |
| [...pkg/beam/runners/dataflow/dataflowlib/translate.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL3RyYW5zbGF0ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/fn\_arity.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbl9hcml0eS5nbw==) | | |
| [sdks/go/pkg/beam/io/databaseio/mapper.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhYmFzZWlvL21hcHBlci5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy91dGlsLmdv) | | |
| [sdks/go/pkg/beam/transforms/stats/count.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL2NvdW50Lmdv) | | |
| [sdks/go/pkg/beam/core/metrics/store.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL21ldHJpY3Mvc3RvcmUuZ28=) | | |
| [sdks/go/pkg/beam/util/starcgenx/starcgenx.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL3N0YXJjZ2VueC9zdGFyY2dlbnguZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/coderx/int.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L2ludC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/clone.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L2Nsb25lLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...e87c4bb](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r798090588
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -325,17 +325,24 @@ public static ReadRows readRows() {
*
* @param <T> Type of the data to be read.
*/
- public static <T> ReadWithPartitions<T> readWithPartitions() {
- return new AutoValue_JdbcIO_ReadWithPartitions.Builder<T>()
+ public static <T, PartitionColumnT> ReadWithPartitions<T, PartitionColumnT> readWithPartitions(
Review comment:
you're right. Indeed, I wanted to avoid breaking as much as possible - so this change does not break the existing integration tests (see JdbcIOIT), while adding support for any type partitioning - and the transform is fairly new, so I think we may be able to pull this off : ) - WDYT?
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -999,59 +1015,84 @@ public void populateDisplayData(DisplayData.Builder builder) {
/** Implementation of {@link #readWithPartitions}. */
@AutoValue
- public abstract static class ReadWithPartitions<T> extends PTransform<PBegin, PCollection<T>> {
+ public abstract static class ReadWithPartitions<T, PartitionColumnT>
+ extends PTransform<PBegin, PCollection<T>> {
abstract @Nullable SerializableFunction<Void, DataSource> getDataSourceProviderFn();
abstract @Nullable RowMapper<T> getRowMapper();
abstract @Nullable Coder<T> getCoder();
- abstract Integer getNumPartitions();
+ abstract @Nullable Integer getNumPartitions();
abstract @Nullable String getPartitionColumn();
- abstract @Nullable Long getLowerBound();
+ abstract @javax.annotation.Nullable @Nullable PartitionColumnT getLowerBound();
- abstract @Nullable Long getUpperBound();
+ abstract @javax.annotation.Nullable @Nullable PartitionColumnT getUpperBound();
abstract @Nullable String getTable();
- abstract Builder<T> toBuilder();
+ abstract TypeDescriptor<PartitionColumnT> getPartitionColumnType();
+
+ abstract Builder<T, PartitionColumnT> toBuilder();
+
+ /**
+ * A helper for {@link ReadWithPartitions} that handles range calculations.
+ *
+ * @param <PartitionT>
+ */
+ public interface JdbcReadWithPartitionsHelper<PartitionT>
Review comment:
I think this makes sense - we can leave this internal for now, and in the future consider any changes to the types that we use, and externalize for users if we want to. Done,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1030425091
> > The issue is that the query with the full range would be executing in the database, so we may duplicate work on the database - so I'm not sure how bad this is or not : )
>
> I was thinking about this as well. When just traversing the index tree that's probably fine if it doesn't happen too aggressively. But if it involves sorting and throwing a significant part of that work away all the time it will certainly impact the database badly.
If I have time, I'll try to write a SDF implementation for this next week, and make sure that the documentation is clear about what it does, and considerations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r800937170
##########
File path: sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java
##########
@@ -41,4 +53,126 @@ public void testGetPreparedStatementSetCaller() throws Exception {
String expectedStmt = "INSERT INTO test_table(col1, col2, col3) VALUES(?, ?, ?)";
assertEquals(expectedStmt, generatedStmt);
}
+
+ @Test
+ public void testStringPartitioningWithSingleKeyFn() {
+ JdbcReadWithPartitionsHelper<String> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(TypeDescriptors.strings());
+ List<KV<String, String>> expectedRanges =
Review comment:
oh great catch - I'd like to support strings, but I am fine to leave for a follow up change (maybe use a specialized range object rather than a KV, huh?)
I'll remove the string support for now, and follow up in a different PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r800950689
##########
File path: sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java
##########
@@ -41,4 +53,126 @@ public void testGetPreparedStatementSetCaller() throws Exception {
String expectedStmt = "INSERT INTO test_table(col1, col2, col3) VALUES(?, ?, ?)";
assertEquals(expectedStmt, generatedStmt);
}
+
+ @Test
+ public void testStringPartitioningWithSingleKeyFn() {
+ JdbcReadWithPartitionsHelper<String> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(TypeDescriptors.strings());
+ List<KV<String, String>> expectedRanges =
+ Lists.<KV<String, String>>newArrayList(KV.of("a", "b"));
+ List<KV<String, String>> ranges = Lists.newArrayList(helper.calculateRanges("a", "a", 10L));
+ // It is not possible to generate any more than one range, because the lower and upper range are
+ // exactly the same.
+ // The range is "a" to the very next element after it, which would be "a"+1 -> "b".
+ // Because the query's filter statement is : WHERE column >= lowerBound AND column < upperBound.
+ assertEquals(1, ranges.size());
+ assertArrayEquals(expectedRanges.toArray(), ranges.toArray());
+ }
+
+ @Test
+ public void testStringPartitioningWithSingleKeyMultiletterFn() {
+ JdbcReadWithPartitionsHelper<String> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(TypeDescriptors.strings());
+ List<KV<String, String>> expectedRanges =
+ Lists.<KV<String, String>>newArrayList(KV.of("afar", "afas"));
+ List<KV<String, String>> ranges =
+ Lists.newArrayList(helper.calculateRanges("afar", "afar", 10L));
+ // It is not possible to generate any more than one range, because the lower and upper range are
+ // exactly the same.
+ // The range is "afar" to the very next element after it, which would be "afar"+1 -> "afas".
+ // Because the query's filter statement is : WHERE column >= lowerBound AND column < upperBound.
+ assertEquals(1, ranges.size());
+ assertArrayEquals(expectedRanges.toArray(), ranges.toArray());
+ }
+
+ @Test
+ public void testStringPartitioningWithMultiletter() {
+ JdbcReadWithPartitionsHelper<String> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(TypeDescriptors.strings());
+ List<KV<String, String>> ranges =
+ Lists.newArrayList(helper.calculateRanges("afarisade", "zfastaridoteaf", 10L));
+ // The upper bound is "zfastaridoteaf" to the very next element after it, which would
+ // be "zfastaridoteaf"+1 -> "zfastaridoteaf".
+ // Because the query's filter statement is : WHERE column >= lowerBound AND column < upperBound.
+ assertEquals(13L, ranges.size());
+ assertThat(
+ ranges,
+ containsInAnyOrder(
+ KV.of("afarisade", "c"),
+ KV.of("c", "e"),
+ KV.of("e", "g"),
+ KV.of("g", "i"),
+ KV.of("i", "k"),
+ KV.of("k", "m"),
+ KV.of("m", "o"),
+ KV.of("o", "q"),
+ KV.of("q", "s"),
+ KV.of("s", "u"),
+ KV.of("u", "w"),
+ KV.of("w", "y"),
+ KV.of("y", "zfastaridoteag")));
+ }
+
+ @Test
+ public void testDatetimePartitioningWithSingleKey() {
+ JdbcReadWithPartitionsHelper<DateTime> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(
+ TypeDescriptor.of(DateTime.class));
+ DateTime onlyPoint = DateTime.now();
+ List<KV<DateTime, DateTime>> expectedRanges =
+ Lists.newArrayList(KV.of(onlyPoint, onlyPoint.plusMillis(1)));
+ List<KV<DateTime, DateTime>> ranges =
+ Lists.newArrayList(helper.calculateRanges(onlyPoint, onlyPoint, 10L));
+ // It is not possible to generate any more than one range, because the lower and upper range are
+ // exactly the same.
+ // The range goes from the current DateTime to ONE MILISECOND AFTER.
+ // Because the query's filter statement is : WHERE column >= lowerBound AND column < upperBound.
+ assertEquals(1, ranges.size());
+ assertArrayEquals(expectedRanges.toArray(), ranges.toArray());
+ }
+
+ @Test
+ public void testDatetimePartitioningWithMultiKey() {
+ JdbcReadWithPartitionsHelper<DateTime> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(
+ TypeDescriptor.of(DateTime.class));
+ DateTime lastPoint = DateTime.now();
+ // At least 10ms in the past, or more.
+ DateTime firstPoint = lastPoint.minusMillis(10 + new Random().nextInt(Integer.MAX_VALUE));
+ List<KV<DateTime, DateTime>> ranges =
+ Lists.newArrayList(helper.calculateRanges(firstPoint, lastPoint, 10L));
+ // DateTime ranges are able to work out 10-11 ranges because they split in miliseconds which is
+ // very small granularity.
+ assertThat(Double.valueOf(ranges.size()), closeTo(10, 1));
+ }
+
+ @Test
+ public void testLongPartitioningWithSingleKey() {
+ JdbcReadWithPartitionsHelper<Long> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(TypeDescriptors.longs());
+ List<KV<Long, Long>> expectedRanges = Lists.newArrayList(KV.of(12L, 13L));
+ List<KV<Long, Long>> ranges = Lists.newArrayList(helper.calculateRanges(12L, 12L, 10L));
+ // It is not possible to generate any more than one range, because the lower and upper range are
+ // exactly the same.
+ // The range goes from the current Long element to ONE ELEMENT AFTER.
+ // Because the query's filter statement is : WHERE column >= lowerBound AND column < upperBound.
+ assertEquals(1, ranges.size());
+ assertArrayEquals(expectedRanges.toArray(), ranges.toArray());
+ }
+
+ @Test
+ public void testLongPartitioningNotEnoughRanges() {
+ JdbcReadWithPartitionsHelper<Long> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(TypeDescriptors.longs());
+ // The minimum stride is one, which is what causes this sort of partitioning.
+ List<KV<Long, Long>> expectedRanges =
+ Lists.newArrayList(KV.of(12L, 14L), KV.of(14L, 16L), KV.of(16L, 18L), KV.of(18L, 21L));
+ List<KV<Long, Long>> ranges = Lists.newArrayList(helper.calculateRanges(12L, 20L, 10L));
+ // It is not possible to generate any more than one range, because the lower and upper range are
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (800a566) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52008 +42819
- Misses 9612 10160 +548
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...kg/beam/core/runtime/graphx/schema/logicaltypes.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NjaGVtYS9sb2dpY2FsdHlwZXMuZ28=) | | |
| [sdks/go/pkg/beam/core/funcx/sideinput.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L3NpZGVpbnB1dC5nbw==) | | |
| [sdks/go/pkg/beam/transforms/stats/max.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21heC5nbw==) | | |
| [sdks/go/pkg/beam/core/funcx/fn.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L2ZuLmdv) | | |
| [sdks/go/pkg/beam/windowing.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS93aW5kb3dpbmcuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/coderx/varint.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L3ZhcmludC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/data.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kYXRhLmdv) | | |
| [sdks/go/pkg/beam/io/databaseio/database.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhYmFzZWlvL2RhdGFiYXNlLmdv) | | |
| [sdks/go/pkg/beam/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvbWV0cmljc3guZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...800a566](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (800a566) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52008 +42819
- Misses 9612 10160 +548
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/util/reflectx/calls.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvY2FsbHMuZ28=) | | |
| [sdks/go/pkg/beam/option.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9vcHRpb24uZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/cogbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9jb2diay5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/varint.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3ZhcmludC5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/bind.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2JpbmQuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/coderx/string.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L3N0cmluZy5nbw==) | | |
| [sdks/go/pkg/beam/core/funcx/signature.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L3NpZ25hdHVyZS5nbw==) | | |
| [.../pkg/beam/runners/dataflow/dataflowlib/messages.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL21lc3NhZ2VzLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/symbols.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvc3ltYm9scy5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/registry.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3JlZ2lzdHJ5Lmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...800a566](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (800a566) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52008 +42819
- Misses 9612 10160 +548
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/xlangx/namespace.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L25hbWVzcGFjZS5nbw==) | | |
| [sdks/go/pkg/beam/beam.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9iZWFtLnNoaW1zLmdv) | | |
| [sdks/go/pkg/beam/transforms/top/top.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3RvcC90b3AuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/types.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvdHlwZXMuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/data.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kYXRhLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/registry.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3JlZ2lzdHJ5Lmdv) | | |
| [sdks/go/pkg/beam/io/filesystem/memfs/memory.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL21lbWZzL21lbW9yeS5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/map.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL21hcC5nbw==) | | |
| [sdks/go/pkg/beam/gbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9nYmsuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/max.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21heC5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...800a566](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1032293644
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1032989016
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1033588847
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e87c4bb) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
> :exclamation: Current head e87c4bb differs from pull request most recent head 0512486. Consider uploading reports for the commit 0512486 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62161 +42374
===========================================
+ Hits 9189 51998 +42809
- Misses 9612 10163 +551
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/window.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy93aW5kb3cuZ28=) | | |
| [...ks/go/pkg/beam/core/runtime/harness/cache\_hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9jYWNoZV9ob29rcy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/sdf\_invokers.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zZGZfaW52b2tlcnMuZ28=) | | |
| [sdks/go/pkg/beam/io/filesystem/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/coder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L2NvZGVyLmdv) | | |
| [sdks/go/pkg/beam/testing/passert/equals.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvZXF1YWxzLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/varint.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3ZhcmludC5nbw==) | | |
| [...o/pkg/beam/io/rtrackers/offsetrange/offsetrange.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9ydHJhY2tlcnMvb2Zmc2V0cmFuZ2Uvb2Zmc2V0cmFuZ2UuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/coder/time.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3RpbWUuZ28=) | | |
| [sdks/go/pkg/beam/core/funcx/signature.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L3NpZ25hdHVyZS5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...0512486](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1030426864
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (df9a12c) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.19%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 51997 +42808
- Misses 9612 10171 +559
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/io/databaseio/mapper.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhYmFzZWlvL21hcHBlci5nbw==) | | |
| [sdks/go/pkg/beam/transforms/stats/max\_switch.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21heF9zd2l0Y2guZ28=) | | |
| [...kg/beam/core/runtime/graphx/schema/logicaltypes.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NjaGVtYS9sb2dpY2FsdHlwZXMuZ28=) | | |
| [...dks/go/pkg/beam/core/runtime/harness/monitoring.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9tb25pdG9yaW5nLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/window.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy93aW5kb3cuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/sdf\_invokers.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zZGZfaW52b2tlcnMuZ28=) | | |
| [...o/pkg/beam/runners/dataflow/dataflowlib/metrics.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL21ldHJpY3MuZ28=) | | |
| [sdks/go/pkg/beam/core/util/reflectx/functions.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvZnVuY3Rpb25zLmdv) | | |
| [sdks/go/pkg/beam/testing/passert/sum.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvc3VtLmdv) | | |
| [sdks/go/pkg/beam/flatten.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9mbGF0dGVuLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...df9a12c](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1031794085
I filed these two issues to track the work items:
- https://issues.apache.org/jira/browse/BEAM-13846
- https://issues.apache.org/jira/browse/BEAM-13847
I moved the string partitioner to a prototype. I'll work on that perhaps this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r801978211
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
##########
@@ -333,31 +343,154 @@ private static Calendar withTimestampAndTimezone(DateTime dateTime) {
return calendar;
}
+ /**
+ * A helper for {@link ReadWithPartitions} that handles range calculations.
+ *
+ * @param <PartitionT>
+ */
+ interface JdbcReadWithPartitionsHelper<PartitionT>
+ extends PreparedStatementSetter<KV<PartitionT, PartitionT>>,
+ RowMapper<KV<Long, KV<PartitionT, PartitionT>>> {
+ static <T> JdbcReadWithPartitionsHelper<T> getPartitionsHelper(TypeDescriptor<T> type) {
+ // This cast is unchecked, thus this is a small type-checking risk. We just need
+ // to make sure that all preset helpers in `JdbcUtil.PRESET_HELPERS` are matched
+ // in type from their Key and their Value.
+ return (JdbcReadWithPartitionsHelper<T>) PRESET_HELPERS.get(type.getRawType());
+ }
+
+ Iterable<KV<PartitionT, PartitionT>> calculateRanges(
+ PartitionT lowerBound, PartitionT upperBound, Long partitions);
+
+ @Override
+ void setParameters(KV<PartitionT, PartitionT> element, PreparedStatement preparedStatement);
+
+ @Override
+ KV<Long, KV<PartitionT, PartitionT>> mapRow(ResultSet resultSet) throws Exception;
+ }
+
/** Create partitions on a table. */
- static class PartitioningFn extends DoFn<KV<Integer, KV<Long, Long>>, KV<String, Long>> {
+ static class PartitioningFn<T> extends DoFn<KV<Long, KV<T, T>>, KV<T, T>> {
+ private static final Logger LOG = LoggerFactory.getLogger(PartitioningFn.class);
+ final TypeDescriptor<T> partitioningColumnType;
+
+ PartitioningFn(TypeDescriptor<T> partitioningColumnType) {
+ this.partitioningColumnType = partitioningColumnType;
+ }
+
@ProcessElement
public void processElement(ProcessContext c) {
- Integer numPartitions = c.element().getKey();
- Long lowerBound = c.element().getValue().getKey();
- Long upperBound = c.element().getValue().getValue();
- if (lowerBound > upperBound) {
- throw new RuntimeException(
- String.format(
- "Lower bound [%s] is higher than upper bound [%s]", lowerBound, upperBound));
- }
- long stride = (upperBound - lowerBound) / numPartitions + 1;
- for (long i = lowerBound; i < upperBound - stride; i += stride) {
- String range = String.format("%s,%s", i, i + stride);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
- }
- if (upperBound - lowerBound > stride * (numPartitions - 1)) {
- long indexFrom = (numPartitions - 1) * stride;
- long indexTo = upperBound + 1;
- String range = String.format("%s,%s", indexFrom, indexTo);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
+ T lowerBound = c.element().getValue().getKey();
+ T upperBound = c.element().getValue().getValue();
+ JdbcReadWithPartitionsHelper<T> helper =
+ JdbcReadWithPartitionsHelper.getPartitionsHelper(partitioningColumnType);
+ List<KV<T, T>> ranges =
+ Lists.newArrayList(helper.calculateRanges(lowerBound, upperBound, c.element().getKey()));
+ LOG.warn("Total of {} ranges: {}", ranges.size(), ranges);
+ for (KV<T, T> e : ranges) {
+ c.output(e);
}
}
}
+
+ public static final Map<Class<?>, JdbcReadWithPartitionsHelper<?>> PRESET_HELPERS =
+ ImmutableMap.of(
+ Long.class,
+ new JdbcReadWithPartitionsHelper<Long>() {
+ @Override
+ public Iterable<KV<Long, Long>> calculateRanges(
+ Long lowerBound, Long upperBound, Long partitions) {
+ List<KV<Long, Long>> ranges = new ArrayList<>();
+ // We divide by partitions FIRST to make sure that we can cover the whole LONG range.
+ // If we substract first, then we may end up with Long.MAX - Long.MIN, which is 2*MAX,
+ // and we'd have trouble with the pipeline.
+ long stride = (upperBound / partitions - lowerBound / partitions) + 1;
+ long highest = lowerBound;
+ for (long i = lowerBound; i < upperBound - stride; i += stride) {
+ ranges.add(KV.of(i, i + stride));
+ highest = i + stride;
+ }
+ if (upperBound - lowerBound > stride * (ranges.size() - 1)) {
+ long indexFrom = highest;
+ long indexTo = upperBound + 1;
+ ranges.add(KV.of(indexFrom, indexTo));
+ }
+ return ranges;
+ }
+
+ @Override
+ public void setParameters(KV<Long, Long> element, PreparedStatement preparedStatement) {
+ try {
+ preparedStatement.setLong(1, element.getKey());
+ preparedStatement.setLong(2, element.getValue());
+ } catch (SQLException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Override
+ public KV<Long, KV<Long, Long>> mapRow(ResultSet resultSet) throws Exception {
+ if (resultSet.getMetaData().getColumnCount() == 3) {
+ return KV.of(
+ resultSet.getLong(3), KV.of(resultSet.getLong(1), resultSet.getLong(2)));
+ } else {
+ return KV.of(0L, KV.of(resultSet.getLong(1), resultSet.getLong(2)));
+ }
+ }
+ },
+ DateTime.class,
+ new JdbcReadWithPartitionsHelper<DateTime>() {
+ @Override
+ public Iterable<KV<DateTime, DateTime>> calculateRanges(
+ DateTime lowerBound, DateTime upperBound, Long partitions) {
+ final List<KV<DateTime, DateTime>> result = new ArrayList<>();
+
+ final long intervalMillis = upperBound.getMillis() - lowerBound.getMillis();
+ final long strideMillis =
Review comment:
done. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (014888e) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.19%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19%
===========================================
Files 201 452 +251
Lines 19787 62175 +42388
===========================================
+ Hits 9189 52002 +42813
- Misses 9612 10173 +561
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/transforms/stats/sum\_switch.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL3N1bV9zd2l0Y2guZ28=) | | |
| [sdks/go/pkg/beam/core/util/hooks/hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvaG9va3MvaG9va3MuZ28=) | | |
| [...kg/beam/core/runtime/xlangx/expansionx/download.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuc2lvbngvZG93bmxvYWQuZ28=) | | |
| [sdks/go/pkg/beam/partition.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wYXJ0aXRpb24uZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/count.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL2NvdW50Lmdv) | | |
| [sdks/go/pkg/beam/core/runtime/metricsx/urns.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvdXJucy5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/coder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2NvZGVyLmdv) | | |
| [sdks/go/pkg/beam/runners/direct/direct.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9kaXJlY3QuZ28=) | | |
| [sdks/go/pkg/beam/testing/passert/passert.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvcGFzc2VydC5nbw==) | | |
| [...s/go/pkg/beam/core/runtime/graphx/schema/schema.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NjaGVtYS9zY2hlbWEuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...014888e](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f155b81) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.61% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62252 +42465
===========================================
+ Hits 9189 52050 +42861
- Misses 9612 10202 +590
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/data.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kYXRhLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/sdf\_invokers.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zZGZfaW52b2tlcnMuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/coder/registry.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3JlZ2lzdHJ5Lmdv) | | |
| [sdks/go/pkg/beam/core/funcx/fn.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L2ZuLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/payload.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L3BheWxvYWQuZ28=) | | |
| [sdks/go/pkg/beam/transforms/filter/filter.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL2ZpbHRlci9maWx0ZXIuc2hpbXMuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/coder/windows.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3dpbmRvd3MuZ28=) | | |
| [sdks/go/pkg/beam/util/harnessopts/sampler.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL2hhcm5lc3NvcHRzL3NhbXBsZXIuZ28=) | | |
| [sdks/go/pkg/beam/testing/passert/passert.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvcGFzc2VydC5zaGltcy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/cogbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9jb2diay5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...f155b81](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (014888e) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.19%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19%
===========================================
Files 201 452 +251
Lines 19787 62175 +42388
===========================================
+ Hits 9189 52002 +42813
- Misses 9612 10173 +561
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/harness/datamgr.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9kYXRhbWdyLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/harness/sampler.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zYW1wbGVyLmdv) | | |
| [...ks/go/pkg/beam/core/runtime/harness/cache\_hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9jYWNoZV9ob29rcy5nbw==) | | |
| [sdks/go/pkg/beam/testing/teststream/teststream.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Rlc3RzdHJlYW0vdGVzdHN0cmVhbS5nbw==) | | |
| [sdks/go/pkg/beam/windowing.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS93aW5kb3dpbmcuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/quantiles.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL3F1YW50aWxlcy5nbw==) | | |
| [sdks/go/pkg/beam/testing/passert/passert.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvcGFzc2VydC5zaGltcy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/namespace.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L25hbWVzcGFjZS5nbw==) | | |
| [sdks/go/pkg/beam/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/harness/session.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zZXNzaW9uLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...014888e](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1024821813
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027385833
thanks @mosche ! The last commit I've pushed makes the test suite run with MySQL and Postgres test containers. Thanks again!
See the tests running against both: https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/912/testReport/org.apache.beam.sdk.io.jdbc/JdbcIOAutoPartitioningIT/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8412e56) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51984 +42795
- Misses 9612 10162 +550
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/util/gcsx/gcs.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL2djc3gvZ2NzLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/user.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3VzZXIuZ28=) | | |
| [sdks/go/pkg/beam/io/textio/sdf.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby90ZXh0aW8vc2RmLmdv) | | |
| [sdks/go/pkg/beam/core/util/symtab/symtab.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvc3ltdGFiL3N5bXRhYi5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/iterable.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2l0ZXJhYmxlLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/serialize.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NlcmlhbGl6ZS5nbw==) | | |
| [...ks/go/pkg/beam/runners/dataflow/dataflowlib/job.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL2pvYi5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/windows.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3dpbmRvd3MuZ28=) | | |
| [sdks/go/pkg/beam/io/synthetic/source.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9zeW50aGV0aWMvc291cmNlLmdv) | | |
| [sdks/go/pkg/beam/core/util/reflectx/types.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvdHlwZXMuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...8412e56](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e87c4bb) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62161 +42374
===========================================
+ Hits 9189 51998 +42809
- Misses 9612 10163 +551
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/io/filesystem/local/local.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL2xvY2FsL2xvY2FsLmdv) | | |
| [sdks/go/pkg/beam/transforms/top/top.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3RvcC90b3AuZ28=) | | |
| [sdks/go/pkg/beam/runners/direct/direct.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9kaXJlY3QuZ28=) | | |
| [sdks/go/pkg/beam/core/util/reflectx/tags.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvdGFncy5nbw==) | | |
| [.../go/pkg/beam/core/graph/coder/testutil/testutil.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3Rlc3R1dGlsL3Rlc3R1dGlsLmdv) | | |
| [sdks/go/pkg/beam/transforms/stats/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/coder/map.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL21hcC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/harness/harness.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9oYXJuZXNzLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/int.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2ludC5nbw==) | | |
| [.../go/pkg/beam/runners/dataflow/dataflowlib/stage.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL3N0YWdlLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...e87c4bb](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r799531475
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
##########
@@ -333,31 +343,206 @@ private static Calendar withTimestampAndTimezone(DateTime dateTime) {
return calendar;
}
+ /**
+ * A helper for {@link ReadWithPartitions} that handles range calculations.
+ *
+ * @param <PartitionT>
+ */
+ interface JdbcReadWithPartitionsHelper<PartitionT>
+ extends PreparedStatementSetter<KV<PartitionT, PartitionT>>,
+ RowMapper<KV<Long, KV<PartitionT, PartitionT>>> {
+ Iterable<KV<PartitionT, PartitionT>> calculateRanges(
+ PartitionT lowerBound, PartitionT upperBound, Long partitions);
+
+ @Override
+ void setParameters(KV<PartitionT, PartitionT> element, PreparedStatement preparedStatement);
+
+ @Override
+ KV<Long, KV<PartitionT, PartitionT>> mapRow(ResultSet resultSet) throws Exception;
+ }
+
/** Create partitions on a table. */
- static class PartitioningFn extends DoFn<KV<Integer, KV<Long, Long>>, KV<String, Long>> {
+ static class PartitioningFn<T> extends DoFn<KV<Long, KV<T, T>>, KV<T, T>> {
+ private static final Logger LOG = LoggerFactory.getLogger(PartitioningFn.class);
+ final TypeDescriptor<T> partitioningColumnType;
+
+ PartitioningFn(TypeDescriptor<T> partitioningColumnType) {
+ this.partitioningColumnType = partitioningColumnType;
+ }
+
@ProcessElement
public void processElement(ProcessContext c) {
- Integer numPartitions = c.element().getKey();
- Long lowerBound = c.element().getValue().getKey();
- Long upperBound = c.element().getValue().getValue();
- if (lowerBound > upperBound) {
- throw new RuntimeException(
- String.format(
- "Lower bound [%s] is higher than upper bound [%s]", lowerBound, upperBound));
- }
- long stride = (upperBound - lowerBound) / numPartitions + 1;
- for (long i = lowerBound; i < upperBound - stride; i += stride) {
- String range = String.format("%s,%s", i, i + stride);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
- }
- if (upperBound - lowerBound > stride * (numPartitions - 1)) {
- long indexFrom = (numPartitions - 1) * stride;
- long indexTo = upperBound + 1;
- String range = String.format("%s,%s", indexFrom, indexTo);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
+ T lowerBound = c.element().getValue().getKey();
+ T upperBound = c.element().getValue().getValue();
+ JdbcReadWithPartitionsHelper<T> helper =
+ (JdbcReadWithPartitionsHelper<T>) PRESET_HELPERS.get(partitioningColumnType.getRawType());
+ List<KV<T, T>> ranges =
+ Lists.newArrayList(helper.calculateRanges(lowerBound, upperBound, c.element().getKey()));
+ LOG.warn("Total of {} ranges: {}", ranges.size(), ranges);
+ for (KV<T, T> e : ranges) {
+ c.output(e);
}
}
}
+
+ public static final Map<Class<?>, JdbcReadWithPartitionsHelper<?>> PRESET_HELPERS =
Review comment:
That would be great I think, thanks. I personally think such tests can serve as the best possible documentation for other devs and the ones in JdbcIOTest are already on a fairly high level as they run a pipeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (014888e) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.19%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19%
===========================================
Files 201 452 +251
Lines 19787 62175 +42388
===========================================
+ Hits 9189 52002 +42813
- Misses 9612 10173 +561
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/sideinput.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zaWRlaW5wdXQuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/mean.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21lYW4uZ28=) | | |
| [sdks/go/pkg/beam/io/textio/textio.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby90ZXh0aW8vdGV4dGlvLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/harness/harness.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9oYXJuZXNzLmdv) | | |
| [...ks/go/pkg/beam/runners/dataflow/dataflowlib/job.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL2pvYi5nbw==) | | |
| [sdks/go/pkg/beam/transforms/stats/count.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL2NvdW50Lmdv) | | |
| [sdks/go/pkg/beam/core/runtime/harness/logging.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9sb2dnaW5nLmdv) | | |
| [sdks/go/pkg/beam/io/databaseio/mapper.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhYmFzZWlvL21hcHBlci5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/datasink.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kYXRhc2luay5nbw==) | | |
| [.../go/pkg/beam/runners/dataflow/dataflowlib/fixup.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL2ZpeHVwLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...014888e](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0512486) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52004 +42815
- Misses 9612 10164 +552
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/io/databaseio/mapper.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhYmFzZWlvL21hcHBlci5nbw==) | | |
| [sdks/go/pkg/beam/transforms/stats/max\_switch.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21heF9zd2l0Y2guZ28=) | | |
| [...kg/beam/core/runtime/graphx/schema/logicaltypes.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NjaGVtYS9sb2dpY2FsdHlwZXMuZ28=) | | |
| [...dks/go/pkg/beam/core/runtime/harness/monitoring.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9tb25pdG9yaW5nLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/window.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy93aW5kb3cuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/sdf\_invokers.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zZGZfaW52b2tlcnMuZ28=) | | |
| [...o/pkg/beam/runners/dataflow/dataflowlib/metrics.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL21ldHJpY3MuZ28=) | | |
| [sdks/go/pkg/beam/core/util/reflectx/functions.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvZnVuY3Rpb25zLmdv) | | |
| [sdks/go/pkg/beam/testing/passert/sum.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvc3VtLmdv) | | |
| [sdks/go/pkg/beam/flatten.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9mbGF0dGVuLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...0512486](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1031932986
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (800a566) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52008 +42819
- Misses 9612 10160 +548
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/transforms/stats/mean.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21lYW4uZ28=) | | |
| [sdks/go/pkg/beam/provision/provision.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wcm92aXNpb24vcHJvdmlzaW9uLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/clone.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L2Nsb25lLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/bool.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2Jvb2wuZ28=) | | |
| [...beam/core/runtime/harness/statecache/statecache.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZWNhY2hlL3N0YXRlY2FjaGUuZ28=) | | |
| [sdks/go/pkg/beam/util/shimx/generate.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL3NoaW14L2dlbmVyYXRlLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/map.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL21hcC5nbw==) | | |
| [sdks/go/pkg/beam/option.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9vcHRpb24uZ28=) | | |
| [sdks/go/pkg/beam/xlang.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS94bGFuZy5nbw==) | | |
| [sdks/go/pkg/beam/core/util/protox/any.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcHJvdG94L2FueS5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...800a566](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f155b81) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.61% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62252 +42465
===========================================
+ Hits 9189 52050 +42861
- Misses 9612 10202 +590
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/util/protox/any.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcHJvdG94L2FueS5nbw==) | | |
| [sdks/go/pkg/beam/io/filesystem/local/local.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL2xvY2FsL2xvY2FsLmdv) | | |
| [sdks/go/pkg/beam/core/funcx/output.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L291dHB1dC5nbw==) | | |
| [sdks/go/pkg/beam/core/metrics/sampler.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL21ldHJpY3Mvc2FtcGxlci5nbw==) | | |
| [...o/pkg/beam/runners/dataflow/dataflowlib/metrics.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL21ldHJpY3MuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/min\_switch.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21pbl9zd2l0Y2guZ28=) | | |
| [sdks/go/pkg/beam/testing/passert/passert.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvcGFzc2VydC5zaGltcy5nbw==) | | |
| [sdks/go/pkg/beam/transforms/stats/max\_switch.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21heF9zd2l0Y2guZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/fn\_arity.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbl9hcml0eS5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/varint.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3ZhcmludC5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...f155b81](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1033125346
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0512486) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 0512486 differs from pull request most recent head df9a12c. Consider uploading reports for the commit df9a12c to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52004 +42815
- Misses 9612 10164 +552
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/transforms/stats/mean.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21lYW4uZ28=) | | |
| [sdks/go/pkg/beam/runners/vet/vet.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL3ZldC92ZXQuZ28=) | | |
| [sdks/go/pkg/beam/schema.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9zY2hlbWEuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/datasource.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kYXRhc291cmNlLmdv) | | |
| [sdks/go/pkg/beam/transforms/stats/max.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21heC5nbw==) | | |
| [sdks/go/pkg/beam/core/funcx/output.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L291dHB1dC5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/row\_encoder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3Jvd19lbmNvZGVyLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | | |
| [sdks/go/pkg/beam/combine.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb21iaW5lLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/sideinput.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zaWRlaW5wdXQuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...df9a12c](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0512486) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52004 +42815
- Misses 9612 10164 +552
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/datasource.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kYXRhc291cmNlLmdv) | | |
| [sdks/go/pkg/beam/runners/direct/gbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9nYmsuZ28=) | | |
| [sdks/go/pkg/beam/core/util/hooks/hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvaG9va3MvaG9va3MuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/clone.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L2Nsb25lLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/reshuffle.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9yZXNodWZmbGUuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/stats.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL3N0YXRzLnNoaW1zLmdv) | | |
| [sdks/go/pkg/beam/io/bigqueryio/bigquery.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9iaWdxdWVyeWlvL2JpZ3F1ZXJ5Lmdv) | | |
| [...ks/go/pkg/beam/core/runtime/harness/cache\_hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9jYWNoZV9ob29rcy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/status.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zdGF0dXMuZ28=) | | |
| [sdks/go/pkg/beam/io/databaseio/mapper.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhYmFzZWlvL21hcHBlci5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...0512486](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f155b81) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.61% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62252 +42465
===========================================
+ Hits 9189 52050 +42861
- Misses 9612 10202 +590
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/decode.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kZWNvZGUuZ28=) | | |
| [sdks/go/pkg/beam/runners/direct/impulse.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9pbXB1bHNlLmdv) | | |
| [sdks/go/pkg/beam/combine.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb21iaW5lLmdv) | | |
| [sdks/go/pkg/beam/artifact/materialize.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9hcnRpZmFjdC9tYXRlcmlhbGl6ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/serialize.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NlcmlhbGl6ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/coder/row\_encoder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3Jvd19lbmNvZGVyLmdv) | | |
| [sdks/go/pkg/beam/io/datastoreio/datastore.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhc3RvcmVpby9kYXRhc3RvcmUuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/coderx/int.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L2ludC5nbw==) | | |
| [sdks/go/pkg/beam/transforms/stats/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL3V0aWwuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...f155b81](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0512486) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52004 +42815
- Misses 9612 10164 +552
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/status.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zdGF0dXMuZ28=) | | |
| [...kg/beam/core/runtime/xlangx/expansionx/download.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuc2lvbngvZG93bmxvYWQuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/resolve.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L3Jlc29sdmUuZ28=) | | |
| [sdks/go/pkg/beam/util/harnessopts/sampler.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL2hhcm5lc3NvcHRzL3NhbXBsZXIuZ28=) | | |
| [.../pkg/beam/runners/universal/extworker/extworker.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL3VuaXZlcnNhbC9leHR3b3JrZXIvZXh0d29ya2VyLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/discard.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kaXNjYXJkLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/flatten.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbGF0dGVuLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/expand.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuZC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/decode.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kZWNvZGUuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/symbols.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvc3ltYm9scy5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...0512486](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (df9a12c) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.19%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 51997 +42808
- Misses 9612 10171 +559
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/io/filesystem/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/io/textio/textio.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby90ZXh0aW8vdGV4dGlvLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/xlang.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3hsYW5nLmdv) | | |
| [sdks/go/pkg/beam/transforms/stats/max\_switch.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21heF9zd2l0Y2guZ28=) | | |
| [sdks/go/pkg/beam/core/graph/coder/double.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2RvdWJsZS5nbw==) | | |
| [...kg/beam/core/runtime/graphx/schema/logicaltypes.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NjaGVtYS9sb2dpY2FsdHlwZXMuZ28=) | | |
| [sdks/go/pkg/beam/artifact/materialize.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9hcnRpZmFjdC9tYXRlcmlhbGl6ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/cogbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9jb2diay5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/input.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9pbnB1dC5nbw==) | | |
| [sdks/go/pkg/beam/forward.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9mb3J3YXJkLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...df9a12c](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (014888e) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.19%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19%
===========================================
Files 201 452 +251
Lines 19787 62175 +42388
===========================================
+ Hits 9189 52002 +42813
- Misses 9612 10173 +561
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/graph/coder/time.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3RpbWUuZ28=) | | |
| [sdks/go/pkg/beam/runners/direct/buffer.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9idWZmZXIuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/coderx/int.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L2ludC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/coderx/varint.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L3ZhcmludC5nbw==) | | |
| [sdks/go/pkg/beam/coder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb2Rlci5nbw==) | | |
| [sdks/go/pkg/beam/core/funcx/signature.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L3NpZ25hdHVyZS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/expand.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuZC5nbw==) | | |
| [sdks/go/pkg/beam/io/databaseio/database.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhYmFzZWlvL2RhdGFiYXNlLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/core/metrics/sampler.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL21ldHJpY3Mvc2FtcGxlci5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...014888e](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8412e56) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51984 +42795
- Misses 9612 10162 +550
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/typex/fulltype.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3R5cGV4L2Z1bGx0eXBlLmdv) | | |
| [...beam/core/runtime/harness/statecache/statecache.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZWNhY2hlL3N0YXRlY2FjaGUuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/symbols.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvc3ltYm9scy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/user.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3VzZXIuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/metricsx/urns.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvdXJucy5nbw==) | | |
| [sdks/go/pkg/beam/testing/passert/sum.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvc3VtLmdv) | | |
| [sdks/go/pkg/beam/transforms/stats/mean.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21lYW4uZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/tree.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3RyZWUuZ28=) | | |
| [sdks/go/pkg/beam/core/metrics/sampler.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL21ldHJpY3Mvc2FtcGxlci5nbw==) | | |
| [sdks/go/pkg/beam/core/typex/class.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3R5cGV4L2NsYXNzLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...8412e56](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1026430662
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027385833
thanks @mosche ! The last commit I've pushed makes the test suite run with MySQL and Postgres test containers. Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (19b9a69) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51979 +42790
- Misses 9612 10167 +555
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/graph/coder/row\_decoder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3Jvd19kZWNvZGVyLmdv) | | |
| [sdks/go/pkg/beam/core/funcx/signature.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L3NpZ25hdHVyZS5nbw==) | | |
| [...o/pkg/beam/runners/dataflow/dataflowlib/execute.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL2V4ZWN1dGUuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/coder/bytes.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2J5dGVzLmdv) | | |
| [sdks/go/pkg/beam/transforms/stats/min.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21pbi5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/xlang.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3hsYW5nLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/cogbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9jb2diay5nbw==) | | |
| [sdks/go/pkg/beam/artifact/materialize.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9hcnRpZmFjdC9tYXRlcmlhbGl6ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/row\_encoder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3Jvd19lbmNvZGVyLmdv) | | |
| [sdks/go/pkg/beam/runners/direct/impulse.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9pbXB1bHNlLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...19b9a69](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1028621597
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r797695823
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -999,59 +1015,84 @@ public void populateDisplayData(DisplayData.Builder builder) {
/** Implementation of {@link #readWithPartitions}. */
@AutoValue
- public abstract static class ReadWithPartitions<T> extends PTransform<PBegin, PCollection<T>> {
+ public abstract static class ReadWithPartitions<T, PartitionColumnT>
+ extends PTransform<PBegin, PCollection<T>> {
abstract @Nullable SerializableFunction<Void, DataSource> getDataSourceProviderFn();
abstract @Nullable RowMapper<T> getRowMapper();
abstract @Nullable Coder<T> getCoder();
- abstract Integer getNumPartitions();
+ abstract @Nullable Integer getNumPartitions();
abstract @Nullable String getPartitionColumn();
- abstract @Nullable Long getLowerBound();
+ abstract @javax.annotation.Nullable @Nullable PartitionColumnT getLowerBound();
Review comment:
Wondering, why also adding javax Nullable here?
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -1071,74 +1112,136 @@ public void populateDisplayData(DisplayData.Builder builder) {
* strides for generated WHERE clause expressions used to split the column withPartitionColumn
* evenly. When the input is less than 1, the number is set to 1.
*/
- public ReadWithPartitions<T> withNumPartitions(int numPartitions) {
+ public ReadWithPartitions<T, PartitionColumnT> withNumPartitions(int numPartitions) {
checkArgument(numPartitions > 0, "numPartitions can not be less than 1");
return toBuilder().setNumPartitions(numPartitions).build();
}
/** The name of a column of numeric type that will be used for partitioning. */
- public ReadWithPartitions<T> withPartitionColumn(String partitionColumn) {
+ public ReadWithPartitions<T, PartitionColumnT> withPartitionColumn(String partitionColumn) {
checkNotNull(partitionColumn, "partitionColumn can not be null");
return toBuilder().setPartitionColumn(partitionColumn).build();
}
- public ReadWithPartitions<T> withLowerBound(Long lowerBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withLowerBound(PartitionColumnT lowerBound) {
return toBuilder().setLowerBound(lowerBound).build();
}
- public ReadWithPartitions<T> withUpperBound(Long upperBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withUpperBound(PartitionColumnT upperBound) {
return toBuilder().setUpperBound(upperBound).build();
}
/** Name of the table in the external database. Can be used to pass a user-defined subqery. */
- public ReadWithPartitions<T> withTable(String tableName) {
+ public ReadWithPartitions<T, PartitionColumnT> withTable(String tableName) {
checkNotNull(tableName, "table can not be null");
return toBuilder().setTable(tableName).build();
}
+ private static final int EQUAL = 0;
+
@Override
public PCollection<T> expand(PBegin input) {
checkNotNull(getRowMapper(), "withRowMapper() is required");
checkNotNull(
getDataSourceProviderFn(),
"withDataSourceConfiguration() or withDataSourceProviderFn() is required");
checkNotNull(getPartitionColumn(), "withPartitionColumn() is required");
- checkArgument(
- getUpperBound() != null && getLowerBound() != null,
- "Upper and lower bounds are mandatory parameters for JdbcIO.readWithPartitions");
checkNotNull(getTable(), "withTable() is required");
+ if (getLowerBound() != null && getLowerBound() instanceof Comparable<?>) {
Review comment:
Validation currently doesn't check that `(getLowerBound() != null) == (getUpperBound() != null)`, this may break the if condition further down ... especially if not implementing `Comparable`.
Imho, this is an example where something like `withBounds(lower, upper)` provides much more guidance, but also makes validation a lot easier.
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -1071,74 +1112,136 @@ public void populateDisplayData(DisplayData.Builder builder) {
* strides for generated WHERE clause expressions used to split the column withPartitionColumn
* evenly. When the input is less than 1, the number is set to 1.
*/
- public ReadWithPartitions<T> withNumPartitions(int numPartitions) {
+ public ReadWithPartitions<T, PartitionColumnT> withNumPartitions(int numPartitions) {
checkArgument(numPartitions > 0, "numPartitions can not be less than 1");
return toBuilder().setNumPartitions(numPartitions).build();
}
/** The name of a column of numeric type that will be used for partitioning. */
- public ReadWithPartitions<T> withPartitionColumn(String partitionColumn) {
+ public ReadWithPartitions<T, PartitionColumnT> withPartitionColumn(String partitionColumn) {
checkNotNull(partitionColumn, "partitionColumn can not be null");
return toBuilder().setPartitionColumn(partitionColumn).build();
}
- public ReadWithPartitions<T> withLowerBound(Long lowerBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withLowerBound(PartitionColumnT lowerBound) {
return toBuilder().setLowerBound(lowerBound).build();
}
- public ReadWithPartitions<T> withUpperBound(Long upperBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withUpperBound(PartitionColumnT upperBound) {
return toBuilder().setUpperBound(upperBound).build();
}
/** Name of the table in the external database. Can be used to pass a user-defined subqery. */
- public ReadWithPartitions<T> withTable(String tableName) {
+ public ReadWithPartitions<T, PartitionColumnT> withTable(String tableName) {
checkNotNull(tableName, "table can not be null");
return toBuilder().setTable(tableName).build();
}
+ private static final int EQUAL = 0;
+
@Override
public PCollection<T> expand(PBegin input) {
checkNotNull(getRowMapper(), "withRowMapper() is required");
checkNotNull(
getDataSourceProviderFn(),
"withDataSourceConfiguration() or withDataSourceProviderFn() is required");
checkNotNull(getPartitionColumn(), "withPartitionColumn() is required");
- checkArgument(
- getUpperBound() != null && getLowerBound() != null,
- "Upper and lower bounds are mandatory parameters for JdbcIO.readWithPartitions");
checkNotNull(getTable(), "withTable() is required");
+ if (getLowerBound() != null && getLowerBound() instanceof Comparable<?>) {
+ // Not all partition types are comparable. For example, LocalDateTime, which is a valid
+ // partitioning type, is not Comparable, so we can't enforce this for all sorts of
+ // partitioning.
+ checkArgument(
+ ((Comparable<PartitionColumnT>) getLowerBound()).compareTo(getUpperBound()) < EQUAL,
+ "The lower bound of partitioning column is larger or equal than the upper bound");
+ }
checkArgument(
- getLowerBound() < getUpperBound(),
- "The lower bound of partitioning column is larger or equal than the upper bound");
- checkArgument(
- getUpperBound() - getLowerBound() >= getNumPartitions(),
- "The specified number of partitions is more than the difference between upper bound and lower bound");
+ JdbcUtil.PRESET_HELPERS.containsKey(getPartitionColumnType().getRawType()),
+ "readWithPartitions only supports the following types: %s",
+ JdbcUtil.PRESET_HELPERS.keySet());
+
+ PCollection<KV<Integer, KV<PartitionColumnT, PartitionColumnT>>> params;
+
+ if (getLowerBound() == null && getUpperBound() == null) {
+ String query =
+ String.format(
+ "SELECT min(%s), max(%s) FROM %s",
+ getPartitionColumn(), getPartitionColumn(), getTable());
+ if (getNumPartitions() == null) {
+ query =
+ String.format(
+ "SELECT min(%s), max(%s), count(*) FROM %s",
+ getPartitionColumn(), getPartitionColumn(), getTable());
+ }
+ params =
+ input
+ .apply(
+ JdbcIO.<KV<Integer, KV<PartitionColumnT, PartitionColumnT>>>read()
+ .withQuery(query)
+ .withDataSourceProviderFn(getDataSourceProviderFn())
+ .withRowMapper(
+ (JdbcReadWithPartitionsHelper<PartitionColumnT>)
+ JdbcUtil.PRESET_HELPERS.get(getPartitionColumnType().getRawType())))
+ .apply(
+ MapElements.via(
+ new SimpleFunction<
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>>,
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>>>() {
+ @Override
+ public KV<Integer, KV<PartitionColumnT, PartitionColumnT>> apply(
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>> input) {
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>> result;
+ if (getNumPartitions() == null) {
+ // In this case, we use the table row count to infer a number of
+ // partitions.
+ // We take the square root of the number of rows, and divide it by 5
+ // to keep a relatively low number of partitions, given that an RDBMS
+ // cannot usually accept a very large number of connections.
+ Integer numPartitions =
Review comment:
Wondering, for smallish counts this still creates a fairly large number of partitions ...
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
##########
@@ -334,30 +342,188 @@ private static Calendar withTimestampAndTimezone(DateTime dateTime) {
}
/** Create partitions on a table. */
- static class PartitioningFn extends DoFn<KV<Integer, KV<Long, Long>>, KV<String, Long>> {
+ static class PartitioningFn<T> extends DoFn<KV<Integer, KV<T, T>>, KV<T, T>> {
+ private static final Logger LOG = LoggerFactory.getLogger(PartitioningFn.class);
+ final TypeDescriptor<T> partitioningColumnType;
+
+ PartitioningFn(TypeDescriptor<T> partitioningColumnType) {
+ this.partitioningColumnType = partitioningColumnType;
+ }
+
@ProcessElement
public void processElement(ProcessContext c) {
- Integer numPartitions = c.element().getKey();
- Long lowerBound = c.element().getValue().getKey();
- Long upperBound = c.element().getValue().getValue();
- if (lowerBound > upperBound) {
- throw new RuntimeException(
- String.format(
- "Lower bound [%s] is higher than upper bound [%s]", lowerBound, upperBound));
- }
- long stride = (upperBound - lowerBound) / numPartitions + 1;
- for (long i = lowerBound; i < upperBound - stride; i += stride) {
- String range = String.format("%s,%s", i, i + stride);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
- }
- if (upperBound - lowerBound > stride * (numPartitions - 1)) {
- long indexFrom = (numPartitions - 1) * stride;
- long indexTo = upperBound + 1;
- String range = String.format("%s,%s", indexFrom, indexTo);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
+ T lowerBound = c.element().getValue().getKey();
+ T upperBound = c.element().getValue().getValue();
+ JdbcReadWithPartitionsHelper<T> helper =
+ (JdbcReadWithPartitionsHelper<T>) PRESET_HELPERS.get(partitioningColumnType.getRawType());
+ List<KV<T, T>> ranges =
+ Lists.newArrayList(helper.calculateRanges(lowerBound, upperBound, c.element().getKey()));
+ LOG.warn("Total of {} ranges: {}", ranges.size(), ranges);
+ for (KV<T, T> e : ranges) {
+ c.output(e);
}
}
}
+
+ public static final Map<Class<?>, JdbcReadWithPartitionsHelper<?>> PRESET_HELPERS =
+ ImmutableMap.of(
+ String.class,
Review comment:
To be honest, I'm not sure how to feel about offering equal width binning for strings. From past experience I fear this might trick users into a seemingly simple solution that could quickly turn into a bad surprise. I've rarely seen string columns to be uniformly distributed. Too often there's high skew around certain chars, not even speaking about categorical values... What do you think?
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -1071,74 +1112,136 @@ public void populateDisplayData(DisplayData.Builder builder) {
* strides for generated WHERE clause expressions used to split the column withPartitionColumn
* evenly. When the input is less than 1, the number is set to 1.
*/
- public ReadWithPartitions<T> withNumPartitions(int numPartitions) {
+ public ReadWithPartitions<T, PartitionColumnT> withNumPartitions(int numPartitions) {
checkArgument(numPartitions > 0, "numPartitions can not be less than 1");
return toBuilder().setNumPartitions(numPartitions).build();
}
/** The name of a column of numeric type that will be used for partitioning. */
- public ReadWithPartitions<T> withPartitionColumn(String partitionColumn) {
+ public ReadWithPartitions<T, PartitionColumnT> withPartitionColumn(String partitionColumn) {
checkNotNull(partitionColumn, "partitionColumn can not be null");
return toBuilder().setPartitionColumn(partitionColumn).build();
}
- public ReadWithPartitions<T> withLowerBound(Long lowerBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withLowerBound(PartitionColumnT lowerBound) {
return toBuilder().setLowerBound(lowerBound).build();
}
- public ReadWithPartitions<T> withUpperBound(Long upperBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withUpperBound(PartitionColumnT upperBound) {
return toBuilder().setUpperBound(upperBound).build();
}
/** Name of the table in the external database. Can be used to pass a user-defined subqery. */
- public ReadWithPartitions<T> withTable(String tableName) {
+ public ReadWithPartitions<T, PartitionColumnT> withTable(String tableName) {
checkNotNull(tableName, "table can not be null");
return toBuilder().setTable(tableName).build();
}
+ private static final int EQUAL = 0;
+
@Override
public PCollection<T> expand(PBegin input) {
checkNotNull(getRowMapper(), "withRowMapper() is required");
checkNotNull(
getDataSourceProviderFn(),
"withDataSourceConfiguration() or withDataSourceProviderFn() is required");
checkNotNull(getPartitionColumn(), "withPartitionColumn() is required");
- checkArgument(
- getUpperBound() != null && getLowerBound() != null,
- "Upper and lower bounds are mandatory parameters for JdbcIO.readWithPartitions");
checkNotNull(getTable(), "withTable() is required");
+ if (getLowerBound() != null && getLowerBound() instanceof Comparable<?>) {
+ // Not all partition types are comparable. For example, LocalDateTime, which is a valid
+ // partitioning type, is not Comparable, so we can't enforce this for all sorts of
+ // partitioning.
+ checkArgument(
+ ((Comparable<PartitionColumnT>) getLowerBound()).compareTo(getUpperBound()) < EQUAL,
+ "The lower bound of partitioning column is larger or equal than the upper bound");
+ }
checkArgument(
- getLowerBound() < getUpperBound(),
- "The lower bound of partitioning column is larger or equal than the upper bound");
- checkArgument(
- getUpperBound() - getLowerBound() >= getNumPartitions(),
- "The specified number of partitions is more than the difference between upper bound and lower bound");
+ JdbcUtil.PRESET_HELPERS.containsKey(getPartitionColumnType().getRawType()),
+ "readWithPartitions only supports the following types: %s",
+ JdbcUtil.PRESET_HELPERS.keySet());
+
+ PCollection<KV<Integer, KV<PartitionColumnT, PartitionColumnT>>> params;
+
+ if (getLowerBound() == null && getUpperBound() == null) {
+ String query =
+ String.format(
+ "SELECT min(%s), max(%s) FROM %s",
+ getPartitionColumn(), getPartitionColumn(), getTable());
+ if (getNumPartitions() == null) {
+ query =
+ String.format(
+ "SELECT min(%s), max(%s), count(*) FROM %s",
+ getPartitionColumn(), getPartitionColumn(), getTable());
+ }
+ params =
+ input
+ .apply(
+ JdbcIO.<KV<Integer, KV<PartitionColumnT, PartitionColumnT>>>read()
+ .withQuery(query)
+ .withDataSourceProviderFn(getDataSourceProviderFn())
+ .withRowMapper(
+ (JdbcReadWithPartitionsHelper<PartitionColumnT>)
+ JdbcUtil.PRESET_HELPERS.get(getPartitionColumnType().getRawType())))
+ .apply(
+ MapElements.via(
+ new SimpleFunction<
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>>,
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>>>() {
+ @Override
+ public KV<Integer, KV<PartitionColumnT, PartitionColumnT>> apply(
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>> input) {
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>> result;
+ if (getNumPartitions() == null) {
+ // In this case, we use the table row count to infer a number of
+ // partitions.
+ // We take the square root of the number of rows, and divide it by 5
+ // to keep a relatively low number of partitions, given that an RDBMS
+ // cannot usually accept a very large number of connections.
+ Integer numPartitions =
+ Long.valueOf(
+ Math.round(Math.floor(Math.sqrt(input.getKey()) / 5)))
+ .intValue();
+ if (numPartitions == 0) {
+ numPartitions = 1;
+ }
+ result = KV.of(numPartitions, input.getValue());
+ } else {
+ result = KV.of(getNumPartitions(), input.getValue());
+ }
+ LOG.info(
+ "Inferred min: {} - max: {} - numPartitions: {}",
+ result.getValue().getKey(),
+ result.getValue().getValue(),
+ result.getKey());
+ return result;
+ }
+ }));
+ } else {
+ params =
+ input.apply(
+ Create.of(KV.of(getNumPartitions(), KV.of(getLowerBound(), getUpperBound()))));
+ }
- PCollection<KV<Integer, KV<Long, Long>>> params =
- input.apply(
- Create.of(
- Collections.singletonList(
- KV.of(getNumPartitions(), KV.of(getLowerBound(), getUpperBound())))));
- PCollection<KV<String, Iterable<Long>>> ranges =
+ PCollection<KV<PartitionColumnT, PartitionColumnT>> ranges =
params
- .apply("Partitioning", ParDo.of(new PartitioningFn()))
- .apply("Group partitions", GroupByKey.create());
+ .apply("Partitioning", ParDo.of(new PartitioningFn<>(getPartitionColumnType())))
+ .apply("Reshuffle partitions", Reshuffle.viaRandomKey());
- JdbcIO.ReadAll<KV<String, Iterable<Long>>, T> readAll =
- JdbcIO.<KV<String, Iterable<Long>>, T>readAll()
+ JdbcIO.ReadAll<KV<PartitionColumnT, PartitionColumnT>, T> readAll =
+ JdbcIO.<KV<PartitionColumnT, PartitionColumnT>, T>readAll()
.withDataSourceProviderFn(getDataSourceProviderFn())
.withQuery(
String.format(
"select * from %1$s where %2$s >= ? and %2$s < ?",
getTable(), getPartitionColumn()))
.withRowMapper(getRowMapper())
.withParameterSetter(
- (PreparedStatementSetter<KV<String, Iterable<Long>>>)
- (element, preparedStatement) -> {
- String[] range = element.getKey().split(",", -1);
- preparedStatement.setLong(1, Long.parseLong(range[0]));
- preparedStatement.setLong(2, Long.parseLong(range[1]));
- })
+ // This cast is unchecked, thus this is a small type-checking risk. We just need
Review comment:
Seems fine 👍 How about exposing something like `<T> JdbcReadWithPartitionsHelper<T> getPartitionsHelper(TypeDescriptor<T> type)` in JdbcUtil rather than the preset map itself? With that the cast has to be done in one place only ...
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -999,59 +1015,84 @@ public void populateDisplayData(DisplayData.Builder builder) {
/** Implementation of {@link #readWithPartitions}. */
@AutoValue
- public abstract static class ReadWithPartitions<T> extends PTransform<PBegin, PCollection<T>> {
+ public abstract static class ReadWithPartitions<T, PartitionColumnT>
+ extends PTransform<PBegin, PCollection<T>> {
abstract @Nullable SerializableFunction<Void, DataSource> getDataSourceProviderFn();
abstract @Nullable RowMapper<T> getRowMapper();
abstract @Nullable Coder<T> getCoder();
- abstract Integer getNumPartitions();
+ abstract @Nullable Integer getNumPartitions();
abstract @Nullable String getPartitionColumn();
- abstract @Nullable Long getLowerBound();
+ abstract @javax.annotation.Nullable @Nullable PartitionColumnT getLowerBound();
- abstract @Nullable Long getUpperBound();
+ abstract @javax.annotation.Nullable @Nullable PartitionColumnT getUpperBound();
abstract @Nullable String getTable();
- abstract Builder<T> toBuilder();
+ abstract TypeDescriptor<PartitionColumnT> getPartitionColumnType();
+
+ abstract Builder<T, PartitionColumnT> toBuilder();
+
+ /**
+ * A helper for {@link ReadWithPartitions} that handles range calculations.
+ *
+ * @param <PartitionT>
+ */
+ public interface JdbcReadWithPartitionsHelper<PartitionT>
Review comment:
How about keeping this interface private initially and move it into JdbcUtil? It might not be polished / hardened enough to be exposed to users right away .. Also, to be on the safe treat count as long rather than int?
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -325,17 +325,24 @@ public static ReadRows readRows() {
*
* @param <T> Type of the data to be read.
*/
- public static <T> ReadWithPartitions<T> readWithPartitions() {
- return new AutoValue_JdbcIO_ReadWithPartitions.Builder<T>()
+ public static <T, PartitionColumnT> ReadWithPartitions<T, PartitionColumnT> readWithPartitions(
Review comment:
Hmm, introducing `PartitionColumnT` is a breaking change. Though, thinking about how likely this is going to affect people and what's required to work around it when using AutoValue, this feels like the better alternative ;)
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -999,59 +1015,84 @@ public void populateDisplayData(DisplayData.Builder builder) {
/** Implementation of {@link #readWithPartitions}. */
@AutoValue
- public abstract static class ReadWithPartitions<T> extends PTransform<PBegin, PCollection<T>> {
+ public abstract static class ReadWithPartitions<T, PartitionColumnT>
+ extends PTransform<PBegin, PCollection<T>> {
abstract @Nullable SerializableFunction<Void, DataSource> getDataSourceProviderFn();
abstract @Nullable RowMapper<T> getRowMapper();
abstract @Nullable Coder<T> getCoder();
- abstract Integer getNumPartitions();
+ abstract @Nullable Integer getNumPartitions();
abstract @Nullable String getPartitionColumn();
- abstract @Nullable Long getLowerBound();
+ abstract @javax.annotation.Nullable @Nullable PartitionColumnT getLowerBound();
- abstract @Nullable Long getUpperBound();
+ abstract @javax.annotation.Nullable @Nullable PartitionColumnT getUpperBound();
abstract @Nullable String getTable();
- abstract Builder<T> toBuilder();
+ abstract TypeDescriptor<PartitionColumnT> getPartitionColumnType();
+
+ abstract Builder<T, PartitionColumnT> toBuilder();
+
+ /**
+ * A helper for {@link ReadWithPartitions} that handles range calculations.
+ *
+ * @param <PartitionT>
+ */
+ public interface JdbcReadWithPartitionsHelper<PartitionT>
Review comment:
Would be nice to have a simple `Pair` / `Tuple` value similar to `KV` to avoid the semantic connotation of using `KV` in such cases and the confusion this might create ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1026430779
Run Java_Examples_Dataflow_Java17 PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1023901471
@mosche would you take a look at this change and let me know what you think?
I suggest you look at the new integration tests in `JdbcIOAutoPartitioningIT`, you can see the intended usage: A user would pass the type of the partition column, and we automatically generate partition ranges.
Some follow up changes are:
- Automatically infer the upper and lower bound at pipeline runtime via `select min(col), max(col) from table`, so the user won't have to provide them.
- Dynamic rebalancing: change ReadFn to work as a splittable DoFn that can sub-partition a range if it's going slow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1026348623
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (014888e) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.19%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19%
===========================================
Files 201 452 +251
Lines 19787 62175 +42388
===========================================
+ Hits 9189 52002 +42813
- Misses 9612 10173 +561
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/combine.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb21iaW5lLmdv) | | |
| [sdks/go/pkg/beam/transforms/stats/quantiles.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL3F1YW50aWxlcy5nbw==) | | |
| [sdks/go/pkg/beam/transforms/stats/min\_switch.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21pbl9zd2l0Y2guZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/sideinput.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zaWRlaW5wdXQuZ28=) | | |
| [...ks/go/pkg/beam/core/runtime/harness/cache\_hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9jYWNoZV9ob29rcy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/datasource.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kYXRhc291cmNlLmdv) | | |
| [sdks/go/pkg/beam/core/typex/class.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3R5cGV4L2NsYXNzLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/multiplex.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9tdWx0aXBsZXguZ28=) | | |
| [sdks/go/pkg/beam/partition.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wYXJ0aXRpb24uZ28=) | | |
| [sdks/go/pkg/beam/io/filesystem/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL3V0aWwuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...014888e](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (800a566) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52008 +42819
- Misses 9612 10160 +548
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvbWV0cmljc3guZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/types.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvdHlwZXMuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/registry.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L3JlZ2lzdHJ5Lmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/multiplex.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9tdWx0aXBsZXguZ28=) | | |
| [sdks/go/pkg/beam/io/datastoreio/datastore.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhc3RvcmVpby9kYXRhc3RvcmUuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/fn\_arity.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbl9hcml0eS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/harness/datamgr.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9kYXRhbWdyLmdv) | | |
| [sdks/go/pkg/beam/runners/direct/buffer.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9idWZmZXIuZ28=) | | |
| [sdks/go/pkg/beam/schema.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9zY2hlbWEuZ28=) | | |
| [sdks/go/pkg/beam/runners/dataflow/dataflow.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93Lmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...800a566](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (47aa02a) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.60% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62233 +42446
===========================================
+ Hits 9189 52033 +42844
- Misses 9612 10200 +588
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/pardo.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9wYXJkby5nbw==) | | |
| [sdks/go/pkg/beam/core/util/reflectx/tags.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvdGFncy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/emit.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9lbWl0Lmdv) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/payload.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L3BheWxvYWQuZ28=) | | |
| [sdks/go/pkg/beam/testing/passert/floats.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvZmxvYXRzLmdv) | | |
| [sdks/go/pkg/beam/io/synthetic/source.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9zeW50aGV0aWMvc291cmNlLmdv) | | |
| [sdks/go/pkg/beam/core/funcx/output.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L291dHB1dC5nbw==) | | |
| [sdks/go/pkg/beam/io/datastoreio/datastore.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhc3RvcmVpby9kYXRhc3RvcmUuZ28=) | | |
| [.../go/pkg/beam/runners/dataflow/dataflowlib/fixup.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL2ZpeHVwLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/multiplex.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9tdWx0aXBsZXguZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...47aa02a](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r799902478
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -1071,74 +1112,136 @@ public void populateDisplayData(DisplayData.Builder builder) {
* strides for generated WHERE clause expressions used to split the column withPartitionColumn
* evenly. When the input is less than 1, the number is set to 1.
*/
- public ReadWithPartitions<T> withNumPartitions(int numPartitions) {
+ public ReadWithPartitions<T, PartitionColumnT> withNumPartitions(int numPartitions) {
checkArgument(numPartitions > 0, "numPartitions can not be less than 1");
return toBuilder().setNumPartitions(numPartitions).build();
}
/** The name of a column of numeric type that will be used for partitioning. */
- public ReadWithPartitions<T> withPartitionColumn(String partitionColumn) {
+ public ReadWithPartitions<T, PartitionColumnT> withPartitionColumn(String partitionColumn) {
checkNotNull(partitionColumn, "partitionColumn can not be null");
return toBuilder().setPartitionColumn(partitionColumn).build();
}
- public ReadWithPartitions<T> withLowerBound(Long lowerBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withLowerBound(PartitionColumnT lowerBound) {
return toBuilder().setLowerBound(lowerBound).build();
}
- public ReadWithPartitions<T> withUpperBound(Long upperBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withUpperBound(PartitionColumnT upperBound) {
return toBuilder().setUpperBound(upperBound).build();
}
/** Name of the table in the external database. Can be used to pass a user-defined subqery. */
- public ReadWithPartitions<T> withTable(String tableName) {
+ public ReadWithPartitions<T, PartitionColumnT> withTable(String tableName) {
checkNotNull(tableName, "table can not be null");
return toBuilder().setTable(tableName).build();
}
+ private static final int EQUAL = 0;
+
@Override
public PCollection<T> expand(PBegin input) {
checkNotNull(getRowMapper(), "withRowMapper() is required");
checkNotNull(
getDataSourceProviderFn(),
"withDataSourceConfiguration() or withDataSourceProviderFn() is required");
checkNotNull(getPartitionColumn(), "withPartitionColumn() is required");
- checkArgument(
- getUpperBound() != null && getLowerBound() != null,
- "Upper and lower bounds are mandatory parameters for JdbcIO.readWithPartitions");
checkNotNull(getTable(), "withTable() is required");
+ if (getLowerBound() != null && getLowerBound() instanceof Comparable<?>) {
+ // Not all partition types are comparable. For example, LocalDateTime, which is a valid
+ // partitioning type, is not Comparable, so we can't enforce this for all sorts of
+ // partitioning.
+ checkArgument(
+ ((Comparable<PartitionColumnT>) getLowerBound()).compareTo(getUpperBound()) < EQUAL,
+ "The lower bound of partitioning column is larger or equal than the upper bound");
+ }
checkArgument(
- getLowerBound() < getUpperBound(),
- "The lower bound of partitioning column is larger or equal than the upper bound");
- checkArgument(
- getUpperBound() - getLowerBound() >= getNumPartitions(),
- "The specified number of partitions is more than the difference between upper bound and lower bound");
+ JdbcUtil.PRESET_HELPERS.containsKey(getPartitionColumnType().getRawType()),
+ "readWithPartitions only supports the following types: %s",
+ JdbcUtil.PRESET_HELPERS.keySet());
+
+ PCollection<KV<Integer, KV<PartitionColumnT, PartitionColumnT>>> params;
+
+ if (getLowerBound() == null && getUpperBound() == null) {
+ String query =
+ String.format(
+ "SELECT min(%s), max(%s) FROM %s",
+ getPartitionColumn(), getPartitionColumn(), getTable());
+ if (getNumPartitions() == null) {
+ query =
+ String.format(
+ "SELECT min(%s), max(%s), count(*) FROM %s",
+ getPartitionColumn(), getPartitionColumn(), getTable());
+ }
+ params =
+ input
+ .apply(
+ JdbcIO.<KV<Integer, KV<PartitionColumnT, PartitionColumnT>>>read()
+ .withQuery(query)
+ .withDataSourceProviderFn(getDataSourceProviderFn())
+ .withRowMapper(
+ (JdbcReadWithPartitionsHelper<PartitionColumnT>)
+ JdbcUtil.PRESET_HELPERS.get(getPartitionColumnType().getRawType())))
+ .apply(
+ MapElements.via(
+ new SimpleFunction<
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>>,
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>>>() {
+ @Override
+ public KV<Integer, KV<PartitionColumnT, PartitionColumnT>> apply(
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>> input) {
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>> result;
+ if (getNumPartitions() == null) {
+ // In this case, we use the table row count to infer a number of
+ // partitions.
+ // We take the square root of the number of rows, and divide it by 5
+ // to keep a relatively low number of partitions, given that an RDBMS
+ // cannot usually accept a very large number of connections.
+ Integer numPartitions =
Review comment:
thanks for the graphs! I decided to do `/ 10` for now. LMK WYT : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a change in pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r802477959
##########
File path: sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java
##########
@@ -28,6 +44,66 @@
@RunWith(JUnit4.class)
public class JdbcUtilTest {
+ static final JdbcReadWithPartitionsHelper<String> PROTOTYPE_STRING_PARTITIONER =
Review comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (47aa02a) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 47aa02a differs from pull request most recent head f155b81. Consider uploading reports for the commit f155b81 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.60% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62233 +42446
===========================================
+ Hits 9189 52033 +42844
- Misses 9612 10200 +588
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | | |
| [...kg/beam/core/runtime/xlangx/expansionx/download.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuc2lvbngvZG93bmxvYWQuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/min.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21pbi5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/windows.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3dpbmRvd3MuZ28=) | | |
| [sdks/go/pkg/beam/pardo.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wYXJkby5nbw==) | | |
| [sdks/go/pkg/beam/validate.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS92YWxpZGF0ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/graph.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2dyYXBoLmdv) | | |
| [sdks/go/pkg/beam/transforms/stats/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/count.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL2NvdW50Lmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/cogbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9jb2diay5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...f155b81](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r798090462
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -999,59 +1015,84 @@ public void populateDisplayData(DisplayData.Builder builder) {
/** Implementation of {@link #readWithPartitions}. */
@AutoValue
- public abstract static class ReadWithPartitions<T> extends PTransform<PBegin, PCollection<T>> {
+ public abstract static class ReadWithPartitions<T, PartitionColumnT>
+ extends PTransform<PBegin, PCollection<T>> {
abstract @Nullable SerializableFunction<Void, DataSource> getDataSourceProviderFn();
abstract @Nullable RowMapper<T> getRowMapper();
abstract @Nullable Coder<T> getCoder();
- abstract Integer getNumPartitions();
+ abstract @Nullable Integer getNumPartitions();
abstract @Nullable String getPartitionColumn();
- abstract @Nullable Long getLowerBound();
+ abstract @javax.annotation.Nullable @Nullable PartitionColumnT getLowerBound();
Review comment:
I think this was autocompleted. I've removed it. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8412e56) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51984 +42795
- Misses 9612 10162 +550
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/encoding.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9lbmNvZGluZy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/cogbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9jb2diay5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/row.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3Jvdy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/pcollection.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9wY29sbGVjdGlvbi5nbw==) | | |
| [...o/pkg/beam/io/rtrackers/offsetrange/offsetrange.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9ydHJhY2tlcnMvb2Zmc2V0cmFuZ2Uvb2Zmc2V0cmFuZ2UuZ28=) | | |
| [sdks/go/pkg/beam/io/filesystem/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/core/metrics/metrics.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL21ldHJpY3MvbWV0cmljcy5nbw==) | | |
| [sdks/go/pkg/beam/core/funcx/output.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L291dHB1dC5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/row\_encoder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3Jvd19lbmNvZGVyLmdv) | | |
| [sdks/go/pkg/beam/transforms/stats/min\_switch.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21pbl9zd2l0Y2guZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...8412e56](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8412e56) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51984 +42795
- Misses 9612 10162 +550
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/transforms/stats/count.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL2NvdW50Lmdv) | | |
| [sdks/go/pkg/beam/gbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9nYmsuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/options.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvb3B0aW9ucy5nbw==) | | |
| [sdks/go/pkg/beam/create.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jcmVhdGUuZ28=) | | |
| [sdks/go/pkg/beam/pardo.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wYXJkby5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/reshuffle.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9yZXNodWZmbGUuZ28=) | | |
| [sdks/go/pkg/beam/transforms/top/top.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3RvcC90b3Auc2hpbXMuZ28=) | | |
| [sdks/go/pkg/beam/util/starcgenx/starcgenx.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL3N0YXJjZ2VueC9zdGFyY2dlbnguZ28=) | | |
| [sdks/go/pkg/beam/core/graph/edge.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2VkZ2UuZ28=) | | |
| [sdks/go/pkg/beam/io/synthetic/source.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9zeW50aGV0aWMvc291cmNlLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...8412e56](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (19b9a69) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51979 +42790
- Misses 9612 10167 +555
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/xlangx/namespace.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L25hbWVzcGFjZS5nbw==) | | |
| [sdks/go/pkg/beam/transforms/stats/stats.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL3N0YXRzLnNoaW1zLmdv) | | |
| [...pkg/beam/runners/dataflow/dataflowlib/translate.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL3RyYW5zbGF0ZS5nbw==) | | |
| [sdks/go/pkg/beam/testing/passert/passert.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvcGFzc2VydC5zaGltcy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/emit.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9lbWl0Lmdv) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/xlang.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3hsYW5nLmdv) | | |
| [sdks/go/pkg/beam/beam.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9iZWFtLnNoaW1zLmdv) | | |
| [sdks/go/pkg/beam/transforms/top/top.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3RvcC90b3AuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/harness/sampler.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zYW1wbGVyLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/pcollection.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9wY29sbGVjdGlvbi5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...19b9a69](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1028621461
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1023894252
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a change in pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r801420304
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
##########
@@ -333,31 +343,154 @@ private static Calendar withTimestampAndTimezone(DateTime dateTime) {
return calendar;
}
+ /**
+ * A helper for {@link ReadWithPartitions} that handles range calculations.
+ *
+ * @param <PartitionT>
+ */
+ interface JdbcReadWithPartitionsHelper<PartitionT>
+ extends PreparedStatementSetter<KV<PartitionT, PartitionT>>,
+ RowMapper<KV<Long, KV<PartitionT, PartitionT>>> {
+ static <T> JdbcReadWithPartitionsHelper<T> getPartitionsHelper(TypeDescriptor<T> type) {
+ // This cast is unchecked, thus this is a small type-checking risk. We just need
+ // to make sure that all preset helpers in `JdbcUtil.PRESET_HELPERS` are matched
+ // in type from their Key and their Value.
+ return (JdbcReadWithPartitionsHelper<T>) PRESET_HELPERS.get(type.getRawType());
+ }
+
+ Iterable<KV<PartitionT, PartitionT>> calculateRanges(
+ PartitionT lowerBound, PartitionT upperBound, Long partitions);
+
+ @Override
+ void setParameters(KV<PartitionT, PartitionT> element, PreparedStatement preparedStatement);
+
+ @Override
+ KV<Long, KV<PartitionT, PartitionT>> mapRow(ResultSet resultSet) throws Exception;
+ }
+
/** Create partitions on a table. */
- static class PartitioningFn extends DoFn<KV<Integer, KV<Long, Long>>, KV<String, Long>> {
+ static class PartitioningFn<T> extends DoFn<KV<Long, KV<T, T>>, KV<T, T>> {
+ private static final Logger LOG = LoggerFactory.getLogger(PartitioningFn.class);
+ final TypeDescriptor<T> partitioningColumnType;
+
+ PartitioningFn(TypeDescriptor<T> partitioningColumnType) {
+ this.partitioningColumnType = partitioningColumnType;
+ }
+
@ProcessElement
public void processElement(ProcessContext c) {
- Integer numPartitions = c.element().getKey();
- Long lowerBound = c.element().getValue().getKey();
- Long upperBound = c.element().getValue().getValue();
- if (lowerBound > upperBound) {
- throw new RuntimeException(
- String.format(
- "Lower bound [%s] is higher than upper bound [%s]", lowerBound, upperBound));
- }
- long stride = (upperBound - lowerBound) / numPartitions + 1;
- for (long i = lowerBound; i < upperBound - stride; i += stride) {
- String range = String.format("%s,%s", i, i + stride);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
- }
- if (upperBound - lowerBound > stride * (numPartitions - 1)) {
- long indexFrom = (numPartitions - 1) * stride;
- long indexTo = upperBound + 1;
- String range = String.format("%s,%s", indexFrom, indexTo);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
+ T lowerBound = c.element().getValue().getKey();
+ T upperBound = c.element().getValue().getValue();
+ JdbcReadWithPartitionsHelper<T> helper =
+ JdbcReadWithPartitionsHelper.getPartitionsHelper(partitioningColumnType);
+ List<KV<T, T>> ranges =
+ Lists.newArrayList(helper.calculateRanges(lowerBound, upperBound, c.element().getKey()));
+ LOG.warn("Total of {} ranges: {}", ranges.size(), ranges);
+ for (KV<T, T> e : ranges) {
+ c.output(e);
}
}
}
+
+ public static final Map<Class<?>, JdbcReadWithPartitionsHelper<?>> PRESET_HELPERS =
+ ImmutableMap.of(
+ Long.class,
+ new JdbcReadWithPartitionsHelper<Long>() {
+ @Override
+ public Iterable<KV<Long, Long>> calculateRanges(
+ Long lowerBound, Long upperBound, Long partitions) {
+ List<KV<Long, Long>> ranges = new ArrayList<>();
+ // We divide by partitions FIRST to make sure that we can cover the whole LONG range.
+ // If we substract first, then we may end up with Long.MAX - Long.MIN, which is 2*MAX,
+ // and we'd have trouble with the pipeline.
+ long stride = (upperBound / partitions - lowerBound / partitions) + 1;
+ long highest = lowerBound;
+ for (long i = lowerBound; i < upperBound - stride; i += stride) {
+ ranges.add(KV.of(i, i + stride));
+ highest = i + stride;
+ }
+ if (upperBound - lowerBound > stride * (ranges.size() - 1)) {
+ long indexFrom = highest;
+ long indexTo = upperBound + 1;
+ ranges.add(KV.of(indexFrom, indexTo));
+ }
+ return ranges;
+ }
+
+ @Override
+ public void setParameters(KV<Long, Long> element, PreparedStatement preparedStatement) {
+ try {
+ preparedStatement.setLong(1, element.getKey());
+ preparedStatement.setLong(2, element.getValue());
+ } catch (SQLException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Override
+ public KV<Long, KV<Long, Long>> mapRow(ResultSet resultSet) throws Exception {
+ if (resultSet.getMetaData().getColumnCount() == 3) {
+ return KV.of(
+ resultSet.getLong(3), KV.of(resultSet.getLong(1), resultSet.getLong(2)));
+ } else {
+ return KV.of(0L, KV.of(resultSet.getLong(1), resultSet.getLong(2)));
+ }
+ }
+ },
+ DateTime.class,
+ new JdbcReadWithPartitionsHelper<DateTime>() {
+ @Override
+ public Iterable<KV<DateTime, DateTime>> calculateRanges(
+ DateTime lowerBound, DateTime upperBound, Long partitions) {
+ final List<KV<DateTime, DateTime>> result = new ArrayList<>();
+
+ final long intervalMillis = upperBound.getMillis() - lowerBound.getMillis();
+ final long strideMillis =
Review comment:
nitpick, how about
```java
final Duration stride = Duration.millis(Math.max(1, intervalMillis / partitions))
```
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
##########
@@ -333,31 +343,154 @@ private static Calendar withTimestampAndTimezone(DateTime dateTime) {
return calendar;
}
+ /**
+ * A helper for {@link ReadWithPartitions} that handles range calculations.
+ *
+ * @param <PartitionT>
+ */
+ interface JdbcReadWithPartitionsHelper<PartitionT>
+ extends PreparedStatementSetter<KV<PartitionT, PartitionT>>,
+ RowMapper<KV<Long, KV<PartitionT, PartitionT>>> {
+ static <T> JdbcReadWithPartitionsHelper<T> getPartitionsHelper(TypeDescriptor<T> type) {
+ // This cast is unchecked, thus this is a small type-checking risk. We just need
+ // to make sure that all preset helpers in `JdbcUtil.PRESET_HELPERS` are matched
+ // in type from their Key and their Value.
+ return (JdbcReadWithPartitionsHelper<T>) PRESET_HELPERS.get(type.getRawType());
+ }
+
+ Iterable<KV<PartitionT, PartitionT>> calculateRanges(
+ PartitionT lowerBound, PartitionT upperBound, Long partitions);
+
+ @Override
+ void setParameters(KV<PartitionT, PartitionT> element, PreparedStatement preparedStatement);
+
+ @Override
+ KV<Long, KV<PartitionT, PartitionT>> mapRow(ResultSet resultSet) throws Exception;
+ }
+
/** Create partitions on a table. */
- static class PartitioningFn extends DoFn<KV<Integer, KV<Long, Long>>, KV<String, Long>> {
+ static class PartitioningFn<T> extends DoFn<KV<Long, KV<T, T>>, KV<T, T>> {
+ private static final Logger LOG = LoggerFactory.getLogger(PartitioningFn.class);
+ final TypeDescriptor<T> partitioningColumnType;
+
+ PartitioningFn(TypeDescriptor<T> partitioningColumnType) {
+ this.partitioningColumnType = partitioningColumnType;
+ }
+
@ProcessElement
public void processElement(ProcessContext c) {
- Integer numPartitions = c.element().getKey();
- Long lowerBound = c.element().getValue().getKey();
- Long upperBound = c.element().getValue().getValue();
- if (lowerBound > upperBound) {
- throw new RuntimeException(
- String.format(
- "Lower bound [%s] is higher than upper bound [%s]", lowerBound, upperBound));
- }
- long stride = (upperBound - lowerBound) / numPartitions + 1;
- for (long i = lowerBound; i < upperBound - stride; i += stride) {
- String range = String.format("%s,%s", i, i + stride);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
- }
- if (upperBound - lowerBound > stride * (numPartitions - 1)) {
- long indexFrom = (numPartitions - 1) * stride;
- long indexTo = upperBound + 1;
- String range = String.format("%s,%s", indexFrom, indexTo);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
+ T lowerBound = c.element().getValue().getKey();
+ T upperBound = c.element().getValue().getValue();
+ JdbcReadWithPartitionsHelper<T> helper =
+ JdbcReadWithPartitionsHelper.getPartitionsHelper(partitioningColumnType);
+ List<KV<T, T>> ranges =
+ Lists.newArrayList(helper.calculateRanges(lowerBound, upperBound, c.element().getKey()));
+ LOG.warn("Total of {} ranges: {}", ranges.size(), ranges);
+ for (KV<T, T> e : ranges) {
+ c.output(e);
}
}
}
+
+ public static final Map<Class<?>, JdbcReadWithPartitionsHelper<?>> PRESET_HELPERS =
+ ImmutableMap.of(
+ Long.class,
+ new JdbcReadWithPartitionsHelper<Long>() {
+ @Override
+ public Iterable<KV<Long, Long>> calculateRanges(
+ Long lowerBound, Long upperBound, Long partitions) {
+ List<KV<Long, Long>> ranges = new ArrayList<>();
+ // We divide by partitions FIRST to make sure that we can cover the whole LONG range.
+ // If we substract first, then we may end up with Long.MAX - Long.MIN, which is 2*MAX,
+ // and we'd have trouble with the pipeline.
+ long stride = (upperBound / partitions - lowerBound / partitions) + 1;
+ long highest = lowerBound;
+ for (long i = lowerBound; i < upperBound - stride; i += stride) {
+ ranges.add(KV.of(i, i + stride));
+ highest = i + stride;
+ }
+ if (upperBound - lowerBound > stride * (ranges.size() - 1)) {
+ long indexFrom = highest;
+ long indexTo = upperBound + 1;
+ ranges.add(KV.of(indexFrom, indexTo));
+ }
Review comment:
Nitpick, would suggest to simplify as follows:
```
if(highest < upperBound + 1){
ranges.add(KV.of(highest, upperBound + 1));
}
##########
File path: sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java
##########
@@ -28,6 +44,66 @@
@RunWith(JUnit4.class)
public class JdbcUtilTest {
+ static final JdbcReadWithPartitionsHelper<String> PROTOTYPE_STRING_PARTITIONER =
Review comment:
Are you planning to park the code here? Would suggest to at least add a comment with the follow up ticket
##########
File path: sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOAutoPartitioningIT.java
##########
@@ -0,0 +1,403 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.jdbc;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.Objects;
+import java.util.Random;
+import javax.sql.DataSource;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.io.GenerateSequence;
+import org.apache.beam.sdk.io.common.DatabaseTestHelper;
+import org.apache.beam.sdk.io.jdbc.JdbcIO.RowMapper;
+import org.apache.beam.sdk.metrics.Distribution;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.schemas.JavaFieldSchema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.annotations.SchemaCreate;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.joda.time.DateTime;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TestRule;
+import org.junit.runner.Description;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.model.Statement;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.JdbcDatabaseContainer;
+import org.testcontainers.containers.MySQLContainer;
+import org.testcontainers.containers.PostgreSQLContainer;
+
+/** A test of {@link org.apache.beam.sdk.io.jdbc.JdbcIO} on test containers. */
+@RunWith(Parameterized.class)
+public class JdbcIOAutoPartitioningIT {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcIOAutoPartitioningIT.class);
+
+ public static final Integer NUM_ROWS = 1_000;
+ public static final String TABLE_NAME = "baseTable";
+
+ @ClassRule public static TestPipeline pipelineWrite = TestPipeline.create();
+ @Rule public TestPipeline pipelineRead = TestPipeline.create();
+
+ @Parameterized.Parameters(name = "{0}")
+ public static Iterable<String> params() {
+ return Lists.newArrayList("mysql", "postgres");
+ }
+
+ @Parameterized.Parameter(0)
+ public String dbms;
+
+ public static JdbcDatabaseContainer<?> getDb(String dbName) {
+ if (dbName.equals("mysql")) {
+ return mysql;
+ } else {
+ return postgres;
+ }
+ }
+
+ // We need to implement this retrying rule because we're running ~18 pipelines that connect to
+ // two databases in a few seconds. This may cause the databases to be overwhelmed, and reject
+ // connections or error out. By using this rule, we ensure that each pipeline is trued twice so
+ // that flakiness from databases being overwhelmed can be managed.
+ @Rule
+ public TestRule retryRule =
+ new TestRule() {
+ // We establish max number of retries at 2.
+ public final int maxRetries = 2;
+
+ @Override
+ public Statement apply(Statement base, Description description) {
+ return new Statement() {
+ @Override
+ public void evaluate() throws Throwable {
+ Throwable caughtThrowable = null;
+ // implement retry logic here
+ for (int i = 0; i < maxRetries; i++) {
+ try {
+ pipelineRead.apply(base, description);
+ // base.evaluate();
Review comment:
Be careful @pabloem , with this commented no tests are actually running. Everything just succeeds...
Almost missed this, but was caught by surprise to see the tests for string partitioning pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1032340091
Run PythonDocs PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (df9a12c) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.19%`.
> The diff coverage is `n/a`.
> :exclamation: Current head df9a12c differs from pull request most recent head 800a566. Consider uploading reports for the commit 800a566 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 51997 +42808
- Misses 9612 10171 +559
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/util/protox/any.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcHJvdG94L2FueS5nbw==) | | |
| [.../pkg/beam/runners/dataflow/dataflowlib/messages.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL21lc3NhZ2VzLmdv) | | |
| [...ks/go/pkg/beam/core/runtime/harness/cache\_hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9jYWNoZV9ob29rcy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/cogbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9jb2diay5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/symbols.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvc3ltYm9scy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy91dGlsLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/window.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy93aW5kb3cuZ28=) | | |
| [sdks/go/pkg/beam/io/bigqueryio/bigquery.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9iaWdxdWVyeWlvL2JpZ3F1ZXJ5Lmdv) | | |
| [sdks/go/pkg/beam/core/runtime/metricsx/urns.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvdXJucy5nbw==) | | |
| [sdks/go/pkg/beam/partition.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wYXJ0aXRpb24uZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...800a566](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (800a566) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52008 +42819
- Misses 9612 10160 +548
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/testing/passert/passert.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvcGFzc2VydC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/expand.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuZC5nbw==) | | |
| [sdks/go/pkg/beam/flatten.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9mbGF0dGVuLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/int.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2ludC5nbw==) | | |
| [sdks/go/pkg/beam/impulse.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pbXB1bHNlLmdv) | | |
| [sdks/go/pkg/beam/core/graph/graph.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2dyYXBoLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/double.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2RvdWJsZS5nbw==) | | |
| [sdks/go/pkg/beam/metrics.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9tZXRyaWNzLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/input.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9pbnB1dC5nbw==) | | |
| [sdks/go/pkg/beam/core/util/reflectx/tags.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvdGFncy5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...800a566](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0512486) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52004 +42815
- Misses 9612 10164 +552
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy91dGlsLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/varint.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3ZhcmludC5nbw==) | | |
| [sdks/go/pkg/beam/encoding.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9lbmNvZGluZy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/status.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zdGF0dXMuZ28=) | | |
| [sdks/go/pkg/beam/transforms/top/top.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3RvcC90b3Auc2hpbXMuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/coder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L2NvZGVyLmdv) | | |
| [sdks/go/pkg/beam/io/bigqueryio/bigquery.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9iaWdxdWVyeWlvL2JpZ3F1ZXJ5Lmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/fn\_arity.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbl9hcml0eS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/serialize.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NlcmlhbGl6ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/row\_decoder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3Jvd19kZWNvZGVyLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...0512486](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (47aa02a) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.60% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62233 +42446
===========================================
+ Hits 9189 52033 +42844
- Misses 9612 10200 +588
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/graph/coder/time.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3RpbWUuZ28=) | | |
| [sdks/go/pkg/beam/runners/direct/buffer.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9idWZmZXIuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/coderx/int.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L2ludC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/coderx/varint.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L3ZhcmludC5nbw==) | | |
| [sdks/go/pkg/beam/coder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb2Rlci5nbw==) | | |
| [sdks/go/pkg/beam/core/funcx/signature.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L3NpZ25hdHVyZS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/expand.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuZC5nbw==) | | |
| [sdks/go/pkg/beam/io/databaseio/database.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhYmFzZWlvL2RhdGFiYXNlLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/core/metrics/sampler.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL21ldHJpY3Mvc2FtcGxlci5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...47aa02a](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1034067113
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1033059876
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a change in pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r800633013
##########
File path: sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java
##########
@@ -41,4 +53,126 @@ public void testGetPreparedStatementSetCaller() throws Exception {
String expectedStmt = "INSERT INTO test_table(col1, col2, col3) VALUES(?, ?, ?)";
assertEquals(expectedStmt, generatedStmt);
}
+
+ @Test
+ public void testStringPartitioningWithSingleKeyFn() {
+ JdbcReadWithPartitionsHelper<String> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(TypeDescriptors.strings());
+ List<KV<String, String>> expectedRanges =
+ Lists.<KV<String, String>>newArrayList(KV.of("a", "b"));
+ List<KV<String, String>> ranges = Lists.newArrayList(helper.calculateRanges("a", "a", 10L));
+ // It is not possible to generate any more than one range, because the lower and upper range are
+ // exactly the same.
+ // The range is "a" to the very next element after it, which would be "a"+1 -> "b".
+ // Because the query's filter statement is : WHERE column >= lowerBound AND column < upperBound.
+ assertEquals(1, ranges.size());
+ assertArrayEquals(expectedRanges.toArray(), ranges.toArray());
+ }
+
+ @Test
+ public void testStringPartitioningWithSingleKeyMultiletterFn() {
+ JdbcReadWithPartitionsHelper<String> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(TypeDescriptors.strings());
+ List<KV<String, String>> expectedRanges =
+ Lists.<KV<String, String>>newArrayList(KV.of("afar", "afas"));
+ List<KV<String, String>> ranges =
+ Lists.newArrayList(helper.calculateRanges("afar", "afar", 10L));
+ // It is not possible to generate any more than one range, because the lower and upper range are
+ // exactly the same.
+ // The range is "afar" to the very next element after it, which would be "afar"+1 -> "afas".
+ // Because the query's filter statement is : WHERE column >= lowerBound AND column < upperBound.
+ assertEquals(1, ranges.size());
+ assertArrayEquals(expectedRanges.toArray(), ranges.toArray());
+ }
+
+ @Test
+ public void testStringPartitioningWithMultiletter() {
+ JdbcReadWithPartitionsHelper<String> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(TypeDescriptors.strings());
+ List<KV<String, String>> ranges =
+ Lists.newArrayList(helper.calculateRanges("afarisade", "zfastaridoteaf", 10L));
+ // The upper bound is "zfastaridoteaf" to the very next element after it, which would
+ // be "zfastaridoteaf"+1 -> "zfastaridoteaf".
+ // Because the query's filter statement is : WHERE column >= lowerBound AND column < upperBound.
+ assertEquals(13L, ranges.size());
+ assertThat(
+ ranges,
+ containsInAnyOrder(
+ KV.of("afarisade", "c"),
+ KV.of("c", "e"),
+ KV.of("e", "g"),
+ KV.of("g", "i"),
+ KV.of("i", "k"),
+ KV.of("k", "m"),
+ KV.of("m", "o"),
+ KV.of("o", "q"),
+ KV.of("q", "s"),
+ KV.of("s", "u"),
+ KV.of("u", "w"),
+ KV.of("w", "y"),
+ KV.of("y", "zfastaridoteag")));
+ }
+
+ @Test
+ public void testDatetimePartitioningWithSingleKey() {
+ JdbcReadWithPartitionsHelper<DateTime> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(
+ TypeDescriptor.of(DateTime.class));
+ DateTime onlyPoint = DateTime.now();
+ List<KV<DateTime, DateTime>> expectedRanges =
+ Lists.newArrayList(KV.of(onlyPoint, onlyPoint.plusMillis(1)));
+ List<KV<DateTime, DateTime>> ranges =
+ Lists.newArrayList(helper.calculateRanges(onlyPoint, onlyPoint, 10L));
+ // It is not possible to generate any more than one range, because the lower and upper range are
+ // exactly the same.
+ // The range goes from the current DateTime to ONE MILISECOND AFTER.
+ // Because the query's filter statement is : WHERE column >= lowerBound AND column < upperBound.
+ assertEquals(1, ranges.size());
+ assertArrayEquals(expectedRanges.toArray(), ranges.toArray());
+ }
+
+ @Test
+ public void testDatetimePartitioningWithMultiKey() {
+ JdbcReadWithPartitionsHelper<DateTime> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(
+ TypeDescriptor.of(DateTime.class));
+ DateTime lastPoint = DateTime.now();
+ // At least 10ms in the past, or more.
+ DateTime firstPoint = lastPoint.minusMillis(10 + new Random().nextInt(Integer.MAX_VALUE));
+ List<KV<DateTime, DateTime>> ranges =
+ Lists.newArrayList(helper.calculateRanges(firstPoint, lastPoint, 10L));
+ // DateTime ranges are able to work out 10-11 ranges because they split in miliseconds which is
+ // very small granularity.
+ assertThat(Double.valueOf(ranges.size()), closeTo(10, 1));
+ }
+
+ @Test
+ public void testLongPartitioningWithSingleKey() {
+ JdbcReadWithPartitionsHelper<Long> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(TypeDescriptors.longs());
+ List<KV<Long, Long>> expectedRanges = Lists.newArrayList(KV.of(12L, 13L));
+ List<KV<Long, Long>> ranges = Lists.newArrayList(helper.calculateRanges(12L, 12L, 10L));
+ // It is not possible to generate any more than one range, because the lower and upper range are
+ // exactly the same.
+ // The range goes from the current Long element to ONE ELEMENT AFTER.
+ // Because the query's filter statement is : WHERE column >= lowerBound AND column < upperBound.
+ assertEquals(1, ranges.size());
+ assertArrayEquals(expectedRanges.toArray(), ranges.toArray());
+ }
+
+ @Test
+ public void testLongPartitioningNotEnoughRanges() {
+ JdbcReadWithPartitionsHelper<Long> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(TypeDescriptors.longs());
+ // The minimum stride is one, which is what causes this sort of partitioning.
+ List<KV<Long, Long>> expectedRanges =
+ Lists.newArrayList(KV.of(12L, 14L), KV.of(14L, 16L), KV.of(16L, 18L), KV.of(18L, 21L));
+ List<KV<Long, Long>> ranges = Lists.newArrayList(helper.calculateRanges(12L, 20L, 10L));
+ // It is not possible to generate any more than one range, because the lower and upper range are
Review comment:
pls remove, comment is wrong for this case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a change in pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r800629040
##########
File path: sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java
##########
@@ -41,4 +53,126 @@ public void testGetPreparedStatementSetCaller() throws Exception {
String expectedStmt = "INSERT INTO test_table(col1, col2, col3) VALUES(?, ?, ?)";
assertEquals(expectedStmt, generatedStmt);
}
+
+ @Test
+ public void testStringPartitioningWithSingleKeyFn() {
+ JdbcReadWithPartitionsHelper<String> helper =
+ JdbcUtil.JdbcReadWithPartitionsHelper.getPartitionsHelper(TypeDescriptors.strings());
+ List<KV<String, String>> expectedRanges =
Review comment:
@pabloem Unfortunately the string partitions are wrong, for values of type string (of undefined length) a deterministic next value does not exist: `<= 'a'` is simply not the same as `< 'b'`.
Trying to do this might include values that should be filtered out (e.g. `aa` in the example above) and generate wrong results.
I'm personally still reluctant to support String partitions for already mentioned reasons. If you want to move forward with strings I would recommend to wrap boundary values in a small wrapper that contains an additional flag to specify if the value is inclusive or not. This way you can adjust your queries accordingly and treat intermediate partitions different from the very last one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (800a566) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52008 +42819
- Misses 9612 10160 +548
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/plan.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9wbGFuLmdv) | | |
| [sdks/go/pkg/beam/testing/passert/hash.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvaGFzaC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/data.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kYXRhLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvbWV0cmljc3guZ28=) | | |
| [sdks/go/pkg/beam/beam.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9iZWFtLnNoaW1zLmdv) | | |
| [sdks/go/pkg/beam/core/metrics/store.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL21ldHJpY3Mvc3RvcmUuZ28=) | | |
| [sdks/go/pkg/beam/pardo.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wYXJkby5nbw==) | | |
| [sdks/go/pkg/beam/core/util/protox/base64.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcHJvdG94L2Jhc2U2NC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/input.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9pbnB1dC5nbw==) | | |
| [sdks/go/pkg/beam/io/filesystem/memfs/memory.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL21lbWZzL21lbW9yeS5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...800a566](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (df9a12c) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.19%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 51997 +42808
- Misses 9612 10171 +559
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/window.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy93aW5kb3cuZ28=) | | |
| [...ks/go/pkg/beam/core/runtime/harness/cache\_hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9jYWNoZV9ob29rcy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/sdf\_invokers.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zZGZfaW52b2tlcnMuZ28=) | | |
| [sdks/go/pkg/beam/io/filesystem/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/coder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L2NvZGVyLmdv) | | |
| [sdks/go/pkg/beam/testing/passert/equals.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvZXF1YWxzLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/varint.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3ZhcmludC5nbw==) | | |
| [...o/pkg/beam/io/rtrackers/offsetrange/offsetrange.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9ydHJhY2tlcnMvb2Zmc2V0cmFuZ2Uvb2Zmc2V0cmFuZ2UuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/coder/time.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3RpbWUuZ28=) | | |
| [sdks/go/pkg/beam/core/funcx/signature.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L3NpZ25hdHVyZS5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...df9a12c](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (800a566) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52008 +42819
- Misses 9612 10160 +548
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/io/filesystem/gcs/gcs.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL2djcy9nY3MuZ28=) | | |
| [sdks/go/pkg/beam/runner.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXIuZ28=) | | |
| [sdks/go/pkg/beam/provision/provision.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wcm92aXNpb24vcHJvdmlzaW9uLmdv) | | |
| [sdks/go/pkg/beam/core/util/reflectx/call.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvY2FsbC5nbw==) | | |
| [...s/go/pkg/beam/core/runtime/graphx/schema/schema.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NjaGVtYS9zY2hlbWEuZ28=) | | |
| [sdks/go/pkg/beam/artifact/materialize.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9hcnRpZmFjdC9tYXRlcmlhbGl6ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/encode.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9lbmNvZGUuZ28=) | | |
| [sdks/go/pkg/beam/io/filesystem/memfs/memory.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL21lbWZzL21lbW9yeS5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/fn.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2ZuLmdv) | | |
| [sdks/go/pkg/beam/internal/errors/errors.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pbnRlcm5hbC9lcnJvcnMvZXJyb3JzLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...800a566](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (800a566) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52008 +42819
- Misses 9612 10160 +548
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/util/protox/any.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcHJvdG94L2FueS5nbw==) | | |
| [.../pkg/beam/runners/dataflow/dataflowlib/messages.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL21lc3NhZ2VzLmdv) | | |
| [...ks/go/pkg/beam/core/runtime/harness/cache\_hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9jYWNoZV9ob29rcy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/cogbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9jb2diay5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/symbols.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvc3ltYm9scy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy91dGlsLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/window.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy93aW5kb3cuZ28=) | | |
| [sdks/go/pkg/beam/io/bigqueryio/bigquery.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9iaWdxdWVyeWlvL2JpZ3F1ZXJ5Lmdv) | | |
| [sdks/go/pkg/beam/core/runtime/metricsx/urns.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvdXJucy5nbw==) | | |
| [sdks/go/pkg/beam/partition.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wYXJ0aXRpb24uZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...800a566](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (800a566) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52008 +42819
- Misses 9612 10160 +548
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/beam.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9iZWFtLnNoaW1zLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/hash.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9oYXNoLmdv) | | |
| [...ks/go/pkg/beam/core/runtime/coderx/coderx.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L2NvZGVyeC5zaGltcy5nbw==) | | |
| [sdks/go/pkg/beam/core/util/reflectx/call.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvY2FsbC5nbw==) | | |
| [sdks/go/pkg/beam/core/util/hooks/hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvaG9va3MvaG9va3MuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/datasink.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kYXRhc2luay5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/clone.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L2Nsb25lLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/translate.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3RyYW5zbGF0ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/funcx/signature.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L3NpZ25hdHVyZS5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/scope.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL3Njb3BlLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...800a566](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r798090176
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
##########
@@ -334,30 +342,188 @@ private static Calendar withTimestampAndTimezone(DateTime dateTime) {
}
/** Create partitions on a table. */
- static class PartitioningFn extends DoFn<KV<Integer, KV<Long, Long>>, KV<String, Long>> {
+ static class PartitioningFn<T> extends DoFn<KV<Integer, KV<T, T>>, KV<T, T>> {
+ private static final Logger LOG = LoggerFactory.getLogger(PartitioningFn.class);
+ final TypeDescriptor<T> partitioningColumnType;
+
+ PartitioningFn(TypeDescriptor<T> partitioningColumnType) {
+ this.partitioningColumnType = partitioningColumnType;
+ }
+
@ProcessElement
public void processElement(ProcessContext c) {
- Integer numPartitions = c.element().getKey();
- Long lowerBound = c.element().getValue().getKey();
- Long upperBound = c.element().getValue().getValue();
- if (lowerBound > upperBound) {
- throw new RuntimeException(
- String.format(
- "Lower bound [%s] is higher than upper bound [%s]", lowerBound, upperBound));
- }
- long stride = (upperBound - lowerBound) / numPartitions + 1;
- for (long i = lowerBound; i < upperBound - stride; i += stride) {
- String range = String.format("%s,%s", i, i + stride);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
- }
- if (upperBound - lowerBound > stride * (numPartitions - 1)) {
- long indexFrom = (numPartitions - 1) * stride;
- long indexTo = upperBound + 1;
- String range = String.format("%s,%s", indexFrom, indexTo);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
+ T lowerBound = c.element().getValue().getKey();
+ T upperBound = c.element().getValue().getValue();
+ JdbcReadWithPartitionsHelper<T> helper =
+ (JdbcReadWithPartitionsHelper<T>) PRESET_HELPERS.get(partitioningColumnType.getRawType());
+ List<KV<T, T>> ranges =
+ Lists.newArrayList(helper.calculateRanges(lowerBound, upperBound, c.element().getKey()));
+ LOG.warn("Total of {} ranges: {}", ranges.size(), ranges);
+ for (KV<T, T> e : ranges) {
+ c.output(e);
}
}
}
+
+ public static final Map<Class<?>, JdbcReadWithPartitionsHelper<?>> PRESET_HELPERS =
+ ImmutableMap.of(
+ String.class,
Review comment:
I agree that this will not partition ranges equally, but it's hard to partition intelligently without querying the database ahead of time.
I think the best strategy to improve the automatic partitioning will be to make ReadFn a splittable DoFn where a range can be partitioned at runtime. What do you think about that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1028468770
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r798585058
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -1071,74 +1128,157 @@ public void populateDisplayData(DisplayData.Builder builder) {
* strides for generated WHERE clause expressions used to split the column withPartitionColumn
* evenly. When the input is less than 1, the number is set to 1.
*/
- public ReadWithPartitions<T> withNumPartitions(int numPartitions) {
+ public ReadWithPartitions<T, PartitionColumnT> withNumPartitions(int numPartitions) {
checkArgument(numPartitions > 0, "numPartitions can not be less than 1");
return toBuilder().setNumPartitions(numPartitions).build();
}
/** The name of a column of numeric type that will be used for partitioning. */
- public ReadWithPartitions<T> withPartitionColumn(String partitionColumn) {
+ public ReadWithPartitions<T, PartitionColumnT> withPartitionColumn(String partitionColumn) {
checkNotNull(partitionColumn, "partitionColumn can not be null");
return toBuilder().setPartitionColumn(partitionColumn).build();
}
- public ReadWithPartitions<T> withLowerBound(Long lowerBound) {
+ /** The name of a column of numeric type that will be used for partitioning. */
Review comment:
copy paste? comment looks wrong here
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -196,31 +197,60 @@
* );
* }</code></pre>
*
- * <p>3. To read all data from a table in parallel with partitioning can be done with {@link
- * ReadWithPartitions}:
+ * <h4>Parallel reading from a JDBC datasource</h4>
+ *
+ * <p>Beam supports partitioned reading of all data from a table. Automatic partitioning is
+ * supported for a few data types: {@link Long}, {@link org.joda.time.DateTime}, {@link String}. To
+ * enable this, use {@link ReadWithPartitions}.
Review comment:
```suggestion
* enable this, use {@link JdbcIO#readWithPartitions(TypeDescriptor)}.
```
##########
File path: sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOAutoPartitioningIT.java
##########
@@ -0,0 +1,394 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.jdbc;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.Objects;
+import java.util.Random;
+import javax.sql.DataSource;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.io.GenerateSequence;
+import org.apache.beam.sdk.io.common.DatabaseTestHelper;
+import org.apache.beam.sdk.io.jdbc.JdbcIO.RowMapper;
+import org.apache.beam.sdk.metrics.Distribution;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.schemas.JavaFieldSchema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.annotations.SchemaCreate;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.joda.time.DateTime;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TestRule;
+import org.junit.runner.Description;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.model.Statement;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.JdbcDatabaseContainer;
+import org.testcontainers.containers.MySQLContainer;
+import org.testcontainers.containers.PostgreSQLContainer;
+
+/** A test of {@link org.apache.beam.sdk.io.jdbc.JdbcIO} on test containers. */
+@RunWith(Parameterized.class)
+public class JdbcIOAutoPartitioningIT {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcIOAutoPartitioningIT.class);
+
+ public static final Integer NUM_ROWS = 1_000;
+ public static final String TABLE_NAME = "baseTable";
+
+ @ClassRule public static TestPipeline pipelineWrite = TestPipeline.create();
+ @Rule public TestPipeline pipelineRead = TestPipeline.create();
+
+ @Parameterized.Parameters(name = "{0}")
+ public static Iterable<String> params() {
Review comment:
Awesome 🎉
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
##########
@@ -334,30 +342,188 @@ private static Calendar withTimestampAndTimezone(DateTime dateTime) {
}
/** Create partitions on a table. */
- static class PartitioningFn extends DoFn<KV<Integer, KV<Long, Long>>, KV<String, Long>> {
+ static class PartitioningFn<T> extends DoFn<KV<Integer, KV<T, T>>, KV<T, T>> {
+ private static final Logger LOG = LoggerFactory.getLogger(PartitioningFn.class);
+ final TypeDescriptor<T> partitioningColumnType;
+
+ PartitioningFn(TypeDescriptor<T> partitioningColumnType) {
+ this.partitioningColumnType = partitioningColumnType;
+ }
+
@ProcessElement
public void processElement(ProcessContext c) {
- Integer numPartitions = c.element().getKey();
- Long lowerBound = c.element().getValue().getKey();
- Long upperBound = c.element().getValue().getValue();
- if (lowerBound > upperBound) {
- throw new RuntimeException(
- String.format(
- "Lower bound [%s] is higher than upper bound [%s]", lowerBound, upperBound));
- }
- long stride = (upperBound - lowerBound) / numPartitions + 1;
- for (long i = lowerBound; i < upperBound - stride; i += stride) {
- String range = String.format("%s,%s", i, i + stride);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
- }
- if (upperBound - lowerBound > stride * (numPartitions - 1)) {
- long indexFrom = (numPartitions - 1) * stride;
- long indexTo = upperBound + 1;
- String range = String.format("%s,%s", indexFrom, indexTo);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
+ T lowerBound = c.element().getValue().getKey();
+ T upperBound = c.element().getValue().getValue();
+ JdbcReadWithPartitionsHelper<T> helper =
+ (JdbcReadWithPartitionsHelper<T>) PRESET_HELPERS.get(partitioningColumnType.getRawType());
+ List<KV<T, T>> ranges =
+ Lists.newArrayList(helper.calculateRanges(lowerBound, upperBound, c.element().getKey()));
+ LOG.warn("Total of {} ranges: {}", ranges.size(), ranges);
+ for (KV<T, T> e : ranges) {
+ c.output(e);
}
}
}
+
+ public static final Map<Class<?>, JdbcReadWithPartitionsHelper<?>> PRESET_HELPERS =
+ ImmutableMap.of(
+ String.class,
Review comment:
Interesting to think about SDFs in that context, being able to claim work requires a stable order for sure. So the index is really becoming a must ... But you could gather stats while querying and use that to repartition dynamically. That would be great in deed! :)
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -1071,74 +1128,157 @@ public void populateDisplayData(DisplayData.Builder builder) {
* strides for generated WHERE clause expressions used to split the column withPartitionColumn
* evenly. When the input is less than 1, the number is set to 1.
*/
- public ReadWithPartitions<T> withNumPartitions(int numPartitions) {
+ public ReadWithPartitions<T, PartitionColumnT> withNumPartitions(int numPartitions) {
checkArgument(numPartitions > 0, "numPartitions can not be less than 1");
return toBuilder().setNumPartitions(numPartitions).build();
}
/** The name of a column of numeric type that will be used for partitioning. */
- public ReadWithPartitions<T> withPartitionColumn(String partitionColumn) {
+ public ReadWithPartitions<T, PartitionColumnT> withPartitionColumn(String partitionColumn) {
checkNotNull(partitionColumn, "partitionColumn can not be null");
return toBuilder().setPartitionColumn(partitionColumn).build();
}
- public ReadWithPartitions<T> withLowerBound(Long lowerBound) {
+ /** The name of a column of numeric type that will be used for partitioning. */
+ public ReadWithPartitions<T, PartitionColumnT> withRowOutput() {
+ return toBuilder().setUseBeamSchema(true).build();
+ }
+
+ public ReadWithPartitions<T, PartitionColumnT> withLowerBound(PartitionColumnT lowerBound) {
return toBuilder().setLowerBound(lowerBound).build();
}
- public ReadWithPartitions<T> withUpperBound(Long upperBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withUpperBound(PartitionColumnT upperBound) {
return toBuilder().setUpperBound(upperBound).build();
}
/** Name of the table in the external database. Can be used to pass a user-defined subqery. */
- public ReadWithPartitions<T> withTable(String tableName) {
+ public ReadWithPartitions<T, PartitionColumnT> withTable(String tableName) {
checkNotNull(tableName, "table can not be null");
return toBuilder().setTable(tableName).build();
}
+ private static final int EQUAL = 0;
+
@Override
public PCollection<T> expand(PBegin input) {
checkNotNull(getRowMapper(), "withRowMapper() is required");
checkNotNull(
getDataSourceProviderFn(),
"withDataSourceConfiguration() or withDataSourceProviderFn() is required");
checkNotNull(getPartitionColumn(), "withPartitionColumn() is required");
- checkArgument(
Review comment:
I think this check should be kept and updated to `(getUpperBound() != null) == (getLowerBound() != null)` to make sure either both or no bounds are set ...? Otherwise `params` might be set incorrectly
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -999,69 +1045,80 @@ public void populateDisplayData(DisplayData.Builder builder) {
/** Implementation of {@link #readWithPartitions}. */
@AutoValue
- public abstract static class ReadWithPartitions<T> extends PTransform<PBegin, PCollection<T>> {
+ public abstract static class ReadWithPartitions<T, PartitionColumnT>
+ extends PTransform<PBegin, PCollection<T>> {
abstract @Nullable SerializableFunction<Void, DataSource> getDataSourceProviderFn();
abstract @Nullable RowMapper<T> getRowMapper();
abstract @Nullable Coder<T> getCoder();
- abstract Integer getNumPartitions();
+ abstract @Nullable Integer getNumPartitions();
abstract @Nullable String getPartitionColumn();
- abstract @Nullable Long getLowerBound();
+ abstract @Nullable Boolean getUseBeamSchema();
Review comment:
Any particular reason to have this Nullable / Boolean instead of using a primitive boolean?
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -196,31 +197,60 @@
* );
* }</code></pre>
*
- * <p>3. To read all data from a table in parallel with partitioning can be done with {@link
- * ReadWithPartitions}:
+ * <h4>Parallel reading from a JDBC datasource</h4>
+ *
+ * <p>Beam supports partitioned reading of all data from a table. Automatic partitioning is
+ * supported for a few data types: {@link Long}, {@link org.joda.time.DateTime}, {@link String}. To
+ * enable this, use {@link ReadWithPartitions}.
+ *
+ * <p>The partitioning scheme depends on these parameters, which can be user-provided, or
+ * automatically inferred by Beam (for the supported types):
+ *
+ * <ul>
+ * <li>Upper bound
+ * <li>Lower bound
+ * <li>Number of partitions - when auto-inferred, the number of partitions defaults to the square
+ * root of the number of rows divided by 5 (i.e.: {@code Math.floor(Math.sqrt(numRows) / 5)}).
+ * </ul>
+ *
+ * <p>To trigger auto-inference of these parameters, the user just needs to not provide them. To
+ * infer them automatically, Beam runs either of these statements:
+ *
+ * <ul>
+ * <li>{@code SELECT min(column), max(column), COUNT(*) from table} when none of the parameters is
+ * passed to the transform.
+ * <li>{@code SELECT min(column), max(column) from table} when only number of partitions is
+ * provided, but not upper or lower bounds.
+ * </ul>
+ *
+ * <p><b>Should I use this transform?</b> Consider using this transform in the following situations:
+ *
+ * <ul>
+ * <li>The partitioning column is indexed. This will help speed up the range queries
Review comment:
Is it worth mentioning that auto partitioning only works well if values of the partition column are more or less uniformly distributed?
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
##########
@@ -334,30 +342,188 @@ private static Calendar withTimestampAndTimezone(DateTime dateTime) {
}
/** Create partitions on a table. */
- static class PartitioningFn extends DoFn<KV<Integer, KV<Long, Long>>, KV<String, Long>> {
+ static class PartitioningFn<T> extends DoFn<KV<Integer, KV<T, T>>, KV<T, T>> {
+ private static final Logger LOG = LoggerFactory.getLogger(PartitioningFn.class);
+ final TypeDescriptor<T> partitioningColumnType;
+
+ PartitioningFn(TypeDescriptor<T> partitioningColumnType) {
+ this.partitioningColumnType = partitioningColumnType;
+ }
+
@ProcessElement
public void processElement(ProcessContext c) {
- Integer numPartitions = c.element().getKey();
- Long lowerBound = c.element().getValue().getKey();
- Long upperBound = c.element().getValue().getValue();
- if (lowerBound > upperBound) {
- throw new RuntimeException(
- String.format(
- "Lower bound [%s] is higher than upper bound [%s]", lowerBound, upperBound));
- }
- long stride = (upperBound - lowerBound) / numPartitions + 1;
- for (long i = lowerBound; i < upperBound - stride; i += stride) {
- String range = String.format("%s,%s", i, i + stride);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
- }
- if (upperBound - lowerBound > stride * (numPartitions - 1)) {
- long indexFrom = (numPartitions - 1) * stride;
- long indexTo = upperBound + 1;
- String range = String.format("%s,%s", indexFrom, indexTo);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
+ T lowerBound = c.element().getValue().getKey();
+ T upperBound = c.element().getValue().getValue();
+ JdbcReadWithPartitionsHelper<T> helper =
+ (JdbcReadWithPartitionsHelper<T>) PRESET_HELPERS.get(partitioningColumnType.getRawType());
+ List<KV<T, T>> ranges =
+ Lists.newArrayList(helper.calculateRanges(lowerBound, upperBound, c.element().getKey()));
+ LOG.warn("Total of {} ranges: {}", ranges.size(), ranges);
+ for (KV<T, T> e : ranges) {
+ c.output(e);
}
}
}
+
+ public static final Map<Class<?>, JdbcReadWithPartitionsHelper<?>> PRESET_HELPERS =
+ ImmutableMap.of(
+ String.class,
Review comment:
Yes, without querying significant stats from the DB it's going to be hard to do this well ...
I'm just wondering if user's aren't better off partitioning manually using `readAll` using their domain knowledge rather than offering something automatic we already know isn't going to perform well.
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
##########
@@ -333,31 +343,206 @@ private static Calendar withTimestampAndTimezone(DateTime dateTime) {
return calendar;
}
+ /**
+ * A helper for {@link ReadWithPartitions} that handles range calculations.
+ *
+ * @param <PartitionT>
+ */
+ interface JdbcReadWithPartitionsHelper<PartitionT>
+ extends PreparedStatementSetter<KV<PartitionT, PartitionT>>,
+ RowMapper<KV<Long, KV<PartitionT, PartitionT>>> {
+ Iterable<KV<PartitionT, PartitionT>> calculateRanges(
+ PartitionT lowerBound, PartitionT upperBound, Long partitions);
+
+ @Override
+ void setParameters(KV<PartitionT, PartitionT> element, PreparedStatement preparedStatement);
+
+ @Override
+ KV<Long, KV<PartitionT, PartitionT>> mapRow(ResultSet resultSet) throws Exception;
+ }
+
/** Create partitions on a table. */
- static class PartitioningFn extends DoFn<KV<Integer, KV<Long, Long>>, KV<String, Long>> {
+ static class PartitioningFn<T> extends DoFn<KV<Long, KV<T, T>>, KV<T, T>> {
+ private static final Logger LOG = LoggerFactory.getLogger(PartitioningFn.class);
+ final TypeDescriptor<T> partitioningColumnType;
+
+ PartitioningFn(TypeDescriptor<T> partitioningColumnType) {
+ this.partitioningColumnType = partitioningColumnType;
+ }
+
@ProcessElement
public void processElement(ProcessContext c) {
- Integer numPartitions = c.element().getKey();
- Long lowerBound = c.element().getValue().getKey();
- Long upperBound = c.element().getValue().getValue();
- if (lowerBound > upperBound) {
- throw new RuntimeException(
- String.format(
- "Lower bound [%s] is higher than upper bound [%s]", lowerBound, upperBound));
- }
- long stride = (upperBound - lowerBound) / numPartitions + 1;
- for (long i = lowerBound; i < upperBound - stride; i += stride) {
- String range = String.format("%s,%s", i, i + stride);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
- }
- if (upperBound - lowerBound > stride * (numPartitions - 1)) {
- long indexFrom = (numPartitions - 1) * stride;
- long indexTo = upperBound + 1;
- String range = String.format("%s,%s", indexFrom, indexTo);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
+ T lowerBound = c.element().getValue().getKey();
+ T upperBound = c.element().getValue().getValue();
+ JdbcReadWithPartitionsHelper<T> helper =
+ (JdbcReadWithPartitionsHelper<T>) PRESET_HELPERS.get(partitioningColumnType.getRawType());
+ List<KV<T, T>> ranges =
+ Lists.newArrayList(helper.calculateRanges(lowerBound, upperBound, c.element().getKey()));
+ LOG.warn("Total of {} ranges: {}", ranges.size(), ranges);
+ for (KV<T, T> e : ranges) {
+ c.output(e);
}
}
}
+
+ public static final Map<Class<?>, JdbcReadWithPartitionsHelper<?>> PRESET_HELPERS =
Review comment:
Too me it looks like unit tests have been quite abandoned for this IO, particularly this util class has almost no (direct) test coverage :( I know the partitioning fn is tested further down, but I believe it would be beneficial to have some decent tests for the partitioning logic / range calculations itself without having to run a pipeline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r798993247
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -1071,74 +1112,136 @@ public void populateDisplayData(DisplayData.Builder builder) {
* strides for generated WHERE clause expressions used to split the column withPartitionColumn
* evenly. When the input is less than 1, the number is set to 1.
*/
- public ReadWithPartitions<T> withNumPartitions(int numPartitions) {
+ public ReadWithPartitions<T, PartitionColumnT> withNumPartitions(int numPartitions) {
checkArgument(numPartitions > 0, "numPartitions can not be less than 1");
return toBuilder().setNumPartitions(numPartitions).build();
}
/** The name of a column of numeric type that will be used for partitioning. */
- public ReadWithPartitions<T> withPartitionColumn(String partitionColumn) {
+ public ReadWithPartitions<T, PartitionColumnT> withPartitionColumn(String partitionColumn) {
checkNotNull(partitionColumn, "partitionColumn can not be null");
return toBuilder().setPartitionColumn(partitionColumn).build();
}
- public ReadWithPartitions<T> withLowerBound(Long lowerBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withLowerBound(PartitionColumnT lowerBound) {
return toBuilder().setLowerBound(lowerBound).build();
}
- public ReadWithPartitions<T> withUpperBound(Long upperBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withUpperBound(PartitionColumnT upperBound) {
return toBuilder().setUpperBound(upperBound).build();
}
/** Name of the table in the external database. Can be used to pass a user-defined subqery. */
- public ReadWithPartitions<T> withTable(String tableName) {
+ public ReadWithPartitions<T, PartitionColumnT> withTable(String tableName) {
checkNotNull(tableName, "table can not be null");
return toBuilder().setTable(tableName).build();
}
+ private static final int EQUAL = 0;
+
@Override
public PCollection<T> expand(PBegin input) {
checkNotNull(getRowMapper(), "withRowMapper() is required");
checkNotNull(
getDataSourceProviderFn(),
"withDataSourceConfiguration() or withDataSourceProviderFn() is required");
checkNotNull(getPartitionColumn(), "withPartitionColumn() is required");
- checkArgument(
- getUpperBound() != null && getLowerBound() != null,
- "Upper and lower bounds are mandatory parameters for JdbcIO.readWithPartitions");
checkNotNull(getTable(), "withTable() is required");
+ if (getLowerBound() != null && getLowerBound() instanceof Comparable<?>) {
+ // Not all partition types are comparable. For example, LocalDateTime, which is a valid
+ // partitioning type, is not Comparable, so we can't enforce this for all sorts of
+ // partitioning.
+ checkArgument(
+ ((Comparable<PartitionColumnT>) getLowerBound()).compareTo(getUpperBound()) < EQUAL,
+ "The lower bound of partitioning column is larger or equal than the upper bound");
+ }
checkArgument(
- getLowerBound() < getUpperBound(),
- "The lower bound of partitioning column is larger or equal than the upper bound");
- checkArgument(
- getUpperBound() - getLowerBound() >= getNumPartitions(),
- "The specified number of partitions is more than the difference between upper bound and lower bound");
+ JdbcUtil.PRESET_HELPERS.containsKey(getPartitionColumnType().getRawType()),
+ "readWithPartitions only supports the following types: %s",
+ JdbcUtil.PRESET_HELPERS.keySet());
+
+ PCollection<KV<Integer, KV<PartitionColumnT, PartitionColumnT>>> params;
+
+ if (getLowerBound() == null && getUpperBound() == null) {
+ String query =
+ String.format(
+ "SELECT min(%s), max(%s) FROM %s",
+ getPartitionColumn(), getPartitionColumn(), getTable());
+ if (getNumPartitions() == null) {
+ query =
+ String.format(
+ "SELECT min(%s), max(%s), count(*) FROM %s",
+ getPartitionColumn(), getPartitionColumn(), getTable());
+ }
+ params =
+ input
+ .apply(
+ JdbcIO.<KV<Integer, KV<PartitionColumnT, PartitionColumnT>>>read()
+ .withQuery(query)
+ .withDataSourceProviderFn(getDataSourceProviderFn())
+ .withRowMapper(
+ (JdbcReadWithPartitionsHelper<PartitionColumnT>)
+ JdbcUtil.PRESET_HELPERS.get(getPartitionColumnType().getRawType())))
+ .apply(
+ MapElements.via(
+ new SimpleFunction<
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>>,
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>>>() {
+ @Override
+ public KV<Integer, KV<PartitionColumnT, PartitionColumnT>> apply(
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>> input) {
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>> result;
+ if (getNumPartitions() == null) {
+ // In this case, we use the table row count to infer a number of
+ // partitions.
+ // We take the square root of the number of rows, and divide it by 5
+ // to keep a relatively low number of partitions, given that an RDBMS
+ // cannot usually accept a very large number of connections.
+ Integer numPartitions =
+ Long.valueOf(
+ Math.round(Math.floor(Math.sqrt(input.getKey()) / 5)))
+ .intValue();
+ if (numPartitions == 0) {
+ numPartitions = 1;
+ }
+ result = KV.of(numPartitions, input.getValue());
+ } else {
+ result = KV.of(getNumPartitions(), input.getValue());
+ }
+ LOG.info(
+ "Inferred min: {} - max: {} - numPartitions: {}",
+ result.getValue().getKey(),
+ result.getValue().getValue(),
+ result.getKey());
+ return result;
+ }
+ }));
+ } else {
+ params =
+ input.apply(
+ Create.of(KV.of(getNumPartitions(), KV.of(getLowerBound(), getUpperBound()))));
+ }
- PCollection<KV<Integer, KV<Long, Long>>> params =
- input.apply(
- Create.of(
- Collections.singletonList(
- KV.of(getNumPartitions(), KV.of(getLowerBound(), getUpperBound())))));
- PCollection<KV<String, Iterable<Long>>> ranges =
+ PCollection<KV<PartitionColumnT, PartitionColumnT>> ranges =
params
- .apply("Partitioning", ParDo.of(new PartitioningFn()))
- .apply("Group partitions", GroupByKey.create());
+ .apply("Partitioning", ParDo.of(new PartitioningFn<>(getPartitionColumnType())))
+ .apply("Reshuffle partitions", Reshuffle.viaRandomKey());
- JdbcIO.ReadAll<KV<String, Iterable<Long>>, T> readAll =
- JdbcIO.<KV<String, Iterable<Long>>, T>readAll()
+ JdbcIO.ReadAll<KV<PartitionColumnT, PartitionColumnT>, T> readAll =
+ JdbcIO.<KV<PartitionColumnT, PartitionColumnT>, T>readAll()
.withDataSourceProviderFn(getDataSourceProviderFn())
.withQuery(
String.format(
"select * from %1$s where %2$s >= ? and %2$s < ?",
getTable(), getPartitionColumn()))
.withRowMapper(getRowMapper())
.withParameterSetter(
- (PreparedStatementSetter<KV<String, Iterable<Long>>>)
- (element, preparedStatement) -> {
- String[] range = element.getKey().split(",", -1);
- preparedStatement.setLong(1, Long.parseLong(range[0]));
- preparedStatement.setLong(2, Long.parseLong(range[1]));
- })
+ // This cast is unchecked, thus this is a small type-checking risk. We just need
Review comment:
done!
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -1071,74 +1128,157 @@ public void populateDisplayData(DisplayData.Builder builder) {
* strides for generated WHERE clause expressions used to split the column withPartitionColumn
* evenly. When the input is less than 1, the number is set to 1.
*/
- public ReadWithPartitions<T> withNumPartitions(int numPartitions) {
+ public ReadWithPartitions<T, PartitionColumnT> withNumPartitions(int numPartitions) {
checkArgument(numPartitions > 0, "numPartitions can not be less than 1");
return toBuilder().setNumPartitions(numPartitions).build();
}
/** The name of a column of numeric type that will be used for partitioning. */
- public ReadWithPartitions<T> withPartitionColumn(String partitionColumn) {
+ public ReadWithPartitions<T, PartitionColumnT> withPartitionColumn(String partitionColumn) {
checkNotNull(partitionColumn, "partitionColumn can not be null");
return toBuilder().setPartitionColumn(partitionColumn).build();
}
- public ReadWithPartitions<T> withLowerBound(Long lowerBound) {
+ /** The name of a column of numeric type that will be used for partitioning. */
Review comment:
fixed
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -999,69 +1045,80 @@ public void populateDisplayData(DisplayData.Builder builder) {
/** Implementation of {@link #readWithPartitions}. */
@AutoValue
- public abstract static class ReadWithPartitions<T> extends PTransform<PBegin, PCollection<T>> {
+ public abstract static class ReadWithPartitions<T, PartitionColumnT>
+ extends PTransform<PBegin, PCollection<T>> {
abstract @Nullable SerializableFunction<Void, DataSource> getDataSourceProviderFn();
abstract @Nullable RowMapper<T> getRowMapper();
abstract @Nullable Coder<T> getCoder();
- abstract Integer getNumPartitions();
+ abstract @Nullable Integer getNumPartitions();
abstract @Nullable String getPartitionColumn();
- abstract @Nullable Long getLowerBound();
+ abstract @Nullable Boolean getUseBeamSchema();
Review comment:
fixed
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -325,17 +325,24 @@ public static ReadRows readRows() {
*
* @param <T> Type of the data to be read.
*/
- public static <T> ReadWithPartitions<T> readWithPartitions() {
- return new AutoValue_JdbcIO_ReadWithPartitions.Builder<T>()
+ public static <T, PartitionColumnT> ReadWithPartitions<T, PartitionColumnT> readWithPartitions(
Review comment:
thanks!
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -196,31 +197,60 @@
* );
* }</code></pre>
*
- * <p>3. To read all data from a table in parallel with partitioning can be done with {@link
- * ReadWithPartitions}:
+ * <h4>Parallel reading from a JDBC datasource</h4>
+ *
+ * <p>Beam supports partitioned reading of all data from a table. Automatic partitioning is
+ * supported for a few data types: {@link Long}, {@link org.joda.time.DateTime}, {@link String}. To
+ * enable this, use {@link ReadWithPartitions}.
+ *
+ * <p>The partitioning scheme depends on these parameters, which can be user-provided, or
+ * automatically inferred by Beam (for the supported types):
+ *
+ * <ul>
+ * <li>Upper bound
+ * <li>Lower bound
+ * <li>Number of partitions - when auto-inferred, the number of partitions defaults to the square
+ * root of the number of rows divided by 5 (i.e.: {@code Math.floor(Math.sqrt(numRows) / 5)}).
+ * </ul>
+ *
+ * <p>To trigger auto-inference of these parameters, the user just needs to not provide them. To
+ * infer them automatically, Beam runs either of these statements:
+ *
+ * <ul>
+ * <li>{@code SELECT min(column), max(column), COUNT(*) from table} when none of the parameters is
+ * passed to the transform.
+ * <li>{@code SELECT min(column), max(column) from table} when only number of partitions is
+ * provided, but not upper or lower bounds.
+ * </ul>
+ *
+ * <p><b>Should I use this transform?</b> Consider using this transform in the following situations:
+ *
+ * <ul>
+ * <li>The partitioning column is indexed. This will help speed up the range queries
Review comment:
done!
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -196,31 +197,60 @@
* );
* }</code></pre>
*
- * <p>3. To read all data from a table in parallel with partitioning can be done with {@link
- * ReadWithPartitions}:
+ * <h4>Parallel reading from a JDBC datasource</h4>
+ *
+ * <p>Beam supports partitioned reading of all data from a table. Automatic partitioning is
+ * supported for a few data types: {@link Long}, {@link org.joda.time.DateTime}, {@link String}. To
+ * enable this, use {@link ReadWithPartitions}.
Review comment:
I've included this change in my new commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (800a566) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52008 +42819
- Misses 9612 10160 +548
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/funcx/output.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L291dHB1dC5nbw==) | | |
| [sdks/go/pkg/beam/pcollection.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wY29sbGVjdGlvbi5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/harness/statemgr.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZW1nci5nbw==) | | |
| [...s/go/pkg/beam/core/runtime/harness/sampler\_hook.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zYW1wbGVyX2hvb2suZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/harness/datamgr.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9kYXRhbWdyLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/dataflow.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L2RhdGFmbG93Lmdv) | | |
| [...o/pkg/beam/runners/dataflow/dataflowlib/metrics.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL21ldHJpY3MuZ28=) | | |
| [sdks/go/pkg/beam/option.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9vcHRpb24uZ28=) | | |
| [sdks/go/pkg/beam/metrics.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9tZXRyaWNzLmdv) | | |
| [sdks/go/pkg/beam/io/synthetic/source.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9zeW50aGV0aWMvc291cmNlLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...800a566](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f155b81) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.61% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62252 +42465
===========================================
+ Hits 9189 52050 +42861
- Misses 9612 10202 +590
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/util/reflectx/calls.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvY2FsbHMuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/fn\_arity.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbl9hcml0eS5nbw==) | | |
| [sdks/go/pkg/beam/runners/direct/direct.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9kaXJlY3QuZ28=) | | |
| [...o/pkg/beam/runners/dataflow/dataflowlib/execute.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL2V4ZWN1dGUuZ28=) | | |
| [sdks/go/pkg/beam/combine.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb21iaW5lLmdv) | | |
| [.../pkg/beam/runners/dataflow/dataflowlib/messages.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL21lc3NhZ2VzLmdv) | | |
| [sdks/go/pkg/beam/pardo.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wYXJkby5nbw==) | | |
| [sdks/go/pkg/beam/core/funcx/signature.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L3NpZ25hdHVyZS5nbw==) | | |
| [sdks/go/pkg/beam/testing/passert/passert.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvcGFzc2VydC5zaGltcy5nbw==) | | |
| [...ks/go/pkg/beam/core/runtime/harness/cache\_hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9jYWNoZV9ob29rcy5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...f155b81](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f155b81) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.61% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62252 +42465
===========================================
+ Hits 9189 52050 +42861
- Misses 9612 10202 +590
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/harness/harness.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9oYXJuZXNzLmdv) | | |
| [sdks/go/pkg/beam/impulse.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pbXB1bHNlLmdv) | | |
| [sdks/go/pkg/beam/forward.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9mb3J3YXJkLmdv) | | |
| [sdks/go/pkg/beam/testing/passert/floats.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvZmxvYXRzLmdv) | | |
| [sdks/go/pkg/beam/core/graph/bind.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2JpbmQuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/encode.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9lbmNvZGUuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/pcollection.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9wY29sbGVjdGlvbi5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/harness/session.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zZXNzaW9uLmdv) | | |
| [sdks/go/pkg/beam/beam.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9iZWFtLnNoaW1zLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/tree.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3RyZWUuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...f155b81](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r799871125
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
##########
@@ -333,31 +343,206 @@ private static Calendar withTimestampAndTimezone(DateTime dateTime) {
return calendar;
}
+ /**
+ * A helper for {@link ReadWithPartitions} that handles range calculations.
+ *
+ * @param <PartitionT>
+ */
+ interface JdbcReadWithPartitionsHelper<PartitionT>
+ extends PreparedStatementSetter<KV<PartitionT, PartitionT>>,
+ RowMapper<KV<Long, KV<PartitionT, PartitionT>>> {
+ Iterable<KV<PartitionT, PartitionT>> calculateRanges(
+ PartitionT lowerBound, PartitionT upperBound, Long partitions);
+
+ @Override
+ void setParameters(KV<PartitionT, PartitionT> element, PreparedStatement preparedStatement);
+
+ @Override
+ KV<Long, KV<PartitionT, PartitionT>> mapRow(ResultSet resultSet) throws Exception;
+ }
+
/** Create partitions on a table. */
- static class PartitioningFn extends DoFn<KV<Integer, KV<Long, Long>>, KV<String, Long>> {
+ static class PartitioningFn<T> extends DoFn<KV<Long, KV<T, T>>, KV<T, T>> {
+ private static final Logger LOG = LoggerFactory.getLogger(PartitioningFn.class);
+ final TypeDescriptor<T> partitioningColumnType;
+
+ PartitioningFn(TypeDescriptor<T> partitioningColumnType) {
+ this.partitioningColumnType = partitioningColumnType;
+ }
+
@ProcessElement
public void processElement(ProcessContext c) {
- Integer numPartitions = c.element().getKey();
- Long lowerBound = c.element().getValue().getKey();
- Long upperBound = c.element().getValue().getValue();
- if (lowerBound > upperBound) {
- throw new RuntimeException(
- String.format(
- "Lower bound [%s] is higher than upper bound [%s]", lowerBound, upperBound));
- }
- long stride = (upperBound - lowerBound) / numPartitions + 1;
- for (long i = lowerBound; i < upperBound - stride; i += stride) {
- String range = String.format("%s,%s", i, i + stride);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
- }
- if (upperBound - lowerBound > stride * (numPartitions - 1)) {
- long indexFrom = (numPartitions - 1) * stride;
- long indexTo = upperBound + 1;
- String range = String.format("%s,%s", indexFrom, indexTo);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
+ T lowerBound = c.element().getValue().getKey();
+ T upperBound = c.element().getValue().getValue();
+ JdbcReadWithPartitionsHelper<T> helper =
+ (JdbcReadWithPartitionsHelper<T>) PRESET_HELPERS.get(partitioningColumnType.getRawType());
+ List<KV<T, T>> ranges =
+ Lists.newArrayList(helper.calculateRanges(lowerBound, upperBound, c.element().getKey()));
+ LOG.warn("Total of {} ranges: {}", ranges.size(), ranges);
+ for (KV<T, T> e : ranges) {
+ c.output(e);
}
}
}
+
+ public static final Map<Class<?>, JdbcReadWithPartitionsHelper<?>> PRESET_HELPERS =
Review comment:
Added some unit tests for the partitiones specifically. They helped me find a few bugs in the implementation : ) thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r801980448
##########
File path: sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOAutoPartitioningIT.java
##########
@@ -0,0 +1,403 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.jdbc;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.Objects;
+import java.util.Random;
+import javax.sql.DataSource;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.io.GenerateSequence;
+import org.apache.beam.sdk.io.common.DatabaseTestHelper;
+import org.apache.beam.sdk.io.jdbc.JdbcIO.RowMapper;
+import org.apache.beam.sdk.metrics.Distribution;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.schemas.JavaFieldSchema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.annotations.SchemaCreate;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.joda.time.DateTime;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TestRule;
+import org.junit.runner.Description;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.model.Statement;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.JdbcDatabaseContainer;
+import org.testcontainers.containers.MySQLContainer;
+import org.testcontainers.containers.PostgreSQLContainer;
+
+/** A test of {@link org.apache.beam.sdk.io.jdbc.JdbcIO} on test containers. */
+@RunWith(Parameterized.class)
+public class JdbcIOAutoPartitioningIT {
+ private static final Logger LOG = LoggerFactory.getLogger(JdbcIOAutoPartitioningIT.class);
+
+ public static final Integer NUM_ROWS = 1_000;
+ public static final String TABLE_NAME = "baseTable";
+
+ @ClassRule public static TestPipeline pipelineWrite = TestPipeline.create();
+ @Rule public TestPipeline pipelineRead = TestPipeline.create();
+
+ @Parameterized.Parameters(name = "{0}")
+ public static Iterable<String> params() {
+ return Lists.newArrayList("mysql", "postgres");
+ }
+
+ @Parameterized.Parameter(0)
+ public String dbms;
+
+ public static JdbcDatabaseContainer<?> getDb(String dbName) {
+ if (dbName.equals("mysql")) {
+ return mysql;
+ } else {
+ return postgres;
+ }
+ }
+
+ // We need to implement this retrying rule because we're running ~18 pipelines that connect to
+ // two databases in a few seconds. This may cause the databases to be overwhelmed, and reject
+ // connections or error out. By using this rule, we ensure that each pipeline is trued twice so
+ // that flakiness from databases being overwhelmed can be managed.
+ @Rule
+ public TestRule retryRule =
+ new TestRule() {
+ // We establish max number of retries at 2.
+ public final int maxRetries = 2;
+
+ @Override
+ public Statement apply(Statement base, Description description) {
+ return new Statement() {
+ @Override
+ public void evaluate() throws Throwable {
+ Throwable caughtThrowable = null;
+ // implement retry logic here
+ for (int i = 0; i < maxRetries; i++) {
+ try {
+ pipelineRead.apply(base, description);
+ // base.evaluate();
Review comment:
oh wow incredible catch! thanks so much @mosche - I am so embarrassed that I had left this like that. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (014888e) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.19%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19%
===========================================
Files 201 452 +251
Lines 19787 62175 +42388
===========================================
+ Hits 9189 52002 +42813
- Misses 9612 10173 +561
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/graphx/translate.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3RyYW5zbGF0ZS5nbw==) | | |
| [sdks/go/pkg/beam/runners/direct/impulse.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9pbXB1bHNlLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/harness/logging.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9sb2dnaW5nLmdv) | | |
| [sdks/go/pkg/beam/io/textio/textio.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby90ZXh0aW8vdGV4dGlvLmdv) | | |
| [...o/pkg/beam/runners/dataflow/dataflowlib/metrics.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL21ldHJpY3MuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/coder/bytes.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2J5dGVzLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/data.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kYXRhLmdv) | | |
| [sdks/go/pkg/beam/core/funcx/output.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L291dHB1dC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/sdf.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zZGYuZ28=) | | |
| [sdks/go/pkg/beam/testing/passert/passert.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvcGFzc2VydC5zaGltcy5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...014888e](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (47aa02a) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.60% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62233 +42446
===========================================
+ Hits 9189 52033 +42844
- Misses 9612 10200 +588
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/runners/direct/gbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9nYmsuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/max.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21heC5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/bind.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2JpbmQuZ28=) | | |
| [sdks/go/pkg/beam/combine.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb21iaW5lLmdv) | | |
| [sdks/go/pkg/beam/beam.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9iZWFtLnNoaW1zLmdv) | | |
| [sdks/go/pkg/beam/transforms/filter/filter.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL2ZpbHRlci9maWx0ZXIuZ28=) | | |
| [sdks/go/pkg/beam/util/gcsx/gcs.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL2djc3gvZ2NzLmdv) | | |
| [...dks/go/pkg/beam/core/runtime/harness/monitoring.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9tb25pdG9yaW5nLmdv) | | |
| [sdks/go/pkg/beam/transforms/top/top.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3RvcC90b3Auc2hpbXMuZ28=) | | |
| [sdks/go/pkg/beam/io/databaseio/database.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhYmFzZWlvL2RhdGFiYXNlLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...47aa02a](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1033125770
PostComit tests ran: https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/926/testReport/org.apache.beam.sdk.io.jdbc/JdbcIOAutoPartitioningIT/
You can see that they ran by reviewing logs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1024448196
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r798106399
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -1071,74 +1112,136 @@ public void populateDisplayData(DisplayData.Builder builder) {
* strides for generated WHERE clause expressions used to split the column withPartitionColumn
* evenly. When the input is less than 1, the number is set to 1.
*/
- public ReadWithPartitions<T> withNumPartitions(int numPartitions) {
+ public ReadWithPartitions<T, PartitionColumnT> withNumPartitions(int numPartitions) {
checkArgument(numPartitions > 0, "numPartitions can not be less than 1");
return toBuilder().setNumPartitions(numPartitions).build();
}
/** The name of a column of numeric type that will be used for partitioning. */
- public ReadWithPartitions<T> withPartitionColumn(String partitionColumn) {
+ public ReadWithPartitions<T, PartitionColumnT> withPartitionColumn(String partitionColumn) {
checkNotNull(partitionColumn, "partitionColumn can not be null");
return toBuilder().setPartitionColumn(partitionColumn).build();
}
- public ReadWithPartitions<T> withLowerBound(Long lowerBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withLowerBound(PartitionColumnT lowerBound) {
return toBuilder().setLowerBound(lowerBound).build();
}
- public ReadWithPartitions<T> withUpperBound(Long upperBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withUpperBound(PartitionColumnT upperBound) {
return toBuilder().setUpperBound(upperBound).build();
}
/** Name of the table in the external database. Can be used to pass a user-defined subqery. */
- public ReadWithPartitions<T> withTable(String tableName) {
+ public ReadWithPartitions<T, PartitionColumnT> withTable(String tableName) {
checkNotNull(tableName, "table can not be null");
return toBuilder().setTable(tableName).build();
}
+ private static final int EQUAL = 0;
+
@Override
public PCollection<T> expand(PBegin input) {
checkNotNull(getRowMapper(), "withRowMapper() is required");
checkNotNull(
getDataSourceProviderFn(),
"withDataSourceConfiguration() or withDataSourceProviderFn() is required");
checkNotNull(getPartitionColumn(), "withPartitionColumn() is required");
- checkArgument(
- getUpperBound() != null && getLowerBound() != null,
- "Upper and lower bounds are mandatory parameters for JdbcIO.readWithPartitions");
checkNotNull(getTable(), "withTable() is required");
+ if (getLowerBound() != null && getLowerBound() instanceof Comparable<?>) {
Review comment:
I also like `withBounds`. Unfortunately I suppose it's too late now : ) : (
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r798994206
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
##########
@@ -333,31 +343,206 @@ private static Calendar withTimestampAndTimezone(DateTime dateTime) {
return calendar;
}
+ /**
+ * A helper for {@link ReadWithPartitions} that handles range calculations.
+ *
+ * @param <PartitionT>
+ */
+ interface JdbcReadWithPartitionsHelper<PartitionT>
+ extends PreparedStatementSetter<KV<PartitionT, PartitionT>>,
+ RowMapper<KV<Long, KV<PartitionT, PartitionT>>> {
+ Iterable<KV<PartitionT, PartitionT>> calculateRanges(
+ PartitionT lowerBound, PartitionT upperBound, Long partitions);
+
+ @Override
+ void setParameters(KV<PartitionT, PartitionT> element, PreparedStatement preparedStatement);
+
+ @Override
+ KV<Long, KV<PartitionT, PartitionT>> mapRow(ResultSet resultSet) throws Exception;
+ }
+
/** Create partitions on a table. */
- static class PartitioningFn extends DoFn<KV<Integer, KV<Long, Long>>, KV<String, Long>> {
+ static class PartitioningFn<T> extends DoFn<KV<Long, KV<T, T>>, KV<T, T>> {
+ private static final Logger LOG = LoggerFactory.getLogger(PartitioningFn.class);
+ final TypeDescriptor<T> partitioningColumnType;
+
+ PartitioningFn(TypeDescriptor<T> partitioningColumnType) {
+ this.partitioningColumnType = partitioningColumnType;
+ }
+
@ProcessElement
public void processElement(ProcessContext c) {
- Integer numPartitions = c.element().getKey();
- Long lowerBound = c.element().getValue().getKey();
- Long upperBound = c.element().getValue().getValue();
- if (lowerBound > upperBound) {
- throw new RuntimeException(
- String.format(
- "Lower bound [%s] is higher than upper bound [%s]", lowerBound, upperBound));
- }
- long stride = (upperBound - lowerBound) / numPartitions + 1;
- for (long i = lowerBound; i < upperBound - stride; i += stride) {
- String range = String.format("%s,%s", i, i + stride);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
- }
- if (upperBound - lowerBound > stride * (numPartitions - 1)) {
- long indexFrom = (numPartitions - 1) * stride;
- long indexTo = upperBound + 1;
- String range = String.format("%s,%s", indexFrom, indexTo);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
+ T lowerBound = c.element().getValue().getKey();
+ T upperBound = c.element().getValue().getValue();
+ JdbcReadWithPartitionsHelper<T> helper =
+ (JdbcReadWithPartitionsHelper<T>) PRESET_HELPERS.get(partitioningColumnType.getRawType());
+ List<KV<T, T>> ranges =
+ Lists.newArrayList(helper.calculateRanges(lowerBound, upperBound, c.element().getKey()));
+ LOG.warn("Total of {} ranges: {}", ranges.size(), ranges);
+ for (KV<T, T> e : ranges) {
+ c.output(e);
}
}
}
+
+ public static final Map<Class<?>, JdbcReadWithPartitionsHelper<?>> PRESET_HELPERS =
Review comment:
there's a few unit tests that test the PartitioningFn, which in turn uses these helpers - these are in JdbcIOTest - wdyt?
I'm happy to add a few more if you think it would be valuable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e87c4bb) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62161 +42374
===========================================
+ Hits 9189 51998 +42809
- Misses 9612 10163 +551
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/io/filesystem/gcs/gcs.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL2djcy9nY3MuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/init.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaW5pdC5nbw==) | | |
| [sdks/go/pkg/beam/gbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9nYmsuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/sum.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL3N1bS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/coder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L2NvZGVyLmdv) | | |
| [sdks/go/pkg/beam/core/graph/edge.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2VkZ2UuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy91dGlsLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/bytes.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2J5dGVzLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/coder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2NvZGVyLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/coder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9jb2Rlci5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...e87c4bb](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r798090176
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
##########
@@ -334,30 +342,188 @@ private static Calendar withTimestampAndTimezone(DateTime dateTime) {
}
/** Create partitions on a table. */
- static class PartitioningFn extends DoFn<KV<Integer, KV<Long, Long>>, KV<String, Long>> {
+ static class PartitioningFn<T> extends DoFn<KV<Integer, KV<T, T>>, KV<T, T>> {
+ private static final Logger LOG = LoggerFactory.getLogger(PartitioningFn.class);
+ final TypeDescriptor<T> partitioningColumnType;
+
+ PartitioningFn(TypeDescriptor<T> partitioningColumnType) {
+ this.partitioningColumnType = partitioningColumnType;
+ }
+
@ProcessElement
public void processElement(ProcessContext c) {
- Integer numPartitions = c.element().getKey();
- Long lowerBound = c.element().getValue().getKey();
- Long upperBound = c.element().getValue().getValue();
- if (lowerBound > upperBound) {
- throw new RuntimeException(
- String.format(
- "Lower bound [%s] is higher than upper bound [%s]", lowerBound, upperBound));
- }
- long stride = (upperBound - lowerBound) / numPartitions + 1;
- for (long i = lowerBound; i < upperBound - stride; i += stride) {
- String range = String.format("%s,%s", i, i + stride);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
- }
- if (upperBound - lowerBound > stride * (numPartitions - 1)) {
- long indexFrom = (numPartitions - 1) * stride;
- long indexTo = upperBound + 1;
- String range = String.format("%s,%s", indexFrom, indexTo);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
+ T lowerBound = c.element().getValue().getKey();
+ T upperBound = c.element().getValue().getValue();
+ JdbcReadWithPartitionsHelper<T> helper =
+ (JdbcReadWithPartitionsHelper<T>) PRESET_HELPERS.get(partitioningColumnType.getRawType());
+ List<KV<T, T>> ranges =
+ Lists.newArrayList(helper.calculateRanges(lowerBound, upperBound, c.element().getKey()));
+ LOG.warn("Total of {} ranges: {}", ranges.size(), ranges);
+ for (KV<T, T> e : ranges) {
+ c.output(e);
}
}
}
+
+ public static final Map<Class<?>, JdbcReadWithPartitionsHelper<?>> PRESET_HELPERS =
+ ImmutableMap.of(
+ String.class,
Review comment:
I agree that this will not partition ranges equally, but it's hard to partition intelligently without querying the database ahead of time.
I think the best strategy to improve the automatic partitioning will be to make ReadFn a splittable DoFn where a range can be partitioned at runtime. What do you think about that? (in a follow-up change)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1029488107
regarding your comment in https://github.com/apache/beam/pull/15848#discussion_r798790351
yeah, SDF is interesting here, and I am not 100% sure how to approach it - but yes, we need ordering, so the implementation would be something like:
```
resultset = query.execute();
while(true) {
resultset.next();
if(!tryClaim(resultSet.get(key))) {
return DONE;
}
c.output(format(resultSet));
}
```
and the split would be something like:
```
tryClaim(key) {
this.latestKey = key
}
trySplit() {
ranges = generateRanges(2, latestKey, endOfRange) // Generate two ranges between the current key and the last key
return new SplitResult(ranges.currentRange, ranges.nextRange);
}
```
The issue is that the query with the full range would be executing in the database, so we may duplicate work on the database - so I'm not sure how bad this is or not : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e87c4bb) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62161 +42374
===========================================
+ Hits 9189 51998 +42809
- Misses 9612 10163 +551
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/impulse.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pbXB1bHNlLmdv) | | |
| [sdks/go/pkg/beam/runners/dataflow/dataflow.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93Lmdv) | | |
| [...ks/go/pkg/beam/core/runtime/coderx/coderx.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L2NvZGVyeC5zaGltcy5nbw==) | | |
| [sdks/go/pkg/beam/core/util/protox/base64.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcHJvdG94L2Jhc2U2NC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/serialize.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NlcmlhbGl6ZS5nbw==) | | |
| [.../go/pkg/beam/runners/dataflow/dataflowlib/stage.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL3N0YWdlLmdv) | | |
| [...s/go/pkg/beam/core/runtime/graphx/schema/schema.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NjaGVtYS9zY2hlbWEuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/coder/bytes.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2J5dGVzLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/time.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3RpbWUuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/expand.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuZC5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...e87c4bb](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r799014490
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -1071,74 +1128,157 @@ public void populateDisplayData(DisplayData.Builder builder) {
* strides for generated WHERE clause expressions used to split the column withPartitionColumn
* evenly. When the input is less than 1, the number is set to 1.
*/
- public ReadWithPartitions<T> withNumPartitions(int numPartitions) {
+ public ReadWithPartitions<T, PartitionColumnT> withNumPartitions(int numPartitions) {
checkArgument(numPartitions > 0, "numPartitions can not be less than 1");
return toBuilder().setNumPartitions(numPartitions).build();
}
/** The name of a column of numeric type that will be used for partitioning. */
- public ReadWithPartitions<T> withPartitionColumn(String partitionColumn) {
+ public ReadWithPartitions<T, PartitionColumnT> withPartitionColumn(String partitionColumn) {
checkNotNull(partitionColumn, "partitionColumn can not be null");
return toBuilder().setPartitionColumn(partitionColumn).build();
}
- public ReadWithPartitions<T> withLowerBound(Long lowerBound) {
+ /** The name of a column of numeric type that will be used for partitioning. */
+ public ReadWithPartitions<T, PartitionColumnT> withRowOutput() {
+ return toBuilder().setUseBeamSchema(true).build();
+ }
+
+ public ReadWithPartitions<T, PartitionColumnT> withLowerBound(PartitionColumnT lowerBound) {
return toBuilder().setLowerBound(lowerBound).build();
}
- public ReadWithPartitions<T> withUpperBound(Long upperBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withUpperBound(PartitionColumnT upperBound) {
return toBuilder().setUpperBound(upperBound).build();
}
/** Name of the table in the external database. Can be used to pass a user-defined subqery. */
- public ReadWithPartitions<T> withTable(String tableName) {
+ public ReadWithPartitions<T, PartitionColumnT> withTable(String tableName) {
checkNotNull(tableName, "table can not be null");
return toBuilder().setTable(tableName).build();
}
+ private static final int EQUAL = 0;
+
@Override
public PCollection<T> expand(PBegin input) {
checkNotNull(getRowMapper(), "withRowMapper() is required");
checkNotNull(
getDataSourceProviderFn(),
"withDataSourceConfiguration() or withDataSourceProviderFn() is required");
checkNotNull(getPartitionColumn(), "withPartitionColumn() is required");
- checkArgument(
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r801977527
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
##########
@@ -333,31 +343,154 @@ private static Calendar withTimestampAndTimezone(DateTime dateTime) {
return calendar;
}
+ /**
+ * A helper for {@link ReadWithPartitions} that handles range calculations.
+ *
+ * @param <PartitionT>
+ */
+ interface JdbcReadWithPartitionsHelper<PartitionT>
+ extends PreparedStatementSetter<KV<PartitionT, PartitionT>>,
+ RowMapper<KV<Long, KV<PartitionT, PartitionT>>> {
+ static <T> JdbcReadWithPartitionsHelper<T> getPartitionsHelper(TypeDescriptor<T> type) {
+ // This cast is unchecked, thus this is a small type-checking risk. We just need
+ // to make sure that all preset helpers in `JdbcUtil.PRESET_HELPERS` are matched
+ // in type from their Key and their Value.
+ return (JdbcReadWithPartitionsHelper<T>) PRESET_HELPERS.get(type.getRawType());
+ }
+
+ Iterable<KV<PartitionT, PartitionT>> calculateRanges(
+ PartitionT lowerBound, PartitionT upperBound, Long partitions);
+
+ @Override
+ void setParameters(KV<PartitionT, PartitionT> element, PreparedStatement preparedStatement);
+
+ @Override
+ KV<Long, KV<PartitionT, PartitionT>> mapRow(ResultSet resultSet) throws Exception;
+ }
+
/** Create partitions on a table. */
- static class PartitioningFn extends DoFn<KV<Integer, KV<Long, Long>>, KV<String, Long>> {
+ static class PartitioningFn<T> extends DoFn<KV<Long, KV<T, T>>, KV<T, T>> {
+ private static final Logger LOG = LoggerFactory.getLogger(PartitioningFn.class);
+ final TypeDescriptor<T> partitioningColumnType;
+
+ PartitioningFn(TypeDescriptor<T> partitioningColumnType) {
+ this.partitioningColumnType = partitioningColumnType;
+ }
+
@ProcessElement
public void processElement(ProcessContext c) {
- Integer numPartitions = c.element().getKey();
- Long lowerBound = c.element().getValue().getKey();
- Long upperBound = c.element().getValue().getValue();
- if (lowerBound > upperBound) {
- throw new RuntimeException(
- String.format(
- "Lower bound [%s] is higher than upper bound [%s]", lowerBound, upperBound));
- }
- long stride = (upperBound - lowerBound) / numPartitions + 1;
- for (long i = lowerBound; i < upperBound - stride; i += stride) {
- String range = String.format("%s,%s", i, i + stride);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
- }
- if (upperBound - lowerBound > stride * (numPartitions - 1)) {
- long indexFrom = (numPartitions - 1) * stride;
- long indexTo = upperBound + 1;
- String range = String.format("%s,%s", indexFrom, indexTo);
- KV<String, Long> kvRange = KV.of(range, 1L);
- c.output(kvRange);
+ T lowerBound = c.element().getValue().getKey();
+ T upperBound = c.element().getValue().getValue();
+ JdbcReadWithPartitionsHelper<T> helper =
+ JdbcReadWithPartitionsHelper.getPartitionsHelper(partitioningColumnType);
+ List<KV<T, T>> ranges =
+ Lists.newArrayList(helper.calculateRanges(lowerBound, upperBound, c.element().getKey()));
+ LOG.warn("Total of {} ranges: {}", ranges.size(), ranges);
+ for (KV<T, T> e : ranges) {
+ c.output(e);
}
}
}
+
+ public static final Map<Class<?>, JdbcReadWithPartitionsHelper<?>> PRESET_HELPERS =
+ ImmutableMap.of(
+ Long.class,
+ new JdbcReadWithPartitionsHelper<Long>() {
+ @Override
+ public Iterable<KV<Long, Long>> calculateRanges(
+ Long lowerBound, Long upperBound, Long partitions) {
+ List<KV<Long, Long>> ranges = new ArrayList<>();
+ // We divide by partitions FIRST to make sure that we can cover the whole LONG range.
+ // If we substract first, then we may end up with Long.MAX - Long.MIN, which is 2*MAX,
+ // and we'd have trouble with the pipeline.
+ long stride = (upperBound / partitions - lowerBound / partitions) + 1;
+ long highest = lowerBound;
+ for (long i = lowerBound; i < upperBound - stride; i += stride) {
+ ranges.add(KV.of(i, i + stride));
+ highest = i + stride;
+ }
+ if (upperBound - lowerBound > stride * (ranges.size() - 1)) {
+ long indexFrom = highest;
+ long indexTo = upperBound + 1;
+ ranges.add(KV.of(indexFrom, indexTo));
+ }
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (df9a12c) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.19%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 51997 +42808
- Misses 9612 10171 +559
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/io/textio/textio.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby90ZXh0aW8vdGV4dGlvLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/cogbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9jb2diay5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/discard.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kaXNjYXJkLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/serialize.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NlcmlhbGl6ZS5nbw==) | | |
| [sdks/go/pkg/beam/transforms/stats/sum\_switch.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL3N1bV9zd2l0Y2guZ28=) | | |
| [sdks/go/pkg/beam/io/bigqueryio/bigquery.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9iaWdxdWVyeWlvL2JpZ3F1ZXJ5Lmdv) | | |
| [...s/go/pkg/beam/core/runtime/graphx/schema/schema.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NjaGVtYS9zY2hlbWEuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/replace.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L3JlcGxhY2UuZ28=) | | |
| [.../pkg/beam/runners/universal/extworker/extworker.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL3VuaXZlcnNhbC9leHR3b3JrZXIvZXh0d29ya2VyLmdv) | | |
| [sdks/go/pkg/beam/core/graph/graph.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2dyYXBoLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...df9a12c](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (800a566) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52008 +42819
- Misses 9612 10160 +548
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/xlangx/payload.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L3BheWxvYWQuZ28=) | | |
| [sdks/go/pkg/beam/io/textio/textio.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby90ZXh0aW8vdGV4dGlvLmdv) | | |
| [sdks/go/pkg/beam/core/util/protox/any.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcHJvdG94L2FueS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/pcollection.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9wY29sbGVjdGlvbi5nbw==) | | |
| [sdks/go/pkg/beam/core/funcx/fn.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L2ZuLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/cogbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L2NvZ2JrLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/bytes.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2J5dGVzLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/row\_decoder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3Jvd19kZWNvZGVyLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/registry.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L3JlZ2lzdHJ5Lmdv) | | |
| [sdks/go/pkg/beam/testing/passert/passert.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvcGFzc2VydC5zaGltcy5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...800a566](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (47aa02a) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.60% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62233 +42446
===========================================
+ Hits 9189 52033 +42844
- Misses 9612 10200 +588
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/pardo.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9wYXJkby5nbw==) | | |
| [sdks/go/pkg/beam/core/util/reflectx/tags.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvdGFncy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/emit.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9lbWl0Lmdv) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/payload.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L3BheWxvYWQuZ28=) | | |
| [sdks/go/pkg/beam/testing/passert/floats.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvZmxvYXRzLmdv) | | |
| [sdks/go/pkg/beam/io/synthetic/source.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9zeW50aGV0aWMvc291cmNlLmdv) | | |
| [sdks/go/pkg/beam/core/funcx/output.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L291dHB1dC5nbw==) | | |
| [sdks/go/pkg/beam/io/datastoreio/datastore.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhc3RvcmVpby9kYXRhc3RvcmUuZ28=) | | |
| [.../go/pkg/beam/runners/dataflow/dataflowlib/fixup.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL2ZpeHVwLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/multiplex.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9tdWx0aXBsZXguZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...47aa02a](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (47aa02a) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.60% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62233 +42446
===========================================
+ Hits 9189 52033 +42844
- Misses 9612 10200 +588
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/combine.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9jb21iaW5lLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/edge.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2VkZ2UuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/xlang.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL3hsYW5nLmdv) | | |
| [sdks/go/pkg/beam/util/harnessopts/cache.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL2hhcm5lc3NvcHRzL2NhY2hlLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/flatten.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbGF0dGVuLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/harness/datamgr.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9kYXRhbWdyLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/harness/session.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zZXNzaW9uLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvbWV0cmljc3guZ28=) | | |
| [sdks/go/pkg/beam/core/graph/scope.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL3Njb3BlLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...47aa02a](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r799521812
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -196,31 +197,60 @@
* );
* }</code></pre>
*
- * <p>3. To read all data from a table in parallel with partitioning can be done with {@link
- * ReadWithPartitions}:
+ * <h4>Parallel reading from a JDBC datasource</h4>
+ *
+ * <p>Beam supports partitioned reading of all data from a table. Automatic partitioning is
+ * supported for a few data types: {@link Long}, {@link org.joda.time.DateTime}, {@link String}. To
+ * enable this, use {@link ReadWithPartitions}.
+ *
+ * <p>The partitioning scheme depends on these parameters, which can be user-provided, or
+ * automatically inferred by Beam (for the supported types):
+ *
+ * <ul>
+ * <li>Upper bound
+ * <li>Lower bound
+ * <li>Number of partitions - when auto-inferred, the number of partitions defaults to the square
+ * root of the number of rows divided by 5 (i.e.: {@code Math.floor(Math.sqrt(numRows) / 5)}).
+ * </ul>
+ *
+ * <p>To trigger auto-inference of these parameters, the user just needs to not provide them. To
+ * infer them automatically, Beam runs either of these statements:
+ *
+ * <ul>
+ * <li>{@code SELECT min(column), max(column), COUNT(*) from table} when none of the parameters is
+ * passed to the transform.
+ * <li>{@code SELECT min(column), max(column) from table} when only number of partitions is
+ * provided, but not upper or lower bounds.
+ * </ul>
+ *
+ * <p><b>Should I use this transform?</b> Consider using this transform in the following situations:
+ *
+ * <ul>
+ * <li>The partitioning column is indexed. This will help speed up the range queries
Review comment:
thx 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (800a566) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52008 +42819
- Misses 9612 10160 +548
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/xlangx/namespace.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L25hbWVzcGFjZS5nbw==) | | |
| [sdks/go/pkg/beam/beam.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9iZWFtLnNoaW1zLmdv) | | |
| [sdks/go/pkg/beam/transforms/top/top.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3RvcC90b3AuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/types.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvdHlwZXMuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/data.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kYXRhLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/registry.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3JlZ2lzdHJ5Lmdv) | | |
| [sdks/go/pkg/beam/io/filesystem/memfs/memory.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL21lbWZzL21lbW9yeS5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/map.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL21hcC5nbw==) | | |
| [sdks/go/pkg/beam/gbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9nYmsuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/max.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21heC5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...800a566](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f155b81) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.61% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62252 +42465
===========================================
+ Hits 9189 52050 +42861
- Misses 9612 10202 +590
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/combine.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9jb21iaW5lLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/edge.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2VkZ2UuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/xlang.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL3hsYW5nLmdv) | | |
| [sdks/go/pkg/beam/util/harnessopts/cache.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL2hhcm5lc3NvcHRzL2NhY2hlLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/flatten.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbGF0dGVuLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/harness/datamgr.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9kYXRhbWdyLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/harness/session.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zZXNzaW9uLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvbWV0cmljc3guZ28=) | | |
| [sdks/go/pkg/beam/core/graph/scope.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL3Njb3BlLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...f155b81](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1031794287
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f155b81) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.61% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62252 +42465
===========================================
+ Hits 9189 52050 +42861
- Misses 9612 10202 +590
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/transforms/stats/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/option.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9vcHRpb24uZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/stats.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL3N0YXRzLnNoaW1zLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/metricsx/urns.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvdXJucy5nbw==) | | |
| [sdks/go/pkg/beam/io/synthetic/step.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9zeW50aGV0aWMvc3RlcC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/fn.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbi5nbw==) | | |
| [sdks/go/pkg/beam/validate.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS92YWxpZGF0ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/combine.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9jb21iaW5lLmdv) | | |
| [sdks/go/pkg/beam/schema.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9zY2hlbWEuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...f155b81](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f155b81) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.61% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62252 +42465
===========================================
+ Hits 9189 52050 +42861
- Misses 9612 10202 +590
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/input.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9pbnB1dC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/resolve.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L3Jlc29sdmUuZ28=) | | |
| [sdks/go/pkg/beam/core/metrics/store.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL21ldHJpY3Mvc3RvcmUuZ28=) | | |
| [sdks/go/pkg/beam/io/filesystem/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/core/util/protox/protox.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcHJvdG94L3Byb3RveC5nbw==) | | |
| [sdks/go/pkg/beam/core/metrics/metrics.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL21ldHJpY3MvbWV0cmljcy5nbw==) | | |
| [.../go/pkg/beam/runners/dataflow/dataflowlib/stage.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL3N0YWdlLmdv) | | |
| [sdks/go/pkg/beam/transforms/stats/quantiles.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL3F1YW50aWxlcy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/data.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kYXRhLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/reshuffle.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9yZXNodWZmbGUuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...f155b81](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1034521284
most welcome, @pabloem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (19b9a69) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51979 +42790
- Misses 9612 10167 +555
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/fn\_arity.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbl9hcml0eS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/plan.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9wbGFuLmdv) | | |
| [...ks/go/pkg/beam/core/runtime/harness/cache\_hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9jYWNoZV9ob29rcy5nbw==) | | |
| [sdks/go/pkg/beam/pardo.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wYXJkby5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/coder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2NvZGVyLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/sdf\_invokers.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zZGZfaW52b2tlcnMuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/max.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21heC5nbw==) | | |
| [sdks/go/pkg/beam/core/util/hooks/hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvaG9va3MvaG9va3MuZ28=) | | |
| [...pkg/beam/runners/dataflow/dataflowlib/translate.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL3RyYW5zbGF0ZS5nbw==) | | |
| [sdks/go/pkg/beam/transforms/filter/filter.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL2ZpbHRlci9maWx0ZXIuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...19b9a69](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (19b9a69) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51979 +42790
- Misses 9612 10167 +555
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/graphx/serialize.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NlcmlhbGl6ZS5nbw==) | | |
| [sdks/go/pkg/beam/io/filesystem/gcs/gcs.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL2djcy9nY3MuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/sdf.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zZGYuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/scope.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL3Njb3BlLmdv) | | |
| [...kg/beam/core/runtime/graphx/schema/logicaltypes.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NjaGVtYS9sb2dpY2FsdHlwZXMuZ28=) | | |
| [sdks/go/pkg/beam/runners/vet/vet.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL3ZldC92ZXQuZ28=) | | |
| [sdks/go/pkg/beam/util/harnessopts/sampler.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL2hhcm5lc3NvcHRzL3NhbXBsZXIuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/min\_switch.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21pbl9zd2l0Y2guZ28=) | | |
| [sdks/go/pkg/beam/testing/passert/count.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvY291bnQuZ28=) | | |
| [sdks/go/pkg/beam/io/databaseio/writer.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9kYXRhYmFzZWlvL3dyaXRlci5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...19b9a69](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (19b9a69) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 19b9a69 differs from pull request most recent head e87c4bb. Consider uploading reports for the commit e87c4bb to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51979 +42790
- Misses 9612 10167 +555
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsLmdv) | | |
| [sdks/go/pkg/beam/core/metrics/metrics.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL21ldHJpY3MvbWV0cmljcy5nbw==) | | |
| [sdks/go/pkg/beam/core/metrics/sampler.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL21ldHJpY3Mvc2FtcGxlci5nbw==) | | |
| [sdks/go/pkg/beam/runners/direct/direct.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9kaXJlY3QuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/combine.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9jb21iaW5lLmdv) | | |
| [sdks/go/pkg/beam/core/funcx/signature.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L3NpZ25hdHVyZS5nbw==) | | |
| [sdks/go/pkg/beam/partition.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wYXJ0aXRpb24uZ28=) | | |
| [...ks/go/pkg/beam/core/runtime/harness/cache\_hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9jYWNoZV9ob29rcy5nbw==) | | |
| [sdks/go/pkg/beam/pcollection.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wY29sbGVjdGlvbi5nbw==) | | |
| [sdks/go/pkg/beam/runners/vet/vet.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL3ZldC92ZXQuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...e87c4bb](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1026333148
sorry that I've kept adding code : ) this should be 'done' now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8412e56) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51984 +42795
- Misses 9612 10162 +550
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/coderx/varint.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L3ZhcmludC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/window.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy93aW5kb3cuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/max.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21heC5nbw==) | | |
| [sdks/go/pkg/beam/transforms/filter/filter.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL2ZpbHRlci9maWx0ZXIuc2hpbXMuZ28=) | | |
| [sdks/go/pkg/beam/util/gcsx/gcs.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL2djc3gvZ2NzLmdv) | | |
| [sdks/go/pkg/beam/core/typex/fulltype.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3R5cGV4L2Z1bGx0eXBlLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/input.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9pbnB1dC5nbw==) | | |
| [sdks/go/pkg/beam/core/util/hooks/hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvaG9va3MvaG9va3MuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/hash.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9oYXNoLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/windows.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3dpbmRvd3MuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...8412e56](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027060602
Just started reading looking into the code, so no worries. The examples provided in JdbcIOAutoPartitioningIT are looking really good. Also, thanks for adding testcontainers there 💯 , that was much needed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1026333329
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8412e56) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 8412e56 differs from pull request most recent head cc2922b. Consider uploading reports for the commit cc2922b to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51984 +42795
- Misses 9612 10162 +550
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/pipeline.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9waXBlbGluZS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/reshuffle.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9yZXNodWZmbGUuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L3V0aWwuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/edge.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2VkZ2UuZ28=) | | |
| [sdks/go/pkg/beam/testing/passert/passert.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvcGFzc2VydC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/window.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy93aW5kb3cuZ28=) | | |
| [sdks/go/pkg/beam/io/filesystem/gcs/gcs.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL2djcy9nY3MuZ28=) | | |
| [sdks/go/pkg/beam/util/harnessopts/cache.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL2hhcm5lc3NvcHRzL2NhY2hlLmdv) | | |
| [sdks/go/pkg/beam/testing/teststream/teststream.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Rlc3RzdHJlYW0vdGVzdHN0cmVhbS5nbw==) | | |
| [sdks/go/pkg/beam/testing/passert/hash.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvaGFzaC5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...cc2922b](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (19b9a69) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51979 +42790
- Misses 9612 10167 +555
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/fn.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbi5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/graph.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2dyYXBoLmdv) | | |
| [sdks/go/pkg/beam/transforms/top/top.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3RvcC90b3Auc2hpbXMuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/metricsx/urns.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvbWV0cmljc3gvdXJucy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/coderx/string.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L3N0cmluZy5nbw==) | | |
| [sdks/go/pkg/beam/pcollection.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wY29sbGVjdGlvbi5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/coderx/int.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L2ludC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/combine.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9jb21iaW5lLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/cogbk.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L2NvZ2JrLmdv) | | |
| [sdks/go/pkg/beam/validate.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS92YWxpZGF0ZS5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...19b9a69](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (19b9a69) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.20%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.64% +37.20%
===========================================
Files 201 452 +251
Lines 19787 62146 +42359
===========================================
+ Hits 9189 51979 +42790
- Misses 9612 10167 +555
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/io/filesystem/memfs/memory.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL21lbWZzL21lbW9yeS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/harness/sampler.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zYW1wbGVyLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/pipelinex/clone.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L2Nsb25lLmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/iterable.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2l0ZXJhYmxlLmdv) | | |
| [sdks/go/pkg/beam/core/graph/graph.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2dyYXBoLmdv) | | |
| [sdks/go/pkg/beam/transforms/top/top.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3RvcC90b3AuZ28=) | | |
| [sdks/go/pkg/beam/testing/passert/passert.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvcGFzc2VydC5nbw==) | | |
| [...beam/core/runtime/harness/statecache/statecache.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zdGF0ZWNhY2hlL3N0YXRlY2FjaGUuZ28=) | | |
| [...pkg/beam/core/runtime/xlangx/expansionx/process.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuc2lvbngvcHJvY2Vzcy5nbw==) | | |
| [sdks/go/pkg/beam/core/util/reflectx/structs.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvc3RydWN0cy5nbw==) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...19b9a69](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r798493081
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -325,17 +325,24 @@ public static ReadRows readRows() {
*
* @param <T> Type of the data to be read.
*/
- public static <T> ReadWithPartitions<T> readWithPartitions() {
- return new AutoValue_JdbcIO_ReadWithPartitions.Builder<T>()
+ public static <T, PartitionColumnT> ReadWithPartitions<T, PartitionColumnT> readWithPartitions(
Review comment:
I think this is acceptable and the best possible option 👍 just wanted to raise it ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e87c4bb) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62161 +42374
===========================================
+ Hits 9189 51998 +42809
- Misses 9612 10163 +551
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/transforms/top/top.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3RvcC90b3Auc2hpbXMuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/translate.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3RyYW5zbGF0ZS5nbw==) | | |
| [...pkg/beam/core/runtime/xlangx/expansionx/process.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuc2lvbngvcHJvY2Vzcy5nbw==) | | |
| [sdks/go/pkg/beam/artifact/materialize.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9hcnRpZmFjdC9tYXRlcmlhbGl6ZS5nbw==) | | |
| [sdks/go/pkg/beam/external.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9leHRlcm5hbC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/registry.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L3JlZ2lzdHJ5Lmdv) | | |
| [sdks/go/pkg/beam/core/graph/coder/row\_encoder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3Jvd19lbmNvZGVyLmdv) | | |
| [sdks/go/pkg/beam/testing/passert/count.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90ZXN0aW5nL3Bhc3NlcnQvY291bnQuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/edge.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2VkZ2UuZ28=) | | |
| [sdks/go/pkg/beam/io/bigqueryio/bigquery.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9iaWdxdWVyeWlvL2JpZ3F1ZXJ5Lmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...e87c4bb](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1024419181
@pabloem sure, I'll absolutely give this a look early next week! Sorry, didn't get to it today...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche edited a comment on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027060602
Just started looking into the code, so no worries. The examples provided in JdbcIOAutoPartitioningIT are looking really good. Also, thanks for adding testcontainers there 💯 , that was much needed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (df9a12c) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.19%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 51997 +42808
- Misses 9612 10171 +559
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/schema.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9zY2hlbWEuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/translate.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy90cmFuc2xhdGUuZ28=) | | |
| [sdks/go/pkg/beam/transforms/stats/mean.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21lYW4uZ28=) | | |
| [sdks/go/pkg/beam/artifact/stage.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9hcnRpZmFjdC9zdGFnZS5nbw==) | | |
| [sdks/go/pkg/beam/core/typex/fulltype.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3R5cGV4L2Z1bGx0eXBlLmdv) | | |
| [sdks/go/pkg/beam/runners/direct/buffer.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9idWZmZXIuZ28=) | | |
| [sdks/go/pkg/beam/core/util/reflectx/util.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvdXRpbC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/exec/pardo.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9wYXJkby5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/row.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3Jvdy5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/bool.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2Jvb2wuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...df9a12c](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (014888e) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.19%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.63% +37.19%
===========================================
Files 201 452 +251
Lines 19787 62175 +42388
===========================================
+ Hits 9189 52002 +42813
- Misses 9612 10173 +561
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/pipelinex/replace.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvcGlwZWxpbmV4L3JlcGxhY2UuZ28=) | | |
| [sdks/go/pkg/beam/core/util/reflectx/tags.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvdGFncy5nbw==) | | |
| [sdks/go/pkg/beam/io/synthetic/step.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9zeW50aGV0aWMvc3RlcC5nbw==) | | |
| [sdks/go/pkg/beam/validate.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS92YWxpZGF0ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/util/reflectx/call.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvY2FsbC5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/expand.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuZC5nbw==) | | |
| [sdks/go/pkg/beam/xlang.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS94bGFuZy5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/xlangx/namespace.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L25hbWVzcGFjZS5nbw==) | | |
| [sdks/go/pkg/beam/transforms/top/top.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3RvcC90b3Auc2hpbXMuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/graphx/coder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L2NvZGVyLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...014888e](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (800a566) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.21%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 800a566 differs from pull request most recent head 014888e. Consider uploading reports for the commit 014888e to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.65% +37.21%
===========================================
Files 201 452 +251
Lines 19787 62168 +42381
===========================================
+ Hits 9189 52008 +42819
- Misses 9612 10160 +548
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/util/gcsx/gcs.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS91dGlsL2djc3gvZ2NzLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/coderx/int.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4L2ludC5nbw==) | | |
| [sdks/go/pkg/beam/io/filesystem/memfs/memory.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL21lbWZzL21lbW9yeS5nbw==) | | |
| [sdks/go/pkg/beam/core/metrics/sampler.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL21ldHJpY3Mvc2FtcGxlci5nbw==) | | |
| [sdks/go/pkg/beam/artifact/materialize.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9hcnRpZmFjdC9tYXRlcmlhbGl6ZS5nbw==) | | |
| [sdks/go/pkg/beam/core/runtime/harness/session.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zZXNzaW9uLmdv) | | |
| [sdks/go/pkg/beam/coder.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb2Rlci5nbw==) | | |
| [sdks/go/pkg/beam/core/graph/coder/time.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL3RpbWUuZ28=) | | |
| [sdks/go/pkg/beam/core/graph/graph.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2dyYXBoLmdv) | | |
| [sdks/go/pkg/beam/core/util/reflectx/calls.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvcmVmbGVjdHgvY2FsbHMuZ28=) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...014888e](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15848: [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1027405916
# [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15848](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f155b81) into [master](https://codecov.io/gh/apache/beam/commit/e18e2b9ba26bee41e93c3109318672eb7b1b26bc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e18e2b9) will **increase** coverage by `37.17%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15848 +/- ##
===========================================
+ Coverage 46.43% 83.61% +37.17%
===========================================
Files 201 452 +251
Lines 19787 62252 +42465
===========================================
+ Hits 9189 52050 +42861
- Misses 9612 10202 +590
+ Partials 986 0 -986
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/multiplex.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9tdWx0aXBsZXguZ28=) | | |
| [sdks/go/pkg/beam/core/graph/fn.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2ZuLmdv) | | |
| [sdks/go/pkg/beam/pcollection.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wY29sbGVjdGlvbi5nbw==) | | |
| [sdks/go/pkg/beam/core/util/hooks/hooks.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3V0aWwvaG9va3MvaG9va3MuZ28=) | | |
| [sdks/go/pkg/beam/internal/errors/errors.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pbnRlcm5hbC9lcnJvcnMvZXJyb3JzLmdv) | | |
| [sdks/go/pkg/beam/beam.shims.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9iZWFtLnNoaW1zLmdv) | | |
| [sdks/go/pkg/beam/core/runtime/exec/plan.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9wbGFuLmdv) | | |
| [sdks/go/pkg/beam/transforms/stats/min.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3N0YXRzL21pbi5nbw==) | | |
| [sdks/go/pkg/beam/transforms/top/top.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS90cmFuc2Zvcm1zL3RvcC90b3AuZ28=) | | |
| [sdks/go/pkg/beam/core/runtime/exec/hash.go](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9oYXNoLmdv) | | |
| ... and [643 more](https://codecov.io/gh/apache/beam/pull/15848/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e18e2b9...f155b81](https://codecov.io/gh/apache/beam/pull/15848?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on pull request #15848:
URL: https://github.com/apache/beam/pull/15848#issuecomment-1030064463
> The issue is that the query with the full range would be executing in the database, so we may duplicate work on the database - so I'm not sure how bad this is or not : )
I was thinking about this as well. When just traversing the index tree that's probably fine if it doesn't happen too aggressively. But if it involves sorting and throwing a significant part of that work away all the time it will certainly impact the database badly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a change in pull request #15848: Sketching an any-type implementation for readWithPartitions for JdbcIO
Posted by GitBox <gi...@apache.org>.
mosche commented on a change in pull request #15848:
URL: https://github.com/apache/beam/pull/15848#discussion_r799520225
##########
File path: sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
##########
@@ -1071,74 +1112,136 @@ public void populateDisplayData(DisplayData.Builder builder) {
* strides for generated WHERE clause expressions used to split the column withPartitionColumn
* evenly. When the input is less than 1, the number is set to 1.
*/
- public ReadWithPartitions<T> withNumPartitions(int numPartitions) {
+ public ReadWithPartitions<T, PartitionColumnT> withNumPartitions(int numPartitions) {
checkArgument(numPartitions > 0, "numPartitions can not be less than 1");
return toBuilder().setNumPartitions(numPartitions).build();
}
/** The name of a column of numeric type that will be used for partitioning. */
- public ReadWithPartitions<T> withPartitionColumn(String partitionColumn) {
+ public ReadWithPartitions<T, PartitionColumnT> withPartitionColumn(String partitionColumn) {
checkNotNull(partitionColumn, "partitionColumn can not be null");
return toBuilder().setPartitionColumn(partitionColumn).build();
}
- public ReadWithPartitions<T> withLowerBound(Long lowerBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withLowerBound(PartitionColumnT lowerBound) {
return toBuilder().setLowerBound(lowerBound).build();
}
- public ReadWithPartitions<T> withUpperBound(Long upperBound) {
+ public ReadWithPartitions<T, PartitionColumnT> withUpperBound(PartitionColumnT upperBound) {
return toBuilder().setUpperBound(upperBound).build();
}
/** Name of the table in the external database. Can be used to pass a user-defined subqery. */
- public ReadWithPartitions<T> withTable(String tableName) {
+ public ReadWithPartitions<T, PartitionColumnT> withTable(String tableName) {
checkNotNull(tableName, "table can not be null");
return toBuilder().setTable(tableName).build();
}
+ private static final int EQUAL = 0;
+
@Override
public PCollection<T> expand(PBegin input) {
checkNotNull(getRowMapper(), "withRowMapper() is required");
checkNotNull(
getDataSourceProviderFn(),
"withDataSourceConfiguration() or withDataSourceProviderFn() is required");
checkNotNull(getPartitionColumn(), "withPartitionColumn() is required");
- checkArgument(
- getUpperBound() != null && getLowerBound() != null,
- "Upper and lower bounds are mandatory parameters for JdbcIO.readWithPartitions");
checkNotNull(getTable(), "withTable() is required");
+ if (getLowerBound() != null && getLowerBound() instanceof Comparable<?>) {
+ // Not all partition types are comparable. For example, LocalDateTime, which is a valid
+ // partitioning type, is not Comparable, so we can't enforce this for all sorts of
+ // partitioning.
+ checkArgument(
+ ((Comparable<PartitionColumnT>) getLowerBound()).compareTo(getUpperBound()) < EQUAL,
+ "The lower bound of partitioning column is larger or equal than the upper bound");
+ }
checkArgument(
- getLowerBound() < getUpperBound(),
- "The lower bound of partitioning column is larger or equal than the upper bound");
- checkArgument(
- getUpperBound() - getLowerBound() >= getNumPartitions(),
- "The specified number of partitions is more than the difference between upper bound and lower bound");
+ JdbcUtil.PRESET_HELPERS.containsKey(getPartitionColumnType().getRawType()),
+ "readWithPartitions only supports the following types: %s",
+ JdbcUtil.PRESET_HELPERS.keySet());
+
+ PCollection<KV<Integer, KV<PartitionColumnT, PartitionColumnT>>> params;
+
+ if (getLowerBound() == null && getUpperBound() == null) {
+ String query =
+ String.format(
+ "SELECT min(%s), max(%s) FROM %s",
+ getPartitionColumn(), getPartitionColumn(), getTable());
+ if (getNumPartitions() == null) {
+ query =
+ String.format(
+ "SELECT min(%s), max(%s), count(*) FROM %s",
+ getPartitionColumn(), getPartitionColumn(), getTable());
+ }
+ params =
+ input
+ .apply(
+ JdbcIO.<KV<Integer, KV<PartitionColumnT, PartitionColumnT>>>read()
+ .withQuery(query)
+ .withDataSourceProviderFn(getDataSourceProviderFn())
+ .withRowMapper(
+ (JdbcReadWithPartitionsHelper<PartitionColumnT>)
+ JdbcUtil.PRESET_HELPERS.get(getPartitionColumnType().getRawType())))
+ .apply(
+ MapElements.via(
+ new SimpleFunction<
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>>,
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>>>() {
+ @Override
+ public KV<Integer, KV<PartitionColumnT, PartitionColumnT>> apply(
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>> input) {
+ KV<Integer, KV<PartitionColumnT, PartitionColumnT>> result;
+ if (getNumPartitions() == null) {
+ // In this case, we use the table row count to infer a number of
+ // partitions.
+ // We take the square root of the number of rows, and divide it by 5
+ // to keep a relatively low number of partitions, given that an RDBMS
+ // cannot usually accept a very large number of connections.
+ Integer numPartitions =
Review comment:
i think that makes sense, e.g. `sqrt(x)/10` looks good. or alternatively just subtracting some constant value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org