You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/07/15 15:46:43 UTC
[GitHub] [beam] benWize opened a new pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
benWize opened a new pull request #15174:
URL: https://github.com/apache/beam/pull/15174
Working on related issues
[BEAM-9514] For aggregate return type, use Calcite type inference instead of ZetaSQL. Reference #14392
[BEAM-12100] LongSum uses Math.addExact that handles overflow and underflow sum.
[BEAM-10379] Remove default filter of nulls in AggregationCombineFn and create a DropNull wrapper for the aggregations that rely on dropping nulls.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r683616139
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -246,6 +250,67 @@ private Transform(
validateWindowIsSupported(windowedStream);
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner();
+
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createCombiner() {
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ return createGroupCombiner();
+ }
+ return createGlobalCombiner();
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createGlobalCombiner() {
+ org.apache.beam.sdk.schemas.transforms.Group.Global<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsGlobally<Row> combined = null;
+ for (FieldAggregation fieldAggregation : fieldAggregations) {
+ List<Integer> inputs = fieldAggregation.inputs;
+ CombineFn combineFn = fieldAggregation.combineFn;
+ if (inputs.size() > 1 || inputs.isEmpty()) {
+ // In this path we extract a Row (an empty row if inputs.isEmpty).
+ combined =
+ (combined == null)
+ ? globally.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField)
+ : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
+ } else {
+ // Combining over a single field, so extract just that field.
+ combined =
+ (combined == null)
+ ? globally.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField)
+ : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
+ }
+ }
+
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = combined;
+ if (combiner == null) {
+ // If no field aggregations were specified, we run a constant combiner that always returns
+ // a single empty row for each key. This is used by the SELECT DISTINCT query plan - in this
+ // case a group by is generated to determine unique keys, and a constant null combiner is
+ // used.
+ combiner =
+ globally.aggregateField(
+ "*",
+ AggregationCombineFnAdapter.createConstantCombineFn(),
+ Field.of(
+ "e",
+ FieldType.row(AggregationCombineFnAdapter.EMPTY_SCHEMA).withNullable(true)));
+ ignoreValues = true;
+ }
+ return combiner;
+ }
Review comment:
I created `LongSum` to handle AritmeticExceptions like Sumoverflow and Sumunderflow, without modifying the original Sum.ofLongs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/1c9fff2a5f93b42798b006098bd60aab0877013e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c9fff2) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head b49b254. Consider uploading reports for the commit b49b254 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.80% 83.84% +0.04%
==========================================
Files 441 441
Lines 59539 59726 +187
==========================================
+ Hits 49894 50075 +181
- Misses 9645 9651 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.28% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <ø> (-0.23%)` | :arrow_down: |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.49% <ø> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <ø> (+0.21%)` | :arrow_up: |
| [...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==) | `75.00% <ø> (ø)` | |
| [...dks/python/apache\_beam/examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5) | `62.16% <ø> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `87.58% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `87.41% <ø> (-0.08%)` | :arrow_down: |
| [...on/apache\_beam/ml/gcp/naturallanguageml\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL25hdHVyYWxsYW5ndWFnZW1sX3Rlc3RfaXQucHk=) | `69.23% <ø> (ø)` | |
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `69.76% <ø> (ø)` | |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c9fff2...b49b254](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-895604449
Hi @ibzib, I overrode `identity()` in `LongSum` as you suggested, it was necessary to override `identity()` for the other sum types as well.
I found out that `VarianceFn` and `CovarianceFn` were throwing an ArithmeticException like in https://ci-beam.apache.org/job/beam_PreCommit_SQL_Commit/4013/testReport/junit/org.apache.beam.sdk.extensions.sql/BeamSqlDslSqlStdOperatorsTest/testVARPOP/ so I added a check when count and variance are zero to avoid the division undefined exception.
I am stuck now with a couple of errors related to the incompatibility of Coders like in https://ci-beam.apache.org/job/beam_PreCommit_SQL_Java11_Commit/2966/testReport/junit/org.apache.beam.sdk.extensions.sql.zetasql/ZetaSqlDialectSpecTest/testArrayAggZetasql/ and https://ci-beam.apache.org/job/beam_PreCommit_SQL_Java11_Commit/2966/testReport/junit/org.apache.beam.sdk.extensions.sql.zetasql/ZetaSqlDialectSpecTest/testCountIfZetaSQLDialect/.
Do you have a suggestion of how should I proceed with these?
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r683616139
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -246,6 +250,67 @@ private Transform(
validateWindowIsSupported(windowedStream);
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner();
+
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createCombiner() {
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ return createGroupCombiner();
+ }
+ return createGlobalCombiner();
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createGlobalCombiner() {
+ org.apache.beam.sdk.schemas.transforms.Group.Global<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsGlobally<Row> combined = null;
+ for (FieldAggregation fieldAggregation : fieldAggregations) {
+ List<Integer> inputs = fieldAggregation.inputs;
+ CombineFn combineFn = fieldAggregation.combineFn;
+ if (inputs.size() > 1 || inputs.isEmpty()) {
+ // In this path we extract a Row (an empty row if inputs.isEmpty).
+ combined =
+ (combined == null)
+ ? globally.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField)
+ : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
+ } else {
+ // Combining over a single field, so extract just that field.
+ combined =
+ (combined == null)
+ ? globally.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField)
+ : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
+ }
+ }
+
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = combined;
+ if (combiner == null) {
+ // If no field aggregations were specified, we run a constant combiner that always returns
+ // a single empty row for each key. This is used by the SELECT DISTINCT query plan - in this
+ // case a group by is generated to determine unique keys, and a constant null combiner is
+ // used.
+ combiner =
+ globally.aggregateField(
+ "*",
+ AggregationCombineFnAdapter.createConstantCombineFn(),
+ Field.of(
+ "e",
+ FieldType.row(AggregationCombineFnAdapter.EMPTY_SCHEMA).withNullable(true)));
+ ignoreValues = true;
+ }
+ return combiner;
+ }
Review comment:
I created `LongSum` to handle ArithmeticExceptions like Sumoverflow and Sumunderflow, without modifying the original Sum.ofLongs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-912105136
We just merged #14729 so you will need to rebase this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r682168175
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -246,6 +250,67 @@ private Transform(
validateWindowIsSupported(windowedStream);
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner();
+
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createCombiner() {
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ return createGroupCombiner();
+ }
+ return createGlobalCombiner();
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createGlobalCombiner() {
+ org.apache.beam.sdk.schemas.transforms.Group.Global<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsGlobally<Row> combined = null;
+ for (FieldAggregation fieldAggregation : fieldAggregations) {
+ List<Integer> inputs = fieldAggregation.inputs;
+ CombineFn combineFn = fieldAggregation.combineFn;
+ if (inputs.size() > 1 || inputs.isEmpty()) {
+ // In this path we extract a Row (an empty row if inputs.isEmpty).
+ combined =
+ (combined == null)
+ ? globally.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField)
+ : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
+ } else {
+ // Combining over a single field, so extract just that field.
+ combined =
+ (combined == null)
+ ? globally.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField)
+ : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
+ }
+ }
+
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = combined;
+ if (combiner == null) {
+ // If no field aggregations were specified, we run a constant combiner that always returns
+ // a single empty row for each key. This is used by the SELECT DISTINCT query plan - in this
+ // case a group by is generated to determine unique keys, and a constant null combiner is
+ // used.
+ combiner =
+ globally.aggregateField(
+ "*",
+ AggregationCombineFnAdapter.createConstantCombineFn(),
+ Field.of(
+ "e",
+ FieldType.row(AggregationCombineFnAdapter.EMPTY_SCHEMA).withNullable(true)));
+ ignoreValues = true;
+ }
+ return combiner;
+ }
Review comment:
I created this method to process aggregations without "GROUP BY".
Some test related to aggregations on empty tables are passing now, but different tests like this https://github.com/apache/beam/blob/bdee3bd963d3213c207ea124a6bfd8256deb220b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlDialectSpecTest.java#L2216 are failing with this change, I am not sure why, but I when this [transform](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Group.java#L555 ) is applied, the data is dropped and does not execute the combineFn aggregation methods..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2a9182b) into [master](https://codecov.io/gh/apache/beam/commit/823ab5a1bee062b917cb8811633579614622e1e1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (823ab5a) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2a9182b differs from pull request most recent head fc64931. Consider uploading reports for the commit fc64931 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
=======================================
Coverage 83.74% 83.74%
=======================================
Files 443 443
Lines 60077 60077
=======================================
+ Hits 50312 50313 +1
+ Misses 9765 9764 -1
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.39% <0.00%> (+0.12%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [762a619...fc64931](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-919487157
Thanks, @ibzib I fixed null handling for BitXor with your suggestion, and for the case of `array_agg` I tried overriding type to
`x -> createSqlType(SqlTypeName.ARRAY, true)`
https://github.com/apache/beam/blob/dcc1812ab1cee26e09c3d6c3c6d3fe2496ac4f65/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SqlOperators.java#L92
but an assert fails with `java.lang.AssertionError: use createArrayType() instead`
I looked into `createArrayType()` implementation, and it creates an `ArraySqlType` with `nullable = false`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-919570065
> ArraySqlType has a public constructor, so can't we just call it directly with nullable = true? https://calcite.apache.org/javadocAggregate/org/apache/calcite/sql/type/ArraySqlType.html#%3Cinit%3E(org.apache.calcite.rel.type.RelDataType,boolean)
Yes, it is fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/1c9fff2a5f93b42798b006098bd60aab0877013e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c9fff2) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head b49b254. Consider uploading reports for the commit b49b254 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.80% 83.84% +0.04%
==========================================
Files 441 441
Lines 59539 59726 +187
==========================================
+ Hits 49894 50075 +181
- Misses 9645 9651 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.28% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <ø> (-0.23%)` | :arrow_down: |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.49% <ø> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <ø> (+0.21%)` | :arrow_up: |
| [...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==) | `75.00% <ø> (ø)` | |
| [...dks/python/apache\_beam/examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5) | `62.16% <ø> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `87.58% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `87.41% <ø> (-0.08%)` | :arrow_down: |
| [...on/apache\_beam/ml/gcp/naturallanguageml\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL25hdHVyYWxsYW5ndWFnZW1sX3Rlc3RfaXQucHk=) | `69.23% <ø> (ø)` | |
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `69.76% <ø> (ø)` | |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c9fff2...b49b254](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-907408347
> > I changed the condition to validate if the PCollection is in the global window.
>
> @benWize Sorry I haven't been able to review this promptly. I'm planning on reviewing and merging #14729 first, since it's critical we upgrade Calcite. Unfortunately there's a chance that could break this PR. Just a heads up. cc @kileys
Ok Kyle, thanks for the heads up!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/3a7b8e757d0c220b5838ca6a0805ee463b24f85d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3a7b8e7) will **increase** coverage by `0.09%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head 2a9182b. Consider uploading reports for the commit 2a9182b to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.74% 83.84% +0.09%
==========================================
Files 442 441 -1
Lines 60050 59726 -324
==========================================
- Hits 50290 50075 -215
+ Misses 9760 9651 -109
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache\_beam/runners/dataflow/internal/apiclient.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9hcGljbGllbnQucHk=) | `74.66% <0.00%> (-1.80%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.70% <0.00%> (-0.38%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/trigger.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy90cmlnZ2VyLnB5) | `89.27% <0.00%> (-0.35%)` | :arrow_down: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <0.00%> (-0.19%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/parquetio.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vcGFycXVldGlvLnB5) | `95.95% <0.00%> (-0.14%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.89% <0.00%> (-0.11%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `93.60% <0.00%> (-0.11%)` | :arrow_down: |
| [...thon/apache\_beam/runners/worker/sdk\_worker\_main.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlcl9tYWluLnB5) | `72.26% <0.00%> (-0.09%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `86.15% <0.00%> (-0.07%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/filebasedsource.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWJhc2Vkc291cmNlLnB5) | `98.75% <0.00%> (-0.04%)` | :arrow_down: |
| ... and [28 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2b4aaa0...2a9182b](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2a9182b) into [master](https://codecov.io/gh/apache/beam/commit/3a7b8e757d0c220b5838ca6a0805ee463b24f85d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3a7b8e7) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
=======================================
Coverage 83.74% 83.74%
=======================================
Files 442 443 +1
Lines 60050 60077 +27
=======================================
+ Hits 50290 50313 +23
- Misses 9760 9764 +4
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.39% <0.00%> (-0.13%)` | :arrow_down: |
| [...he\_beam/portability/api/external\_transforms\_pb2.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2V4dGVybmFsX3RyYW5zZm9ybXNfcGIyLnB5) | `100.00% <0.00%> (ø)` | |
| [...am/portability/api/external\_transforms\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2V4dGVybmFsX3RyYW5zZm9ybXNfcGIyX3VybnMucHk=) | `0.00% <0.00%> (ø)` | |
| [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `92.52% <0.00%> (+1.86%)` | :arrow_up: |
| [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `93.02% <0.00%> (+4.65%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2b4aaa0...2a9182b](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r683018811
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -246,6 +250,67 @@ private Transform(
validateWindowIsSupported(windowedStream);
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner();
+
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createCombiner() {
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ return createGroupCombiner();
+ }
+ return createGlobalCombiner();
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createGlobalCombiner() {
+ org.apache.beam.sdk.schemas.transforms.Group.Global<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsGlobally<Row> combined = null;
+ for (FieldAggregation fieldAggregation : fieldAggregations) {
+ List<Integer> inputs = fieldAggregation.inputs;
+ CombineFn combineFn = fieldAggregation.combineFn;
+ if (inputs.size() > 1 || inputs.isEmpty()) {
+ // In this path we extract a Row (an empty row if inputs.isEmpty).
+ combined =
+ (combined == null)
+ ? globally.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField)
+ : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
+ } else {
+ // Combining over a single field, so extract just that field.
+ combined =
+ (combined == null)
+ ? globally.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField)
+ : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
+ }
+ }
+
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = combined;
+ if (combiner == null) {
+ // If no field aggregations were specified, we run a constant combiner that always returns
+ // a single empty row for each key. This is used by the SELECT DISTINCT query plan - in this
+ // case a group by is generated to determine unique keys, and a constant null combiner is
+ // used.
+ combiner =
+ globally.aggregateField(
+ "*",
+ AggregationCombineFnAdapter.createConstantCombineFn(),
+ Field.of(
+ "e",
+ FieldType.row(AggregationCombineFnAdapter.EMPTY_SCHEMA).withNullable(true)));
+ ignoreValues = true;
+ }
+ return combiner;
+ }
Review comment:
It looks like the incorrect null value is coming from here: https://github.com/apache/beam/blob/39cf3fccbb3950b9fd0c8938301d192c3ca85cce/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java#L515
You can fix it by overriding the `identity()` method in `LongSum`. Or maybe you could change it back to `Sum.ofLongs`, I'm not sure why it needed a separate definition here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r699544775
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Group.java
##########
@@ -364,13 +369,29 @@
}
}
+ public interface AggregateCombinerInterface<InputT> {
Review comment:
Nit: don't include `Interface` in the name of an interface.
```suggestion
public interface AggregateCombiner<InputT> {
```
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Group.java
##########
@@ -364,13 +369,29 @@
}
}
+ public interface AggregateCombinerInterface<InputT> {
+ <CombineInputT, AccumT, CombineOutputT> AggregateCombinerInterface<InputT> aggregateField(
+ int inputFielId, CombineFn<CombineInputT, AccumT, CombineOutputT> fn, Field outputField);
Review comment:
typo
```suggestion
int inputFieldId, CombineFn<CombineInputT, AccumT, CombineOutputT> fn, Field outputField);
```
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -243,55 +247,72 @@ private Transform(
if (windowFn != null) {
windowedStream = assignTimestampsAndWindow(upstream);
}
-
validateWindowIsSupported(windowedStream);
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> byFields =
+ org.apache.beam.sdk.schemas.transforms.Group.byFieldIds(keyFieldsIds);
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner(byFields);
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(
+ mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner(globally);
+ return windowedStream.apply(combiner).setRowSchema(outputSchema);
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createCombiner(
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row>
+ initialCombiner) {
- org.apache.beam.sdk.schemas.transforms.Group.ByFields<Row> byFields =
- org.apache.beam.sdk.schemas.transforms.Group.byFieldIds(keyFieldsIds);
- org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsByFields<Row> combined = null;
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> combined = null;
for (FieldAggregation fieldAggregation : fieldAggregations) {
List<Integer> inputs = fieldAggregation.inputs;
CombineFn combineFn = fieldAggregation.combineFn;
- if (inputs.size() > 1 || inputs.isEmpty()) {
- // In this path we extract a Row (an empty row if inputs.isEmpty).
+ if (inputs.size() == 1) {
+ // Combining over a single field, so extract just that field.
combined =
(combined == null)
- ? byFields.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField)
- : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
+ ? initialCombiner.aggregateField(
+ inputs.get(0), combineFn, fieldAggregation.outputField)
+ : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
} else {
- // Combining over a single field, so extract just that field.
+ // In this path we extract a Row (an empty row if inputs.isEmpty).
combined =
(combined == null)
- ? byFields.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField)
- : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
+ ? initialCombiner.aggregateFieldsById(
+ inputs, combineFn, fieldAggregation.outputField)
+ : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
}
}
- PTransform<PCollection<Row>, PCollection<Row>> combiner = combined;
- boolean ignoreValues = false;
+ PTransform<PCollection<Row>, PCollection<Row>> combiner =
+ (PTransform<PCollection<Row>, PCollection<Row>>) combined;
Review comment:
These casts indicate that `AggregateCombinerInterface` should probably extend PTransform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r698875183
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
##########
@@ -376,29 +445,45 @@ public BigDecimal toBigDecimal(BigDecimal record) {
}
}
- static class BitOr<T extends Number> extends CombineFn<T, Long, Long> {
+ static class BitOr<T extends Number> extends CombineFn<T, BitOr.Accum, Long> {
+ static class Accum implements Serializable {
+ /** True if no inputs have been seen yet. */
+ boolean isEmpty = true;
+ /** The bitwise-and of the inputs seen so far. */
+ long bitOr = 0L;
+ }
+
@Override
- public Long createAccumulator() {
- return 0L;
+ public Accum createAccumulator() {
+ return new Accum();
}
@Override
- public Long addInput(Long accum, T input) {
- return accum | input.longValue();
+ public Accum addInput(Accum accum, T input) {
+ accum.bitOr |= input.longValue();
Review comment:
I added a validation similar to BitAnd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r683018811
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -246,6 +250,67 @@ private Transform(
validateWindowIsSupported(windowedStream);
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner();
+
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createCombiner() {
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ return createGroupCombiner();
+ }
+ return createGlobalCombiner();
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createGlobalCombiner() {
+ org.apache.beam.sdk.schemas.transforms.Group.Global<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsGlobally<Row> combined = null;
+ for (FieldAggregation fieldAggregation : fieldAggregations) {
+ List<Integer> inputs = fieldAggregation.inputs;
+ CombineFn combineFn = fieldAggregation.combineFn;
+ if (inputs.size() > 1 || inputs.isEmpty()) {
+ // In this path we extract a Row (an empty row if inputs.isEmpty).
+ combined =
+ (combined == null)
+ ? globally.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField)
+ : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
+ } else {
+ // Combining over a single field, so extract just that field.
+ combined =
+ (combined == null)
+ ? globally.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField)
+ : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
+ }
+ }
+
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = combined;
+ if (combiner == null) {
+ // If no field aggregations were specified, we run a constant combiner that always returns
+ // a single empty row for each key. This is used by the SELECT DISTINCT query plan - in this
+ // case a group by is generated to determine unique keys, and a constant null combiner is
+ // used.
+ combiner =
+ globally.aggregateField(
+ "*",
+ AggregationCombineFnAdapter.createConstantCombineFn(),
+ Field.of(
+ "e",
+ FieldType.row(AggregationCombineFnAdapter.EMPTY_SCHEMA).withNullable(true)));
+ ignoreValues = true;
+ }
+ return combiner;
+ }
Review comment:
It looks like the incorrect null value is coming from here: https://github.com/apache/beam/blob/39cf3fccbb3950b9fd0c8938301d192c3ca85cce/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java#L515
You can fix it by overriding the `identity()` method in `LongSum`. Or maybe you could change it back to `Sum.ofLongs`, I'm not sure why it needed a separate definition here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-896394325
> I am stuck now with a couple of errors related to the incompatibility of Coders like in https://ci-beam.apache.org/job/beam_PreCommit_SQL_Java11_Commit/2966/testReport/junit/org.apache.beam.sdk.extensions.sql.zetasql/ZetaSqlDialectSpecTest/testArrayAggZetasql/ and https://ci-beam.apache.org/job/beam_PreCommit_SQL_Java11_Commit/2966/testReport/junit/org.apache.beam.sdk.extensions.sql.zetasql/ZetaSqlDialectSpecTest/testCountIfZetaSQLDialect/.
> Do you have a suggestion of how should I proceed with these?
I'm guessing the reason the error message mentions "elements of type class java.lang.Object" is because of the type signature of DropNullFn. `DropNullFn<T> extends CombineFn<T, Object, Object>` DropNullFn should use generics for all three types, not just the input type. Try changing it to `DropNullFn<InputT, AccumT, OutputT> extends CombineFn<InputT, AccumT, OutputT>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-880835480
Hi @ibzib, I am working on a group of related issues using some of your work in #14392. Could you help me to take a look at the progress so far and provide some guidance on the next steps?
I am planning to include [BEAM-12098](https://issues.apache.org/jira/browse/BEAM-12098) and [BEAM-12099](https://issues.apache.org/jira/browse/BEAM-12099) in this PR, but for those, I have a problem, in the transformation related to UNNEST, when it receives an empty array, does not return anything so it fails before aggregation transforms (COUNT, BIT_OR).
I think the problem could be in this part https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/unnest/BeamZetaSqlUncollectRel.java#L115 because when the array is empty, this transform does not output anything.
Any guidance on this would be appreciated.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/1c9fff2a5f93b42798b006098bd60aab0877013e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c9fff2) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head 072a47e. Consider uploading reports for the commit 072a47e to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.80% 83.84% +0.04%
==========================================
Files 441 441
Lines 59539 59726 +187
==========================================
+ Hits 49894 50075 +181
- Misses 9645 9651 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.28% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <ø> (-0.23%)` | :arrow_down: |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.49% <ø> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <ø> (+0.21%)` | :arrow_up: |
| [...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==) | `75.00% <ø> (ø)` | |
| [...dks/python/apache\_beam/examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5) | `62.16% <ø> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `87.58% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `87.41% <ø> (-0.08%)` | :arrow_down: |
| [...on/apache\_beam/ml/gcp/naturallanguageml\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL25hdHVyYWxsYW5ndWFnZW1sX3Rlc3RfaXQucHk=) | `69.23% <ø> (ø)` | |
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `69.76% <ø> (ø)` | |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c9fff2...072a47e](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-902302298
> @benWize the other option it suggested in the error message was to use `Combine.globally().asSingletonView()`. Did you try that?
Yes, I tried but it causes a compilation error because of the return type of `.asSingletonView()`. I tried setting ` .withoutDefaults()` just for Unbounded PCollections, and it is working now. https://github.com/apache/beam/pull/15174/commits/7473306d414322f284fb3b29f16e280108f9e2d5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r700532707
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -243,55 +247,72 @@ private Transform(
if (windowFn != null) {
windowedStream = assignTimestampsAndWindow(upstream);
}
-
validateWindowIsSupported(windowedStream);
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> byFields =
+ org.apache.beam.sdk.schemas.transforms.Group.byFieldIds(keyFieldsIds);
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner(byFields);
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(
+ mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner(globally);
+ return windowedStream.apply(combiner).setRowSchema(outputSchema);
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createCombiner(
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row>
+ initialCombiner) {
- org.apache.beam.sdk.schemas.transforms.Group.ByFields<Row> byFields =
- org.apache.beam.sdk.schemas.transforms.Group.byFieldIds(keyFieldsIds);
- org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsByFields<Row> combined = null;
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> combined = null;
for (FieldAggregation fieldAggregation : fieldAggregations) {
List<Integer> inputs = fieldAggregation.inputs;
CombineFn combineFn = fieldAggregation.combineFn;
- if (inputs.size() > 1 || inputs.isEmpty()) {
- // In this path we extract a Row (an empty row if inputs.isEmpty).
+ if (inputs.size() == 1) {
+ // Combining over a single field, so extract just that field.
combined =
(combined == null)
- ? byFields.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField)
- : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
+ ? initialCombiner.aggregateField(
+ inputs.get(0), combineFn, fieldAggregation.outputField)
+ : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
} else {
- // Combining over a single field, so extract just that field.
+ // In this path we extract a Row (an empty row if inputs.isEmpty).
combined =
(combined == null)
- ? byFields.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField)
- : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
+ ? initialCombiner.aggregateFieldsById(
+ inputs, combineFn, fieldAggregation.outputField)
+ : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
}
}
- PTransform<PCollection<Row>, PCollection<Row>> combiner = combined;
- boolean ignoreValues = false;
+ PTransform<PCollection<Row>, PCollection<Row>> combiner =
+ (PTransform<PCollection<Row>, PCollection<Row>>) combined;
Review comment:
I tried to extend from PTransform, but I got several conflicts because of the different output types for `Global`, `CombineFieldsGlobally` and `CombineFieldsByFields` classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r698874840
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
##########
@@ -56,12 +58,13 @@
BUILTIN_AGGREGATOR_FACTORIES =
ImmutableMap.<String, Function<Schema.FieldType, CombineFn<?, ?, ?>>>builder()
.put("ANY_VALUE", typeName -> Sample.anyValueCombineFn())
- .put("COUNT", typeName -> Count.combineFn())
- .put("MAX", BeamBuiltinAggregations::createMax)
- .put("MIN", BeamBuiltinAggregations::createMin)
- .put("SUM", BeamBuiltinAggregations::createSum)
- .put("$SUM0", BeamBuiltinAggregations::createSum)
- .put("AVG", BeamBuiltinAggregations::createAvg)
+ // Drop null elements for these aggregations BEAM-10379
Review comment:
Comment changed
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
##########
@@ -229,27 +232,93 @@ public T apply(T left, T right) {
public Short apply(Short left, Short right) {
return (short) (left + right);
}
+
+ @Override
+ public @Nullable Short identity() {
+ return 0;
+ }
}
static class ByteSum extends Combine.BinaryCombineFn<Byte> {
@Override
public Byte apply(Byte left, Byte right) {
return (byte) (left + right);
}
+
+ @Override
+ public @Nullable Byte identity() {
+ return 0;
+ }
}
static class FloatSum extends Combine.BinaryCombineFn<Float> {
@Override
public Float apply(Float left, Float right) {
return left + right;
}
+
+ @Override
+ public @Nullable Float identity() {
+ return 0F;
+ }
+ }
+
+ static class LongSum extends Combine.BinaryCombineFn<Long> {
+ @Override
+ public Long apply(Long left, Long right) {
+ return Math.addExact(left, right);
+ }
+
+ @Override
+ public @Nullable Long identity() {
+ return 0L;
+ }
}
static class BigDecimalSum extends Combine.BinaryCombineFn<BigDecimal> {
@Override
public BigDecimal apply(BigDecimal left, BigDecimal right) {
return left.add(right);
}
+
+ @Override
+ public @Nullable BigDecimal identity() {
+ return BigDecimal.ZERO;
+ }
+ }
+
+ static class DropNullFn<InputT, AccumT, OutputT> extends CombineFn<InputT, AccumT, OutputT> {
+ CombineFn<InputT, AccumT, OutputT> combineFn;
Review comment:
Yes, fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r706264231
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Group.java
##########
@@ -364,19 +365,43 @@
}
}
+ public abstract static class AggregateCombiner<InputT>
Review comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r702199547
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -243,55 +247,72 @@ private Transform(
if (windowFn != null) {
windowedStream = assignTimestampsAndWindow(upstream);
}
-
validateWindowIsSupported(windowedStream);
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> byFields =
+ org.apache.beam.sdk.schemas.transforms.Group.byFieldIds(keyFieldsIds);
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner(byFields);
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(
+ mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner(globally);
+ return windowedStream.apply(combiner).setRowSchema(outputSchema);
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createCombiner(
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row>
+ initialCombiner) {
- org.apache.beam.sdk.schemas.transforms.Group.ByFields<Row> byFields =
- org.apache.beam.sdk.schemas.transforms.Group.byFieldIds(keyFieldsIds);
- org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsByFields<Row> combined = null;
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> combined = null;
for (FieldAggregation fieldAggregation : fieldAggregations) {
List<Integer> inputs = fieldAggregation.inputs;
CombineFn combineFn = fieldAggregation.combineFn;
- if (inputs.size() > 1 || inputs.isEmpty()) {
- // In this path we extract a Row (an empty row if inputs.isEmpty).
+ if (inputs.size() == 1) {
+ // Combining over a single field, so extract just that field.
combined =
(combined == null)
- ? byFields.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField)
- : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
+ ? initialCombiner.aggregateField(
+ inputs.get(0), combineFn, fieldAggregation.outputField)
+ : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
} else {
- // Combining over a single field, so extract just that field.
+ // In this path we extract a Row (an empty row if inputs.isEmpty).
combined =
(combined == null)
- ? byFields.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField)
- : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
+ ? initialCombiner.aggregateFieldsById(
+ inputs, combineFn, fieldAggregation.outputField)
+ : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
}
}
- PTransform<PCollection<Row>, PCollection<Row>> combiner = combined;
- boolean ignoreValues = false;
+ PTransform<PCollection<Row>, PCollection<Row>> combiner =
+ (PTransform<PCollection<Row>, PCollection<Row>>) combined;
Review comment:
I did this, but I had to create a new method to make an instance of `CombineFieldsGlobally` with null `schemaAggregateFn` because that param is initialized here https://github.com/apache/beam/blob/22e2adbc05cdb3cb2f9800508108926df67f829b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java#L284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/3a7b8e757d0c220b5838ca6a0805ee463b24f85d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3a7b8e7) will **increase** coverage by `0.09%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head f43f241. Consider uploading reports for the commit f43f241 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.74% 83.84% +0.09%
==========================================
Files 442 441 -1
Lines 60050 59726 -324
==========================================
- Hits 50290 50075 -215
+ Misses 9760 9651 -109
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache\_beam/runners/dataflow/internal/apiclient.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9hcGljbGllbnQucHk=) | `74.66% <0.00%> (-1.80%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.70% <0.00%> (-0.38%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/trigger.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy90cmlnZ2VyLnB5) | `89.27% <0.00%> (-0.35%)` | :arrow_down: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <0.00%> (-0.19%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/parquetio.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vcGFycXVldGlvLnB5) | `95.95% <0.00%> (-0.14%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.89% <0.00%> (-0.11%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `93.60% <0.00%> (-0.11%)` | :arrow_down: |
| [...thon/apache\_beam/runners/worker/sdk\_worker\_main.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlcl9tYWluLnB5) | `72.26% <0.00%> (-0.09%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `86.15% <0.00%> (-0.07%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/filebasedsource.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWJhc2Vkc291cmNlLnB5) | `98.75% <0.00%> (-0.04%)` | :arrow_down: |
| ... and [28 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2b4aaa0...f43f241](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-902120571
> > I am stuck now with a couple of errors related to the incompatibility of Coders like in https://ci-beam.apache.org/job/beam_PreCommit_SQL_Java11_Commit/2966/testReport/junit/org.apache.beam.sdk.extensions.sql.zetasql/ZetaSqlDialectSpecTest/testArrayAggZetasql/ and https://ci-beam.apache.org/job/beam_PreCommit_SQL_Java11_Commit/2966/testReport/junit/org.apache.beam.sdk.extensions.sql.zetasql/ZetaSqlDialectSpecTest/testCountIfZetaSQLDialect/.
> > Do you have a suggestion of how should I proceed with these?
>
> I'm guessing the reason the error message mentions "elements of type class java.lang.Object" is because of the type signature of DropNullFn. `DropNullFn<T> extends CombineFn<T, Object, Object>` DropNullFn should use generics for all three types, not just the input type. Try changing it to `DropNullFn<InputT, AccumT, OutputT> extends CombineFn<InputT, AccumT, OutputT>`
I used generics in `DropNullFn`, I did the same in `ArrayAgg` and that fixed `testArrayAggZetasql`.
For `testCountIfZetaSQLDialect` the problem was with the Coder for the accumulator class, implementing `Accum` as `Serializable` fixed the problem. Now the only failing test is this https://ci-beam.apache.org/job/beam_PreCommit_SQL_Java11_Commit/2992/testReport/junit/org.apache.beam.sdk.extensions.sql/BeamSqlDslAggregationTest/testAggregationNonGroupedFunctionsWithUnbounded/.
I've tried adding `.withoutDefaults() in `https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Group.java#L556 but that causes a failure in https://github.com/apache/beam/blob/072a47e519130c38e4cc1b5ac23f352d6e41d85a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlDialectSpecTest.java#L3843 because of the default value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r702197338
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -216,6 +216,8 @@ public RelWriter explainTerms(RelWriter pw) {
private WindowFn<Row, IntervalWindow> windowFn;
private int windowFieldIndex;
private List<FieldAggregation> fieldAggregations;
+ private final int groupSetCount;
+ private boolean ignoreValues;
Review comment:
`ignoreValues` is modified latter https://github.com/apache/beam/blob/77a11a135a6aafc8878af6554f7a4cc799c47804/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java#L312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r702210901
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -243,55 +247,72 @@ private Transform(
if (windowFn != null) {
windowedStream = assignTimestampsAndWindow(upstream);
}
-
validateWindowIsSupported(windowedStream);
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> byFields =
+ org.apache.beam.sdk.schemas.transforms.Group.byFieldIds(keyFieldsIds);
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner(byFields);
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(
+ mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner(globally);
+ return windowedStream.apply(combiner).setRowSchema(outputSchema);
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createCombiner(
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row>
+ initialCombiner) {
- org.apache.beam.sdk.schemas.transforms.Group.ByFields<Row> byFields =
- org.apache.beam.sdk.schemas.transforms.Group.byFieldIds(keyFieldsIds);
- org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsByFields<Row> combined = null;
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> combined = null;
for (FieldAggregation fieldAggregation : fieldAggregations) {
List<Integer> inputs = fieldAggregation.inputs;
CombineFn combineFn = fieldAggregation.combineFn;
- if (inputs.size() > 1 || inputs.isEmpty()) {
- // In this path we extract a Row (an empty row if inputs.isEmpty).
+ if (inputs.size() == 1) {
+ // Combining over a single field, so extract just that field.
combined =
(combined == null)
- ? byFields.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField)
- : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
+ ? initialCombiner.aggregateField(
+ inputs.get(0), combineFn, fieldAggregation.outputField)
+ : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
} else {
- // Combining over a single field, so extract just that field.
+ // In this path we extract a Row (an empty row if inputs.isEmpty).
combined =
(combined == null)
- ? byFields.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField)
- : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
+ ? initialCombiner.aggregateFieldsById(
+ inputs, combineFn, fieldAggregation.outputField)
+ : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
}
}
- PTransform<PCollection<Row>, PCollection<Row>> combiner = combined;
- boolean ignoreValues = false;
+ PTransform<PCollection<Row>, PCollection<Row>> combiner =
+ (PTransform<PCollection<Row>, PCollection<Row>>) combined;
Review comment:
If we always turn the null `schemaAggregateFn` into `SchemaAggregateFn.create()`, why not initialize with `SchemaAggregateFn.create()` instead of null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-917101080
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-917140185
I copied the test and fixed the error in `bitAnd`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/3a7b8e757d0c220b5838ca6a0805ee463b24f85d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3a7b8e7) will **increase** coverage by `0.09%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head 899f28e. Consider uploading reports for the commit 899f28e to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.74% 83.84% +0.09%
==========================================
Files 442 441 -1
Lines 60050 59726 -324
==========================================
- Hits 50290 50075 -215
+ Misses 9760 9651 -109
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache\_beam/runners/dataflow/internal/apiclient.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9hcGljbGllbnQucHk=) | `74.66% <0.00%> (-1.80%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.70% <0.00%> (-0.38%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/trigger.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy90cmlnZ2VyLnB5) | `89.27% <0.00%> (-0.35%)` | :arrow_down: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <0.00%> (-0.19%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/parquetio.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vcGFycXVldGlvLnB5) | `95.95% <0.00%> (-0.14%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.89% <0.00%> (-0.11%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `93.60% <0.00%> (-0.11%)` | :arrow_down: |
| [...thon/apache\_beam/runners/worker/sdk\_worker\_main.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlcl9tYWluLnB5) | `72.26% <0.00%> (-0.09%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `86.15% <0.00%> (-0.07%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/filebasedsource.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWJhc2Vkc291cmNlLnB5) | `98.75% <0.00%> (-0.04%)` | :arrow_down: |
| ... and [28 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2b4aaa0...899f28e](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-902231762
@benWize the other option it suggested in the error message was to use `Combine.globally().asSingletonView()`. Did you try that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r702995062
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -243,55 +247,72 @@ private Transform(
if (windowFn != null) {
windowedStream = assignTimestampsAndWindow(upstream);
}
-
validateWindowIsSupported(windowedStream);
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> byFields =
+ org.apache.beam.sdk.schemas.transforms.Group.byFieldIds(keyFieldsIds);
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner(byFields);
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(
+ mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner(globally);
+ return windowedStream.apply(combiner).setRowSchema(outputSchema);
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createCombiner(
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row>
+ initialCombiner) {
- org.apache.beam.sdk.schemas.transforms.Group.ByFields<Row> byFields =
- org.apache.beam.sdk.schemas.transforms.Group.byFieldIds(keyFieldsIds);
- org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsByFields<Row> combined = null;
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> combined = null;
for (FieldAggregation fieldAggregation : fieldAggregations) {
List<Integer> inputs = fieldAggregation.inputs;
CombineFn combineFn = fieldAggregation.combineFn;
- if (inputs.size() > 1 || inputs.isEmpty()) {
- // In this path we extract a Row (an empty row if inputs.isEmpty).
+ if (inputs.size() == 1) {
+ // Combining over a single field, so extract just that field.
combined =
(combined == null)
- ? byFields.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField)
- : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
+ ? initialCombiner.aggregateField(
+ inputs.get(0), combineFn, fieldAggregation.outputField)
+ : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
} else {
- // Combining over a single field, so extract just that field.
+ // In this path we extract a Row (an empty row if inputs.isEmpty).
combined =
(combined == null)
- ? byFields.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField)
- : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
+ ? initialCombiner.aggregateFieldsById(
+ inputs, combineFn, fieldAggregation.outputField)
+ : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
}
}
- PTransform<PCollection<Row>, PCollection<Row>> combiner = combined;
- boolean ignoreValues = false;
+ PTransform<PCollection<Row>, PCollection<Row>> combiner =
+ (PTransform<PCollection<Row>, PCollection<Row>>) combined;
Review comment:
I made the change and it worked, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/1c9fff2a5f93b42798b006098bd60aab0877013e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c9fff2) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head b49b254. Consider uploading reports for the commit b49b254 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.80% 83.84% +0.04%
==========================================
Files 441 441
Lines 59539 59726 +187
==========================================
+ Hits 49894 50075 +181
- Misses 9645 9651 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.28% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <ø> (-0.23%)` | :arrow_down: |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.49% <ø> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <ø> (+0.21%)` | :arrow_up: |
| [...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==) | `75.00% <ø> (ø)` | |
| [...dks/python/apache\_beam/examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5) | `62.16% <ø> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `87.58% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `87.41% <ø> (-0.08%)` | :arrow_down: |
| [...on/apache\_beam/ml/gcp/naturallanguageml\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL25hdHVyYWxsYW5ndWFnZW1sX3Rlc3RfaXQucHk=) | `69.23% <ø> (ø)` | |
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `69.76% <ø> (ø)` | |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c9fff2...b49b254](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2a9182b) into [master](https://codecov.io/gh/apache/beam/commit/823ab5a1bee062b917cb8811633579614622e1e1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (823ab5a) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2a9182b differs from pull request most recent head dcc1812. Consider uploading reports for the commit dcc1812 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
=======================================
Coverage 83.74% 83.74%
=======================================
Files 443 443
Lines 60077 60077
=======================================
+ Hits 50312 50313 +1
+ Misses 9765 9764 -1
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.39% <0.00%> (+0.12%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [762a619...dcc1812](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-917201142
@benWize Thanks, that fixed the error. The tests are no longer crashing, but 6 failures remain:
- aggregation_queries_test:aggregation_array_agg_1
- aggregation_queries_test:array_agg_empty_input
- aggregation_queries_test:aggregation_36
- aggregation_queries_test:aggregation_53
- aggregation_queries_test:aggregation_58
- aggregation_queries_test:aggregation_62
https://github.com/google/zetasql/blob/dd883180de6387ad80bcf7534b5aae8191e66621/zetasql/compliance/testdata/aggregation_queries.test
I took a quick look at the errors, and it seems they are all related to null handling. (I'm guessing these are all test cases where the SQL query failed to compile before, but now they're compiling and giving wrong answers.)
- Some functions are expected to return null when the input is empty.
- BIT_AND, BIT_OR, etc. are expected to ignore null input mixed with non-null inputs. I think the current implementations of these functions in Beam are incorrect. The compliance tests are the source of truth.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-917164324
Run Java_Examples_Dataflow PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-919517308
> Thanks, @ibzib I fixed null handling for BitXor with your suggestion, and for the case of `array_agg` I tried overriding type to
> `x -> createSqlType(SqlTypeName.ARRAY, true)`
> https://github.com/apache/beam/blob/dcc1812ab1cee26e09c3d6c3c6d3fe2496ac4f65/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SqlOperators.java#L92
>
>
> but an assert fails with `java.lang.AssertionError: use createArrayType() instead`
> I looked into `createArrayType()` implementation, and it creates an `ArraySqlType` with `nullable = false`.
ArraySqlType has a public constructor, so can't we just call it directly with nullable = true? https://calcite.apache.org/javadocAggregate/org/apache/calcite/sql/type/ArraySqlType.html#%3Cinit%3E(org.apache.calcite.rel.type.RelDataType,boolean)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-918647393
@ibzib I copied most of the tests to check the fixes, I was able to fix null handling for `BitOr` and `BitAnd`, but in the case of `BitXOr` and `Array_Agg` it shows an error because the `inferedType` by Calcite is BIGINT NOT NULL, so when trying to return `null` (Like in an empty input) fails by type mismatch (`ARRAY<INT64 NOT NULL> NOT NULL is not nullable`). https://github.com/apache/beam/blob/8adc4370e462ce9c1c0eb7472aea53be707d7cd5/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/AggregateScanConverter.java#L259
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2a9182b) into [master](https://codecov.io/gh/apache/beam/commit/823ab5a1bee062b917cb8811633579614622e1e1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (823ab5a) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2a9182b differs from pull request most recent head 384750a. Consider uploading reports for the commit 384750a to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
=======================================
Coverage 83.74% 83.74%
=======================================
Files 443 443
Lines 60077 60077
=======================================
+ Hits 50312 50313 +1
+ Misses 9765 9764 -1
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.39% <0.00%> (+0.12%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [762a619...384750a](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r699544775
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Group.java
##########
@@ -364,13 +369,29 @@
}
}
+ public interface AggregateCombinerInterface<InputT> {
Review comment:
Nit: don't include `Interface` in the name of an interface.
```suggestion
public interface AggregateCombiner<InputT> {
```
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Group.java
##########
@@ -364,13 +369,29 @@
}
}
+ public interface AggregateCombinerInterface<InputT> {
+ <CombineInputT, AccumT, CombineOutputT> AggregateCombinerInterface<InputT> aggregateField(
+ int inputFielId, CombineFn<CombineInputT, AccumT, CombineOutputT> fn, Field outputField);
Review comment:
typo
```suggestion
int inputFieldId, CombineFn<CombineInputT, AccumT, CombineOutputT> fn, Field outputField);
```
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -243,55 +247,72 @@ private Transform(
if (windowFn != null) {
windowedStream = assignTimestampsAndWindow(upstream);
}
-
validateWindowIsSupported(windowedStream);
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> byFields =
+ org.apache.beam.sdk.schemas.transforms.Group.byFieldIds(keyFieldsIds);
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner(byFields);
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(
+ mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner(globally);
+ return windowedStream.apply(combiner).setRowSchema(outputSchema);
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createCombiner(
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row>
+ initialCombiner) {
- org.apache.beam.sdk.schemas.transforms.Group.ByFields<Row> byFields =
- org.apache.beam.sdk.schemas.transforms.Group.byFieldIds(keyFieldsIds);
- org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsByFields<Row> combined = null;
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> combined = null;
for (FieldAggregation fieldAggregation : fieldAggregations) {
List<Integer> inputs = fieldAggregation.inputs;
CombineFn combineFn = fieldAggregation.combineFn;
- if (inputs.size() > 1 || inputs.isEmpty()) {
- // In this path we extract a Row (an empty row if inputs.isEmpty).
+ if (inputs.size() == 1) {
+ // Combining over a single field, so extract just that field.
combined =
(combined == null)
- ? byFields.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField)
- : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
+ ? initialCombiner.aggregateField(
+ inputs.get(0), combineFn, fieldAggregation.outputField)
+ : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
} else {
- // Combining over a single field, so extract just that field.
+ // In this path we extract a Row (an empty row if inputs.isEmpty).
combined =
(combined == null)
- ? byFields.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField)
- : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
+ ? initialCombiner.aggregateFieldsById(
+ inputs, combineFn, fieldAggregation.outputField)
+ : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
}
}
- PTransform<PCollection<Row>, PCollection<Row>> combiner = combined;
- boolean ignoreValues = false;
+ PTransform<PCollection<Row>, PCollection<Row>> combiner =
+ (PTransform<PCollection<Row>, PCollection<Row>>) combined;
Review comment:
These casts indicate that `AggregateCombinerInterface` should probably extend PTransform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2a9182b) into [master](https://codecov.io/gh/apache/beam/commit/823ab5a1bee062b917cb8811633579614622e1e1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (823ab5a) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2a9182b differs from pull request most recent head 1c4ad22. Consider uploading reports for the commit 1c4ad22 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
=======================================
Coverage 83.74% 83.74%
=======================================
Files 443 443
Lines 60077 60077
=======================================
+ Hits 50312 50313 +1
+ Misses 9765 9764 -1
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.39% <0.00%> (+0.12%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [762a619...1c4ad22](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2a9182b) into [master](https://codecov.io/gh/apache/beam/commit/823ab5a1bee062b917cb8811633579614622e1e1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (823ab5a) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2a9182b differs from pull request most recent head 1c4ad22. Consider uploading reports for the commit 1c4ad22 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
=======================================
Coverage 83.74% 83.74%
=======================================
Files 443 443
Lines 60077 60077
=======================================
+ Hits 50312 50313 +1
+ Misses 9765 9764 -1
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.39% <0.00%> (+0.12%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [762a619...1c4ad22](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r682170667
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -246,6 +250,67 @@ private Transform(
validateWindowIsSupported(windowedStream);
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner();
+
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createCombiner() {
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ return createGroupCombiner();
+ }
+ return createGlobalCombiner();
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createGlobalCombiner() {
+ org.apache.beam.sdk.schemas.transforms.Group.Global<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsGlobally<Row> combined = null;
+ for (FieldAggregation fieldAggregation : fieldAggregations) {
+ List<Integer> inputs = fieldAggregation.inputs;
+ CombineFn combineFn = fieldAggregation.combineFn;
+ if (inputs.size() > 1 || inputs.isEmpty()) {
+ // In this path we extract a Row (an empty row if inputs.isEmpty).
+ combined =
+ (combined == null)
+ ? globally.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField)
+ : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
+ } else {
+ // Combining over a single field, so extract just that field.
+ combined =
+ (combined == null)
+ ? globally.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField)
+ : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
+ }
+ }
+
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = combined;
+ if (combiner == null) {
+ // If no field aggregations were specified, we run a constant combiner that always returns
+ // a single empty row for each key. This is used by the SELECT DISTINCT query plan - in this
+ // case a group by is generated to determine unique keys, and a constant null combiner is
+ // used.
+ combiner =
+ globally.aggregateField(
+ "*",
+ AggregationCombineFnAdapter.createConstantCombineFn(),
+ Field.of(
+ "e",
+ FieldType.row(AggregationCombineFnAdapter.EMPTY_SCHEMA).withNullable(true)));
+ ignoreValues = true;
+ }
+ return combiner;
+ }
Review comment:
I left the stack trace for reference https://ci-beam.apache.org/job/beam_PreCommit_SQL_Commit/4013/testReport/junit/org.apache.beam.sdk.extensions.sql.zetasql/ZetaSqlDialectSpecTest/testWithQueryEight/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/1c9fff2a5f93b42798b006098bd60aab0877013e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c9fff2) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head 22e2adb. Consider uploading reports for the commit 22e2adb to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.80% 83.84% +0.04%
==========================================
Files 441 441
Lines 59539 59726 +187
==========================================
+ Hits 49894 50075 +181
- Misses 9645 9651 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.28% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <ø> (-0.23%)` | :arrow_down: |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.49% <ø> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <ø> (+0.21%)` | :arrow_up: |
| [...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==) | `75.00% <ø> (ø)` | |
| [...dks/python/apache\_beam/examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5) | `62.16% <ø> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `87.58% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `87.41% <ø> (-0.08%)` | :arrow_down: |
| [...on/apache\_beam/ml/gcp/naturallanguageml\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL25hdHVyYWxsYW5ndWFnZW1sX3Rlc3RfaXQucHk=) | `69.23% <ø> (ø)` | |
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `69.76% <ø> (ø)` | |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c9fff2...22e2adb](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r698874206
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -216,6 +216,8 @@ public RelWriter explainTerms(RelWriter pw) {
private WindowFn<Row, IntervalWindow> windowFn;
private int windowFieldIndex;
private List<FieldAggregation> fieldAggregations;
+ private int groupSetCount;
Review comment:
Yes, fixed!
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -243,9 +247,70 @@ private Transform(
if (windowFn != null) {
windowedStream = assignTimestampsAndWindow(upstream);
}
-
validateWindowIsSupported(windowedStream);
+ return createCombiner(pinput, windowedStream);
+ }
+
+ private PCollection<Row> createCombiner(
+ PCollectionList<Row> pinput, PCollection<Row> windowedStream) {
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createGroupCombiner();
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(
+ mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createGlobalCombiner();
+ return windowedStream.apply(combiner).setRowSchema(outputSchema);
+ }
+ private PTransform<PCollection<Row>, PCollection<Row>> createGlobalCombiner() {
+ org.apache.beam.sdk.schemas.transforms.Group.Global<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsGlobally<Row> combined = null;
+ for (FieldAggregation fieldAggregation : fieldAggregations) {
+ List<Integer> inputs = fieldAggregation.inputs;
+ CombineFn combineFn = fieldAggregation.combineFn;
+ if (inputs.size() > 1 || inputs.isEmpty()) {
Review comment:
Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-902754019
> I think what we really want is to check whether or not the PCollection is in the global window, which is orthogonal to whether or not it's bounded. (In the test data, the bounded input is globally windowed while the unbounded input is not, but that's not always true generally.)
>
> I'll try to do a full review of this PR tomorrow, but I might not get to it till next week.
I changed the condition to validate if the PCollection is in the global window.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r697732974
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -216,6 +216,8 @@ public RelWriter explainTerms(RelWriter pw) {
private WindowFn<Row, IntervalWindow> windowFn;
private int windowFieldIndex;
private List<FieldAggregation> fieldAggregations;
+ private int groupSetCount;
Review comment:
Should these be `final`?
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -243,9 +247,70 @@ private Transform(
if (windowFn != null) {
windowedStream = assignTimestampsAndWindow(upstream);
}
-
validateWindowIsSupported(windowedStream);
+ return createCombiner(pinput, windowedStream);
+ }
+
+ private PCollection<Row> createCombiner(
+ PCollectionList<Row> pinput, PCollection<Row> windowedStream) {
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createGroupCombiner();
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(
+ mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createGlobalCombiner();
+ return windowedStream.apply(combiner).setRowSchema(outputSchema);
+ }
+ private PTransform<PCollection<Row>, PCollection<Row>> createGlobalCombiner() {
Review comment:
`createGroupCombiner` and `createGlobalCombiner` are mostly the same. Can we find a way to reduce code duplication?
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -243,9 +247,70 @@ private Transform(
if (windowFn != null) {
windowedStream = assignTimestampsAndWindow(upstream);
}
-
validateWindowIsSupported(windowedStream);
+ return createCombiner(pinput, windowedStream);
+ }
+
+ private PCollection<Row> createCombiner(
+ PCollectionList<Row> pinput, PCollection<Row> windowedStream) {
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createGroupCombiner();
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(
+ mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createGlobalCombiner();
+ return windowedStream.apply(combiner).setRowSchema(outputSchema);
+ }
+ private PTransform<PCollection<Row>, PCollection<Row>> createGlobalCombiner() {
+ org.apache.beam.sdk.schemas.transforms.Group.Global<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsGlobally<Row> combined = null;
+ for (FieldAggregation fieldAggregation : fieldAggregations) {
+ List<Integer> inputs = fieldAggregation.inputs;
+ CombineFn combineFn = fieldAggregation.combineFn;
+ if (inputs.size() > 1 || inputs.isEmpty()) {
Review comment:
Nit: this logic is simpler if we put the single field case first.
```
if (inputs.size() == 1) {
} else {
}
```
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
##########
@@ -229,27 +232,93 @@ public T apply(T left, T right) {
public Short apply(Short left, Short right) {
return (short) (left + right);
}
+
+ @Override
+ public @Nullable Short identity() {
+ return 0;
+ }
}
static class ByteSum extends Combine.BinaryCombineFn<Byte> {
@Override
public Byte apply(Byte left, Byte right) {
return (byte) (left + right);
}
+
+ @Override
+ public @Nullable Byte identity() {
+ return 0;
+ }
}
static class FloatSum extends Combine.BinaryCombineFn<Float> {
@Override
public Float apply(Float left, Float right) {
return left + right;
}
+
+ @Override
+ public @Nullable Float identity() {
+ return 0F;
+ }
+ }
+
+ static class LongSum extends Combine.BinaryCombineFn<Long> {
+ @Override
+ public Long apply(Long left, Long right) {
+ return Math.addExact(left, right);
+ }
+
+ @Override
+ public @Nullable Long identity() {
+ return 0L;
+ }
}
static class BigDecimalSum extends Combine.BinaryCombineFn<BigDecimal> {
@Override
public BigDecimal apply(BigDecimal left, BigDecimal right) {
return left.add(right);
}
+
+ @Override
+ public @Nullable BigDecimal identity() {
+ return BigDecimal.ZERO;
+ }
+ }
+
+ static class DropNullFn<InputT, AccumT, OutputT> extends CombineFn<InputT, AccumT, OutputT> {
Review comment:
Should this be private?
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
##########
@@ -56,12 +58,13 @@
BUILTIN_AGGREGATOR_FACTORIES =
ImmutableMap.<String, Function<Schema.FieldType, CombineFn<?, ?, ?>>>builder()
.put("ANY_VALUE", typeName -> Sample.anyValueCombineFn())
- .put("COUNT", typeName -> Count.combineFn())
- .put("MAX", BeamBuiltinAggregations::createMax)
- .put("MIN", BeamBuiltinAggregations::createMin)
- .put("SUM", BeamBuiltinAggregations::createSum)
- .put("$SUM0", BeamBuiltinAggregations::createSum)
- .put("AVG", BeamBuiltinAggregations::createAvg)
+ // Drop null elements for these aggregations BEAM-10379
Review comment:
It might be confusing to link to that jira here, since it's referring to functions that _don't_ want to drop null elements.
```suggestion
// Drop null elements for these aggregations.
```
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
##########
@@ -376,29 +445,45 @@ public BigDecimal toBigDecimal(BigDecimal record) {
}
}
- static class BitOr<T extends Number> extends CombineFn<T, Long, Long> {
+ static class BitOr<T extends Number> extends CombineFn<T, BitOr.Accum, Long> {
+ static class Accum implements Serializable {
+ /** True if no inputs have been seen yet. */
+ boolean isEmpty = true;
+ /** The bitwise-and of the inputs seen so far. */
+ long bitOr = 0L;
+ }
+
@Override
- public Long createAccumulator() {
- return 0L;
+ public Accum createAccumulator() {
+ return new Accum();
}
@Override
- public Long addInput(Long accum, T input) {
- return accum | input.longValue();
+ public Accum addInput(Accum accum, T input) {
+ accum.bitOr |= input.longValue();
Review comment:
What if `input` is null?
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn.java
##########
@@ -124,6 +124,11 @@ private BigDecimal getCovariance(CovarianceAccumulator covariance) {
BigDecimal adjustedCount =
this.isSample ? covariance.count().subtract(BigDecimal.ONE) : covariance.count();
+ // Avoid ArithmeticException: Division is undefined when adjustedCount and covariance are 0
Review comment:
Isn't division undefined whenever the denominator is 0, regardless of the numerator?
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
##########
@@ -229,27 +232,93 @@ public T apply(T left, T right) {
public Short apply(Short left, Short right) {
return (short) (left + right);
}
+
+ @Override
+ public @Nullable Short identity() {
+ return 0;
+ }
}
static class ByteSum extends Combine.BinaryCombineFn<Byte> {
@Override
public Byte apply(Byte left, Byte right) {
return (byte) (left + right);
}
+
+ @Override
+ public @Nullable Byte identity() {
+ return 0;
+ }
}
static class FloatSum extends Combine.BinaryCombineFn<Float> {
@Override
public Float apply(Float left, Float right) {
return left + right;
}
+
+ @Override
+ public @Nullable Float identity() {
+ return 0F;
+ }
+ }
+
+ static class LongSum extends Combine.BinaryCombineFn<Long> {
+ @Override
+ public Long apply(Long left, Long right) {
+ return Math.addExact(left, right);
+ }
+
+ @Override
+ public @Nullable Long identity() {
+ return 0L;
+ }
}
static class BigDecimalSum extends Combine.BinaryCombineFn<BigDecimal> {
@Override
public BigDecimal apply(BigDecimal left, BigDecimal right) {
return left.add(right);
}
+
+ @Override
+ public @Nullable BigDecimal identity() {
+ return BigDecimal.ZERO;
+ }
+ }
+
+ static class DropNullFn<InputT, AccumT, OutputT> extends CombineFn<InputT, AccumT, OutputT> {
+ CombineFn<InputT, AccumT, OutputT> combineFn;
Review comment:
Can this be `private final`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r698874206
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -216,6 +216,8 @@ public RelWriter explainTerms(RelWriter pw) {
private WindowFn<Row, IntervalWindow> windowFn;
private int windowFieldIndex;
private List<FieldAggregation> fieldAggregations;
+ private int groupSetCount;
Review comment:
Yes, fixed!
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -243,9 +247,70 @@ private Transform(
if (windowFn != null) {
windowedStream = assignTimestampsAndWindow(upstream);
}
-
validateWindowIsSupported(windowedStream);
+ return createCombiner(pinput, windowedStream);
+ }
+
+ private PCollection<Row> createCombiner(
+ PCollectionList<Row> pinput, PCollection<Row> windowedStream) {
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createGroupCombiner();
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(
+ mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createGlobalCombiner();
+ return windowedStream.apply(combiner).setRowSchema(outputSchema);
+ }
+ private PTransform<PCollection<Row>, PCollection<Row>> createGlobalCombiner() {
+ org.apache.beam.sdk.schemas.transforms.Group.Global<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsGlobally<Row> combined = null;
+ for (FieldAggregation fieldAggregation : fieldAggregations) {
+ List<Integer> inputs = fieldAggregation.inputs;
+ CombineFn combineFn = fieldAggregation.combineFn;
+ if (inputs.size() > 1 || inputs.isEmpty()) {
Review comment:
Changed
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -243,9 +247,70 @@ private Transform(
if (windowFn != null) {
windowedStream = assignTimestampsAndWindow(upstream);
}
-
validateWindowIsSupported(windowedStream);
+ return createCombiner(pinput, windowedStream);
+ }
+
+ private PCollection<Row> createCombiner(
+ PCollectionList<Row> pinput, PCollection<Row> windowedStream) {
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createGroupCombiner();
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(
+ mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createGlobalCombiner();
+ return windowedStream.apply(combiner).setRowSchema(outputSchema);
+ }
+ private PTransform<PCollection<Row>, PCollection<Row>> createGlobalCombiner() {
Review comment:
I created a new interface in order to make the code reusable for both `global` and `byFields` combiners.
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
##########
@@ -56,12 +58,13 @@
BUILTIN_AGGREGATOR_FACTORIES =
ImmutableMap.<String, Function<Schema.FieldType, CombineFn<?, ?, ?>>>builder()
.put("ANY_VALUE", typeName -> Sample.anyValueCombineFn())
- .put("COUNT", typeName -> Count.combineFn())
- .put("MAX", BeamBuiltinAggregations::createMax)
- .put("MIN", BeamBuiltinAggregations::createMin)
- .put("SUM", BeamBuiltinAggregations::createSum)
- .put("$SUM0", BeamBuiltinAggregations::createSum)
- .put("AVG", BeamBuiltinAggregations::createAvg)
+ // Drop null elements for these aggregations BEAM-10379
Review comment:
Comment changed
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
##########
@@ -229,27 +232,93 @@ public T apply(T left, T right) {
public Short apply(Short left, Short right) {
return (short) (left + right);
}
+
+ @Override
+ public @Nullable Short identity() {
+ return 0;
+ }
}
static class ByteSum extends Combine.BinaryCombineFn<Byte> {
@Override
public Byte apply(Byte left, Byte right) {
return (byte) (left + right);
}
+
+ @Override
+ public @Nullable Byte identity() {
+ return 0;
+ }
}
static class FloatSum extends Combine.BinaryCombineFn<Float> {
@Override
public Float apply(Float left, Float right) {
return left + right;
}
+
+ @Override
+ public @Nullable Float identity() {
+ return 0F;
+ }
+ }
+
+ static class LongSum extends Combine.BinaryCombineFn<Long> {
+ @Override
+ public Long apply(Long left, Long right) {
+ return Math.addExact(left, right);
+ }
+
+ @Override
+ public @Nullable Long identity() {
+ return 0L;
+ }
}
static class BigDecimalSum extends Combine.BinaryCombineFn<BigDecimal> {
@Override
public BigDecimal apply(BigDecimal left, BigDecimal right) {
return left.add(right);
}
+
+ @Override
+ public @Nullable BigDecimal identity() {
+ return BigDecimal.ZERO;
+ }
+ }
+
+ static class DropNullFn<InputT, AccumT, OutputT> extends CombineFn<InputT, AccumT, OutputT> {
+ CombineFn<InputT, AccumT, OutputT> combineFn;
Review comment:
Yes, fixed
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
##########
@@ -229,27 +232,93 @@ public T apply(T left, T right) {
public Short apply(Short left, Short right) {
return (short) (left + right);
}
+
+ @Override
+ public @Nullable Short identity() {
+ return 0;
+ }
}
static class ByteSum extends Combine.BinaryCombineFn<Byte> {
@Override
public Byte apply(Byte left, Byte right) {
return (byte) (left + right);
}
+
+ @Override
+ public @Nullable Byte identity() {
+ return 0;
+ }
}
static class FloatSum extends Combine.BinaryCombineFn<Float> {
@Override
public Float apply(Float left, Float right) {
return left + right;
}
+
+ @Override
+ public @Nullable Float identity() {
+ return 0F;
+ }
+ }
+
+ static class LongSum extends Combine.BinaryCombineFn<Long> {
+ @Override
+ public Long apply(Long left, Long right) {
+ return Math.addExact(left, right);
+ }
+
+ @Override
+ public @Nullable Long identity() {
+ return 0L;
+ }
}
static class BigDecimalSum extends Combine.BinaryCombineFn<BigDecimal> {
@Override
public BigDecimal apply(BigDecimal left, BigDecimal right) {
return left.add(right);
}
+
+ @Override
+ public @Nullable BigDecimal identity() {
+ return BigDecimal.ZERO;
+ }
+ }
+
+ static class DropNullFn<InputT, AccumT, OutputT> extends CombineFn<InputT, AccumT, OutputT> {
Review comment:
Yes, fixed
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
##########
@@ -376,29 +445,45 @@ public BigDecimal toBigDecimal(BigDecimal record) {
}
}
- static class BitOr<T extends Number> extends CombineFn<T, Long, Long> {
+ static class BitOr<T extends Number> extends CombineFn<T, BitOr.Accum, Long> {
+ static class Accum implements Serializable {
+ /** True if no inputs have been seen yet. */
+ boolean isEmpty = true;
+ /** The bitwise-and of the inputs seen so far. */
+ long bitOr = 0L;
+ }
+
@Override
- public Long createAccumulator() {
- return 0L;
+ public Accum createAccumulator() {
+ return new Accum();
}
@Override
- public Long addInput(Long accum, T input) {
- return accum | input.longValue();
+ public Accum addInput(Accum accum, T input) {
+ accum.bitOr |= input.longValue();
Review comment:
I added a validation similar to BitAnd
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn.java
##########
@@ -124,6 +124,11 @@ private BigDecimal getCovariance(CovarianceAccumulator covariance) {
BigDecimal adjustedCount =
this.isSample ? covariance.count().subtract(BigDecimal.ONE) : covariance.count();
+ // Avoid ArithmeticException: Division is undefined when adjustedCount and covariance are 0
Review comment:
Yes, I changed the validation and comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r706264370
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Group.java
##########
@@ -364,19 +365,43 @@
}
}
+ public abstract static class AggregateCombiner<InputT>
+ extends PTransform<PCollection<InputT>, PCollection<Row>> {
+ public abstract <CombineInputT, AccumT, CombineOutputT>
+ AggregateCombiner<InputT> aggregateField(
+ int inputFieldId,
+ CombineFn<CombineInputT, AccumT, CombineOutputT> fn,
+ Field outputField);
+
+ public abstract <CombineInputT, AccumT, CombineOutputT>
+ AggregateCombiner<InputT> aggregateField(
+ String inputFieldName,
+ CombineFn<CombineInputT, AccumT, CombineOutputT> fn,
+ Field outputField);
+
+ public abstract <CombineInputT, AccumT, CombineOutputT>
+ AggregateCombiner<InputT> aggregateFieldsById(
+ List<Integer> inputFieldIds,
+ CombineFn<CombineInputT, AccumT, CombineOutputT> fn,
+ Field outputField);
+ }
+
/**
* a {@link PTransform} that does a global combine using an aggregation built up by calls to
* aggregateField and aggregateFields. The output of this transform will have a schema that is
* determined by the output types of all the composed combiners.
*/
- public static class CombineFieldsGlobally<InputT>
- extends PTransform<PCollection<InputT>, PCollection<Row>> {
+ public static class CombineFieldsGlobally<InputT> extends AggregateCombiner<InputT> {
private final SchemaAggregateFn.Inner schemaAggregateFn;
CombineFieldsGlobally(SchemaAggregateFn.Inner schemaAggregateFn) {
this.schemaAggregateFn = schemaAggregateFn;
}
+ public static CombineFieldsGlobally initialCombiner() {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/1c9fff2a5f93b42798b006098bd60aab0877013e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c9fff2) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head 813c6f3. Consider uploading reports for the commit 813c6f3 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.80% 83.84% +0.04%
==========================================
Files 441 441
Lines 59539 59726 +187
==========================================
+ Hits 49894 50075 +181
- Misses 9645 9651 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.28% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <ø> (-0.23%)` | :arrow_down: |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.49% <ø> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <ø> (+0.21%)` | :arrow_up: |
| [...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==) | `75.00% <ø> (ø)` | |
| [...dks/python/apache\_beam/examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5) | `62.16% <ø> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `87.58% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `87.41% <ø> (-0.08%)` | :arrow_down: |
| [...on/apache\_beam/ml/gcp/naturallanguageml\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL25hdHVyYWxsYW5ndWFnZW1sX3Rlc3RfaXQucHk=) | `69.23% <ø> (ø)` | |
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `69.76% <ø> (ø)` | |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c9fff2...813c6f3](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2a9182b) into [master](https://codecov.io/gh/apache/beam/commit/823ab5a1bee062b917cb8811633579614622e1e1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (823ab5a) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2a9182b differs from pull request most recent head 1c4ad22. Consider uploading reports for the commit 1c4ad22 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
=======================================
Coverage 83.74% 83.74%
=======================================
Files 443 443
Lines 60077 60077
=======================================
+ Hits 50312 50313 +1
+ Misses 9765 9764 -1
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.39% <0.00%> (+0.12%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [762a619...1c4ad22](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-917116597
@benWize I tried running Google compliance tests on this PR again. Most tests pass, but this test crashes:
https://github.com/google/zetasql/blob/dd883180de6387ad80bcf7534b5aae8191e66621/zetasql/compliance/testdata/aggregation_queries.test#L1591
```
java.lang.RuntimeException: Could not construct an accumulator Coder for Inner{inputSchema=Fields:
Field{name=$col1, description=, type=INT64 NOT NULL, options={{}}}
Encoding positions:
{$col1=0}
Options:{{}}UUID: 0e4f58b4-f7ac-4531-a10d-df770ec2f974, outputSchema=Fields:
Field{name=$col1, description=, type=INT64, options={{}}}
Encoding positions:
{$col1=0}
Options:{{}}UUID: null, composedCombineFn=org.apache.beam.sdk.transforms.CombineFns$ComposedCombineFn@46d36c10, fieldAggregations=[org.apache.beam.sdk.schemas.transforms.SchemaAggregateFn$Inner$FieldAggregation@2e410f14]}. Accumulator Coder for a Combine may be null, but may not throw an exception
at org.apache.beam.runners.direct.MultiStepCombine$1.isApplicable(MultiStepCombine.java:114)
at org.apache.beam.runners.direct.MultiStepCombine$1.matches(MultiStepCombine.java:83)
at org.apache.beam.sdk.Pipeline$2.enterCompositeTransform(Pipeline.java:278)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:581)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:585)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:585)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:585)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:585)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$500(TransformHierarchy.java:240)
at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:214)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:469)
at org.apache.beam.sdk.Pipeline.replace(Pipeline.java:268)
at org.apache.beam.sdk.Pipeline.replaceAll(Pipeline.java:218)
at org.apache.beam.runners.direct.DirectRunner.performRewrites(DirectRunner.java:246)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:175)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:323)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:309)
at org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:217)
at org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:236)
at org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:163)
at org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:139)
at cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:309)
at com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
Caused by: org.apache.beam.sdk.coders.CannotProvideCoderException: Cannot infer coder for type parameter AccumT
at org.apache.beam.sdk.coders.CoderRegistry.getCoder(CoderRegistry.java:328)
at org.apache.beam.sdk.transforms.CombineFnBase$AbstractGlobalCombineFn.getAccumulatorCoder(CombineFnBase.java:119)
at org.apache.beam.sdk.transforms.Combine$CombineFn.getAccumulatorCoder(Combine.java:389)
at org.apache.beam.sdk.extensions.sql.impl.transform.agg.AggregationCombineFnAdapter$WrappedCombinerBase.getAccumulatorCoder(AggregationCombineFnAdapter.java:72)
at org.apache.beam.sdk.transforms.CombineFns$ComposedCombineFn.getAccumulatorCoder(CombineFns.java:430)
at org.apache.beam.sdk.schemas.transforms.SchemaAggregateFn$Inner.getAccumulatorCoder(SchemaAggregateFn.java:335)
at org.apache.beam.runners.direct.MultiStepCombine$1.isApplicable(MultiStepCombine.java:106)
... 31 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2a9182b) into [master](https://codecov.io/gh/apache/beam/commit/823ab5a1bee062b917cb8811633579614622e1e1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (823ab5a) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2a9182b differs from pull request most recent head 1c4ad22. Consider uploading reports for the commit 1c4ad22 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
=======================================
Coverage 83.74% 83.74%
=======================================
Files 443 443
Lines 60077 60077
=======================================
+ Hits 50312 50313 +1
+ Misses 9765 9764 -1
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.39% <0.00%> (+0.12%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [762a619...1c4ad22](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-907405010
> I changed the condition to validate if the PCollection is in the global window.
@benWize Sorry I haven't been able to review this promptly. I'm planning on reviewing and merging #14729 first, since it's critical we upgrade Calcite. Unfortunately there's a chance that could break this PR. Just a heads up. cc @kileys
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-902327635
> Yes, I tried but it causes a compilation error because of the return type of `.asSingletonView()`. I tried setting ` .withoutDefaults()` just for Unbounded PCollections, and it is working now. [7473306](https://github.com/apache/beam/commit/7473306d414322f284fb3b29f16e280108f9e2d5)
I think what we really want is to check whether or not the PCollection is in the global window, which is orthogonal to whether or not it's bounded. (In the test data, the bounded input is globally windowed while the unbounded input is not, but that's not always true generally.)
I'll try to do a full review of this PR tomorrow, but I might not get to it till next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r682168175
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -246,6 +250,67 @@ private Transform(
validateWindowIsSupported(windowedStream);
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner();
+
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createCombiner() {
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ return createGroupCombiner();
+ }
+ return createGlobalCombiner();
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createGlobalCombiner() {
+ org.apache.beam.sdk.schemas.transforms.Group.Global<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsGlobally<Row> combined = null;
+ for (FieldAggregation fieldAggregation : fieldAggregations) {
+ List<Integer> inputs = fieldAggregation.inputs;
+ CombineFn combineFn = fieldAggregation.combineFn;
+ if (inputs.size() > 1 || inputs.isEmpty()) {
+ // In this path we extract a Row (an empty row if inputs.isEmpty).
+ combined =
+ (combined == null)
+ ? globally.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField)
+ : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
+ } else {
+ // Combining over a single field, so extract just that field.
+ combined =
+ (combined == null)
+ ? globally.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField)
+ : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
+ }
+ }
+
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = combined;
+ if (combiner == null) {
+ // If no field aggregations were specified, we run a constant combiner that always returns
+ // a single empty row for each key. This is used by the SELECT DISTINCT query plan - in this
+ // case a group by is generated to determine unique keys, and a constant null combiner is
+ // used.
+ combiner =
+ globally.aggregateField(
+ "*",
+ AggregationCombineFnAdapter.createConstantCombineFn(),
+ Field.of(
+ "e",
+ FieldType.row(AggregationCombineFnAdapter.EMPTY_SCHEMA).withNullable(true)));
+ ignoreValues = true;
+ }
+ return combiner;
+ }
Review comment:
I created this method to process aggregations without "GROUP BY".
Some tests related to aggregations on empty tables are passing now, but different tests like this https://github.com/apache/beam/blob/bdee3bd963d3213c207ea124a6bfd8256deb220b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlDialectSpecTest.java#L2216 are failing with this change, I am not sure why, but when this [transform](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Group.java#L555 ) is applied, the data is dropped and does not execute the combineFn aggregation methods..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2a9182b) into [master](https://codecov.io/gh/apache/beam/commit/823ab5a1bee062b917cb8811633579614622e1e1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (823ab5a) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2a9182b differs from pull request most recent head a996e62. Consider uploading reports for the commit a996e62 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
=======================================
Coverage 83.74% 83.74%
=======================================
Files 443 443
Lines 60077 60077
=======================================
+ Hits 50312 50313 +1
+ Misses 9765 9764 -1
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.39% <0.00%> (+0.12%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [762a619...a996e62](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib merged pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib merged pull request #15174:
URL: https://github.com/apache/beam/pull/15174
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-910668690
> There is one ZetaSQL compliance test failure. (ZetaSQL is mostly open source but its compliance tests are not yet, so we usually end up duplicating the tests in Beam where necessary.)
>
> It seems we are not handling this case properly: Returns NULL if the input contains only NULLs. (from https://github.com/google/zetasql/blob/master/docs/aggregate_functions.md#sum)
>
> Can you write a test for this case and fix it?
I added the test, and fix the null handling case, but I had to branch `SUM` and `SUM0` logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r701442197
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -216,6 +216,8 @@ public RelWriter explainTerms(RelWriter pw) {
private WindowFn<Row, IntervalWindow> windowFn;
private int windowFieldIndex;
private List<FieldAggregation> fieldAggregations;
+ private final int groupSetCount;
+ private boolean ignoreValues;
Review comment:
Nit: this can be `final` too.
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
##########
@@ -163,6 +167,32 @@ static CombineFn createSum(Schema.FieldType fieldType) {
}
}
+ /**
+ * {@link CombineFn} for Sum0 where sum of null returns 0 based on {@link Sum} and {@link
+ * Combine.BinaryCombineFn}.
+ */
+ static CombineFn createSum0(Schema.FieldType fieldType) {
+ switch (fieldType.getTypeName()) {
+ case INT32:
+ return new IntegerSum0();
+ case INT16:
+ return new ShortSum0();
+ case BYTE:
+ return new ByteSum0();
+ case INT64:
+ return new LongSum0();
+ case FLOAT:
+ return new FloatSum0();
+ case DOUBLE:
+ return new DoubleSum0();
+ case DECIMAL:
+ return new BigDecimalSum0();
+ default:
+ throw new UnsupportedOperationException(
+ String.format("[%s] is not supported in SUM", fieldType));
Review comment:
```suggestion
String.format("[%s] is not supported in SUM0", fieldType));
```
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -243,55 +247,72 @@ private Transform(
if (windowFn != null) {
windowedStream = assignTimestampsAndWindow(upstream);
}
-
validateWindowIsSupported(windowedStream);
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> byFields =
+ org.apache.beam.sdk.schemas.transforms.Group.byFieldIds(keyFieldsIds);
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner(byFields);
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(
+ mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createCombiner(globally);
+ return windowedStream.apply(combiner).setRowSchema(outputSchema);
+ }
+
+ private PTransform<PCollection<Row>, PCollection<Row>> createCombiner(
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row>
+ initialCombiner) {
- org.apache.beam.sdk.schemas.transforms.Group.ByFields<Row> byFields =
- org.apache.beam.sdk.schemas.transforms.Group.byFieldIds(keyFieldsIds);
- org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsByFields<Row> combined = null;
+ org.apache.beam.sdk.schemas.transforms.Group.AggregateCombinerInterface<Row> combined = null;
for (FieldAggregation fieldAggregation : fieldAggregations) {
List<Integer> inputs = fieldAggregation.inputs;
CombineFn combineFn = fieldAggregation.combineFn;
- if (inputs.size() > 1 || inputs.isEmpty()) {
- // In this path we extract a Row (an empty row if inputs.isEmpty).
+ if (inputs.size() == 1) {
+ // Combining over a single field, so extract just that field.
combined =
(combined == null)
- ? byFields.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField)
- : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
+ ? initialCombiner.aggregateField(
+ inputs.get(0), combineFn, fieldAggregation.outputField)
+ : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
} else {
- // Combining over a single field, so extract just that field.
+ // In this path we extract a Row (an empty row if inputs.isEmpty).
combined =
(combined == null)
- ? byFields.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField)
- : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField);
+ ? initialCombiner.aggregateFieldsById(
+ inputs, combineFn, fieldAggregation.outputField)
+ : combined.aggregateFieldsById(inputs, combineFn, fieldAggregation.outputField);
}
}
- PTransform<PCollection<Row>, PCollection<Row>> combiner = combined;
- boolean ignoreValues = false;
+ PTransform<PCollection<Row>, PCollection<Row>> combiner =
+ (PTransform<PCollection<Row>, PCollection<Row>>) combined;
Review comment:
It looks like the problem is that `CombineFieldsGlobally` and `CombineFieldsByFields` extend `PTransform<PCollection<InputT>, PCollection<Row>>`, but `Global` extends `PTransform<PCollection<InputT>, PCollection<Iterable<InputT>>>`. However, it looks like you only use `Global.globally().aggregateField(...)`, which returns a `CombineFieldsGlobally` transform anyway. https://github.com/apache/beam/blob/3a7b8e757d0c220b5838ca6a0805ee463b24f85d/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Group.java#L171
So can you try using `CombineFieldsGlobally` as the initial combiner instead of `Global`? Then `Global` doesn't have to extend `AggregateCombiner` at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-920277026
> All compliance tests are passing now. Thanks for all your hard work!
>
> LGTM
Thank you for all your guidance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/1c9fff2a5f93b42798b006098bd60aab0877013e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c9fff2) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head f85cef3. Consider uploading reports for the commit f85cef3 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.80% 83.84% +0.04%
==========================================
Files 441 441
Lines 59539 59726 +187
==========================================
+ Hits 49894 50075 +181
- Misses 9645 9651 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.28% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <ø> (-0.23%)` | :arrow_down: |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.49% <ø> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <ø> (+0.21%)` | :arrow_up: |
| [...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==) | `75.00% <ø> (ø)` | |
| [...dks/python/apache\_beam/examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5) | `62.16% <ø> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `87.58% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `87.41% <ø> (-0.08%)` | :arrow_down: |
| [...on/apache\_beam/ml/gcp/naturallanguageml\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL25hdHVyYWxsYW5ndWFnZW1sX3Rlc3RfaXQucHk=) | `69.23% <ø> (ø)` | |
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `69.76% <ø> (ø)` | |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c9fff2...f85cef3](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/1c9fff2a5f93b42798b006098bd60aab0877013e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c9fff2) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head 22e2adb. Consider uploading reports for the commit 22e2adb to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.80% 83.84% +0.04%
==========================================
Files 441 441
Lines 59539 59726 +187
==========================================
+ Hits 49894 50075 +181
- Misses 9645 9651 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.28% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <ø> (-0.23%)` | :arrow_down: |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.49% <ø> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <ø> (+0.21%)` | :arrow_up: |
| [...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==) | `75.00% <ø> (ø)` | |
| [...dks/python/apache\_beam/examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5) | `62.16% <ø> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `87.58% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `87.41% <ø> (-0.08%)` | :arrow_down: |
| [...on/apache\_beam/ml/gcp/naturallanguageml\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL25hdHVyYWxsYW5ndWFnZW1sX3Rlc3RfaXQucHk=) | `69.23% <ø> (ø)` | |
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `69.76% <ø> (ø)` | |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c9fff2...22e2adb](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r698874702
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -243,9 +247,70 @@ private Transform(
if (windowFn != null) {
windowedStream = assignTimestampsAndWindow(upstream);
}
-
validateWindowIsSupported(windowedStream);
+ return createCombiner(pinput, windowedStream);
+ }
+
+ private PCollection<Row> createCombiner(
+ PCollectionList<Row> pinput, PCollection<Row> windowedStream) {
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createGroupCombiner();
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(
+ mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createGlobalCombiner();
+ return windowedStream.apply(combiner).setRowSchema(outputSchema);
+ }
+ private PTransform<PCollection<Row>, PCollection<Row>> createGlobalCombiner() {
Review comment:
I created a new interface in order to make the code reusable for both `global` and `byFields` combiners.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/1c9fff2a5f93b42798b006098bd60aab0877013e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c9fff2) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head 63f1e7a. Consider uploading reports for the commit 63f1e7a to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.80% 83.84% +0.04%
==========================================
Files 441 441
Lines 59539 59726 +187
==========================================
+ Hits 49894 50075 +181
- Misses 9645 9651 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.28% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <ø> (-0.23%)` | :arrow_down: |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.49% <ø> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <ø> (+0.21%)` | :arrow_up: |
| [...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==) | `75.00% <ø> (ø)` | |
| [...dks/python/apache\_beam/examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5) | `62.16% <ø> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `87.58% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `87.41% <ø> (-0.08%)` | :arrow_down: |
| [...on/apache\_beam/ml/gcp/naturallanguageml\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL25hdHVyYWxsYW5ndWFnZW1sX3Rlc3RfaXQucHk=) | `69.23% <ø> (ø)` | |
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `69.76% <ø> (ø)` | |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c9fff2...63f1e7a](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r698875353
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn.java
##########
@@ -124,6 +124,11 @@ private BigDecimal getCovariance(CovarianceAccumulator covariance) {
BigDecimal adjustedCount =
this.isSample ? covariance.count().subtract(BigDecimal.ONE) : covariance.count();
+ // Avoid ArithmeticException: Division is undefined when adjustedCount and covariance are 0
Review comment:
Yes, I changed the validation and comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r700527449
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Group.java
##########
@@ -364,13 +369,29 @@
}
}
+ public interface AggregateCombinerInterface<InputT> {
Review comment:
Changed
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Group.java
##########
@@ -364,13 +369,29 @@
}
}
+ public interface AggregateCombinerInterface<InputT> {
+ <CombineInputT, AccumT, CombineOutputT> AggregateCombinerInterface<InputT> aggregateField(
+ int inputFielId, CombineFn<CombineInputT, AccumT, CombineOutputT> fn, Field outputField);
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-917148386
Run SQL PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/1c9fff2a5f93b42798b006098bd60aab0877013e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c9fff2) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head b49b254. Consider uploading reports for the commit b49b254 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.80% 83.84% +0.04%
==========================================
Files 441 441
Lines 59539 59726 +187
==========================================
+ Hits 49894 50075 +181
- Misses 9645 9651 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.28% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <ø> (-0.23%)` | :arrow_down: |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.49% <ø> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <ø> (+0.21%)` | :arrow_up: |
| [...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==) | `75.00% <ø> (ø)` | |
| [...dks/python/apache\_beam/examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5) | `62.16% <ø> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `87.58% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `87.41% <ø> (-0.08%)` | :arrow_down: |
| [...on/apache\_beam/ml/gcp/naturallanguageml\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL25hdHVyYWxsYW5ndWFnZW1sX3Rlc3RfaXQucHk=) | `69.23% <ø> (ø)` | |
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `69.76% <ø> (ø)` | |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c9fff2...b49b254](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r703740164
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Group.java
##########
@@ -364,19 +365,43 @@
}
}
+ public abstract static class AggregateCombiner<InputT>
+ extends PTransform<PCollection<InputT>, PCollection<Row>> {
+ public abstract <CombineInputT, AccumT, CombineOutputT>
+ AggregateCombiner<InputT> aggregateField(
+ int inputFieldId,
+ CombineFn<CombineInputT, AccumT, CombineOutputT> fn,
+ Field outputField);
+
+ public abstract <CombineInputT, AccumT, CombineOutputT>
+ AggregateCombiner<InputT> aggregateField(
+ String inputFieldName,
+ CombineFn<CombineInputT, AccumT, CombineOutputT> fn,
+ Field outputField);
+
+ public abstract <CombineInputT, AccumT, CombineOutputT>
+ AggregateCombiner<InputT> aggregateFieldsById(
+ List<Integer> inputFieldIds,
+ CombineFn<CombineInputT, AccumT, CombineOutputT> fn,
+ Field outputField);
+ }
+
/**
* a {@link PTransform} that does a global combine using an aggregation built up by calls to
* aggregateField and aggregateFields. The output of this transform will have a schema that is
* determined by the output types of all the composed combiners.
*/
- public static class CombineFieldsGlobally<InputT>
- extends PTransform<PCollection<InputT>, PCollection<Row>> {
+ public static class CombineFieldsGlobally<InputT> extends AggregateCombiner<InputT> {
private final SchemaAggregateFn.Inner schemaAggregateFn;
CombineFieldsGlobally(SchemaAggregateFn.Inner schemaAggregateFn) {
this.schemaAggregateFn = schemaAggregateFn;
}
+ public static CombineFieldsGlobally initialCombiner() {
Review comment:
Add a javadoc comment for this method.
Also consider renaming it to something more standard. Factory methods like this are usually just called `create` (like `SchemaAggregateFn.create()`) so we can follow that pattern here.
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Group.java
##########
@@ -364,19 +365,43 @@
}
}
+ public abstract static class AggregateCombiner<InputT>
Review comment:
Add a javadoc comment for this interface and its methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-884585032
> Hi @ibzib, I am working on a group of related issues using some of your work in #14392. Could you help me to take a look at the progress so far and provide some guidance on the next steps?
>
> I am planning to include [BEAM-12098](https://issues.apache.org/jira/browse/BEAM-12098) and [BEAM-12099](https://issues.apache.org/jira/browse/BEAM-12099) in this PR, but for those, I have a problem, in the transformation related to UNNEST, when it receives an empty array, does not return anything so it fails before aggregation transforms (COUNT, BIT_OR).
>
> I think the problem could be in this part https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/unnest/BeamZetaSqlUncollectRel.java#L115 because when the array is empty, this transform does not output anything.
>
> Any guidance on this would be appreciated.
> Thanks!
Hi Benjamin, thanks for working on this.
If I understand correctly, `UNNEST([])` returning no values (an empty pcollection) is intended behavior.
I experimented with other queries and found that `SELECT COUNT(*) FROM table_empty` also returned no values. So I think the problem is Beam SQL's aggregation implementation. Aggregations on empty pcollections are supposed to return some default value, for example Count on an empty pcollection is expected to return 0. I filed a bug for this: BEAM-12647
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r702196988
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
##########
@@ -163,6 +167,32 @@ static CombineFn createSum(Schema.FieldType fieldType) {
}
}
+ /**
+ * {@link CombineFn} for Sum0 where sum of null returns 0 based on {@link Sum} and {@link
+ * Combine.BinaryCombineFn}.
+ */
+ static CombineFn createSum0(Schema.FieldType fieldType) {
+ switch (fieldType.getTypeName()) {
+ case INT32:
+ return new IntegerSum0();
+ case INT16:
+ return new ShortSum0();
+ case BYTE:
+ return new ByteSum0();
+ case INT64:
+ return new LongSum0();
+ case FLOAT:
+ return new FloatSum0();
+ case DOUBLE:
+ return new DoubleSum0();
+ case DECIMAL:
+ return new BigDecimalSum0();
+ default:
+ throw new UnsupportedOperationException(
+ String.format("[%s] is not supported in SUM", fieldType));
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-918719763
Hi @benWize the operators for BIT_OR etc. are taken from Calcite's SqlStdOperatorTable.
https://github.com/apache/beam/blob/be083918108579d07780a23f179e2bd62b3abcfe/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SqlOperatorMappingTable.java#L79
To override their return type inference, we will need replace the operators from SqlStdOperatorTable with custom operators defined in SqlOperators, and make sure nullability is set correctly. For example, BIT_XOR already has a custom operator, but it looks like the return type is non-null.
https://github.com/apache/beam/blob/be083918108579d07780a23f179e2bd62b3abcfe/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SqlOperators.java#L164
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r702208741
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -216,6 +216,8 @@ public RelWriter explainTerms(RelWriter pw) {
private WindowFn<Row, IntervalWindow> windowFn;
private int windowFieldIndex;
private List<FieldAggregation> fieldAggregations;
+ private final int groupSetCount;
+ private boolean ignoreValues;
Review comment:
Oops, I missed that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2a9182b) into [master](https://codecov.io/gh/apache/beam/commit/823ab5a1bee062b917cb8811633579614622e1e1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (823ab5a) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 2a9182b differs from pull request most recent head 8adc437. Consider uploading reports for the commit 8adc437 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
=======================================
Coverage 83.74% 83.74%
=======================================
Files 443 443
Lines 60077 60077
=======================================
+ Hits 50312 50313 +1
+ Misses 9765 9764 -1
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.39% <0.00%> (+0.12%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [762a619...8adc437](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/3a7b8e757d0c220b5838ca6a0805ee463b24f85d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3a7b8e7) will **increase** coverage by `0.09%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head e3665c9. Consider uploading reports for the commit e3665c9 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.74% 83.84% +0.09%
==========================================
Files 442 441 -1
Lines 60050 59726 -324
==========================================
- Hits 50290 50075 -215
+ Misses 9760 9651 -109
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache\_beam/runners/dataflow/internal/apiclient.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9hcGljbGllbnQucHk=) | `74.66% <0.00%> (-1.80%)` | :arrow_down: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `90.70% <0.00%> (-0.38%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/trigger.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy90cmlnZ2VyLnB5) | `89.27% <0.00%> (-0.35%)` | :arrow_down: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <0.00%> (-0.19%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/parquetio.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vcGFycXVldGlvLnB5) | `95.95% <0.00%> (-0.14%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.89% <0.00%> (-0.11%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `93.60% <0.00%> (-0.11%)` | :arrow_down: |
| [...thon/apache\_beam/runners/worker/sdk\_worker\_main.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlcl9tYWluLnB5) | `72.26% <0.00%> (-0.09%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `86.15% <0.00%> (-0.07%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/filebasedsource.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWJhc2Vkc291cmNlLnB5) | `98.75% <0.00%> (-0.04%)` | :arrow_down: |
| ... and [28 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2b4aaa0...e3665c9](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/1c9fff2a5f93b42798b006098bd60aab0877013e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c9fff2) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head 77a11a1. Consider uploading reports for the commit 77a11a1 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.80% 83.84% +0.04%
==========================================
Files 441 441
Lines 59539 59726 +187
==========================================
+ Hits 49894 50075 +181
- Misses 9645 9651 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.28% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <ø> (-0.23%)` | :arrow_down: |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.49% <ø> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <ø> (+0.21%)` | :arrow_up: |
| [...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==) | `75.00% <ø> (ø)` | |
| [...dks/python/apache\_beam/examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5) | `62.16% <ø> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `87.58% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `87.41% <ø> (-0.08%)` | :arrow_down: |
| [...on/apache\_beam/ml/gcp/naturallanguageml\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL25hdHVyYWxsYW5ndWFnZW1sX3Rlc3RfaXQucHk=) | `69.23% <ø> (ø)` | |
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `69.76% <ø> (ø)` | |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c9fff2...77a11a1](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r698874998
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
##########
@@ -229,27 +232,93 @@ public T apply(T left, T right) {
public Short apply(Short left, Short right) {
return (short) (left + right);
}
+
+ @Override
+ public @Nullable Short identity() {
+ return 0;
+ }
}
static class ByteSum extends Combine.BinaryCombineFn<Byte> {
@Override
public Byte apply(Byte left, Byte right) {
return (byte) (left + right);
}
+
+ @Override
+ public @Nullable Byte identity() {
+ return 0;
+ }
}
static class FloatSum extends Combine.BinaryCombineFn<Float> {
@Override
public Float apply(Float left, Float right) {
return left + right;
}
+
+ @Override
+ public @Nullable Float identity() {
+ return 0F;
+ }
+ }
+
+ static class LongSum extends Combine.BinaryCombineFn<Long> {
+ @Override
+ public Long apply(Long left, Long right) {
+ return Math.addExact(left, right);
+ }
+
+ @Override
+ public @Nullable Long identity() {
+ return 0L;
+ }
}
static class BigDecimalSum extends Combine.BinaryCombineFn<BigDecimal> {
@Override
public BigDecimal apply(BigDecimal left, BigDecimal right) {
return left.add(right);
}
+
+ @Override
+ public @Nullable BigDecimal identity() {
+ return BigDecimal.ZERO;
+ }
+ }
+
+ static class DropNullFn<InputT, AccumT, OutputT> extends CombineFn<InputT, AccumT, OutputT> {
Review comment:
Yes, fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15174: [WIP][BEAM-12100][BEAM-10379][BEAM-9514] AssertionError type mismatch from AggregateScanConverter
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15174:
URL: https://github.com/apache/beam/pull/15174#issuecomment-893071114
# [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15174](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39cf3fc) into [master](https://codecov.io/gh/apache/beam/commit/1c9fff2a5f93b42798b006098bd60aab0877013e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c9fff2) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 39cf3fc differs from pull request most recent head 7473306. Consider uploading reports for the commit 7473306 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15174 +/- ##
==========================================
+ Coverage 83.80% 83.84% +0.04%
==========================================
Files 441 441
Lines 59539 59726 +187
==========================================
+ Hits 49894 50075 +181
- Misses 9645 9651 +6
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `95.28% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVycy5weQ==) | `87.80% <ø> (-0.23%)` | :arrow_down: |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.49% <ø> (+0.01%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `94.87% <ø> (+0.21%)` | :arrow_up: |
| [...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==) | `75.00% <ø> (ø)` | |
| [...dks/python/apache\_beam/examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5) | `62.16% <ø> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `87.58% <ø> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `87.41% <ø> (-0.08%)` | :arrow_down: |
| [...on/apache\_beam/ml/gcp/naturallanguageml\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL25hdHVyYWxsYW5ndWFnZW1sX3Rlc3RfaXQucHk=) | `69.23% <ø> (ø)` | |
| [...n/apache\_beam/ml/gcp/recommendations\_ai\_test\_it.py](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3JlY29tbWVuZGF0aW9uc19haV90ZXN0X2l0LnB5) | `69.76% <ø> (ø)` | |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/15174/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1c9fff2...7473306](https://codecov.io/gh/apache/beam/pull/15174?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org