You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/08/30 23:28:42 UTC
[GitHub] [beam] benWize commented on a change in pull request #15174: [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099] AssertionError type mismatch from AggregateScanConverter
benWize commented on a change in pull request #15174:
URL: https://github.com/apache/beam/pull/15174#discussion_r698874206
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -216,6 +216,8 @@ public RelWriter explainTerms(RelWriter pw) {
private WindowFn<Row, IntervalWindow> windowFn;
private int windowFieldIndex;
private List<FieldAggregation> fieldAggregations;
+ private int groupSetCount;
Review comment:
Yes, fixed!
##########
File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
##########
@@ -243,9 +247,70 @@ private Transform(
if (windowFn != null) {
windowedStream = assignTimestampsAndWindow(upstream);
}
-
validateWindowIsSupported(windowedStream);
+ return createCombiner(pinput, windowedStream);
+ }
+
+ private PCollection<Row> createCombiner(
+ PCollectionList<Row> pinput, PCollection<Row> windowedStream) {
+ // Check if have fields to be grouped
+ if (groupSetCount > 0) {
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createGroupCombiner();
+ boolean verifyRowValues =
+ pinput.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getVerifyRowValues();
+ return windowedStream
+ .apply(combiner)
+ .apply(
+ "mergeRecord",
+ ParDo.of(
+ mergeRecord(outputSchema, windowFieldIndex, ignoreValues, verifyRowValues)))
+ .setRowSchema(outputSchema);
+ }
+ PTransform<PCollection<Row>, PCollection<Row>> combiner = createGlobalCombiner();
+ return windowedStream.apply(combiner).setRowSchema(outputSchema);
+ }
+ private PTransform<PCollection<Row>, PCollection<Row>> createGlobalCombiner() {
+ org.apache.beam.sdk.schemas.transforms.Group.Global<Row> globally =
+ org.apache.beam.sdk.schemas.transforms.Group.globally();
+ org.apache.beam.sdk.schemas.transforms.Group.CombineFieldsGlobally<Row> combined = null;
+ for (FieldAggregation fieldAggregation : fieldAggregations) {
+ List<Integer> inputs = fieldAggregation.inputs;
+ CombineFn combineFn = fieldAggregation.combineFn;
+ if (inputs.size() > 1 || inputs.isEmpty()) {
Review comment:
Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org