You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "razvanculea (via GitHub)" <gi...@apache.org> on 2025/03/17 10:21:47 UTC
[PR] Add 3 examples in playground SQL transform, schema transform and Composite Combine [beam]
razvanculea opened a new pull request, #34322:
URL: https://github.com/apache/beam/pull/34322
3 simple examples of applying multiple combiners on the input PCollection (Min,Max,Sum).
3 different ways of achieving this in Beam
* SQL transform, using the powerful SQL syntax
* Schema transformations, leveraging schemas and Group.byFilelds
* Composed Combiners, same results without the need to use Rows or Schema transforms and with full control over the combiner logic
------------------------
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add 3 examples in playground SQL transform, schema transform and Composite Combine [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #34322:
URL: https://github.com/apache/beam/pull/34322#discussion_r2031494064
##########
examples/java/build.gradle:
##########
@@ -64,6 +64,7 @@ dependencies {
implementation project(":sdks:java:extensions:avro")
implementation project(":sdks:java:extensions:google-cloud-platform-core")
implementation project(":sdks:java:extensions:python")
+ implementation project(":sdks:java:extensions:sql")
Review Comment:
discussed offline, the goal is to add a Beam SQL example to plaground, where SQL dependencies already present (added by go/beampr/26227).
Rearranging example jars is nontrivial to playground, so good for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add 3 examples in playground SQL transform, schema transform and Composite Combine [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #34322:
URL: https://github.com/apache/beam/pull/34322#issuecomment-2787410478
The error message is:
```
* What went wrong:
Execution failed for task ':examples:java:sql:analyzeClassesDependencies'.
> Dependency analysis found issues.
unusedDeclaredArtifacts
- beam.sdks.java.extensions:google-cloud-platform-core:2.65.0-SNAPSHOT@jar
- beam.sdks.java.extensions:ml:2.65.0-SNAPSHOT@jar
- beam.sdks.java.extensions:python:2.65.0-SNAPSHOT@jar
- beam.sdks.java.io:google-cloud-platform:2.65.0-SNAPSHOT@jar
- beam.sdks.java:managed:2.65.0-SNAPSHOT@jar
- com.fasterxml.jackson.core:jackson-annotations:2.15.4@jar
- com.fasterxml.jackson.core:jackson-core:2.15.4@jar
- com.fasterxml.jackson.core:jackson-databind:2.15.4@jar
- com.google.api-client:google-api-client:2.0.0@jar
- com.google.api.grpc:proto-google-cloud-bigtable-v2:2.56.0@jar
- com.google.api.grpc:proto-google-cloud-datastore-v1:0.118.1@jar
- com.google.api.grpc:proto-google-cloud-language-v1:2.61.0@jar
- com.google.apis:google-api-services-bigquery:v2-rev20250313-2.0.0@jar
- com.google.apis:google-api-services-pubsub:v1-rev20220904-2.0.0@jar
- com.google.auth:google-auth-library-credentials:1.33.1@jar
- com.google.auth:google-auth-library-oauth2-http:1.33.1@jar
- com.google.cloud.bigdataoss:util:2.2.26@jar
- com.google.cloud.datastore:datastore-v1-proto-client:2.27.1@jar
- com.google.code.gson:gson:2.10.1@jar
- com.google.http-client:google-http-client:1.46.3@jar
- com.google.oauth-client:google-oauth-client:1.34.1@jar
- com.google.protobuf:protobuf-java:4.29.4@jar
- commons-io:commons-io:2.16.1@jar
- joda-time:joda-time:2.10.14@jar
- org.apache.beam:beam-vendor-grpc-1_69_0:0.1@jar
- org.apache.beam:beam-vendor-guava-32_1_2-jre:0.1@jar
- org.apache.commons:commons-csv:1.8@jar
- org.apache.commons:commons-lang3:3.14.0@jar
- org.apache.httpcomponents:httpclient:4.5.13@jar
- org.apache.httpcomponents:httpcore:4.4.14@jar
- org.apache.kafka:kafka-clients:2.4.1@jar
```
need to clean up build.gradle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add 3 examples in playground SQL transform, schema transform and Composite Combine [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34322:
URL: https://github.com/apache/beam/pull/34322#issuecomment-2778519862
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @kennknowles for label java.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add 3 examples in playground SQL transform, schema transform and Composite Combine [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #34322:
URL: https://github.com/apache/beam/pull/34322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add 3 examples in playground SQL transform, schema transform and Composite Combine [beam]
Posted by "razvanculea (via GitHub)" <gi...@apache.org>.
razvanculea commented on PR #34322:
URL: https://github.com/apache/beam/pull/34322#issuecomment-2740391758
@damondouglas this [blog post](https://medium.com/@razvan.culea/efficiently-applying-multiple-combiners-on-the-same-input-in-apache-beam-fd20a37dba35) will explain how to apply multiple combiners to an input PColl in different ways with Beam and link to these 3 examples
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add 3 examples in playground SQL transform, schema transform and Composite Combine [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34322:
URL: https://github.com/apache/beam/pull/34322#issuecomment-2729089074
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add 3 examples in playground SQL transform, schema transform and Composite Combine [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34322:
URL: https://github.com/apache/beam/pull/34322#issuecomment-2729984432
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @damondouglas for label java.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add 3 examples in playground SQL transform, schema transform and Composite Combine [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #34322:
URL: https://github.com/apache/beam/pull/34322#discussion_r2031489947
##########
examples/java/src/main/java/org/apache/beam/examples/SqlTransformExample.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples;
+
+// beam-playground:
+// name: SqlTransform
+// description: Demonstration of SQL transform usage.
+// multifile: false
+// default_example: false
+// context_line: 60
+// categories:
+// - SQL
Review Comment:
Use "Beam SQL" per https://github.com/apache/beam/blob/master/playground/categories.yaml#L33
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add 3 examples in playground SQL transform, schema transform and Composite Combine [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #34322:
URL: https://github.com/apache/beam/pull/34322#discussion_r2023215098
##########
examples/java/build.gradle:
##########
@@ -64,6 +64,7 @@ dependencies {
implementation project(":sdks:java:extensions:avro")
implementation project(":sdks:java:extensions:google-cloud-platform-core")
implementation project(":sdks:java:extensions:python")
+ implementation project(":sdks:java:extensions:sql")
Review Comment:
This would introduce the heavy beam-sql dependencies to examples-java. Consider create a subproject under example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add 3 examples in playground SQL transform, schema transform and Composite Combine [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34322:
URL: https://github.com/apache/beam/pull/34322#issuecomment-2761192176
Reminder, please take a look at this pr: @damondouglas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add 3 examples in playground SQL transform, schema transform and Composite Combine [beam]
Posted by "razvanculea (via GitHub)" <gi...@apache.org>.
razvanculea commented on PR #34322:
URL: https://github.com/apache/beam/pull/34322#issuecomment-2765314032
assign to next reviewer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add 3 examples in playground SQL transform, schema transform and Composite Combine [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #34322:
URL: https://github.com/apache/beam/pull/34322#issuecomment-2786593726
Please add the new subproject to https://github.com/apache/beam/blob/master/build.gradle.kts#L254 to include the build in precommit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org