You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by vt...@apache.org on 2025/06/04 07:09:43 UTC
(beam) branch fix-playground created (now 48b849e926d)
This is an automated email from the ASF dual-hosted git repository.
vterentev pushed a change to branch fix-playground
in repository https://gitbox.apache.org/repos/asf/beam.git
at 48b849e926d Fix Sql examples for playground
This branch includes the following new commits:
new 48b849e926d Fix Sql examples for playground
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(beam) 01/01: Fix Sql examples for playground
Posted by vt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
vterentev pushed a commit to branch fix-playground
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 48b849e926de27c09e82523b3a8b7ff1e3ba076e
Author: Vitaly Terentyev <vi...@akvelon.com>
AuthorDate: Wed Jun 4 11:09:34 2025 +0400
Fix Sql examples for playground
---
examples/java/build.gradle | 1 +
examples/java/sql/README.md | 51 ---------
examples/java/sql/build.gradle | 122 ---------------------
.../beam/examples}/SchemaTransformExample.java | 4 +-
.../apache/beam/examples}/SqlTransformExample.java | 2 +-
5 files changed, 4 insertions(+), 176 deletions(-)
diff --git a/examples/java/build.gradle b/examples/java/build.gradle
index e10d4fd6b1d..9546e9270a3 100644
--- a/examples/java/build.gradle
+++ b/examples/java/build.gradle
@@ -64,6 +64,7 @@ dependencies {
implementation project(":sdks:java:extensions:avro")
implementation project(":sdks:java:extensions:google-cloud-platform-core")
implementation project(":sdks:java:extensions:python")
+ implementation project(":sdks:java:extensions:sql")
implementation project(":sdks:java:io:google-cloud-platform")
implementation project(":sdks:java:io:kafka")
runtimeOnly project(":sdks:java:io:iceberg")
diff --git a/examples/java/sql/README.md b/examples/java/sql/README.md
deleted file mode 100644
index d27b8ce7025..00000000000
--- a/examples/java/sql/README.md
+++ /dev/null
@@ -1,51 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Example Pipelines for Beam SQL and Schema Transforms
-
-The examples included in this module serve to demonstrate the basic
-functionality of Apache Beam SQL, and act as starting points for
-the development of more complex pipelines.
-
-## SQL transform
-
-An example that leverage the powerful SQL syntax in Beam [SqlTransform](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/sql/SqlTransform.html) directly in your Beam pipelines.
-
-
-[`SqlTransformExample`](https://github.com/apache/beam/blob/master/examples/java/sql/src/main/java/org/apache/beam/examples/sql/SqlTransformExample.java) is a simple pipeline that calculates multiple metrics per key : Min, Max and Sum.
-
-
-## Schema transform
-
-[Beam Schemas](https://beam.apache.org/documentation/programming-guide/#schemas) offer a flexible way of writing in code the same operations that are so easy to express in SQL.
-
-[`SchemaTransformExample`](https://github.com/apache/beam/blob/master/examples/java/sql/src/main/java/org/apache/beam/examples/sql/SchemaTransformExample.java) is a simple pipeline that calculates multiple metrics per key : Min, Max and Sum.
-
-## Running Examples
-
-See [Apache Beam WordCount Example](https://beam.apache.org/get-started/wordcount-example/) for information on running these examples.
-
-Gradle cmd line can be similar to:
-
-`./gradlew clean :examples:java:sql:execute --args="--runner=DirectRunner" -Pdirect-runner -PmainClass=org.apache.beam.examples.sql.SqlTransformExample`
-
-## Beyond SQL and Schemas
-
-Both SQL and Schema Transforms leverage Row type.
-The same results can be achieved directly using Beam transforms on a KV input PCollection. See [Composed Cobiners example](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/CoCombineTransformExample.java).
diff --git a/examples/java/sql/build.gradle b/examples/java/sql/build.gradle
deleted file mode 100644
index 466e2d0f429..00000000000
--- a/examples/java/sql/build.gradle
+++ /dev/null
@@ -1,122 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import groovy.json.JsonOutput
-
-plugins {
- id 'java'
- id 'org.apache.beam.module'
- id 'com.github.johnrengelman.shadow'
-}
-
-applyJavaNature(
- exportJavadoc: false,
- automaticModuleName: 'org.apache.beam.examples.sql',
-)
-provideIntegrationTestingDependencies()
-enableJavaPerformanceTesting()
-
-description = "Apache Beam :: Examples :: Java"
-ext.summary = """Apache Beam SDK provides a simple, Java-based
-interface for processing virtually any size data. This
-artifact includes all Apache Beam Java SDK examples."""
-
-/** Define the list of runners which execute a precommit test.
- * Some runners are run from separate projects, see the preCommit task below
- * for details.
- */
-def preCommitRunners = ["directRunner", "flinkRunner", "sparkRunner"]
-for (String runner : preCommitRunners) {
- configurations.create(runner + "PreCommit")
-}
-configurations.sparkRunnerPreCommit {
- // Ban certain dependencies to prevent a StackOverflow within Spark
- // because JUL -> SLF4J -> JUL, and similarly JDK14 -> SLF4J -> JDK14
- exclude group: "org.slf4j", module: "jul-to-slf4j"
- exclude group: "org.slf4j", module: "slf4j-jdk14"
-}
-
-dependencies {
- implementation enforcedPlatform(library.java.google_cloud_platform_libraries_bom)
- implementation project(path: ":sdks:java:core", configuration: "shadow")
- implementation project(":sdks:java:extensions:sql")
- implementation library.java.slf4j_api
- runtimeOnly project(path: ":runners:direct-java", configuration: "shadow")
- runtimeOnly library.java.hadoop_client
- runtimeOnly library.java.bigdataoss_gcs_connector
- testImplementation project(path: ":runners:direct-java", configuration: "shadow")
- testImplementation library.java.hamcrest
- testImplementation library.java.junit
- testImplementation library.java.mockito_core
- testImplementation library.java.testcontainers_gcloud
-
- // Add dependencies for the PreCommit configurations
- // For each runner a project level dependency on the examples project.
- for (String runner : preCommitRunners) {
- delegate.add(runner + "PreCommit", project(path: ":examples:java", configuration: "testRuntimeMigration"))
- }
- directRunnerPreCommit project(path: ":runners:direct-java", configuration: "shadow")
- flinkRunnerPreCommit project(":runners:flink:${project.ext.latestFlinkVersion}")
- sparkRunnerPreCommit project(":runners:spark:3")
- sparkRunnerPreCommit project(":sdks:java:io:hadoop-file-system")
-
- // Add dependency if requested on command line for runner
- if (project.hasProperty("runnerDependency")) {
- runtimeOnly project(path: project.getProperty("runnerDependency"))
- }
-}
-
-/*
- * Create a ${runner}PreCommit task for each runner which runs a set
- * of integration tests for WordCount and WindowedWordCount.
- */
-def preCommitRunnerClass = [
- directRunner: "org.apache.beam.runners.direct.DirectRunner",
- flinkRunner: "org.apache.beam.runners.flink.TestFlinkRunner",
- sparkRunner: "org.apache.beam.runners.spark.TestSparkRunner",
-]
-def gcpProject = project.findProperty('gcpProject') ?: 'apache-beam-testing'
-def gcsTempRoot = project.findProperty('gcsTempRoot') ?: 'gs://temp-storage-for-end-to-end-tests/'
-
-for (String runner : preCommitRunners) {
- tasks.create(name: runner + "PreCommit", type: Test) {
- def preCommitBeamTestPipelineOptions = [
- "--project=${gcpProject}",
- "--tempRoot=${gcsTempRoot}",
- "--runner=" + preCommitRunnerClass[runner],
- ]
- classpath = configurations."${runner}PreCommit"
- forkEvery 1
- maxParallelForks 4
- systemProperty "beamTestPipelineOptions", JsonOutput.toJson(preCommitBeamTestPipelineOptions)
- }
-}
-
-/* Define a common precommit task which depends on all the individual precommits. */
-task preCommit() {
- for (String runner : preCommitRunners) {
- dependsOn runner + "PreCommit"
- }
-}
-
-tasks.create(name:"execute", type:JavaExec) {
- main = project.hasProperty("mainClass") ? project.getProperty("mainClass") : "NONE"
- classpath = sourceSets.main.runtimeClasspath
- systemProperties System.getProperties()
- args project.hasProperty("exec.args") ? project.getProperty("exec.args").split() : []
-}
\ No newline at end of file
diff --git a/examples/java/sql/src/main/java/org/apache/beam/examples/sql/SchemaTransformExample.java b/examples/java/src/main/java/org/apache/beam/examples/SchemaTransformExample.java
similarity index 98%
rename from examples/java/sql/src/main/java/org/apache/beam/examples/sql/SchemaTransformExample.java
rename to examples/java/src/main/java/org/apache/beam/examples/SchemaTransformExample.java
index d50490c91b0..6f7825f8caa 100644
--- a/examples/java/sql/src/main/java/org/apache/beam/examples/sql/SchemaTransformExample.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/SchemaTransformExample.java
@@ -15,10 +15,10 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.examples.sql;
+package org.apache.beam.examples;
// beam-playground:
-// name: Group.ByFields
+// name: SchemaTransformGroupByFields
// description: Demonstration of Schema transform usage.
// multifile: false
// default_example: false
diff --git a/examples/java/sql/src/main/java/org/apache/beam/examples/sql/SqlTransformExample.java b/examples/java/src/main/java/org/apache/beam/examples/SqlTransformExample.java
similarity index 99%
rename from examples/java/sql/src/main/java/org/apache/beam/examples/sql/SqlTransformExample.java
rename to examples/java/src/main/java/org/apache/beam/examples/SqlTransformExample.java
index 1f8ae1973b4..62839f7a485 100644
--- a/examples/java/sql/src/main/java/org/apache/beam/examples/sql/SqlTransformExample.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/SqlTransformExample.java
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.examples.sql;
+package org.apache.beam.examples;
// beam-playground:
// name: SqlTransform