You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "chamikaramj (via GitHub)" <gi...@apache.org> on 2023/03/29 01:12:19 UTC
[GitHub] [beam] chamikaramj opened a new pull request, #26023: A Transform Service that uses Docker Compose
chamikaramj opened a new pull request, #26023:
URL: https://github.com/apache/beam/pull/26023
A Docker Compose based transform service
This can be started locally to use portable transforms offered by Beam. Additionally this can be used to discover schema-aware transforms available in Beam.
To use this:
* Install Docker
* Build containers (not needed after containers are released)
$./gradlew :transform-service:controller:docker
$ ./gradlew :sdks:java:expansion-service:container:docker
* Run following from the directory that containers docker-compose.yml
$ docker-compose up
* Now the transforms service will be available at "localhost:5001"
Please see https://s.apache.org/beam-transform-service for the design.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1521223285
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj merged pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj merged PR #26023:
URL: https://github.com/apache/beam/pull/26023
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1537296713
Run Java_GCP_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a diff in pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #26023:
URL: https://github.com/apache/beam/pull/26023#discussion_r1186622414
##########
transform-service/controller/build.gradle:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar
+
+plugins {
+ id 'org.apache.beam.module'
+ id 'com.github.johnrengelman.shadow'
+}
+
+apply plugin: 'org.apache.beam.module'
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.transformservice.controller',
+)
+
+applyDockerNature()
+applyGoNature()
+
+description = "Apache Beam :: Transform Service :: Controller"
+
+configurations {
+ dockerDependency
+}
+
+dependencies {
+ dockerDependency project(path: ":transform-service:controller", configuration: "shadow")
+ implementation project(path: ":model:pipeline", configuration: "shadow")
+ implementation project(path: ":model:job-management", configuration: "shadow")
+ implementation project(path: ":sdks:java:core", configuration: "shadow")
+ implementation project(path: ":runners:core-construction-java")
+ implementation project(path: ":sdks:java:fn-execution")
+ implementation library.java.vendored_grpc_1_54_0
+ implementation library.java.vendored_guava_26_0_jre
+ implementation library.java.jackson_annotations
+ implementation library.java.jackson_databind
+ implementation library.java.jackson_dataformat_yaml
+ testImplementation library.java.junit
+ testImplementation library.java.mockito_core
+ testImplementation project(path: ":runners:java-fn-execution")
+}
+
+goBuild {
+ goTargets = '*.go' // only build the immediate directory.
+ outputLocation = './build/target/launcher/${GOOS}_${GOARCH}/boot'
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ from configurations.dockerDependency
+ rename 'beam-transform-service-controller-.*.jar', 'beam-transform-service-controller.jar'
+ setDuplicatesStrategy(DuplicatesStrategy.INCLUDE)
+ into "build/target"
+}
+
+task copyConfigFile(type: Copy) {
+ from "transform_service_config.yml"
+ into "build/target"
+}
+
+docker {
+ name containerImageName(
+ name: project.docker_image_default_repo_prefix + "transform_service_controller",
+ root: project.rootProject.hasProperty(["docker-repository-root"]) ?
+ project.rootProject["docker-repository-root"] :
+ project.docker_image_default_repo_root,
+ tag: project.rootProject.hasProperty(["docker-tag"]) ?
+ project.rootProject["docker-tag"] : project.sdk_version)
+ // tags used by dockerTag task
+ tags containerImageTags()
+ files "./build"
+}
+
+dockerPrepare.dependsOn goBuild
+dockerPrepare.dependsOn copyConfigFile
+dockerPrepare.dependsOn copyDockerfileDependencies
+
+if (project.rootProject.hasProperty(["docker-pull-licenses"])) {
+ def copyGolangLicenses = tasks.register("copyGolangLicenses", Copy) {
+ from "${project(':release:go-licenses:py').buildDir}/output"
+ into "build/target/go-licenses"
+ dependsOn ':release:go-licenses:py:createLicenses'
+ }
+ dockerPrepare.dependsOn copyGolangLicenses
+} else {
+ def skipPullLicenses = tasks.register("skipPullLicenses", Exec) {
+ executable "sh"
+ // Touch a dummy file to ensure the directory exists.
+ args "-c", "mkdir -p build/target/go-licenses && touch build/target/go-licenses/skip"
+ }
+ dockerPrepare.dependsOn skipPullLicenses
+}
+
+task pushAll {
+ dependsOn ":sdks:java:expansion-service:container:dockerPush"
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1537296729
Run Java_Pulsar_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1502761426
R: @robertwb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1487824745
## [Codecov](https://codecov.io/gh/apache/beam/pull/26023?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#26023](https://codecov.io/gh/apache/beam/pull/26023?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6dcdc52) into [master](https://codecov.io/gh/apache/beam/commit/fabd1f6d7a6967fb6387530b7fd849cf5aafe7a7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fabd1f6) will **not change** coverage.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #26023 +/- ##
=======================================
Coverage 72.79% 72.79%
=======================================
Files 775 775
Lines 102863 102863
=======================================
Hits 74877 74877
Misses 26532 26532
Partials 1454 1454
```
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1522204969
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1537296692
Run CommunityMetrics PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1502559457
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @AnandInguva for label python.
R: @Abacn for label java.
R: @damccorm for label build.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #26023: A Transform Service that uses Docker Compose
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #26023:
URL: https://github.com/apache/beam/pull/26023#discussion_r1178514688
##########
transform-service/controller/go.mod:
##########
Review Comment:
If the goal is to keep the released instance of this service in sync with released beam, then you do not want to create a new Go module at this level (the go.mod and go.sum files.)
Otherwise this becomes another thing to try and keep up to date.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #26023: A Transform Service that uses Docker Compose
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #26023:
URL: https://github.com/apache/beam/pull/26023#discussion_r1185491866
##########
sdks/python/apache_beam/runners/portability/expansion_service_main.py:
##########
@@ -59,7 +59,7 @@ def main(argv):
artifact_service.ArtifactRetrievalService(
artifact_service.BeamFilesystemHandler(None).file_reader),
server)
- server.add_insecure_port('localhost:{}'.format(known_args.port))
+ server.add_insecure_port('[::]:{}'.format(known_args.port))
Review Comment:
Thanks, initiated javausingpython XVR tests just to confirm this setting not affecting current expansion service infra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1537045145
Thanks. Updated to support both amd64 and arm64.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1537283697
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #26023: A Transform Service that uses Docker Compose
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #26023:
URL: https://github.com/apache/beam/pull/26023#discussion_r1179434083
##########
sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/Dependency.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.expansion.service;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.auto.value.AutoValue;
+
+//// TODO: generalize to support other types of dependencies
Review Comment:
Add a GitHub Issue link to this TODO?
Is it possible to not suppress nullness warning here and below new codes? (received this request from other reviewers before)
##########
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExpansionServiceClient.java:
##########
@@ -22,4 +22,7 @@
/** A high-level client for a cross-language expansion service. */
public interface ExpansionServiceClient extends AutoCloseable {
ExpansionApi.ExpansionResponse expand(ExpansionApi.ExpansionRequest request);
+
+ ExpansionApi.DiscoverSchemaTransformResponse discover(
Review Comment:
Just confirm, is this interface deemed for internal usage only, so adding a new method would not break classes that implemented it?
##########
transform-service/controller/src/main/java/org/apache/beam/transformservice/controller/ExpansionService.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.transformservice.controller;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.model.expansion.v1.ExpansionApi;
+import org.apache.beam.model.expansion.v1.ExpansionServiceGrpc;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.runners.core.construction.DefaultExpansionServiceClientFactory;
+import org.apache.beam.runners.core.construction.ExpansionServiceClientFactory;
+import org.apache.beam.vendor.grpc.v1p54p0.io.grpc.ManagedChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p54p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class ExpansionService extends ExpansionServiceGrpc.ExpansionServiceImplBase
+ implements AutoCloseable {
+
+ private static final ExpansionServiceClientFactory EXPANSION_SERVICE_CLIENT_FACTORY =
+ DefaultExpansionServiceClientFactory.create(
+ endPoint -> ManagedChannelBuilder.forTarget(endPoint.getUrl()).usePlaintext().build());
+
+ private final ExpansionServiceClientFactory expansionServiceClientFactory;
+
+ final List<Endpoints.ApiServiceDescriptor> endpoints;
+
+ ExpansionService(
+ List<Endpoints.ApiServiceDescriptor> endpoints,
+ @Nullable ExpansionServiceClientFactory clientFactory) {
+ this.endpoints = endpoints;
+ this.expansionServiceClientFactory =
+ clientFactory != null ? clientFactory : EXPANSION_SERVICE_CLIENT_FACTORY;
+ }
+
+ @Override
+ public void expand(
+ ExpansionApi.ExpansionRequest request,
+ StreamObserver<ExpansionApi.ExpansionResponse> responseObserver) {
+ try {
+ responseObserver.onNext(processExpand(request));
+ responseObserver.onCompleted();
+ } catch (RuntimeException exn) {
+ responseObserver.onNext(
Review Comment:
Is it fine to suppress RuntimeException here, if yes may consider add any log, same below
##########
transform-service/controller/src/main/java/org/apache/beam/transformservice/controller/ExpansionService.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.transformservice.controller;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.model.expansion.v1.ExpansionApi;
+import org.apache.beam.model.expansion.v1.ExpansionServiceGrpc;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.runners.core.construction.DefaultExpansionServiceClientFactory;
+import org.apache.beam.runners.core.construction.ExpansionServiceClientFactory;
+import org.apache.beam.vendor.grpc.v1p54p0.io.grpc.ManagedChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p54p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class ExpansionService extends ExpansionServiceGrpc.ExpansionServiceImplBase
+ implements AutoCloseable {
+
+ private static final ExpansionServiceClientFactory EXPANSION_SERVICE_CLIENT_FACTORY =
Review Comment:
Naming: `DEFAULT_EXPANSION_SERVICE_CLIENT_FACTORY` or `DEFAULT_FACTORY` ?
##########
transform-service/controller/build.gradle:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar
+
+plugins {
+ id 'org.apache.beam.module'
+ id 'com.github.johnrengelman.shadow'
+}
+
+apply plugin: 'org.apache.beam.module'
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.transformservice.controller',
+)
+
+applyDockerNature()
+applyGoNature()
+
+description = "Apache Beam :: Transform Service :: Controller"
+
+configurations {
+ dockerDependency
+}
+
+dependencies {
+ dockerDependency project(path: ":transform-service:controller", configuration: "shadow")
+ implementation project(path: ":model:pipeline", configuration: "shadow")
+ implementation project(path: ":model:job-management", configuration: "shadow")
+ implementation project(path: ":sdks:java:core", configuration: "shadow")
+ implementation project(path: ":runners:core-construction-java")
+ implementation project(path: ":sdks:java:fn-execution")
+ implementation library.java.vendored_grpc_1_54_0
+ implementation library.java.vendored_guava_26_0_jre
+ implementation library.java.jackson_annotations
+ implementation library.java.jackson_databind
+ implementation library.java.jackson_dataformat_yaml
+ testImplementation library.java.junit
+ testImplementation library.java.mockito_core
+ testImplementation project(path: ":runners:java-fn-execution")
+}
+
+goBuild {
+ goTargets = '*.go' // only build the immediate directory.
+ outputLocation = './build/target/launcher/${GOOS}_${GOARCH}/boot'
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ from configurations.dockerDependency
+ rename 'beam-transform-service-controller-.*.jar', 'beam-transform-service-controller.jar'
+ setDuplicatesStrategy(DuplicatesStrategy.INCLUDE)
+ into "build/target"
+}
+
+task copyConfigFile(type: Copy) {
+ from "transform_service_config.yml"
+ into "build/target"
+}
+
+docker {
+ name containerImageName(
+ name: project.docker_image_default_repo_prefix + "transform_service_controller",
+ root: project.rootProject.hasProperty(["docker-repository-root"]) ?
+ project.rootProject["docker-repository-root"] :
+ project.docker_image_default_repo_root,
+ tag: project.rootProject.hasProperty(["docker-tag"]) ?
+ project.rootProject["docker-tag"] : project.sdk_version)
+ // tags used by dockerTag task
+ tags containerImageTags()
+ files "./build"
+}
+
+dockerPrepare.dependsOn goBuild
+dockerPrepare.dependsOn copyConfigFile
+dockerPrepare.dependsOn copyDockerfileDependencies
+
+if (project.rootProject.hasProperty(["docker-pull-licenses"])) {
+ def copyGolangLicenses = tasks.register("copyGolangLicenses", Copy) {
+ from "${project(':release:go-licenses:py').buildDir}/output"
+ into "build/target/go-licenses"
+ dependsOn ':release:go-licenses:py:createLicenses'
+ }
+ dockerPrepare.dependsOn copyGolangLicenses
+} else {
+ def skipPullLicenses = tasks.register("skipPullLicenses", Exec) {
+ executable "sh"
+ // Touch a dummy file to ensure the directory exists.
+ args "-c", "mkdir -p build/target/go-licenses && touch build/target/go-licenses/skip"
+ }
+ dockerPrepare.dependsOn skipPullLicenses
+}
+
+task pushAll {
+ dependsOn ":sdks:java:expansion-service:container:dockerPush"
Review Comment:
push all pushes a different project (`:sdks:java:expansion-service:container`) is this correct?
##########
transform-service/controller/build.gradle:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar
Review Comment:
Is this unused import?
##########
transform-service/controller/src/main/java/org/apache/beam/transformservice/controller/Controller.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.transformservice.controller;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.vendor.grpc.v1p54p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p54p0.io.grpc.ServerBuilder;
+
+public class Controller {
+
+ List<Endpoints.ApiServiceDescriptor> endpoints;
+
+ private final TransformServiceOptions options;
+
+ public Controller(String[] args) {
+ // We use PipelineOptions just a library for parsing arguments here.
+ this(PipelineOptionsFactory.fromArgs(args).create());
+ for (String expansionService : options.getTransformServiceConfig().getExpansionservices()) {
Review Comment:
Should endpoints be initialized in `Controller(PipelineOptions opts)`, otherwise `Controller(String[] args)` has an initialized endpoints while `Controller(PipelineOptions opts)` has an empty endpoints. Both constructor are public.
also, any javadoc is nice for this public class and is executable.
##########
sdks/python/expansion-service-container/Dockerfile:
##########
@@ -0,0 +1,72 @@
+###############################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+###############################################################################
+
+# We just need to support one Python version supported by Beam.
+# Picking the current default Beam Python version which is Python 3.8.
+FROM python:3.8-bullseye as expansion-service
Review Comment:
I see other dockerfile in Beam repo uses `FROM python:3.8-slim` base image. Does slim base image suffice here?
##########
transform-service/controller/src/main/java/org/apache/beam/transformservice/controller/package-info.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * TODO: add Javadoc.
+ *
+ * <p>TODO: details
Review Comment:
Note: TODO here
##########
transform-service/controller/go.mod:
##########
@@ -0,0 +1,20 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+module gin
+
+go 1.17
Review Comment:
Same, then could avoid the need of changing these version number here on SDK or go version bump
##########
transform-service/build.gradle:
##########
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins {
+ id 'org.apache.beam.module'
+}
+
+description = "Apache Beam :: Transform Service"
Review Comment:
It is an empty build file for now. Are there going to be content? If not could this build file be dropped?
##########
sdks/python/apache_beam/runners/portability/expansion_service_main.py:
##########
@@ -59,7 +59,7 @@ def main(argv):
artifact_service.ArtifactRetrievalService(
artifact_service.BeamFilesystemHandler(None).file_reader),
server)
- server.add_insecure_port('localhost:{}'.format(known_args.port))
+ server.add_insecure_port('[::]:{}'.format(known_args.port))
Review Comment:
For curiosity, what is `[::]` and why "localhost" no longer working?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1521223396
Run Spotless PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #26023: A Transform Service that uses Docker Compose
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #26023:
URL: https://github.com/apache/beam/pull/26023#discussion_r1185503222
##########
transform-service/controller/build.gradle:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar
+
+plugins {
+ id 'org.apache.beam.module'
+ id 'com.github.johnrengelman.shadow'
+}
+
+apply plugin: 'org.apache.beam.module'
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.transformservice.controller',
+)
+
+applyDockerNature()
+applyGoNature()
+
+description = "Apache Beam :: Transform Service :: Controller"
+
+configurations {
+ dockerDependency
+}
+
+dependencies {
+ dockerDependency project(path: ":transform-service:controller", configuration: "shadow")
+ implementation project(path: ":model:pipeline", configuration: "shadow")
+ implementation project(path: ":model:job-management", configuration: "shadow")
+ implementation project(path: ":sdks:java:core", configuration: "shadow")
+ implementation project(path: ":runners:core-construction-java")
+ implementation project(path: ":sdks:java:fn-execution")
+ implementation library.java.vendored_grpc_1_54_0
+ implementation library.java.vendored_guava_26_0_jre
+ implementation library.java.jackson_annotations
+ implementation library.java.jackson_databind
+ implementation library.java.jackson_dataformat_yaml
+ testImplementation library.java.junit
+ testImplementation library.java.mockito_core
+ testImplementation project(path: ":runners:java-fn-execution")
+}
+
+goBuild {
+ goTargets = '*.go' // only build the immediate directory.
+ outputLocation = './build/target/launcher/${GOOS}_${GOARCH}/boot'
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ from configurations.dockerDependency
+ rename 'beam-transform-service-controller-.*.jar', 'beam-transform-service-controller.jar'
+ setDuplicatesStrategy(DuplicatesStrategy.INCLUDE)
+ into "build/target"
+}
+
+task copyConfigFile(type: Copy) {
+ from "transform_service_config.yml"
+ into "build/target"
+}
+
+docker {
+ name containerImageName(
+ name: project.docker_image_default_repo_prefix + "transform_service_controller",
+ root: project.rootProject.hasProperty(["docker-repository-root"]) ?
+ project.rootProject["docker-repository-root"] :
+ project.docker_image_default_repo_root,
+ tag: project.rootProject.hasProperty(["docker-tag"]) ?
+ project.rootProject["docker-tag"] : project.sdk_version)
+ // tags used by dockerTag task
+ tags containerImageTags()
+ files "./build"
+}
+
+dockerPrepare.dependsOn goBuild
+dockerPrepare.dependsOn copyConfigFile
+dockerPrepare.dependsOn copyDockerfileDependencies
+
+if (project.rootProject.hasProperty(["docker-pull-licenses"])) {
+ def copyGolangLicenses = tasks.register("copyGolangLicenses", Copy) {
+ from "${project(':release:go-licenses:py').buildDir}/output"
+ into "build/target/go-licenses"
+ dependsOn ':release:go-licenses:py:createLicenses'
+ }
+ dockerPrepare.dependsOn copyGolangLicenses
+} else {
+ def skipPullLicenses = tasks.register("skipPullLicenses", Exec) {
+ executable "sh"
+ // Touch a dummy file to ensure the directory exists.
+ args "-c", "mkdir -p build/target/go-licenses && touch build/target/go-licenses/skip"
+ }
+ dockerPrepare.dependsOn skipPullLicenses
+}
+
+task pushAll {
+ dependsOn ":sdks:java:expansion-service:container:dockerPush"
Review Comment:
can just be
dependsOn ":sdks:java:transform-service:controller-container:dockerPush" -> dependsOn dockerPush
since they are in same project
same for `sdks/java/expansion-service/container/build.gradle`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #26023: A Transform Service that uses Docker Compose
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #26023:
URL: https://github.com/apache/beam/pull/26023#discussion_r1185494569
##########
sdks/python/expansion-service-container/Dockerfile:
##########
@@ -0,0 +1,72 @@
+###############################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+###############################################################################
+
+# We just need to support one Python version supported by Beam.
+# Picking the current default Beam Python version which is Python 3.8.
+FROM python:3.8-bullseye as expansion-service
Review Comment:
Thanks, `beam/sdks/python/container/Dockerfile` indeed uses bullseye; the one used slim is `metrics/*/Dockerfile` not relevant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1513597815
@robertwb friendly ping.
Please note that I hope to add some advanced features (for example, sharing auth credentials between the local machine and the Docker containers) in future PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1522092126
Also added support for sharing credentials between the local machine and the expansion service containers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1522148659
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a diff in pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #26023:
URL: https://github.com/apache/beam/pull/26023#discussion_r1179474305
##########
transform-service/controller/go.mod:
##########
Review Comment:
Ack. Probably we can push the transform service under "sdks/java" to avoid adding a new Go module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1535382907
Run XVR_JavaUsingPython_Dataflow PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1509874608
Run Java_GCP_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1509874569
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1537296424
Run Community Metrics Prober
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a diff in pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #26023:
URL: https://github.com/apache/beam/pull/26023#discussion_r1184290326
##########
transform-service/controller/go.mod:
##########
Review Comment:
These files were removed.
##########
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExpansionServiceClient.java:
##########
@@ -22,4 +22,7 @@
/** A high-level client for a cross-language expansion service. */
public interface ExpansionServiceClient extends AutoCloseable {
ExpansionApi.ExpansionResponse expand(ExpansionApi.ExpansionRequest request);
+
+ ExpansionApi.DiscoverSchemaTransformResponse discover(
Review Comment:
Yes. External customers should not need to implement this.
##########
transform-service/controller/src/main/java/org/apache/beam/transformservice/controller/package-info.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * TODO: add Javadoc.
+ *
+ * <p>TODO: details
Review Comment:
Updated.
##########
sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/Dependency.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.expansion.service;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.auto.value.AutoValue;
+
+//// TODO: generalize to support other types of dependencies
Review Comment:
Added a Github issue and removed nullness suppression.
##########
transform-service/controller/build.gradle:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar
+
+plugins {
+ id 'org.apache.beam.module'
+ id 'com.github.johnrengelman.shadow'
+}
+
+apply plugin: 'org.apache.beam.module'
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.transformservice.controller',
+)
+
+applyDockerNature()
+applyGoNature()
+
+description = "Apache Beam :: Transform Service :: Controller"
+
+configurations {
+ dockerDependency
+}
+
+dependencies {
+ dockerDependency project(path: ":transform-service:controller", configuration: "shadow")
+ implementation project(path: ":model:pipeline", configuration: "shadow")
+ implementation project(path: ":model:job-management", configuration: "shadow")
+ implementation project(path: ":sdks:java:core", configuration: "shadow")
+ implementation project(path: ":runners:core-construction-java")
+ implementation project(path: ":sdks:java:fn-execution")
+ implementation library.java.vendored_grpc_1_54_0
+ implementation library.java.vendored_guava_26_0_jre
+ implementation library.java.jackson_annotations
+ implementation library.java.jackson_databind
+ implementation library.java.jackson_dataformat_yaml
+ testImplementation library.java.junit
+ testImplementation library.java.mockito_core
+ testImplementation project(path: ":runners:java-fn-execution")
+}
+
+goBuild {
+ goTargets = '*.go' // only build the immediate directory.
+ outputLocation = './build/target/launcher/${GOOS}_${GOARCH}/boot'
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ from configurations.dockerDependency
+ rename 'beam-transform-service-controller-.*.jar', 'beam-transform-service-controller.jar'
+ setDuplicatesStrategy(DuplicatesStrategy.INCLUDE)
+ into "build/target"
+}
+
+task copyConfigFile(type: Copy) {
+ from "transform_service_config.yml"
+ into "build/target"
+}
+
+docker {
+ name containerImageName(
+ name: project.docker_image_default_repo_prefix + "transform_service_controller",
+ root: project.rootProject.hasProperty(["docker-repository-root"]) ?
+ project.rootProject["docker-repository-root"] :
+ project.docker_image_default_repo_root,
+ tag: project.rootProject.hasProperty(["docker-tag"]) ?
+ project.rootProject["docker-tag"] : project.sdk_version)
+ // tags used by dockerTag task
+ tags containerImageTags()
+ files "./build"
+}
+
+dockerPrepare.dependsOn goBuild
+dockerPrepare.dependsOn copyConfigFile
+dockerPrepare.dependsOn copyDockerfileDependencies
+
+if (project.rootProject.hasProperty(["docker-pull-licenses"])) {
+ def copyGolangLicenses = tasks.register("copyGolangLicenses", Copy) {
+ from "${project(':release:go-licenses:py').buildDir}/output"
+ into "build/target/go-licenses"
+ dependsOn ':release:go-licenses:py:createLicenses'
+ }
+ dockerPrepare.dependsOn copyGolangLicenses
+} else {
+ def skipPullLicenses = tasks.register("skipPullLicenses", Exec) {
+ executable "sh"
+ // Touch a dummy file to ensure the directory exists.
+ args "-c", "mkdir -p build/target/go-licenses && touch build/target/go-licenses/skip"
+ }
+ dockerPrepare.dependsOn skipPullLicenses
+}
+
+task pushAll {
+ dependsOn ":sdks:java:expansion-service:container:dockerPush"
Review Comment:
Fixed.
##########
sdks/python/expansion-service-container/Dockerfile:
##########
@@ -0,0 +1,72 @@
+###############################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+###############################################################################
+
+# We just need to support one Python version supported by Beam.
+# Picking the current default Beam Python version which is Python 3.8.
+FROM python:3.8-bullseye as expansion-service
Review Comment:
Actually, Python SDK harness container uses "${py_version}"-bullseye.
3.8-slim may or may not work but I'd like to keep base images consistent across Beam containers.
##########
sdks/python/apache_beam/runners/portability/expansion_service_main.py:
##########
@@ -59,7 +59,7 @@ def main(argv):
artifact_service.ArtifactRetrievalService(
artifact_service.BeamFilesystemHandler(None).file_reader),
server)
- server.add_insecure_port('localhost:{}'.format(known_args.port))
+ server.add_insecure_port('[::]:{}'.format(known_args.port))
Review Comment:
This represents any IPv6 address:
https://www.ibm.com/docs/en/zos/2.4.0?topic=applications-special-ipv6-addresses
Docker networking failed for "localhost".
##########
transform-service/controller/build.gradle:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar
Review Comment:
Removed.
##########
transform-service/controller/go.mod:
##########
@@ -0,0 +1,20 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+module gin
+
+go 1.17
Review Comment:
This file was removed.
##########
transform-service/controller/src/main/java/org/apache/beam/transformservice/controller/Controller.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.transformservice.controller;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.vendor.grpc.v1p54p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p54p0.io.grpc.ServerBuilder;
+
+public class Controller {
+
+ List<Endpoints.ApiServiceDescriptor> endpoints;
+
+ private final TransformServiceOptions options;
+
+ public Controller(String[] args) {
+ // We use PipelineOptions just a library for parsing arguments here.
+ this(PipelineOptionsFactory.fromArgs(args).create());
+ for (String expansionService : options.getTransformServiceConfig().getExpansionservices()) {
Review Comment:
Updated and added a Javadoc.
##########
transform-service/build.gradle:
##########
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins {
+ id 'org.apache.beam.module'
+}
+
+description = "Apache Beam :: Transform Service"
Review Comment:
Code was refactored and this file is not empty now.
##########
transform-service/controller/src/main/java/org/apache/beam/transformservice/controller/ExpansionService.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.transformservice.controller;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.model.expansion.v1.ExpansionApi;
+import org.apache.beam.model.expansion.v1.ExpansionServiceGrpc;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.runners.core.construction.DefaultExpansionServiceClientFactory;
+import org.apache.beam.runners.core.construction.ExpansionServiceClientFactory;
+import org.apache.beam.vendor.grpc.v1p54p0.io.grpc.ManagedChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p54p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class ExpansionService extends ExpansionServiceGrpc.ExpansionServiceImplBase
+ implements AutoCloseable {
+
+ private static final ExpansionServiceClientFactory EXPANSION_SERVICE_CLIENT_FACTORY =
Review Comment:
Renamed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1535382267
Run XVR_Direct PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #26023: A Transform Service that uses Docker Compose
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #26023:
URL: https://github.com/apache/beam/pull/26023#discussion_r1185503222
##########
transform-service/controller/build.gradle:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar
+
+plugins {
+ id 'org.apache.beam.module'
+ id 'com.github.johnrengelman.shadow'
+}
+
+apply plugin: 'org.apache.beam.module'
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.transformservice.controller',
+)
+
+applyDockerNature()
+applyGoNature()
+
+description = "Apache Beam :: Transform Service :: Controller"
+
+configurations {
+ dockerDependency
+}
+
+dependencies {
+ dockerDependency project(path: ":transform-service:controller", configuration: "shadow")
+ implementation project(path: ":model:pipeline", configuration: "shadow")
+ implementation project(path: ":model:job-management", configuration: "shadow")
+ implementation project(path: ":sdks:java:core", configuration: "shadow")
+ implementation project(path: ":runners:core-construction-java")
+ implementation project(path: ":sdks:java:fn-execution")
+ implementation library.java.vendored_grpc_1_54_0
+ implementation library.java.vendored_guava_26_0_jre
+ implementation library.java.jackson_annotations
+ implementation library.java.jackson_databind
+ implementation library.java.jackson_dataformat_yaml
+ testImplementation library.java.junit
+ testImplementation library.java.mockito_core
+ testImplementation project(path: ":runners:java-fn-execution")
+}
+
+goBuild {
+ goTargets = '*.go' // only build the immediate directory.
+ outputLocation = './build/target/launcher/${GOOS}_${GOARCH}/boot'
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ from configurations.dockerDependency
+ rename 'beam-transform-service-controller-.*.jar', 'beam-transform-service-controller.jar'
+ setDuplicatesStrategy(DuplicatesStrategy.INCLUDE)
+ into "build/target"
+}
+
+task copyConfigFile(type: Copy) {
+ from "transform_service_config.yml"
+ into "build/target"
+}
+
+docker {
+ name containerImageName(
+ name: project.docker_image_default_repo_prefix + "transform_service_controller",
+ root: project.rootProject.hasProperty(["docker-repository-root"]) ?
+ project.rootProject["docker-repository-root"] :
+ project.docker_image_default_repo_root,
+ tag: project.rootProject.hasProperty(["docker-tag"]) ?
+ project.rootProject["docker-tag"] : project.sdk_version)
+ // tags used by dockerTag task
+ tags containerImageTags()
+ files "./build"
+}
+
+dockerPrepare.dependsOn goBuild
+dockerPrepare.dependsOn copyConfigFile
+dockerPrepare.dependsOn copyDockerfileDependencies
+
+if (project.rootProject.hasProperty(["docker-pull-licenses"])) {
+ def copyGolangLicenses = tasks.register("copyGolangLicenses", Copy) {
+ from "${project(':release:go-licenses:py').buildDir}/output"
+ into "build/target/go-licenses"
+ dependsOn ':release:go-licenses:py:createLicenses'
+ }
+ dockerPrepare.dependsOn copyGolangLicenses
+} else {
+ def skipPullLicenses = tasks.register("skipPullLicenses", Exec) {
+ executable "sh"
+ // Touch a dummy file to ensure the directory exists.
+ args "-c", "mkdir -p build/target/go-licenses && touch build/target/go-licenses/skip"
+ }
+ dockerPrepare.dependsOn skipPullLicenses
+}
+
+task pushAll {
+ dependsOn ":sdks:java:expansion-service:container:dockerPush"
Review Comment:
can just be
dependsOn ":sdks:java:transform-service:controller-container:dockerPush" -> dependsOn dockerPush
since they are in same project
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1537297070
Run Java_Amazon-Web-Services2_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1537296937
Run Java_Amazon-Web-Services_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1537283992
Run Spotless PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1502762533
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #26023: A Transform Service that uses Docker Compose
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #26023:
URL: https://github.com/apache/beam/pull/26023#issuecomment-1505666701
Tested by running both following pipelines against the same transform service.
* A Python pipeline that uses Java Kafka read and write transforms.
* A Java pipeline that uses the Python Dataframe transform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org