You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "olehborysevych (via GitHub)" <gi...@apache.org> on 2023/01/31 14:54:42 UTC
[GitHub] [beam] olehborysevych opened a new pull request, #25230: [DO NOT MERGE] fixing dependency management for java playground runner 2.44
olehborysevych opened a new pull request, #25230:
URL: https://github.com/apache/beam/pull/25230
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25230: [DO NOT MERGE] fixing dependency management for java playground runner 2.44
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25230:
URL: https://github.com/apache/beam/pull/25230#discussion_r1093057760
##########
playground/backend/containers/java/Dockerfile:
##########
@@ -57,48 +68,30 @@ COPY --from=build /go/bin/server_java_backend /opt/playground/backend/
COPY --from=build /go/src/playground/backend/configs /opt/playground/backend/configs/
COPY --from=build /go/src/playground/backend/logging.properties /opt/playground/backend/
COPY --from=build /go/src/playground/backend/datasets /opt/playground/backend/datasets/
+COPY --from=dep /pipeline-dependencies/target/dependency/ /opt/apache/beam/jars/
-# Install Beam DirectRunner
-RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-runners-direct-java/$BEAM_VERSION/beam-runners-direct-java-$BEAM_VERSION.jar &&\
- mv beam-runners-direct-java-$BEAM_VERSION.jar /opt/apache/beam/jars/beam-runners-direct.jar
-# Install Beam SDK Core
-RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-sdks-java-core/$BEAM_VERSION/beam-sdks-java-core-$BEAM_VERSION.jar &&\
- mv beam-sdks-java-core-$BEAM_VERSION.jar /opt/apache/beam/jars/beam-sdks-java-core.jar
# Install Beam Examples
RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-examples-java/$BEAM_VERSION/beam-examples-java-$BEAM_VERSION.jar &&\
mv beam-examples-java-$BEAM_VERSION.jar /opt/apache/beam/jars/beam-examples-java.jar
# Install jars for Playground graphs
RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-runners-core-construction-java/$BEAM_VERSION/beam-runners-core-construction-java-$BEAM_VERSION.jar &&\
mv beam-runners-core-construction-java-$BEAM_VERSION.jar /opt/apache/beam/jars/beam-runners-core-construction-java-$BEAM_VERSION.jar
-RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-model-pipeline/$BEAM_VERSION/beam-model-pipeline-$BEAM_VERSION.jar &&\
- mv beam-model-pipeline-$BEAM_VERSION.jar /opt/apache/beam/jars/beam-model-pipeline-$BEAM_VERSION.jar
-# Install Beam SDK Core tests
+# Install jars for core tests
RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-sdks-java-core/$BEAM_VERSION/beam-sdks-java-core-$BEAM_VERSION-tests.jar &&\
mv beam-sdks-java-core-$BEAM_VERSION-tests.jar /opt/apache/beam/jars/beam-sdks-java-core-tests.jar
-# Install Hamcrest
-RUN wget https://repo1.maven.org/maven2/org/hamcrest/hamcrest-all/$HAMCREST_VERSION/hamcrest-all-$HAMCREST_VERSION.jar &&\
- mv hamcrest-all-$HAMCREST_VERSION.jar /opt/apache/beam/jars/hamcrest-all.jar
-# Install JUnit
-RUN wget https://repo1.maven.org/maven2/junit/junit/$JUNIT_VERSION/junit-$JUNIT_VERSION.jar &&\
- mv junit-$JUNIT_VERSION.jar /opt/apache/beam/jars/junit.jar
-# Install Beam Vendor gRPC
-RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-vendor-grpc-$GRPC_VERSION/$BEAM_VENDOR_GRPC_VERSION/beam-vendor-grpc-$GRPC_VERSION-$BEAM_VENDOR_GRPC_VERSION.jar &&\
- mv beam-vendor-grpc-$GRPC_VERSION-$BEAM_VENDOR_GRPC_VERSION.jar /opt/apache/beam/jars/beam-vendor-grpc.jar
-# Install Kafka IO
-RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-sdks-java-io-kafka/$BEAM_VERSION/beam-sdks-java-io-kafka-$BEAM_VERSION.jar &&\
- mv beam-sdks-java-io-kafka-$BEAM_VERSION.jar /opt/apache/beam/jars/beam-sdks-java-io-kafka.jar
-# Install Apache Kafka Clients
+# Install kafka client
RUN wget https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/$KAFKA_CLIENTS_VERSION/kafka-clients-$KAFKA_CLIENTS_VERSION.jar &&\
Review Comment:
Please add comment that we are explicitly downgrading Kafka due to provided version not working with our emulator
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25230: [DO NOT MERGE] fixing dependency management for java playground runner 2.44
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25230:
URL: https://github.com/apache/beam/pull/25230#discussion_r1093058816
##########
playground/backend/containers/java/build.gradle:
##########
@@ -69,9 +69,9 @@ docker {
project.docker_image_default_repo_root)
files "./build/"
tags containerImageTags()
- buildArgs(['BASE_IMAGE': project.rootProject.hasProperty(["base-image"]) ?
- project.rootProject["base-image"] :
- "apache/beam_java8_sdk" ])
+ buildArgs(['BEAM_VERSION': project.rootProject.hasProperty(["sdk-tag"]) ?
+ project.rootProject["sdk-tag"] :
+ "2.44.0" ])
Review Comment:
Can we avoid hard-coding the version in gradle task? We already have default version specified in the Dockerfile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] olehborysevych commented on pull request #25230: [DO NOT MERGE] fixing dependency management for java playground runner 2.44
Posted by "olehborysevych (via GitHub)" <gi...@apache.org>.
olehborysevych commented on PR #25230:
URL: https://github.com/apache/beam/pull/25230#issuecomment-1410627369
R: @rshamunov @TSultanov
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm merged pull request #25230: [Tour of Beam] [Task] Fix dependency management for 2.44 Playground java runner
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #25230:
URL: https://github.com/apache/beam/pull/25230
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] olehborysevych commented on a diff in pull request #25230: [DO NOT MERGE] fixing dependency management for java playground runner 2.44
Posted by "olehborysevych (via GitHub)" <gi...@apache.org>.
olehborysevych commented on code in PR #25230:
URL: https://github.com/apache/beam/pull/25230#discussion_r1093485613
##########
playground/backend/containers/java/build.gradle:
##########
@@ -69,9 +69,9 @@ docker {
project.docker_image_default_repo_root)
files "./build/"
tags containerImageTags()
- buildArgs(['BASE_IMAGE': project.rootProject.hasProperty(["base-image"]) ?
- project.rootProject["base-image"] :
- "apache/beam_java8_sdk" ])
+ buildArgs(['BEAM_VERSION': project.rootProject.hasProperty(["sdk-tag"]) ?
+ project.rootProject["sdk-tag"] :
+ "2.44.0" ])
Review Comment:
I didn't find good way to do this. Our approach for java runner is different from other runners that we can only use release beam version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #25230: [Tour of Beam] [Task] Fix dependency management for 2.44 Playground java runner
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #25230:
URL: https://github.com/apache/beam/pull/25230#discussion_r1093588625
##########
playground/backend/containers/java/build.gradle:
##########
@@ -69,9 +69,9 @@ docker {
project.docker_image_default_repo_root)
files "./build/"
tags containerImageTags()
- buildArgs(['BASE_IMAGE': project.rootProject.hasProperty(["base-image"]) ?
- project.rootProject["base-image"] :
- "apache/beam_java8_sdk" ])
+ buildArgs(['BEAM_VERSION': project.rootProject.hasProperty(["sdk-tag"]) ?
+ project.rootProject["sdk-tag"] :
+ "2.44.0" ])
Review Comment:
Could we at least pull it out as a constant at the top of the file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25230: [DO NOT MERGE] fixing dependency management for java playground runner 2.44
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25230:
URL: https://github.com/apache/beam/pull/25230#discussion_r1093058816
##########
playground/backend/containers/java/build.gradle:
##########
@@ -69,9 +69,9 @@ docker {
project.docker_image_default_repo_root)
files "./build/"
tags containerImageTags()
- buildArgs(['BASE_IMAGE': project.rootProject.hasProperty(["base-image"]) ?
- project.rootProject["base-image"] :
- "apache/beam_java8_sdk" ])
+ buildArgs(['BEAM_VERSION': project.rootProject.hasProperty(["sdk-tag"]) ?
+ project.rootProject["sdk-tag"] :
+ "2.44.0" ])
Review Comment:
Can we avoid hard-coding the version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] olehborysevych commented on a diff in pull request #25230: [DO NOT MERGE] fixing dependency management for java playground runner 2.44
Posted by "olehborysevych (via GitHub)" <gi...@apache.org>.
olehborysevych commented on code in PR #25230:
URL: https://github.com/apache/beam/pull/25230#discussion_r1093441459
##########
playground/backend/containers/java/Dockerfile:
##########
@@ -57,48 +68,30 @@ COPY --from=build /go/bin/server_java_backend /opt/playground/backend/
COPY --from=build /go/src/playground/backend/configs /opt/playground/backend/configs/
COPY --from=build /go/src/playground/backend/logging.properties /opt/playground/backend/
COPY --from=build /go/src/playground/backend/datasets /opt/playground/backend/datasets/
+COPY --from=dep /pipeline-dependencies/target/dependency/ /opt/apache/beam/jars/
-# Install Beam DirectRunner
-RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-runners-direct-java/$BEAM_VERSION/beam-runners-direct-java-$BEAM_VERSION.jar &&\
- mv beam-runners-direct-java-$BEAM_VERSION.jar /opt/apache/beam/jars/beam-runners-direct.jar
-# Install Beam SDK Core
-RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-sdks-java-core/$BEAM_VERSION/beam-sdks-java-core-$BEAM_VERSION.jar &&\
- mv beam-sdks-java-core-$BEAM_VERSION.jar /opt/apache/beam/jars/beam-sdks-java-core.jar
# Install Beam Examples
RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-examples-java/$BEAM_VERSION/beam-examples-java-$BEAM_VERSION.jar &&\
mv beam-examples-java-$BEAM_VERSION.jar /opt/apache/beam/jars/beam-examples-java.jar
# Install jars for Playground graphs
RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-runners-core-construction-java/$BEAM_VERSION/beam-runners-core-construction-java-$BEAM_VERSION.jar &&\
mv beam-runners-core-construction-java-$BEAM_VERSION.jar /opt/apache/beam/jars/beam-runners-core-construction-java-$BEAM_VERSION.jar
-RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-model-pipeline/$BEAM_VERSION/beam-model-pipeline-$BEAM_VERSION.jar &&\
- mv beam-model-pipeline-$BEAM_VERSION.jar /opt/apache/beam/jars/beam-model-pipeline-$BEAM_VERSION.jar
-# Install Beam SDK Core tests
+# Install jars for core tests
RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-sdks-java-core/$BEAM_VERSION/beam-sdks-java-core-$BEAM_VERSION-tests.jar &&\
mv beam-sdks-java-core-$BEAM_VERSION-tests.jar /opt/apache/beam/jars/beam-sdks-java-core-tests.jar
-# Install Hamcrest
-RUN wget https://repo1.maven.org/maven2/org/hamcrest/hamcrest-all/$HAMCREST_VERSION/hamcrest-all-$HAMCREST_VERSION.jar &&\
- mv hamcrest-all-$HAMCREST_VERSION.jar /opt/apache/beam/jars/hamcrest-all.jar
-# Install JUnit
-RUN wget https://repo1.maven.org/maven2/junit/junit/$JUNIT_VERSION/junit-$JUNIT_VERSION.jar &&\
- mv junit-$JUNIT_VERSION.jar /opt/apache/beam/jars/junit.jar
-# Install Beam Vendor gRPC
-RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-vendor-grpc-$GRPC_VERSION/$BEAM_VENDOR_GRPC_VERSION/beam-vendor-grpc-$GRPC_VERSION-$BEAM_VENDOR_GRPC_VERSION.jar &&\
- mv beam-vendor-grpc-$GRPC_VERSION-$BEAM_VENDOR_GRPC_VERSION.jar /opt/apache/beam/jars/beam-vendor-grpc.jar
-# Install Kafka IO
-RUN wget https://repo1.maven.org/maven2/org/apache/beam/beam-sdks-java-io-kafka/$BEAM_VERSION/beam-sdks-java-io-kafka-$BEAM_VERSION.jar &&\
- mv beam-sdks-java-io-kafka-$BEAM_VERSION.jar /opt/apache/beam/jars/beam-sdks-java-io-kafka.jar
-# Install Apache Kafka Clients
+# Install kafka client
RUN wget https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/$KAFKA_CLIENTS_VERSION/kafka-clients-$KAFKA_CLIENTS_VERSION.jar &&\
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] olehborysevych commented on pull request #25230: [DO NOT MERGE] fixing dependency management for java playground runner 2.44
Posted by "olehborysevych (via GitHub)" <gi...@apache.org>.
olehborysevych commented on PR #25230:
URL: https://github.com/apache/beam/pull/25230#issuecomment-1412513707
R: @damccorm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org