You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org> on 2023/02/16 10:10:57 UTC
[GitHub] [beam] MakarkinSAkvelon opened a new pull request, #25508: [Playground] Playground network policy+MiTM proxy
MakarkinSAkvelon opened a new pull request, #25508:
URL: https://github.com/apache/beam/pull/25508
1. Configure Network Policy for GKE Cluster
2. Configure MiTM proxy
3. Deny direct internet access for backend pods
4. Add NetworkPolicy.yaml file
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1109642164
##########
playground/backend/containers/mitmproxy/build.gradle:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+apply plugin: 'org.apache.beam.module'
+apply plugin: 'base'
+applyDockerNature()
+
+def playgroundJobServerProject = "${project.path.replace('-container', '')}"
+
+description = project(playgroundJobServerProject).description + " :: Container"
+
+configurations {
+ dockerDependency
+}
+
+dependencies {
+ dockerDependency project(path: playgroundJobServerProject, configuration: "shadow")
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ copy {
+ from '../../../backend/'
+ into 'build/src'
+ exclude 'containers'
+ }
+ copy {
+ from 'entrypoint.sh'
+ into 'build/'
+ }
+ copy {
+ from 'config.yaml'
+ into 'build/'
+ }
+ copy {
+ from '../../../infrastructure/proxy/allow_list.py'
+ into 'build/'
+ }
+ copy {
+ from '../../../infrastructure/proxy/allow_list_proxy.py'
+ into 'build/'
+ }
+ copy {
+ from '../../../playground'
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25508:
URL: https://github.com/apache/beam/pull/25508#issuecomment-1446627516
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1126142734
##########
playground/backend/containers/go/Dockerfile:
##########
@@ -58,17 +58,6 @@ ENV PROPERTY_PATH=/opt/playground/backend/properties.yaml
## Copy build result
COPY src/configs /opt/playground/backend/configs/
Review Comment:
These "configs" define how to construct command lines to run the Beam examples - I don't see any reason why anybody might need to change them without rebuilding the application and/or container
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1124987371
##########
playground/docker-compose.local.yaml:
##########
@@ -87,10 +101,13 @@ services:
CACHE_TYPE: remote
CACHE_ADDRESS: redis:6379
SERVER_PORT: 8090
+ PLAYGROUND_MITM_SERVICE_HOST: mitmproxy
+ PLAYGROUND_MITM_SERVICE_PORT: 8080
ports:
- "8090:8090"
depends_on:
- redis
+ - mitmproxy
frontend:
image: apache/beam_playground-frontend
Review Comment:
We're updating the playground to implement MiTM proxy as agreed in the meeting.
This approach sounds like a design change that will impact development and scope. If you would like let's discuss this in the coming meeting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on PR #25508:
URL: https://github.com/apache/beam/pull/25508#issuecomment-1434851558
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1118630457
##########
playground/backend/containers/mitmproxy/build.gradle:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+apply plugin: 'org.apache.beam.module'
+apply plugin: 'base'
+applyDockerNature()
+
+def playgroundJobServerProject = "${project.path.replace('-container', '')}"
+
+description = project(playgroundJobServerProject).description + " :: Container"
+
+configurations {
+ dockerDependency
+}
+
+dependencies {
+ dockerDependency project(path: playgroundJobServerProject, configuration: "shadow")
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ copy {
+ from 'entrypoint.sh'
+ into 'build/'
+ }
+ copy {
+ from 'config.yaml'
+ into 'build/'
+ }
+ copy {
+ from '../../../infrastructure/proxy/allow_list.py'
+ into 'build/'
+ }
+ copy {
+ from '../../../infrastructure/proxy/allow_list_proxy.py'
+ into 'build/'
+ }
+
+}
+
+docker {
+ name containerImageName(
+ name: project.docker_image_default_repo_prefix + "playground-backend-mitmproxy",
+ root: project.rootProject.hasProperty(["docker-repository-root"]) ?
+ project.rootProject["docker-repository-root"] :
+ project.docker_image_default_repo_root)
+ files "./build/"
+ tags containerImageTags()
+}
+
+// Ensure that we build the required resources and copy and file dependencies from related projects
+dockerPrepare.dependsOn copyDockerfileDependencies
Review Comment:
Please add new line to end of file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1108619054
##########
playground/backend/containers/mitmproxy/build.gradle:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+apply plugin: 'org.apache.beam.module'
+apply plugin: 'base'
+applyDockerNature()
+
+def playgroundJobServerProject = "${project.path.replace('-container', '')}"
+
+description = project(playgroundJobServerProject).description + " :: Container"
+
+configurations {
+ dockerDependency
+}
+
+dependencies {
+ dockerDependency project(path: playgroundJobServerProject, configuration: "shadow")
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ copy {
Review Comment:
Do we need these files in the container?
##########
playground/backend/containers/mitmproxy/build.gradle:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+apply plugin: 'org.apache.beam.module'
+apply plugin: 'base'
+applyDockerNature()
+
+def playgroundJobServerProject = "${project.path.replace('-container', '')}"
+
+description = project(playgroundJobServerProject).description + " :: Container"
+
+configurations {
+ dockerDependency
+}
+
+dependencies {
+ dockerDependency project(path: playgroundJobServerProject, configuration: "shadow")
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ copy {
+ from '../../../backend/'
+ into 'build/src'
+ exclude 'containers'
+ }
+ copy {
+ from 'entrypoint.sh'
+ into 'build/'
+ }
+ copy {
+ from 'config.yaml'
+ into 'build/'
+ }
+ copy {
+ from '../../../infrastructure/proxy/allow_list.py'
+ into 'build/'
+ }
+ copy {
+ from '../../../infrastructure/proxy/allow_list_proxy.py'
+ into 'build/'
+ }
+ copy {
+ from '../../../playground'
Review Comment:
Do we need these files in the container?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on PR #25508:
URL: https://github.com/apache/beam/pull/25508#issuecomment-1495615875
Replaced by #25994
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1112951362
##########
playground/infrastructure/helm-playground/templates/network-policy.yaml:
##########
@@ -0,0 +1,66 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+ name: egress-deny
+spec:
+ podSelector:
+ matchExpressions:
+ - key: app
+ operator: In
+ values:
+ - backend-go
Review Comment:
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1109821318
##########
playground/infrastructure/helm-playground/templates/deployment-java.yml:
##########
@@ -54,11 +54,11 @@ spec:
periodSeconds: {{ .Values.healthcheck.readPeriodSeconds }}
resources:
Review Comment:
Amend values for c2-standard-16 machines for all containers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1118631066
##########
playground/infrastructure/helm-playground/templates/autoscaling-python.yaml:
##########
@@ -12,15 +12,27 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-apiVersion: autoscaling/v1
+apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: playground-python
spec:
- maxReplicas: 10
- minReplicas: 1
+ maxReplicas: 4
+ minReplicas: 2
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: playground-python
- targetCPUUtilizationPercentage: 90
+ metrics:
+ - type: Resource
+ resource:
+ name: cpu
+ target:
+ type: Utilization
+ averageUtilization: 95
+ - type: Resource
+ resource:
+ name: memory
+ target:
+ type: Utilization
+ averageUtilization: 80
Review Comment:
Please add a new line to the end of file
##########
playground/infrastructure/helm-playground/templates/autoscaling-scio.yaml:
##########
@@ -12,15 +12,21 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-apiVersion: autoscaling/v1
+apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: playground-scio
spec:
- maxReplicas: 10
- minReplicas: 2
+ maxReplicas: 4
+ minReplicas: 2
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: playground-scio
- targetCPUUtilizationPercentage: 90
+ metrics:
+ - type: Resource
+ resource:
+ name: memory
+ target:
+ type: Utilization
+ averageUtilization: 80
Review Comment:
Please add a new line to the end of file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1109640776
##########
playground/backend/containers/mitmproxy/build.gradle:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+apply plugin: 'org.apache.beam.module'
+apply plugin: 'base'
+applyDockerNature()
+
+def playgroundJobServerProject = "${project.path.replace('-container', '')}"
+
+description = project(playgroundJobServerProject).description + " :: Container"
+
+configurations {
+ dockerDependency
+}
+
+dependencies {
+ dockerDependency project(path: playgroundJobServerProject, configuration: "shadow")
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ copy {
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1108463913
##########
playground/backend/containers/scio/entrypoint.sh:
##########
@@ -14,13 +14,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-nohup /opt/mitmproxy/mitmdump -s /opt/mitmproxy/allow_list_proxy.py -p 8081 &
-while [ ! -f /home/appuser/.mitmproxy/mitmproxy-ca.pem ] ;
-do
- sleep 2
-done
-openssl x509 -in /home/appuser/.mitmproxy/mitmproxy-ca.pem -inform PEM -out /home/appuser/.mitmproxy/mitmproxy-ca.crt
-cp /home/appuser/.mitmproxy/mitmproxy-ca.crt /usr/local/share/ca-certificates/extra/
-update-ca-certificates
+export http_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
+export https_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
+export JAVA_TOOL_OPTIONS="-Dhttp.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttp.proxyPort:$PLAYGROUND_MITM_SERVICE_PORT -Dhttps.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttps.proxyPort=$PLAYGROUND_MITM_SERVICE_PORT"
+export SBT_OPTS="-Dhttp.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttp.proxyPort:$PLAYGROUND_MITM_SERVICE_PORT -Dhttps.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttps.proxyPort=$PLAYGROUND_MITM_SERVICE_PORT"
Review Comment:
Replace with
```shell
export SBT_OPTS="$SBT_OPTS -Dhttp.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttp.proxyPort:$PLAYGROUND_MITM_SERVICE_PORT -Dhttps.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttps.proxyPort=$PLAYGROUND_MITM_SERVICE_PORT"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1109318419
##########
playground/backend/containers/mitmproxy/config.yaml:
##########
@@ -0,0 +1,22 @@
+/*
Review Comment:
YAML syntax is not correct. You can validate with yamllint
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1124950528
##########
playground/backend/containers/java/build.gradle:
##########
@@ -43,14 +43,6 @@ task copyDockerfileDependencies(type: Copy) {
from 'entrypoint.sh'
into 'build/'
}
- copy {
- from '../../../infrastructure/proxy/allow_list.py'
- into 'build/'
- }
- copy {
- from '../../../infrastructure/proxy/allow_list_proxy.py'
- into 'build/'
- }
copy {
Review Comment:
This file is in another directory. And we just copy it to the same directory where Dockerfile is. According to the link provided by you, there is a rule:
"COPY obeys the following rules:
The <src> path must be inside the context of the build; you cannot COPY ../something /something, because the first step of a docker build is to send the context directory (and subdirectories) to the docker daemon.
If <src> is a directory, the entire contents of the directory are copied, including filesystem metadata."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1112955772
##########
playground/infrastructure/helm-playground/templates/network-policy.yaml:
##########
@@ -0,0 +1,66 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+ name: egress-deny
+spec:
+ podSelector:
+ matchExpressions:
+ - key: app
+ operator: In
+ values:
+ - backend-go
+ - backend-java
+ - backend-python
+ - backend-scio
+ policyTypes:
+ - Egress
+ egress:
+ - to:
+ - namespaceSelector:
+ matchLabels:
+ kubernetes.io/metadata.name: kube-system
+ podSelector:
+ matchLabels:
+ k8s-app: kube-dns
+ ports:
+ - port: 53
+ protocol: UDP
+ - port: 53
+ protocol: TCP
+ - to:
+ - ipBlock:
+ cidr: 10.0.0.0/8
+ ports:
+ - protocol: TCP
+ port: 6379
+ - protocol: TCP
+ port: 8080
+ - to:
+ - ipBlock:
+ cidr: 169.254.169.252/32
+ ports:
+ - protocol: TCP
+ port: 988
+ - to:
+ - ipBlock:
+ cidr: 169.254.169.254/32
Review Comment:
Comment from:cloud.google.com
"If you use network policy with Workload Identity, you must allow egress to the following IP addresses so your Pods can communicate with the [GKE metadata server](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#gke_mds)."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1108462580
##########
playground/backend/containers/java/Dockerfile:
##########
@@ -94,16 +94,7 @@ RUN cd /go/src/katas &&\
./gradlew jar &&\
mv util/build/libs/util.jar /opt/apache/beam/jars/util.jar
-# Install mitmpoxy
-RUN mkdir /opt/mitmproxy &&\
- cd /opt/mitmproxy &&\
- wget https://snapshots.mitmproxy.org/7.0.4/mitmproxy-7.0.4-linux.tar.gz &&\
- tar -zxvf mitmproxy-7.0.4-linux.tar.gz &&\
- mkdir /usr/local/share/ca-certificates/extra
-COPY allow_list_proxy.py /opt/mitmproxy/
-COPY allow_list.py /opt/mitmproxy/
-ENV HTTP_PROXY="http://127.0.0.1:8081"
-ENV HTTPS_PROXY="http://127.0.0.1:8081"
+RUN mkdir /usr/local/share/ca-certificates/extra
Review Comment:
Please remove
##########
playground/backend/containers/scio/Dockerfile:
##########
@@ -55,17 +56,7 @@ echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | tee /etc/apt/sources.l
curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | apt-key add
RUN apt-get update && apt-get install -y sbt
-
-## Install mitmpoxy
-RUN mkdir /opt/mitmproxy &&\
- cd /opt/mitmproxy &&\
- wget https://snapshots.mitmproxy.org/7.0.4/mitmproxy-7.0.4-linux.tar.gz &&\
- tar -zxvf mitmproxy-7.0.4-linux.tar.gz &&\
- mkdir /usr/local/share/ca-certificates/extra
-COPY allow_list_proxy.py /opt/mitmproxy/
-COPY allow_list.py /opt/mitmproxy/
-ENV HTTP_PROXY="http://127.0.0.1:8081"
-ENV HTTPS_PROXY="http://127.0.0.1:8081"
+RUN mkdir /usr/local/share/ca-certificates/extra
Review Comment:
Please remove
##########
playground/backend/containers/mitmproxy/settings.gradle:
##########
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+rootProject.name = 'apache-beam-playground-backend-mitmproxy'
Review Comment:
Please add empty line to the end of file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1108461210
##########
playground/backend/containers/go/entrypoint.sh:
##########
@@ -14,13 +14,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-nohup /opt/mitmproxy/mitmdump -s /opt/mitmproxy/allow_list_proxy.py -p 8081 &
-while [ ! -f /home/appuser/.mitmproxy/mitmproxy-ca.pem ] ;
-do
- sleep 2
-done
-openssl x509 -in /home/appuser/.mitmproxy/mitmproxy-ca.pem -inform PEM -out /home/appuser/.mitmproxy/mitmproxy-ca.crt
-cp /home/appuser/.mitmproxy/mitmproxy-ca.crt /usr/local/share/ca-certificates/extra/
-update-ca-certificates
-
-/opt/playground/backend/server_go_backend
+export http_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
+export https_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
+/opt/playground/backend/server_go_backend
Review Comment:
Please add empty line to the end of the file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1108464582
##########
playground/backend/new_scio_project.sh:
##########
@@ -16,3 +16,6 @@
# limitations under the License.
yes | sbt new spotify/scio-template.g8
+
+mkdir y/lib
+cp /opt/google-api-client.jar y/lib/
Review Comment:
Remove
##########
playground/backend/new_scio_project.sh:
##########
@@ -16,3 +16,6 @@
# limitations under the License.
yes | sbt new spotify/scio-template.g8
+
+mkdir y/lib
Review Comment:
Remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov closed pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov closed pull request #25508: [Playground] Playground network policy+MiTM proxy
URL: https://github.com/apache/beam/pull/25508
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damondouglas commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "damondouglas (via GitHub)" <gi...@apache.org>.
damondouglas commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1124921815
##########
playground/backend/containers/mitmproxy/Dockerfile:
##########
@@ -0,0 +1,27 @@
+###############################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+###############################################################################
+FROM mitmproxy/mitmproxy:9.0.1
Review Comment:
Could we have one or two sentences describing the purpose of mtmproxy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1109875319
##########
playground/backend/containers/scio/Dockerfile:
##########
@@ -36,6 +36,7 @@ RUN go mod download &&\
go build -o /go/bin/server_scio_backend
FROM $BASE_IMAGE
+ARG GOOGLE_API_CLIENT_VERSION=1.34.1
Review Comment:
Why do we need this ARG ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1111860449
##########
playground/infrastructure/helm-playground/templates/network-policy.yaml:
##########
@@ -0,0 +1,66 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+ name: egress-deny
+spec:
+ podSelector:
+ matchExpressions:
+ - key: app
+ operator: In
+ values:
+ - backend-go
+ - backend-java
+ - backend-python
+ - backend-scio
+ policyTypes:
+ - Egress
+ egress:
+ - to:
+ - namespaceSelector:
+ matchLabels:
+ kubernetes.io/metadata.name: kube-system
+ podSelector:
+ matchLabels:
+ k8s-app: kube-dns
+ ports:
+ - port: 53
+ protocol: UDP
+ - port: 53
+ protocol: TCP
+ - to:
+ - ipBlock:
+ cidr: 10.0.0.0/8
+ ports:
+ - protocol: TCP
+ port: 6379
+ - protocol: TCP
+ port: 8080
+ - to:
+ - ipBlock:
+ cidr: 169.254.169.252/32
+ ports:
+ - protocol: TCP
+ port: 988
+ - to:
+ - ipBlock:
+ cidr: 169.254.169.254/32
Review Comment:
This is a GKE metadata server address. It should be hardcoded here. @MakarkinSAkvelon, can you please add commentary about why this address is specified here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1124955043
##########
playground/infrastructure/helm-playground/templates/deployment-mitmproxy.yaml:
##########
@@ -0,0 +1,39 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: playground-mitm
+ labels:
+ app: backend-mitm
+ type: backend
+spec:
+ template:
+ metadata:
+ name: playground
+ labels:
+ app: backend-mitm
+ type: backend
+ spec:
+ containers:
+ - name: playground-backend-mitm
+ image: "{{ .Values.registry}}/{{ .Values.image.mitm_image }}:{{ .Values.tag }}"
+ imagePullPolicy: {{ .Values.image.pullPolicy }}
+ replicas: {{ .Values.replicaCount }}
+ selector:
+ matchLabels:
+ app: backend-mitm
Review Comment:
MiTM proxy container. Increase security for playground
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1127635717
##########
playground/backend/containers/java/entrypoint.sh:
##########
@@ -14,13 +14,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-nohup /opt/mitmproxy/mitmdump -s /opt/mitmproxy/allow_list_proxy.py -p 8081 &
-while [ ! -f /home/appuser/.mitmproxy/mitmproxy-ca.pem ] ;
-do
- sleep 2
-done
-openssl x509 -in /home/appuser/.mitmproxy/mitmproxy-ca.pem -inform PEM -out /home/appuser/.mitmproxy/mitmproxy-ca.crt
-cp /home/appuser/.mitmproxy/mitmproxy-ca.crt /usr/local/share/ca-certificates/extra/
-update-ca-certificates
-
+if [[ -n "$PLAYGROUND_MITM_SERVICE_HOST" ]] && [[ -n "$PLAYGROUND_MITM_SERVICE_PORT" ]]
+then
+ export http_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
+ export https_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
+ export JAVA_TOOL_OPTIONS="$JAVA_TOOL_OPTIONS -Dhttp.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttp.proxyPort=$PLAYGROUND_MITM_SERVICE_PORT -Dhttps.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttps.proxyPort=$PLAYGROUND_MITM_SERVICE_PORT"
Review Comment:
These variables are dynamic variables and they are set by GKE during deployment. ConfigMap works with Static variables only
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1124953958
##########
playground/infrastructure/helm-playground/templates/network-policy.yaml:
##########
@@ -0,0 +1,66 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+ name: egress-deny
+spec:
+ podSelector:
+ matchExpressions:
+ - key: app
+ operator: In
+ values:
+ - backend-go
+ - backend-java
+ - backend-python
+ - backend-scio
+ policyTypes:
+ - Egress
+ egress:
+ - to:
+ - namespaceSelector:
+ matchLabels:
+ kubernetes.io/metadata.name: kube-system
+ podSelector:
+ matchLabels:
+ k8s-app: kube-dns
+ ports:
+ - port: 53
+ protocol: UDP
+ - port: 53
+ protocol: TCP
+ - to:
+ - ipBlock:
+ cidr: 10.0.0.0/8
+ ports:
+ - protocol: TCP
+ port: 6379
+ - protocol: TCP
+ port: 8080
+ - to:
+ - ipBlock:
+ cidr: 169.254.169.252/32
+ ports:
+ - protocol: TCP
+ port: 988
+ - to:
+ - ipBlock:
+ cidr: 169.254.169.254/32
+ ports:
+ - protocol: TCP
+ port: 80
Review Comment:
Deny direct access to the internet for backend runners. Only thru the MiTM container. Here we define NetworkPolicy for Kubernetes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1127561479
##########
playground/backend/containers/java/entrypoint.sh:
##########
@@ -14,13 +14,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-nohup /opt/mitmproxy/mitmdump -s /opt/mitmproxy/allow_list_proxy.py -p 8081 &
-while [ ! -f /home/appuser/.mitmproxy/mitmproxy-ca.pem ] ;
-do
- sleep 2
-done
-openssl x509 -in /home/appuser/.mitmproxy/mitmproxy-ca.pem -inform PEM -out /home/appuser/.mitmproxy/mitmproxy-ca.crt
-cp /home/appuser/.mitmproxy/mitmproxy-ca.crt /usr/local/share/ca-certificates/extra/
-update-ca-certificates
-
+if [[ -n "$PLAYGROUND_MITM_SERVICE_HOST" ]] && [[ -n "$PLAYGROUND_MITM_SERVICE_PORT" ]]
+then
+ export http_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
+ export https_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
Review Comment:
These are variables to specify proxy settings for programs. So that, they can send requests to the proxy server to access the Internet. We forbid direct access to the external network from runners.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1127637304
##########
playground/backend/containers/mitmproxy/Dockerfile:
##########
@@ -0,0 +1,27 @@
+###############################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+###############################################################################
+FROM mitmproxy/mitmproxy:9.0.1
Review Comment:
To be added after meeting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1108646740
##########
playground/infrastructure/proxy/allow_list.py:
##########
@@ -22,12 +22,11 @@
"datastore.googleapis.com",
"oauth2.googleapis.com",
"storage.googleapis.com",
- "repo1.maven.org"
-]
-
-# ALLOWED_BUCKET_LIST contains all public Google Cloud Storage buckets
-# that are allowed to make requests from Beam Playground
-ALLOWED_BUCKET_LIST = [
- "dataflow-samples",
- "beam-samples"
+ "www.googleapis.com",
+ "repo1.maven.org",
+ "github.com",
+ "packages.confluent.io",
+ "bigquery.googleapis.com",
+ "metadata.google.internal",
+ "169.254.169.254",
Review Comment:
What is this for? Add a comment with explanation to the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1108645144
##########
playground/infrastructure/proxy/allow_list.py:
##########
@@ -22,12 +22,11 @@
"datastore.googleapis.com",
Review Comment:
Do we still need "localhost" and "127.0.0.1" in this list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1108466134
##########
playground/terraform/build.gradle.kts:
##########
@@ -498,5 +499,4 @@ tasks.register("gkebackend") {
pushFrontTask.mustRunAfter(pushBackTask)
indexcreateTask.mustRunAfter(pushFrontTask)
helmTask.mustRunAfter(indexcreateTask)
-}
-
+}
Review Comment:
add empty line to the end of the file
##########
playground/infrastructure/helm-playground/templates/service-mitm.yaml:
##########
@@ -0,0 +1,35 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: v1
+kind: Service
+metadata:
+ labels:
+ app: backend-mitm
+ type: backend
+ name: playground-mitm
+ annotations:
+ cloud.google.com/backend-config: '{"default": "bkend-health"}'
+ cloud.google.com/neg: '{"ingress": true}'
+spec:
+ ports:
+ - name: tcp
+ port: 8080
+ protocol: TCP
+ targetPort: {{ .Values.service.targetPort }}
+ selector:
+ app: backend-mitm
+ type: {{ .Values.service.type }}
Review Comment:
Add empty line to the end of the file
##########
playground/infrastructure/helm-playground/templates/deployment-mitmproxy.yaml:
##########
@@ -0,0 +1,39 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: playground-mitm
+ labels:
+ app: backend-mitm
+ type: backend
+spec:
+ template:
+ metadata:
+ name: playground
+ labels:
+ app: backend-mitm
+ type: backend
+ spec:
+ containers:
+ - name: playground-backend-mitm
+ image: "{{ .Values.registry}}/{{ .Values.image.mitm_image }}:{{ .Values.tag }}"
+ imagePullPolicy: {{ .Values.image.pullPolicy }}
+ replicas: {{ .Values.replicaCount }}
+ selector:
+ matchLabels:
+ app: backend-mitm
Review Comment:
Add empty line to the end of the file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1108461210
##########
playground/backend/containers/go/entrypoint.sh:
##########
@@ -14,13 +14,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-nohup /opt/mitmproxy/mitmdump -s /opt/mitmproxy/allow_list_proxy.py -p 8081 &
-while [ ! -f /home/appuser/.mitmproxy/mitmproxy-ca.pem ] ;
-do
- sleep 2
-done
-openssl x509 -in /home/appuser/.mitmproxy/mitmproxy-ca.pem -inform PEM -out /home/appuser/.mitmproxy/mitmproxy-ca.crt
-cp /home/appuser/.mitmproxy/mitmproxy-ca.crt /usr/local/share/ca-certificates/extra/
-update-ca-certificates
-
-/opt/playground/backend/server_go_backend
+export http_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
+export https_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
+/opt/playground/backend/server_go_backend
Review Comment:
Please add empty line to the end of the file
##########
playground/backend/containers/scio/Dockerfile:
##########
@@ -55,17 +56,7 @@ echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | tee /etc/apt/sources.l
curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | apt-key add
RUN apt-get update && apt-get install -y sbt
-
-## Install mitmpoxy
-RUN mkdir /opt/mitmproxy &&\
- cd /opt/mitmproxy &&\
- wget https://snapshots.mitmproxy.org/7.0.4/mitmproxy-7.0.4-linux.tar.gz &&\
- tar -zxvf mitmproxy-7.0.4-linux.tar.gz &&\
- mkdir /usr/local/share/ca-certificates/extra
-COPY allow_list_proxy.py /opt/mitmproxy/
-COPY allow_list.py /opt/mitmproxy/
-ENV HTTP_PROXY="http://127.0.0.1:8081"
-ENV HTTPS_PROXY="http://127.0.0.1:8081"
+RUN mkdir /usr/local/share/ca-certificates/extra
Review Comment:
Please remove
##########
playground/backend/containers/go/Dockerfile:
##########
@@ -58,16 +58,7 @@ ENV PROPERTY_PATH=/opt/playground/backend/properties.yaml
## Copy build result
COPY src/configs /opt/playground/backend/configs/
-# Install mitmpoxy
-RUN mkdir /opt/mitmproxy &&\
- cd /opt/mitmproxy &&\
- wget -q https://snapshots.mitmproxy.org/7.0.4/mitmproxy-7.0.4-linux.tar.gz &&\
- tar -zxvf mitmproxy-7.0.4-linux.tar.gz &&\
- mkdir /usr/local/share/ca-certificates/extra
-COPY allow_list_proxy.py /opt/mitmproxy/
-COPY allow_list.py /opt/mitmproxy/
-ENV HTTP_PROXY="http://127.0.0.1:8081"
-ENV HTTPS_PROXY="http://127.0.0.1:8081"
+RUN mkdir /usr/local/share/ca-certificates/extra
Review Comment:
Please remove
##########
playground/backend/containers/java/Dockerfile:
##########
@@ -94,16 +94,7 @@ RUN cd /go/src/katas &&\
./gradlew jar &&\
mv util/build/libs/util.jar /opt/apache/beam/jars/util.jar
-# Install mitmpoxy
-RUN mkdir /opt/mitmproxy &&\
- cd /opt/mitmproxy &&\
- wget https://snapshots.mitmproxy.org/7.0.4/mitmproxy-7.0.4-linux.tar.gz &&\
- tar -zxvf mitmproxy-7.0.4-linux.tar.gz &&\
- mkdir /usr/local/share/ca-certificates/extra
-COPY allow_list_proxy.py /opt/mitmproxy/
-COPY allow_list.py /opt/mitmproxy/
-ENV HTTP_PROXY="http://127.0.0.1:8081"
-ENV HTTPS_PROXY="http://127.0.0.1:8081"
+RUN mkdir /usr/local/share/ca-certificates/extra
Review Comment:
Please remove
##########
playground/backend/new_scio_project.sh:
##########
@@ -16,3 +16,6 @@
# limitations under the License.
yes | sbt new spotify/scio-template.g8
+
+mkdir y/lib
Review Comment:
Remove
##########
playground/infrastructure/helm-playground/templates/service-mitm.yaml:
##########
@@ -0,0 +1,35 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: v1
+kind: Service
+metadata:
+ labels:
+ app: backend-mitm
+ type: backend
+ name: playground-mitm
+ annotations:
+ cloud.google.com/backend-config: '{"default": "bkend-health"}'
+ cloud.google.com/neg: '{"ingress": true}'
+spec:
+ ports:
+ - name: tcp
+ port: 8080
+ protocol: TCP
+ targetPort: {{ .Values.service.targetPort }}
+ selector:
+ app: backend-mitm
+ type: {{ .Values.service.type }}
Review Comment:
Add empty line to the end of the file
##########
playground/backend/containers/scio/entrypoint.sh:
##########
@@ -14,13 +14,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-nohup /opt/mitmproxy/mitmdump -s /opt/mitmproxy/allow_list_proxy.py -p 8081 &
-while [ ! -f /home/appuser/.mitmproxy/mitmproxy-ca.pem ] ;
-do
- sleep 2
-done
-openssl x509 -in /home/appuser/.mitmproxy/mitmproxy-ca.pem -inform PEM -out /home/appuser/.mitmproxy/mitmproxy-ca.crt
-cp /home/appuser/.mitmproxy/mitmproxy-ca.crt /usr/local/share/ca-certificates/extra/
-update-ca-certificates
+export http_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
+export https_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
+export JAVA_TOOL_OPTIONS="-Dhttp.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttp.proxyPort:$PLAYGROUND_MITM_SERVICE_PORT -Dhttps.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttps.proxyPort=$PLAYGROUND_MITM_SERVICE_PORT"
+export SBT_OPTS="-Dhttp.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttp.proxyPort:$PLAYGROUND_MITM_SERVICE_PORT -Dhttps.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttps.proxyPort=$PLAYGROUND_MITM_SERVICE_PORT"
Review Comment:
Replace with
```shell
export SBT_OPTS="$SBT_OPTS -Dhttp.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttp.proxyPort:$PLAYGROUND_MITM_SERVICE_PORT -Dhttps.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttps.proxyPort=$PLAYGROUND_MITM_SERVICE_PORT"
```
##########
playground/infrastructure/helm-playground/templates/deployment-mitmproxy.yaml:
##########
@@ -0,0 +1,39 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: playground-mitm
+ labels:
+ app: backend-mitm
+ type: backend
+spec:
+ template:
+ metadata:
+ name: playground
+ labels:
+ app: backend-mitm
+ type: backend
+ spec:
+ containers:
+ - name: playground-backend-mitm
+ image: "{{ .Values.registry}}/{{ .Values.image.mitm_image }}:{{ .Values.tag }}"
+ imagePullPolicy: {{ .Values.image.pullPolicy }}
+ replicas: {{ .Values.replicaCount }}
+ selector:
+ matchLabels:
+ app: backend-mitm
Review Comment:
Add empty line to the end of the file
##########
playground/terraform/build.gradle.kts:
##########
@@ -498,5 +499,4 @@ tasks.register("gkebackend") {
pushFrontTask.mustRunAfter(pushBackTask)
indexcreateTask.mustRunAfter(pushFrontTask)
helmTask.mustRunAfter(indexcreateTask)
-}
-
+}
Review Comment:
add empty line to the end of the file
##########
playground/backend/containers/mitmproxy/settings.gradle:
##########
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+rootProject.name = 'apache-beam-playground-backend-mitmproxy'
Review Comment:
Please add empty line to the end of file
##########
playground/backend/new_scio_project.sh:
##########
@@ -16,3 +16,6 @@
# limitations under the License.
yes | sbt new spotify/scio-template.g8
+
+mkdir y/lib
+cp /opt/google-api-client.jar y/lib/
Review Comment:
Remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1118630845
##########
playground/infrastructure/helm-playground/templates/autoscaling-go.yaml:
##########
@@ -12,15 +12,27 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-apiVersion: autoscaling/v1
+apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: playground-go
spec:
- maxReplicas: 10
- minReplicas: 1
+ maxReplicas: 4
+ minReplicas: 2
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: playground-go
- targetCPUUtilizationPercentage: 90
+ metrics:
+ - type: Resource
+ resource:
+ name: cpu
+ target:
+ type: Utilization
+ averageUtilization: 95
+ - type: Resource
+ resource:
+ name: memory
+ target:
+ type: Utilization
+ averageUtilization: 80
Review Comment:
Please add a new line to the end of file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1124958704
##########
playground/backend/containers/mitmproxy/build.gradle:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+apply plugin: 'org.apache.beam.module'
+apply plugin: 'base'
+applyDockerNature()
+
+def playgroundJobServerProject = "${project.path.replace('-container', '')}"
+
+description = project(playgroundJobServerProject).description + " :: Container"
+
+configurations {
+ dockerDependency
+}
+
+dependencies {
+ dockerDependency project(path: playgroundJobServerProject, configuration: "shadow")
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ copy {
+ from 'entrypoint.sh'
+ into 'build/'
+ }
+ copy {
+ from 'config.yaml'
+ into 'build/'
+ }
+ copy {
+ from '../../../infrastructure/proxy/allow_list.py'
+ into 'build/'
+ }
+ copy {
+ from '../../../infrastructure/proxy/allow_list_proxy.py'
+ into 'build/'
+ }
Review Comment:
This file is in another directory. And we just copy it to the same directory where Dockerfile is. According to the link provided by you, there is a rule:
"COPY obeys the following rules:
The path must be inside the context of the build; you cannot COPY ../something /something, because the first step of a docker build is to send the context directory (and subdirectories) to the docker daemon.
If is a directory, the entire contents of the directory are copied, including filesystem metadata."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1124935539
##########
playground/backend/containers/go/entrypoint.sh:
##########
@@ -14,13 +14,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-nohup /opt/mitmproxy/mitmdump -s /opt/mitmproxy/allow_list_proxy.py -p 8081 &
-while [ ! -f /home/appuser/.mitmproxy/mitmproxy-ca.pem ] ;
-do
- sleep 2
-done
-openssl x509 -in /home/appuser/.mitmproxy/mitmproxy-ca.pem -inform PEM -out /home/appuser/.mitmproxy/mitmproxy-ca.crt
-cp /home/appuser/.mitmproxy/mitmproxy-ca.crt /usr/local/share/ca-certificates/extra/
-update-ca-certificates
-
+if [[ -n "$PLAYGROUND_MITM_SERVICE_HOST" ]] && [[ -n "$PLAYGROUND_MITM_SERVICE_PORT" ]]
+then
+ export http_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
Review Comment:
Direct internet connection for backend runners will be closed, so it will be more securely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1119818317
##########
playground/infrastructure/helm-playground/templates/autoscaling-scio.yaml:
##########
@@ -12,15 +12,21 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-apiVersion: autoscaling/v1
+apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: playground-scio
spec:
- maxReplicas: 10
- minReplicas: 2
+ maxReplicas: 4
+ minReplicas: 2
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: playground-scio
- targetCPUUtilizationPercentage: 90
+ metrics:
+ - type: Resource
+ resource:
+ name: memory
+ target:
+ type: Utilization
+ averageUtilization: 80
Review Comment:
Please also remove trailing spaces (yamllint for hints)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damondouglas commented on pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "damondouglas (via GitHub)" <gi...@apache.org>.
damondouglas commented on PR #25508:
URL: https://github.com/apache/beam/pull/25508#issuecomment-1462425071
@MakarkinSAkvelon (cc: @pabloem ) Did some research on this issue. To accomplish the same goals in Google Cloud:
1) Create a private kubernetes cluster (See: https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters)
2) Create a NAT and Router (See: https://cloud.google.com/nat/docs/overview)
3) Enable private Google access (See: https://cloud.google.com/vpc/docs/private-google-access)
4) Create firewall policy that combines a deny all egress with an allow using FQDN of the outside Google frontend domains (i.e. not google APIs; you can already access google APIs via private google access in the subnetwork). The priority of the allow rule must proceed that of the deny all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1127632799
##########
playground/backend/containers/go/Dockerfile:
##########
@@ -58,17 +58,6 @@ ENV PROPERTY_PATH=/opt/playground/backend/properties.yaml
## Copy build result
COPY src/configs /opt/playground/backend/configs/
-# Install mitmpoxy
-RUN mkdir /opt/mitmproxy &&\
- cd /opt/mitmproxy &&\
- wget -q https://snapshots.mitmproxy.org/7.0.4/mitmproxy-7.0.4-linux.tar.gz &&\
- tar -zxvf mitmproxy-7.0.4-linux.tar.gz &&\
- mkdir /usr/local/share/ca-certificates/extra
-COPY allow_list_proxy.py /opt/mitmproxy/
-COPY allow_list.py /opt/mitmproxy/
-ENV HTTP_PROXY="http://127.0.0.1:8081"
-ENV HTTPS_PROXY="http://127.0.0.1:8081"
-
COPY src/properties.yaml /opt/playground/backend/properties.yaml
Review Comment:
This file contains the settings which would not be changed after GKE deployment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #25508:
URL: https://github.com/apache/beam/pull/25508#issuecomment-1432879564
# [Codecov](https://codecov.io/gh/apache/beam/pull/25508?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#25508](https://codecov.io/gh/apache/beam/pull/25508?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7ccfa36) into [master](https://codecov.io/gh/apache/beam/commit/88da381cc3361d551800868372afb2f737e973b5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (88da381) will **not change** coverage.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #25508 +/- ##
=======================================
Coverage 72.96% 72.96%
=======================================
Files 745 745
Lines 99191 99191
=======================================
Hits 72371 72371
Misses 25455 25455
Partials 1365 1365
```
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1108622289
##########
playground/backend/containers/mitmproxy/build.gradle:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+apply plugin: 'org.apache.beam.module'
+apply plugin: 'base'
+applyDockerNature()
+
+def playgroundJobServerProject = "${project.path.replace('-container', '')}"
+
+description = project(playgroundJobServerProject).description + " :: Container"
+
+configurations {
+ dockerDependency
+}
+
+dependencies {
+ dockerDependency project(path: playgroundJobServerProject, configuration: "shadow")
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ copy {
+ from '../../../backend/'
+ into 'build/src'
+ exclude 'containers'
+ }
+ copy {
+ from 'entrypoint.sh'
+ into 'build/'
+ }
+ copy {
+ from 'config.yaml'
+ into 'build/'
+ }
+ copy {
+ from '../../../infrastructure/proxy/allow_list.py'
+ into 'build/'
+ }
+ copy {
+ from '../../../infrastructure/proxy/allow_list_proxy.py'
+ into 'build/'
+ }
+ copy {
+ from '../../../playground'
+ into 'build/playground'
+ }
+}
+
+docker {
+ name containerImageName(
+ name: project.docker_image_default_repo_prefix + "playground-backend-mitmproxy",
+ root: project.rootProject.hasProperty(["docker-repository-root"]) ?
+ project.rootProject["docker-repository-root"] :
+ project.docker_image_default_repo_root)
+ files "./build/"
+ tags containerImageTags()
+ buildArgs(['GO_BASE_IMAGE': project.rootProject.hasProperty(["go-base-image"]) ?
Review Comment:
These arguments not used by mitm container
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1108462769
##########
playground/backend/containers/go/Dockerfile:
##########
@@ -58,16 +58,7 @@ ENV PROPERTY_PATH=/opt/playground/backend/properties.yaml
## Copy build result
COPY src/configs /opt/playground/backend/configs/
-# Install mitmpoxy
-RUN mkdir /opt/mitmproxy &&\
- cd /opt/mitmproxy &&\
- wget -q https://snapshots.mitmproxy.org/7.0.4/mitmproxy-7.0.4-linux.tar.gz &&\
- tar -zxvf mitmproxy-7.0.4-linux.tar.gz &&\
- mkdir /usr/local/share/ca-certificates/extra
-COPY allow_list_proxy.py /opt/mitmproxy/
-COPY allow_list.py /opt/mitmproxy/
-ENV HTTP_PROXY="http://127.0.0.1:8081"
-ENV HTTPS_PROXY="http://127.0.0.1:8081"
+RUN mkdir /usr/local/share/ca-certificates/extra
Review Comment:
Please remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1109643954
##########
playground/backend/containers/mitmproxy/build.gradle:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+apply plugin: 'org.apache.beam.module'
+apply plugin: 'base'
+applyDockerNature()
+
+def playgroundJobServerProject = "${project.path.replace('-container', '')}"
+
+description = project(playgroundJobServerProject).description + " :: Container"
+
+configurations {
+ dockerDependency
+}
+
+dependencies {
+ dockerDependency project(path: playgroundJobServerProject, configuration: "shadow")
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ copy {
+ from '../../../backend/'
+ into 'build/src'
+ exclude 'containers'
+ }
+ copy {
+ from 'entrypoint.sh'
+ into 'build/'
+ }
+ copy {
+ from 'config.yaml'
+ into 'build/'
+ }
+ copy {
+ from '../../../infrastructure/proxy/allow_list.py'
+ into 'build/'
+ }
+ copy {
+ from '../../../infrastructure/proxy/allow_list_proxy.py'
+ into 'build/'
+ }
+ copy {
+ from '../../../playground'
+ into 'build/playground'
+ }
+}
+
+docker {
+ name containerImageName(
+ name: project.docker_image_default_repo_prefix + "playground-backend-mitmproxy",
+ root: project.rootProject.hasProperty(["docker-repository-root"]) ?
+ project.rootProject["docker-repository-root"] :
+ project.docker_image_default_repo_root)
+ files "./build/"
+ tags containerImageTags()
+ buildArgs(['GO_BASE_IMAGE': project.rootProject.hasProperty(["go-base-image"]) ?
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damondouglas commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "damondouglas (via GitHub)" <gi...@apache.org>.
damondouglas commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1124912067
##########
playground/backend/containers/go/Dockerfile:
##########
@@ -58,17 +58,6 @@ ENV PROPERTY_PATH=/opt/playground/backend/properties.yaml
## Copy build result
COPY src/configs /opt/playground/backend/configs/
Review Comment:
Why are "configs" embeded in a Dockerfile? Given the kubernetes context? Why not rely on a ConfigMap instead of coupling the binary with configuration?
##########
playground/backend/containers/go/Dockerfile:
##########
@@ -58,17 +58,6 @@ ENV PROPERTY_PATH=/opt/playground/backend/properties.yaml
## Copy build result
COPY src/configs /opt/playground/backend/configs/
-# Install mitmpoxy
-RUN mkdir /opt/mitmproxy &&\
- cd /opt/mitmproxy &&\
- wget -q https://snapshots.mitmproxy.org/7.0.4/mitmproxy-7.0.4-linux.tar.gz &&\
- tar -zxvf mitmproxy-7.0.4-linux.tar.gz &&\
- mkdir /usr/local/share/ca-certificates/extra
-COPY allow_list_proxy.py /opt/mitmproxy/
-COPY allow_list.py /opt/mitmproxy/
-ENV HTTP_PROXY="http://127.0.0.1:8081"
-ENV HTTPS_PROXY="http://127.0.0.1:8081"
-
COPY src/properties.yaml /opt/playground/backend/properties.yaml
Review Comment:
Again, why not rely on a ConfigMap and mount that to the container, decoupling configuration from binary? If we are still stuck with appengine, terraform has a way to provide environment variables via the app_engine_flexible_app_version block https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/app_engine_flexible_app_version#env_variables
##########
playground/backend/containers/java/build.gradle:
##########
@@ -43,14 +43,6 @@ task copyDockerfileDependencies(type: Copy) {
from 'entrypoint.sh'
into 'build/'
}
- copy {
- from '../../../infrastructure/proxy/allow_list.py'
- into 'build/'
- }
- copy {
- from '../../../infrastructure/proxy/allow_list_proxy.py'
- into 'build/'
- }
copy {
Review Comment:
Why can't we use https://docs.docker.com/engine/reference/builder/#copy in the Dockerfile? Why does this need to happen in gradle? Isn't gradle designed for building code and automating test execution? Why are we coupling gradle with deployment operations when we have these kinds of tasks as part of the docker ecosystem?
##########
playground/backend/containers/go/entrypoint.sh:
##########
@@ -14,13 +14,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-nohup /opt/mitmproxy/mitmdump -s /opt/mitmproxy/allow_list_proxy.py -p 8081 &
-while [ ! -f /home/appuser/.mitmproxy/mitmproxy-ca.pem ] ;
-do
- sleep 2
-done
-openssl x509 -in /home/appuser/.mitmproxy/mitmproxy-ca.pem -inform PEM -out /home/appuser/.mitmproxy/mitmproxy-ca.crt
-cp /home/appuser/.mitmproxy/mitmproxy-ca.crt /usr/local/share/ca-certificates/extra/
-update-ca-certificates
-
+if [[ -n "$PLAYGROUND_MITM_SERVICE_HOST" ]] && [[ -n "$PLAYGROUND_MITM_SERVICE_PORT" ]]
+then
+ export http_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
Review Comment:
What does this proxy achieve?
##########
playground/backend/containers/java/entrypoint.sh:
##########
@@ -14,13 +14,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-nohup /opt/mitmproxy/mitmdump -s /opt/mitmproxy/allow_list_proxy.py -p 8081 &
-while [ ! -f /home/appuser/.mitmproxy/mitmproxy-ca.pem ] ;
-do
- sleep 2
-done
-openssl x509 -in /home/appuser/.mitmproxy/mitmproxy-ca.pem -inform PEM -out /home/appuser/.mitmproxy/mitmproxy-ca.crt
-cp /home/appuser/.mitmproxy/mitmproxy-ca.crt /usr/local/share/ca-certificates/extra/
-update-ca-certificates
-
+if [[ -n "$PLAYGROUND_MITM_SERVICE_HOST" ]] && [[ -n "$PLAYGROUND_MITM_SERVICE_PORT" ]]
+then
+ export http_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
+ export https_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
Review Comment:
Again why do we need this?
##########
playground/backend/containers/java/entrypoint.sh:
##########
@@ -14,13 +14,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-nohup /opt/mitmproxy/mitmdump -s /opt/mitmproxy/allow_list_proxy.py -p 8081 &
-while [ ! -f /home/appuser/.mitmproxy/mitmproxy-ca.pem ] ;
-do
- sleep 2
-done
-openssl x509 -in /home/appuser/.mitmproxy/mitmproxy-ca.pem -inform PEM -out /home/appuser/.mitmproxy/mitmproxy-ca.crt
-cp /home/appuser/.mitmproxy/mitmproxy-ca.crt /usr/local/share/ca-certificates/extra/
-update-ca-certificates
-
+if [[ -n "$PLAYGROUND_MITM_SERVICE_HOST" ]] && [[ -n "$PLAYGROUND_MITM_SERVICE_PORT" ]]
+then
+ export http_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
+ export https_proxy=http://$PLAYGROUND_MITM_SERVICE_HOST:$PLAYGROUND_MITM_SERVICE_PORT
+ export JAVA_TOOL_OPTIONS="$JAVA_TOOL_OPTIONS -Dhttp.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttp.proxyPort=$PLAYGROUND_MITM_SERVICE_PORT -Dhttps.proxyHost=$PLAYGROUND_MITM_SERVICE_HOST -Dhttps.proxyPort=$PLAYGROUND_MITM_SERVICE_PORT"
Review Comment:
Again, why are we setting an environment variable in a script embedded in the Dockerfile? Why can't we for example use a ConfigMap and derive environment variables from this? If we are still stuck with appengine, terraform has a way to provide environment variables via the app_engine_flexible_app_version block https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/app_engine_flexible_app_version#env_variables
##########
playground/backend/containers/mitmproxy/Dockerfile:
##########
@@ -0,0 +1,27 @@
+###############################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+###############################################################################
+FROM mitmproxy/mitmproxy:9.0.1
Review Comment:
Document the point of mitmproxy
##########
playground/backend/containers/mitmproxy/build.gradle:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+apply plugin: 'org.apache.beam.module'
+apply plugin: 'base'
+applyDockerNature()
+
+def playgroundJobServerProject = "${project.path.replace('-container', '')}"
+
+description = project(playgroundJobServerProject).description + " :: Container"
+
+configurations {
+ dockerDependency
+}
+
+dependencies {
+ dockerDependency project(path: playgroundJobServerProject, configuration: "shadow")
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ copy {
+ from 'entrypoint.sh'
+ into 'build/'
+ }
+ copy {
+ from 'config.yaml'
+ into 'build/'
+ }
+ copy {
+ from '../../../infrastructure/proxy/allow_list.py'
+ into 'build/'
+ }
+ copy {
+ from '../../../infrastructure/proxy/allow_list_proxy.py'
+ into 'build/'
+ }
Review Comment:
Again, why can't we use https://docs.docker.com/engine/reference/builder/#copy in the Dockerfile? Why does this need to happen in gradle? Isn't gradle designed for building code and automating test execution? Why are we coupling gradle with deployment operations when we have these kinds of tasks as part of the docker ecosystem?
##########
playground/infrastructure/helm-playground/templates/deployment-mitmproxy.yaml:
##########
@@ -0,0 +1,39 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: playground-mitm
+ labels:
+ app: backend-mitm
+ type: backend
+spec:
+ template:
+ metadata:
+ name: playground
+ labels:
+ app: backend-mitm
+ type: backend
+ spec:
+ containers:
+ - name: playground-backend-mitm
+ image: "{{ .Values.registry}}/{{ .Values.image.mitm_image }}:{{ .Values.tag }}"
+ imagePullPolicy: {{ .Values.image.pullPolicy }}
+ replicas: {{ .Values.replicaCount }}
+ selector:
+ matchLabels:
+ app: backend-mitm
Review Comment:
Ahain, what does this achieve?
##########
playground/docker-compose.local.yaml:
##########
@@ -87,10 +101,13 @@ services:
CACHE_TYPE: remote
CACHE_ADDRESS: redis:6379
SERVER_PORT: 8090
+ PLAYGROUND_MITM_SERVICE_HOST: mitmproxy
+ PLAYGROUND_MITM_SERVICE_PORT: 8080
ports:
- "8090:8090"
depends_on:
- redis
+ - mitmproxy
frontend:
image: apache/beam_playground-frontend
Review Comment:
Why do we have to maintain both docker compose and kubernetes manifests? What if there is drift from these two artifacts that would lead to errors in production? Why can't we have a single source of truth and someone just setup minikube on their local workstation and deploy the kubernetes manifests instead of docker compose?
##########
playground/infrastructure/helm-playground/templates/network-policy.yaml:
##########
@@ -0,0 +1,66 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+ name: egress-deny
+spec:
+ podSelector:
+ matchExpressions:
+ - key: app
+ operator: In
+ values:
+ - backend-go
+ - backend-java
+ - backend-python
+ - backend-scio
+ policyTypes:
+ - Egress
+ egress:
+ - to:
+ - namespaceSelector:
+ matchLabels:
+ kubernetes.io/metadata.name: kube-system
+ podSelector:
+ matchLabels:
+ k8s-app: kube-dns
+ ports:
+ - port: 53
+ protocol: UDP
+ - port: 53
+ protocol: TCP
+ - to:
+ - ipBlock:
+ cidr: 10.0.0.0/8
+ ports:
+ - protocol: TCP
+ port: 6379
+ - protocol: TCP
+ port: 8080
+ - to:
+ - ipBlock:
+ cidr: 169.254.169.252/32
+ ports:
+ - protocol: TCP
+ port: 988
+ - to:
+ - ipBlock:
+ cidr: 169.254.169.254/32
+ ports:
+ - protocol: TCP
+ port: 80
Review Comment:
Why is this needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] olehborysevych commented on a diff in pull request #25508: [Playground] Playground network policy+MiTM proxy
Posted by "olehborysevych (via GitHub)" <gi...@apache.org>.
olehborysevych commented on code in PR #25508:
URL: https://github.com/apache/beam/pull/25508#discussion_r1111677587
##########
playground/infrastructure/helm-playground/templates/network-policy.yaml:
##########
@@ -0,0 +1,66 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+ name: egress-deny
+spec:
+ podSelector:
+ matchExpressions:
+ - key: app
+ operator: In
+ values:
+ - backend-go
Review Comment:
@MakarkinSAkvelon will this work for grpc-router for ToB?
##########
playground/backend/containers/go/Dockerfile:
##########
@@ -58,17 +58,6 @@ ENV PROPERTY_PATH=/opt/playground/backend/properties.yaml
## Copy build result
COPY src/configs /opt/playground/backend/configs/
-# Install mitmpoxy
-RUN mkdir /opt/mitmproxy &&\
- cd /opt/mitmproxy &&\
- wget -q https://snapshots.mitmproxy.org/7.0.4/mitmproxy-7.0.4-linux.tar.gz &&\
- tar -zxvf mitmproxy-7.0.4-linux.tar.gz &&\
- mkdir /usr/local/share/ca-certificates/extra
-COPY allow_list_proxy.py /opt/mitmproxy/
Review Comment:
@MakarkinSAkvelon since we don't copy those files into containers we need also to remove preparing them in gradle task for all containers.
##########
playground/infrastructure/helm-playground/templates/network-policy.yaml:
##########
@@ -0,0 +1,66 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+ name: egress-deny
+spec:
+ podSelector:
+ matchExpressions:
+ - key: app
+ operator: In
+ values:
+ - backend-go
+ - backend-java
+ - backend-python
+ - backend-scio
+ policyTypes:
+ - Egress
+ egress:
+ - to:
+ - namespaceSelector:
+ matchLabels:
+ kubernetes.io/metadata.name: kube-system
+ podSelector:
+ matchLabels:
+ k8s-app: kube-dns
+ ports:
+ - port: 53
+ protocol: UDP
+ - port: 53
+ protocol: TCP
+ - to:
+ - ipBlock:
+ cidr: 10.0.0.0/8
+ ports:
+ - protocol: TCP
+ port: 6379
+ - protocol: TCP
+ port: 8080
+ - to:
+ - ipBlock:
+ cidr: 169.254.169.252/32
+ ports:
+ - protocol: TCP
+ port: 988
+ - to:
+ - ipBlock:
+ cidr: 169.254.169.254/32
Review Comment:
what is this IP address? should it be hardcoded? @MakarkinSAkvelon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org