You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2023/01/10 13:47:21 UTC

[GitHub] [beam] MakarkinSAkvelon opened a new pull request, #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

MakarkinSAkvelon opened a new pull request, #24958:
URL: https://github.com/apache/beam/pull/24958

   PR request will fix followig things:
   Datastore namespace - allow to change Datastore namespace from command line
   Helm deployment - fix resourse limit
   Ingress timeout - fix SCIO examples upload issue
   REDIS type - change from Standard to Basic
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #24958:
URL: https://github.com/apache/beam/pull/24958#discussion_r1088744965


##########
playground/terraform/infrastructure/variables.tf:
##########
@@ -94,12 +94,12 @@ variable "redis_name" {
 
 variable "redis_tier" {
   description = "Tier of Redis"
-  default     = "STANDARD_HA"

Review Comment:
   Hello, @damondouglas. Even on current Production we're using BASIC Redis version and it's enough



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #24958:
URL: https://github.com/apache/beam/pull/24958#discussion_r1142299726


##########
playground/infrastructure/helm-playground/templates/autoscaling-go.yaml:
##########
@@ -12,15 +12,27 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-apiVersion: autoscaling/v1
+apiVersion: autoscaling/v2
 kind: HorizontalPodAutoscaler
 metadata:
   name: playground-go
 spec:
-  maxReplicas: 10
-  minReplicas: 1
+  maxReplicas: 4

Review Comment:
   Helm variables were created



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] rshamunov commented on a diff in pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #24958:
URL: https://github.com/apache/beam/pull/24958#discussion_r1090418434


##########
playground/terraform/infrastructure/variables.tf:
##########
@@ -94,12 +94,12 @@ variable "redis_name" {
 
 variable "redis_tier" {
   description = "Tier of Redis"
-  default     = "STANDARD_HA"

Review Comment:
   We don't downgrade. Currently we use Basic Redis instance. According to monitoring, we don't utilize it enough to switch to Standard instance.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] rshamunov commented on a diff in pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #24958:
URL: https://github.com/apache/beam/pull/24958#discussion_r1104103312


##########
playground/terraform/infrastructure/variables.tf:
##########
@@ -94,12 +94,12 @@ variable "redis_name" {
 
 variable "redis_tier" {
   description = "Tier of Redis"
-  default     = "STANDARD_HA"

Review Comment:
   @damondouglas, currently Playground use Redis for tasks that don't utilize even Basic tier capacity. We are not at risk of performance issues, even if usage of Playground environment grows significantly. Besides, we can reconfigure Playground to use new Redis instance if we need.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] olehborysevych commented on pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

Posted by GitBox <gi...@apache.org>.
olehborysevych commented on PR #24958:
URL: https://github.com/apache/beam/pull/24958#issuecomment-1381565858

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] volatilemolotov commented on a diff in pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on code in PR #24958:
URL: https://github.com/apache/beam/pull/24958#discussion_r1138849800


##########
playground/infrastructure/helm-playground/templates/autoscaling-go.yaml:
##########
@@ -12,15 +12,27 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-apiVersion: autoscaling/v1
+apiVersion: autoscaling/v2
 kind: HorizontalPodAutoscaler
 metadata:
   name: playground-go
 spec:
-  maxReplicas: 10
-  minReplicas: 1
+  maxReplicas: 4

Review Comment:
   This should be configurable and default should be provided via value file



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] MakarkinSAkvelon commented on pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on PR #24958:
URL: https://github.com/apache/beam/pull/24958#issuecomment-1457907501

   It's a known issue with was fixed before (but after the current PR was raised) so now rebase was completed, please deploy it one more time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] rshamunov commented on pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on PR #24958:
URL: https://github.com/apache/beam/pull/24958#issuecomment-1495615441

   Replaced by #25994


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damondouglas commented on a diff in pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

Posted by "damondouglas (via GitHub)" <gi...@apache.org>.
damondouglas commented on code in PR #24958:
URL: https://github.com/apache/beam/pull/24958#discussion_r1100451527


##########
playground/terraform/infrastructure/variables.tf:
##########
@@ -94,12 +94,12 @@ variable "redis_name" {
 
 variable "redis_tier" {
   description = "Tier of Redis"
-  default     = "STANDARD_HA"

Review Comment:
   @rshamunov What if the usage grows? What is the plan to monitor and respond to capacity issues?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] MakarkinSAkvelon commented on pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on PR #24958:
URL: https://github.com/apache/beam/pull/24958#issuecomment-1383842913

   R @damondouglas 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] MakarkinSAkvelon commented on pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on PR #24958:
URL: https://github.com/apache/beam/pull/24958#issuecomment-1451782040

   Hello, @damondouglas. The only reason why we could get such kind of an issue is just to stop the script during execution that  *.tfstate file didn't contain all the required variables (path to *.tfstate file - "beam/playground/terraform/.terraform") 
   Could you try the following:
   1. Remove "beam/playground/terraform/.terraform" folder
   2. Ensure that "beam/playground/terraform/$environment/terraform.tfvars" contain variables according to Readme.md from the PR
   3. Start the deployment script
   
   Feel free to ping me in any chats (slack or google chat) in case of any issues


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

Posted by "MakarkinSAkvelon (via GitHub)" <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #24958:
URL: https://github.com/apache/beam/pull/24958#discussion_r1093228647


##########
playground/terraform/build.gradle.kts:
##########
@@ -392,13 +392,17 @@ tasks.register("takeConfig") {
    var ipaddrname = ""
    var d_tag = ""
    var dns_name = ""
+   var dstore_name = ""

Review Comment:
   Variable name was fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damondouglas commented on a diff in pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

Posted by "damondouglas (via GitHub)" <gi...@apache.org>.
damondouglas commented on code in PR #24958:
URL: https://github.com/apache/beam/pull/24958#discussion_r1088353872


##########
playground/terraform/infrastructure/variables.tf:
##########
@@ -94,12 +94,12 @@ variable "redis_name" {
 
 variable "redis_tier" {
   description = "Tier of Redis"
-  default     = "STANDARD_HA"

Review Comment:
   Why are we downgrading?



##########
playground/terraform/build.gradle.kts:
##########
@@ -392,13 +392,17 @@ tasks.register("takeConfig") {
    var ipaddrname = ""
    var d_tag = ""
    var dns_name = ""
+   var dstore_name = ""

Review Comment:
   Should we spell out the full `datastore_name`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] rshamunov commented on a diff in pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #24958:
URL: https://github.com/apache/beam/pull/24958#discussion_r1105909185


##########
playground/terraform/README.md:
##########
@@ -116,9 +116,9 @@ gcloud container clusters get-credentials --region `chosen_location` `gke_name`
 ```
 Start the following command from the top level repository folder ("beam") to deploy the Payground infrastructure:
 ```
-./gradlew playground:terraform:gkebackend -Pproject_environment="environment_name" -Pdocker-tag="tag" -Pdns-name="playground.zone" -Psdk-tag=2.43.0
+./gradlew playground:terraform:gkebackend -Pproject_environment="environment_name" -Pdocker-tag="tag" -Pdns-name="playground.zone" -Psdk-tag=2.43.0 -Pdatastore-namespace="dev"
 ```
-Where tag - image tag for backend, playground.zone - chosen DNS for Playground, Psdk-tag - current BEAM version
+Where tag - image tag for backend, playground.zone - chosen DNS for Playground, Psdk-tag - current BEAM version, Pdstore-namespace - namespace for Datastore

Review Comment:
   -Pdatastore_namespace ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] rshamunov closed pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes

Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov closed pull request #24958: [Playground] Datastore namespace, Helm deployment, Ingress timeout, REDIS type fixes
URL: https://github.com/apache/beam/pull/24958


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org