You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/12/23 00:22:14 UTC
[GitHub] [beam] pabloem commented on a change in pull request #16285: [BEAM-13492][Playground] Update backend README file
pabloem commented on a change in pull request #16285:
URL: https://github.com/apache/beam/pull/16285#discussion_r774242474
##########
File path: playground/backend/README.md
##########
@@ -21,47 +21,81 @@
## About
-Apache Beam is an open-source, unified model for defining parallel processing pipelines for batch and streaming data.
-It provides a portable API layer for building sophisticated data-parallel processing pipelines that may be executed
-across a diversity of execution engines, or runners.
-
-Beam Playground helps facilitate trying out and adopting Apache Beam by providing a very quick way for prospective Beam
-users to see and run examples of Apache Beam pipelines in a web interface that requires no setup.
+Beam Playground helps facilitate trying out and adopting [Apache Beam](https://beam.apache.org/) by providing a very
+quick way for prospective Beam users to see and run examples of Apache Beam pipelines in a web interface that requires
+no setup.
## Getting Started
See [playground/README.md](../README.md) for details on requirements and setup.
This section describes what is needed to run the backend application.
-- Generating models from proto file
-- Go commands to run/test application locally
+
+- Go commands to run/test the backend locally
+- Set up environment variables to run the backend locally
+- Running the backend via Docker
### Go commands to run/test application locally
-The following command is used to build and serve the backend locally:
+Go to the backend directory:
+```shell
+$ cd backend
```
-go run ./cmd/server/server.go
+
+The following command is used to build and serve the backend locally:
+
+```shell
+$ go run ./cmd/server/server.go
```
Run the following command to generate a release build file:
-```
-go build ./cmd/server/server.go
+```shell
+$ go build ./cmd/server/server.go
```
Playground tests may be run using this command:
-```
-go test ... -v
+```shell
+$ go test ... -v
```
The full list of commands can be found [here](https://pkg.go.dev/cmd/go).
-## Running the server app
+### Set up environment variables to run the backend locally
+
+These environment variables should be set to run the backend locally:
-To run the server using Docker images there are `Docker` files in the `containers` folder for Go and Java languages.
-Each of them processes the corresponding SDK, so the backend with Go SDK will work with Go examples/katas/tests only.
+- `BEAM_SDK` - is the SDK which backend could process (`SDK_GO` / `SDK_JAVA` / `SDK_PYTHON`)
+- `APP_WORK_DIR` - is the directory where all folders will be placed to process each code processing request
+- `PREPARED_MOD_DIR` - is the directory where prepared go.mod and go.sum files are placed. It is used only for Go SDK
+
+There are also environment variables which are needed for the deployment of Apache Beam Playground. These variables have
+default value and there is no need to set them up to launch locally:
+
+- `SERVER_IP` - is the IP address of the backend server (default value = `localhost`)
+- `SERVER_PORT` - is the PORT of the backend server (default value = `8080`)
+- `CACHE_TYPE` - is a type of the cache service which is used for the backend server. If it is set as a `remote`, then
+ the backend server will use Redis to keep all cache values (default value = `local`)
+- `CACHE_ADDRESS` - is an address of the Redis server. It is used only when `CACHE_TYPE=remote` (default value
+ = `localhost:6379`)
+- `BEAM_PATH` - it is the place where all required for the Java SDK libs are placed
Review comment:
This one would be necessary if we're running Java SDK - even if we run locally, right? Can we move these to their own section? They're mandatory per SDK, right?
##########
File path: playground/backend/README.md
##########
@@ -21,47 +21,81 @@
## About
-Apache Beam is an open-source, unified model for defining parallel processing pipelines for batch and streaming data.
-It provides a portable API layer for building sophisticated data-parallel processing pipelines that may be executed
-across a diversity of execution engines, or runners.
-
-Beam Playground helps facilitate trying out and adopting Apache Beam by providing a very quick way for prospective Beam
-users to see and run examples of Apache Beam pipelines in a web interface that requires no setup.
+Beam Playground helps facilitate trying out and adopting [Apache Beam](https://beam.apache.org/) by providing a very
+quick way for prospective Beam users to see and run examples of Apache Beam pipelines in a web interface that requires
+no setup.
## Getting Started
See [playground/README.md](../README.md) for details on requirements and setup.
This section describes what is needed to run the backend application.
-- Generating models from proto file
-- Go commands to run/test application locally
+
+- Go commands to run/test the backend locally
+- Set up environment variables to run the backend locally
+- Running the backend via Docker
### Go commands to run/test application locally
-The following command is used to build and serve the backend locally:
+Go to the backend directory:
+```shell
+$ cd backend
```
-go run ./cmd/server/server.go
+
+The following command is used to build and serve the backend locally:
+
+```shell
+$ go run ./cmd/server/server.go
```
Run the following command to generate a release build file:
-```
-go build ./cmd/server/server.go
+```shell
+$ go build ./cmd/server/server.go
```
Playground tests may be run using this command:
-```
-go test ... -v
+```shell
+$ go test ... -v
```
The full list of commands can be found [here](https://pkg.go.dev/cmd/go).
-## Running the server app
+### Set up environment variables to run the backend locally
+
+These environment variables should be set to run the backend locally:
-To run the server using Docker images there are `Docker` files in the `containers` folder for Go and Java languages.
-Each of them processes the corresponding SDK, so the backend with Go SDK will work with Go examples/katas/tests only.
+- `BEAM_SDK` - is the SDK which backend could process (`SDK_GO` / `SDK_JAVA` / `SDK_PYTHON`)
Review comment:
can we allow the backend to support all SDKs? - we can start with a per-language deployment, but we may one day discover that we want a single backend - so I think it would be great to have a parameter to allow this (e.g. `SDK_ALL`).
Also, should Scio be added?
##########
File path: playground/backend/README.md
##########
@@ -21,47 +21,81 @@
## About
-Apache Beam is an open-source, unified model for defining parallel processing pipelines for batch and streaming data.
-It provides a portable API layer for building sophisticated data-parallel processing pipelines that may be executed
-across a diversity of execution engines, or runners.
-
-Beam Playground helps facilitate trying out and adopting Apache Beam by providing a very quick way for prospective Beam
-users to see and run examples of Apache Beam pipelines in a web interface that requires no setup.
+Beam Playground helps facilitate trying out and adopting [Apache Beam](https://beam.apache.org/) by providing a very
+quick way for prospective Beam users to see and run examples of Apache Beam pipelines in a web interface that requires
+no setup.
## Getting Started
See [playground/README.md](../README.md) for details on requirements and setup.
This section describes what is needed to run the backend application.
-- Generating models from proto file
-- Go commands to run/test application locally
+
+- Go commands to run/test the backend locally
+- Set up environment variables to run the backend locally
+- Running the backend via Docker
### Go commands to run/test application locally
-The following command is used to build and serve the backend locally:
+Go to the backend directory:
+```shell
+$ cd backend
```
-go run ./cmd/server/server.go
+
+The following command is used to build and serve the backend locally:
+
+```shell
+$ go run ./cmd/server/server.go
```
Run the following command to generate a release build file:
-```
-go build ./cmd/server/server.go
+```shell
+$ go build ./cmd/server/server.go
```
Playground tests may be run using this command:
-```
-go test ... -v
+```shell
+$ go test ... -v
```
The full list of commands can be found [here](https://pkg.go.dev/cmd/go).
-## Running the server app
+### Set up environment variables to run the backend locally
+
+These environment variables should be set to run the backend locally:
-To run the server using Docker images there are `Docker` files in the `containers` folder for Go and Java languages.
-Each of them processes the corresponding SDK, so the backend with Go SDK will work with Go examples/katas/tests only.
+- `BEAM_SDK` - is the SDK which backend could process (`SDK_GO` / `SDK_JAVA` / `SDK_PYTHON`)
+- `APP_WORK_DIR` - is the directory where all folders will be placed to process each code processing request
+- `PREPARED_MOD_DIR` - is the directory where prepared go.mod and go.sum files are placed. It is used only for Go SDK
+
+There are also environment variables which are needed for the deployment of Apache Beam Playground. These variables have
+default value and there is no need to set them up to launch locally:
+
+- `SERVER_IP` - is the IP address of the backend server (default value = `localhost`)
+- `SERVER_PORT` - is the PORT of the backend server (default value = `8080`)
+- `CACHE_TYPE` - is a type of the cache service which is used for the backend server. If it is set as a `remote`, then
+ the backend server will use Redis to keep all cache values (default value = `local`)
+- `CACHE_ADDRESS` - is an address of the Redis server. It is used only when `CACHE_TYPE=remote` (default value
+ = `localhost:6379`)
+- `BEAM_PATH` - it is the place where all required for the Java SDK libs are placed
+ (default value = `/opt/apache/beam/jars/*`)
+- `KEY_EXPIRATION_TIME` - is the expiration time of the keys in the cache (default value = `15 min`)
+- `PIPELINE_EXPIRATION_TIMEOUT` - is the expiration time of the code processing (default value = `15 min`)
+- `PROTOCOL_TYPE` - is the type of the backend server protocol. It could be `TCP` or `HTTP` (default value = `HTTP`)
+- `NUM_PARALLEL_JOBS` - is the max number of the code processing requests which could be processed on the backend server
+ at the same time (default value = `20`). This value is used to check readiness of the backend server. If server
+ processes max number of the code processing requests then need to deploy a new instance.
Review comment:
```suggestion
reaches the max number of concurrent code-processing requests, then it will reject
all other incoming requests with error code XXX.
```
What's the error code? : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org