You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by pa...@apache.org on 2023/06/05 18:11:06 UTC

[beam] branch master updated: Overview of adding snippets to Playground (#25506) (#25507)

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
     new 7af89948ca2 Overview of adding snippets to Playground (#25506) (#25507)
7af89948ca2 is described below

commit 7af89948ca2120b183a26f0551f387777dbee23e
Author: alexeyinkin <le...@inkin.ru>
AuthorDate: Mon Jun 5 22:10:58 2023 +0400

    Overview of adding snippets to Playground (#25506) (#25507)
    
    * Overview of adding snippets to Playground (#25506)
    
    * Minor README changes (#25506)
    
    * Minor README changes (#25506)
    
    * List scanned directories, update example paths (#25506)
    
    * Edit after review answers (#25506)
    
    * Format comments (#25506)
    
    * Clean up (#25506)
    
    * Remove mentions of GCS support for datasets
    
    * Add documentation on how to add examples which use Kafka emulator
    
    * Add `location` field to tag example
    
    * Add license to get_your_code.md document
    
    * Revised iteration, it still is work in progress
    
    * Whitespace and other fixes
    
    * Improve Markdown formatting, fix internal links, fix typos (#25506)
    
    * Changes after internal review (#25506)
    
    * Replace the metadata table with a full YAML example (#25506)
    
    * Update the workflow diagram and the howto for adding an unlisted example (#25506)
    
    * Remove "GCS" variant from DatasetLocation as it's unsupported
    
    * Data access limitations changed
    
    * Note that context_line and categories are optional in example metadata (#25506)
    
    * Update try-beam-playground page with a link to the load_your_code doc (#25506)
    
    * Navigate to load_your_code.md on buttons in Playground (#25506)
    
    * Review and udpates
    
    * Update the diagram (#25506)
    
    * Link to ToB docs, add example URLs (#25506)
    
    * Clean up (#25506)
    
    * Fix links (#25506)
    
    * Playground doc update (#466)
    
    * Add Playground update guide to release-guide.md
    
    * Add SCHEMA.md with datastore schema description
    
    * Add information on when to implement data schema migrations and how
    
    * Add a note about indexes
    
    * Add commented out lines for local backend to backend_urls.dart
    
    * Use default version for python container
    
    * New Readme for playground
    
    * Update release-guide
    
    * Whitespace fix
    
    * Remove TODO related to Python sDK image build
    
    * Clarify TODOs
    
    * Add clarification on Redis usage
    
    * Document dependencies update process
    
    * Add script to fetch SCIO examples
    
    * Add a note on SCIO examples to load_your_code.md
    
    ---------
    
    Co-authored-by: Timur Sultanov <ti...@akvelon.com>
    
    * HOWTO for Flutter upgrade (#25506)
    
    * Clarify what to update in build.gradle for Go runner
    
    * Fix argument name
    
    * Update backend project structure
    
    * Add secription of directories used by the Playground project
    
    * Add link to manual Cloudbuild setup guide
    
    * README file update
    
    * README - HELM steps were added
    
    ---------
    
    Co-authored-by: Timur Sultanov <ti...@akvelon.com>
    Co-authored-by: Alex Kosolapov <al...@akvelon.com>
    Co-authored-by: Rouslan <11...@users.noreply.github.com>
    Co-authored-by: Sergey Makarkin <Se...@akvelon.com>
---
 playground/README.md                               | 405 +++++++-------
 playground/TASKS.md                                | 272 ++++++++++
 playground/backend/CONTRIBUTE.md                   |  65 ++-
 playground/backend/SCHEMA.md                       | 185 +++++++
 playground/backend/containers/python/build.gradle  |   4 +-
 playground/backend/containers/scio/build.gradle    |   2 +-
 .../doc/load_your_code/images/share-my-code.png    | Bin 0 -> 29370 bytes
 playground/doc/load_your_code/images/workflow.png  | Bin 0 -> 58871 bytes
 playground/frontend/lib/constants/links.dart       |  19 -
 .../modules/actions/components/new_example.dart    |   2 +-
 .../examples/examples_dropdown_content.dart        |   3 +-
 .../lib/src/constants/backend_urls.dart            |   8 +
 .../lib/src/constants/links.dart                   |   1 +
 playground/infrastructure/models.py                |  94 +++-
 playground/infrastructure/requirements.txt         |   2 +-
 playground/load_your_code.md                       | 600 +++++++++++++++++++++
 playground/terraform/README.md                     |  94 ++++
 .../site/content/en/contribute/release-guide.md    |   4 +
 .../content/en/get-started/try-beam-playground.md  |  73 +--
 19 files changed, 1495 insertions(+), 338 deletions(-)

diff --git a/playground/README.md b/playground/README.md
index 6d4c153fcab..6f69a59d055 100644
--- a/playground/README.md
+++ b/playground/README.md
@@ -17,214 +17,211 @@
     under the License.
 -->
 
-# Playground
-
 The Beam Playground is a web application to run Beam code snippets in a modern browser. This directory holds code to
 build, test, and deploy the frontend and backend services.
 
-# Development Requirements
-
-The following requirements are needed for development, testing, and deploying.
-
-- [go 1.18+](https://golang.org)
-- [flutter](https://flutter.dev/)
-- Go protobuf dependencies (See [Go gRPC Quickstart](https://grpc.io/docs/languages/go/quickstart/))
-- Dart protobuf dependencies (See [Dart gRPC Quickstart](https://grpc.io/docs/languages/dart/))
-- [buf](https://docs.buf.build/installation)
-- [Docker](https://docs.docker.com/desktop/)
-- [Docker Compose](https://docs.docker.com/compose/install/)
-- [gcloud CLI](https://cloud.google.com/sdk/docs/install)
-- [gcloud Beta Commands](https://cloud.google.com/sdk/gcloud/reference/components/install)
-- [Cloud Datastore Emulator](https://cloud.google.com/datastore/docs/tools/datastore-emulator)
-- [sbt](https://www.scala-sbt.org/1.x/docs/Installing-sbt-on-Linux.html)
-
-### Google Cloud Shell Prerequisites Installation
-Google Cloud Shell already has most of the prerequisites installed. Only few tools need to be installed separately
-
-#### Flutter
+- [Setup development prerequisites](#setup-development-prerequisites)
+- [Common tasks](#common-tasks)
+- [Run Beam Playground locally](#run-beam-playground-locally)
+  - [Configure frontend to use local backend](#configure-frontend-to-use-local-backend)
+  - [Run local deployment using Gradle task](#run-local-deployment-using-gradle-task)
+  - [Deploy examples](#deploy-examples)
+- [How to add your own example](#how-to-add-your-own-example)
+- [Deployment guide](#deployment-guide)
+  - [Manual deployment](#manual-deployment)
+  - [Manual CloudBuild setup](#manual-cloudbuild-setup)
+- [Project structure](#project-structure)
+- [Contribution guide](#contribution-guide)
+
+# Setup development prerequisites
+> ***Google Cloud Shell note***: Google Cloud Shell already has most of the prerequisites installed. Only the following has to be installed manually:
+> - Flutter - run `flutter precache`
+> - Go protobuf dependencies
+> - Dart protobuf dependencies
+> - buf
+> - sbt
+
+1. Install Go 1.18+
+
+    **Ubuntu 22.04 and newer:**
+    ```shell
+    sudo apt install golang`
+    ```
+
+    **Other Linux variants:** Follow manual at https://go.dev/doc/install
+1. Install [flutter](https://docs.flutter.dev/get-started/install/linux)
+
+    **Ubuntu 22.04 or newer:**
+    ```shell
+    sudo apt install flutter
+    ```
+
+    **Other Linux variants:** Follow manual at https://flutter.dev/docs/get-started/install/linux
+
+1. Install [protoc](https://grpc.io/docs/protoc-installation/)
+
+    **Ubuntu 22.04 or newer/Debian 11 or newer:**
+    ```shell
+    sudo apt install protobuf-compiler
+    ```
+    **Other Linux variants:** Follow manual at https://grpc.io/docs/protoc-installation/
+
+1. Install Go protobuf dependencies: [Go gRPC Quickstart](https://grpc.io/docs/languages/go/quickstart/)
+    > Do not forget to update your `PATH` environment variable
+1. Install Dart protobuf dependencies: [Dart gRPC Quickstart](https://grpc.io/docs/languages/dart/quickstart/)
+    > Do not forget to update your `PATH` environment variable
+1. Install npm
+    **Ubuntu 22.04 and newer/Debian 11 and newer:**
+    ```shell
+    sudo apt install npm
+    ```
+    **Other Linux variants:** Follow manual at https://docs.npmjs.com/downloading-and-installing-node-js-and-npm
+1. Install [buf](https://docs.buf.build/installation)
+    ```shell
+    npm install -g @bufbuild/buf
+    ```
+1. Install Docker
+    **Ubuntu 22.04 and newer/Debian 11 and newer:**
+    ```shell
+    sudo apt install docker.io
+    ```
+    **Other Linux variants:** Follow manual at https://docs.docker.com/engine/install/
+
+    To verify your docker installation, run:
+    ```shell
+    docker ps
+    ```
+    It should finish without any errors. If you get a permission denied error, you need to add your user to the docker group:
+    ```shell
+    sudo usermod -aG docker $USER
+    ```
+    Then, log out and log back in to apply the changes.
+1. Install Docker Compose
+    **Ubuntu 22.04 and newer/Debian 11 and newer:**
+    ```shell
+    sudo apt install docker-compose
+    ```
+    **Other Linux variants:** Follow manual at https://docs.docker.com/compose/install/
+1. Install gcloud CLI by following the [manual](https://cloud.google.com/sdk/docs/install) for your system
+1. Install Cloud Datastore Emulator
+    ```shell
+    gcloud components install cloud-datastore-emulator
+    ```
+    or, if you have install gcloud CLI using APT:
+    ```shell
+    sudo apt install google-cloud-cli-datastore-emulator
+    ```
+1. Install `sbt` following instruction at https://www.scala-sbt.org/1.x/docs/Installing-sbt-on-Linux.html
+    > **Optional**: Run `sbt` comamnd once after installation to let it cache its dependencies
+1. Make sure that you have the full installation of Python 3.8 or newer. On Debian or Ubuntu it can be installed using `sudo apt install python3-full`.
+
+# Common tasks
+To get an overview of common tasks, see [TASKS.md](TASKS.md)
+
+# Run Beam Playground locally
+## Configure frontend to use local backend
+> **Note:** Follow this step only if you want to have a local deployment of Playground. Skip this step entirely if you want to deploy Playground to Google Cloud.
+
+Uncommend lines after `// Uncomment the following lines to use local backend.` in [frontend/playground_components/lib/src/constants/backend_urls.dart](/playground/frontend/playground_components/lib/src/constants/backend_urls.dart)
+
+## Run local deployment using Gradle task
+> For more information read the corresponding section in [TASKS.md](./TASKS.md#router-runners-and-frontend)
+
+Run the deployment script:
 ```shell
-git config --global --add safe.directory /google/flutter
-flutter doctor
-```
-
-#### Protobuf
-```shell
-go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28
-go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2
-dart pub global activate protoc_plugin
-npm install -g @bufbuild/buf
-```
-
-#### sbt
-```shell
-echo "deb https://repo.scala-sbt.org/scalasbt/debian all main" | sudo tee /etc/apt/sources.list.d/sbt.list
-echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | sudo tee /etc/apt/sources.list.d/sbt_old.list
-curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | sudo -H gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/scalasbt-release.gpg --import
-sudo chmod 644 /etc/apt/trusted.gpg.d/scalasbt-release.gpg
-sudo apt-get update
-sudo apt-get install sbt
-```
-
-# Available Gradle Tasks
-
-## Perform overall pre-commit checks
-
-```
-cd beam
-./gradlew playgroundPrecommit
-```
-
-## To see available gradle tasks for playground:
-
-```
-cd beam
-./gradlew playground:tasks
-```
-
-## Re-generate protobuf
-
-```
-cd beam
-./gradlew playground:generateProto
-```
-
-## Run local environment using docker compose
-
-### Router only
-
-Start:
-
-```bash
-cd beam
-./gradlew playground:backend:containers:router:dockerComposeLocalUp
-```
-
-Stop:
-
-```bash
-cd beam
-./gradlew playground:backend:containers:router:dockerComposeLocalDown
-```
-
-### Router, runners, and frontend
-
-1. Edit `/playground/frontend/playground_components/lib/src/constants/backend_urls.dart`
-to override backend URLs with yours found in `/playground/docker-compose.local.yaml`.
-2. To start, run:
-
-```bash
-cd beam
-./gradlew playground:dockerComposeLocalUp
-```
-
-3. To stop, run:
-
-```bash
-cd beam
-./gradlew playground:dockerComposeLocalDown
-```
-
-This way of running may not work in all environments because it is not maintained.
-It is used occasionally by the Frontend team to test complex tasks against
-a not-yet-deployed backend.
-The full start may take ~30 minutes and is demanding, so you should likely enable
-only one backend runner for the SDK you need.
-
-If you do not need particular runners, comment out:
-1. Dependencies on them in `/playground/build.gradle.kts` in `dockerComposeLocalUp` task.
-2. Their Docker image configurations in `/playground/docker-compose.local.yaml`.
-
-See also [Backend Lookup](frontend/README.md#backend-lookup) in the Frontend.
-
-## Removing old snippets
-
-Run the method to remove unused code snippets from the Cloud Datastore. Unused snippets are snippets that are out of date. If the last visited date property less or equals than the current date minus dayDiff parameter then a snippet is out of date
-
-```
-cd beam
-./gradlew playground:backend:removeUnusedSnippet -DdayDiff={int} -DprojectId={string} -Dnamespace={datastore namespace}
-```
-
-## Removing a specific snippet
-
-Run the method to remove a specific code snippet from the Cloud Datastore.
-
+./gradlew :playground:dockerComposeLocalUp
 ```
-cd beam
-./gradlew playground:backend:removeSnippet -DsnippetId={string} -DprojectId={string} -Dnamespace={datastore namespace}
-```
-
-## Run playground tests without cache
-
-```
-cd beam
- ./gradlew playground:backend:testWithoutCache
-```
-
-# Deployment
-
-See [terraform](./terraform/README.md) for details on how to build and deploy
-the application and its dependent infrastructure.
-
-# Manual Example deployment
-
-The following requirements are needed for deploying examples manually:
-
-1. GCP project with deployed Playground backend
-2. Python (3.9.x)
-3. Login into GCP (gcloud default login or using service account key)
-
-## Run example deployment script
-Example deployment scripts uses following environment variables:
-
-- GOOGLE_CLOUD_PROJECT    - GCP project id where Playground backend is deployed
-- BEAM_ROOT_DIR           - root folder to search for playground examples
-- SDK_CONFIG              - location of sdk and default example configuration file
-- BEAM_EXAMPLE_CATEGORIES - location of example category configuration file
-- BEAM_USE_WEBGRPC        - use grpc-Web instead of grpc (default)
-- GRPC_TIMEOUT            - timeout for grpc calls (defaults to 10 sec)
-- BEAM_CONCURRENCY        - number of eaxmples to run in parallel (defaults to 10)
-- SERVER_ADDRESS          - address of the backend runnner service for a particular SDK
-
-```
-usage: ci_cd.py [-h] --step {CI,CD} [--namespace NAMESPACE] --datastore-project DATASTORE_PROJECT --sdk {SDK_JAVA,SDK_GO,SDK_PYTHON,SDK_SCIO} --origin {PG_EXAMPLES,TB_EXAMPLES} --subdirs SUBDIRS [SUBDIRS ...]
-
-CI/CD Steps for Playground objects
-
-optional arguments:
-  -h, --help            show this help message and exit
-  --step {CI,CD}        CI step to verify all beam examples/tests/katas. CD step to save all beam examples/tests/katas and their outputs on the GCD
-  --namespace NAMESPACE
-                        Datastore namespace to use when saving data (default: Playground)
-  --datastore-project DATASTORE_PROJECT
-                        Datastore project to use when saving data
-  --sdk {SDK_JAVA,SDK_GO,SDK_PYTHON,SDK_SCIO}
-                        Supported SDKs
-  --origin {PG_EXAMPLES,TB_EXAMPLES}
-                        ORIGIN field of pg_examples/pg_snippets
-  --subdirs SUBDIRS [SUBDIRS ...]
-                        limit sub directories to walk through, relative to BEAM_ROOT_DIR
-```
-
-Helper script to deploy examples for all supported sdk's:
 
+To shut down the playground, run:
 ```shell
-cd playground/infrastructure
-
-export BEAM_ROOT_DIR="../../"
-export SDK_CONFIG="../../playground/sdks.yaml"
-export BEAM_EXAMPLE_CATEGORIES="../categories.yaml"
-export BEAM_USE_WEBGRPC=yes
-export BEAM_CONCURRENCY=4
-export PLAYGROUND_DNS_NAME="your registered dns name for Playground"
-
-for sdk in go java python scio; do
-    export SDK=$sdk &&
-    export SERVER_ADDRESS=https://${SDK}.$PLAYGROUND_DNS_NAME &&
-
-    python3 ci_cd.py --datastore-project $GOOGLE_CLOUD_PROJECT \
-                     --step CD --sdk SDK_${SDK^^} \
-                     --origin PG_EXAMPLES \
-                     --subdirs ./learning/katas ./examples ./sdks
-done
-```
+./gradlew :playground:dockerComposeLocalDown
+```
+
+## Deploy examples
+1. Go to `/playground/infrastucture` directory
+    ```shell
+    cd playground/infrastucture
+    ```
+1. Setup python venv
+    ```shell
+    python3 -m venv venv
+    ```
+1. Activate venv
+    ```shell
+    source venv/bin/activate
+    ```
+1. Install dependencies
+    ```shell
+    pip install -r requirements.txt
+    ```
+1. Setup the environment variables
+    ```shell
+    export BEAM_ROOT_DIR=$(realpath ../../)
+    export SDK_CONFIG="../../playground/sdks.yaml"
+    export BEAM_EXAMPLE_CATEGORIES="../categories.yaml"
+    export BEAM_USE_WEBGRPC=yes
+    export BEAM_CONCURRENCY=4
+    export DATASTORE_EMULATOR_HOST=localhost:8081
+    ```
+
+1. Run the `ci_cd.py` script
+
+    ```shell
+    export SERVER_ADDRESS=<runner_address> # see the note below
+    python ci_cd.py --step CD \
+                    --sdk <SDK> \ # SDK_GO, SDK_JAVA, SDK_PYTHON, SDK_SCIO
+                    --namespace Playground \
+                    --datastore-project test \
+                    --origin PG_EXAMPLES \
+                    --subdirs $BEAM_ROOT_DIR/sdks $BEAM_ROOT_DIR/examples $BEAM_ROOT_DIR/learning/katas # For SCIO examples see note below
+    ```
+    > **Note:** The `SERVER_ADDRESS` variable should be set to the address of the runner server for the particular SDK. For the local deployment the default values are:
+    > | SDK | Address |
+    > | --- | --- |
+    > | Go | `localhost:8084` |
+    > | Java | `localhost:8086` |
+    > | Python | `localhost:8088` |
+    > | SCIO | `localhost:8090` |
+
+    > **Note:** SCIO examples are not committed to the Beam tree. You will need to use [`fetch_scala_examples.py`](infrastructure/fetch_scala_examples.py) script to download them to some directory and then use `--subdirs` option to point to that directory. For example:
+    > ```shell
+    > python fetch_scala_examples.py --output-dir /tmp/scio-examples
+    > python ci_cd.py --step CD \
+    >                 --sdk SDK_SCIO \
+    >                 --namespace Playground \
+    >                 --datastore-project test \
+    >                 --origin PG_EXAMPLES \
+    >                 --subdirs /tmp/scio-examples
+    > ```
+    > See [this section](TASKS.md##obtaining-scio-examples) for more information.
+
+# How to add your own example
+Please refer to [this document](load_your_code.md).
+
+# Deployment guide
+## Manual deployment
+See deployment guide at [terraform/README.md](/playground/terraform/README.md)
+## Manual CloudBuild setup
+To set up CloudBuild triggers manually please refer to [this guide](/playground/terraform/infrastructure/cloudbuild-manual-setup/README.md)
+
+# Project structure
+
+Several directories in this repository are used for the Beam Playground project. The list of the directories used can be seen below.
+
+| Directory | Purpose |
+|-----------|---------|
+| [`/examples`](/examples) | Contains code of the examples for Java SDK. The examples are getting loaded into the main catalog. |
+| [`/learning/beamdoc`](/learning/beamdoc) | Contains code of the examples which should not be available in the main Playground catalog. |
+| [`/learning/katas`](/learning/katas) | Containes small code examples, loaded into main catalog.
+| [`/playground`](/playground) | Root of Playground sources. |
+| [`/playground/api`](/playground/api) | Protobuf definitions for the Playground gRPC API. |
+| [`/playground/backend`](/playground/backend) | Root of Playground backend sources. See [this document](/playground/backend/CONTRIBUTE.md) for detailed description. |
+| [`/playground/frontend`](/playground/frontend) | Root of Playground frontend sources. See [this document](/playground/frontend/CONTRIBUTE.md) for detailed description. |
+| [`/playground/infrastructure`](/playground/infrastructure) | Scripts used in Playground CI/CD pipelines for verifying and uploading examples. |
+| [`/playground/kafka-emulator`](/playground/kafka-emulator) | Sources of a Kafka emulator used for demonstrating KafkaIO examples. |
+| [`/playground/terraform`](/playground/terraform) | Terraform configuration files for Playground deployment to Google Cloud Platform. |
+| [`/sdks`](/sdks) | Source of the BEAM SDKs. Used by Playground as a source of examples for Go and Python SDKs. |
+
+# Contribution guide
+- Backend: see [backend/README.md](/playground/backend/README.md) and [backend/CONTRIBUTE.md](/playground/backend/CONTRIBUTE.md)
+- Frontend: see [frontend/README.md](/playground/frontend/README.md) and [frontend/CONTRIBUTE.md](/playground/frontend/CONTRIBUTE.md)
\ No newline at end of file
diff --git a/playground/TASKS.md b/playground/TASKS.md
new file mode 100644
index 00000000000..6b942fbf314
--- /dev/null
+++ b/playground/TASKS.md
@@ -0,0 +1,272 @@
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+-->
+
+- [Available Gradle Tasks](#available-gradle-tasks)
+  - [Perform overall pre-commit checks](#perform-overall-pre-commit-checks)
+  - [To see available gradle tasks for playground:](#to-see-available-gradle-tasks-for-playground)
+  - [Re-generate protobuf](#re-generate-protobuf)
+  - [Run local environment using docker compose](#run-local-environment-using-docker-compose)
+    - [Router only](#router-only)
+    - [Router, runners, and frontend](#router-runners-and-frontend)
+  - [Removing old snippets](#removing-old-snippets)
+  - [Removing a specific snippet](#removing-a-specific-snippet)
+  - [Run playground tests without cache](#run-playground-tests-without-cache)
+- [Updating dependencies](#updating-dependencies)
+  - [Referenced Beam SDK update](#referenced-beam-sdk-update)
+  - [Update Go version](#update-go-version)
+  - [Update Python version](#update-python-version)
+  - [Update Java version](#update-java-version)
+  - [Update Flutter version](#update-flutter-version)
+- [Deployment](#deployment)
+- [Obtaining SCIO examples](#obtaining-scio-examples)
+- [Manual Example deployment](#manual-example-deployment)
+  - [Run example deployment script](#run-example-deployment-script)
+
+
+# Available Gradle Tasks
+
+## Perform overall pre-commit checks
+
+```
+cd beam
+./gradlew playgroundPrecommit
+```
+
+## To see available gradle tasks for playground:
+
+```
+cd beam
+./gradlew playground:tasks
+```
+
+## Re-generate protobuf
+
+```
+cd beam
+./gradlew playground:generateProto
+```
+
+## Run local environment using docker compose
+
+### Router only
+
+Start:
+
+```bash
+cd beam
+./gradlew playground:backend:containers:router:dockerComposeLocalUp
+```
+
+Stop:
+
+```bash
+cd beam
+./gradlew playground:backend:containers:router:dockerComposeLocalDown
+```
+
+### Router, runners, and frontend
+
+1. Edit `/playground/frontend/playground_components/lib/src/constants/backend_urls.dart`
+to override backend URLs with yours found in `/playground/docker-compose.local.yaml`.
+2. To start, run:
+
+```bash
+cd beam
+./gradlew playground:dockerComposeLocalUp
+```
+
+3. To stop, run:
+
+```bash
+cd beam
+./gradlew playground:dockerComposeLocalDown
+```
+
+This way of running may not work in all environments because it is not maintained.
+It is used occasionally by the Frontend team to test complex tasks against
+a not-yet-deployed backend.
+The full start may take ~30 minutes and is demanding, so you should likely enable
+only one backend runner for the SDK you need.
+
+If you do not need particular runners, comment out:
+1. Dependencies on them in `/playground/build.gradle.kts` in `dockerComposeLocalUp` task.
+2. Their Docker image configurations in `/playground/docker-compose.local.yaml`.
+
+See also [Backend Lookup](frontend/README.md#backend-lookup) in the Frontend.
+
+## Removing old snippets
+
+Run the method to remove unused code snippets from the Cloud Datastore. Unused snippets are snippets that are out of date. If the last visited date property less or equals than the current date minus dayDiff parameter then a snippet is out of date
+
+```
+cd beam
+./gradlew playground:backend:removeUnusedSnippet -DdayDiff={int} -DprojectId={string} -Dnamespace={datastore namespace}
+```
+
+## Removing a specific snippet
+
+Run the method to remove a specific code snippet from the Cloud Datastore.
+
+```
+cd beam
+./gradlew playground:backend:removeSnippet -DsnippetId={string} -DprojectId={string} -Dnamespace={datastore namespace}
+```
+
+## Run playground tests without cache
+
+```
+cd beam
+ ./gradlew playground:backend:testWithoutCache
+```
+
+# Updating dependencies
+## Referenced Beam SDK update
+1. Update default `BEAM_VERSION` values in Java contianer [Dockerfile](/playground/backend/containers/java/Dockerfile)
+2. Update `default_beam_version` in [Java container `build.gradle`](/playground/backend/containers/java/build.gradle)
+3. Update `default_beam_version` in [Python container `build.gradle`](/playground/backend/containers/python/build.gradle)
+4. Update SDK version in CI GitHub workflow:
+    - [playground_examples_ci_reusable.yml](/.github/workflows/playground_examples_ci_reusable.yml)
+5. Update `-Psdk-tag=` in ["Deploy Playground to Kubernetes" section of the deployment guide](/playground/terraform/README.md#deploy-playground-to-kubernetes)
+6. Update `_BEAM_VERSION` in
+    - [playground_ci_stable.yaml](/playground/infrastructure/cloudbuild/playground_ci_stable.yaml)
+    - [playground_cd_stable.yaml](/playground/infrastructure/cloudbuild/playground_cd_stable.yaml)
+
+## Update Go version
+
+1. Update Go version in [`go.mod`](backend/go.mod)
+2. Run `go mod download` and `go mod tidy` in [`backend`](backend) directory
+3. Update `BASE_IMAGE` parameter in [router Dockerfile](backend/containers/router/Dockerfile) and [`build.gradle`](backend/containers/router/build.gradle)
+4. Update `GO_BASE_IMAGE` parameter in [Python Dockerfile](backend/containers/python/Dockerfile) and [`build.gradle`](backend/containers/python/build.gradle)
+5. Update `GO_BASE_IMAGE` parameter in [Java Dockerfile](backend/containers/java/Dockerfile) and [`build.gradle`](backend/containers/java/build.gradle)
+6. Update `GO_BASE_IMAGE` parameter in [SCIO Dockerfile](backend/containers/scio/Dockerfile) and [`build.gradle`](backend/containers/scio/build.gradle)
+7. Update `BASE_IMAGE` for `buildArgs` argument in [Go runner `build.gradle`](backend/containers/go/build.gradle)
+
+## Update Python version
+
+Update `BASE_IMAGE` in [Python Dockerfile](backend/containers/python/Dockerfile)
+
+## Update Java version
+
+1. Update base images version in `FROM maven:3.8.6-openjdk-11 as dep` and `FROM apache/beam_java11_sdk:$BEAM_VERSION` lines in [Java Dockerfile](backend/containers/java/Dockerfile)
+2. Update `BASE_IMAGE` in [SCIO Dockerfile](backend/containers/scio/Dockerfile) and [`build.gradle`](backend/containers/scio/build.gradle)
+
+## Update Flutter version
+
+Update the version in the following locations:
+
+### Project files that declare minimal required versions as we desire, one per Flutter project
+
+- [frontend/playground_components/pubspec.yaml](frontend/playground_components/pubspec.yaml), the line with `flutter: '>=x.x.x'`.
+- [frontend/playground_components_dev/pubspec.yaml](frontend/playground_components_dev/pubspec.yaml), the line with `flutter: '>=x.x.x'`.
+- [frontend/pubspec.yaml](frontend/pubspec.yaml), the line with `flutter: '>=x.x.x'`.
+- In Tour of Beam: [../learning/tour-of-beam/frontend/pubspec.yaml](../learning/tour-of-beam/frontend/pubspec.yaml), the line with `flutter: '>=x.x.x'`.
+
+### Generated project files with computed minimal versions given the package requirements, one per runnable Flutter app
+
+Run `dart pub get` in all of these directories for these files to be updated.
+Verify and merge the changes.
+
+- Playground: [frontend/pubspec.lock](frontend/pubspec.lock), the line with `flutter: ">=x.x.x"`.
+- Tour of Beam: [../learning/tour-of-beam/frontend/pubspec.lock](../learning/tour-of-beam/frontend/pubspec.lock), the line with `flutter: ">=x.x.x"`.
+
+### CI
+
+- [frontend/Dockerfile](frontend/Dockerfile), the line with `ARG FLUTTER_VERSION=x.x.x`.
+- [/.github/workflows/build_playground_frontend.yml](../.github/workflows/build_playground_frontend.yml), the line with `FLUTTER_VERSION: x.x.x`.
+- [/.github/workflows/playground_frontend_test.yml](../.github/workflows/playground_frontend_test.yml), the line with `FLUTTER_VERSION: x.x.x`.
+- [/.github/workflows/tour_of_beam_frontend_test.yml](../.github/workflows/tour_of_beam_frontend_test.yml), the line with `FLUTTER_VERSION: x.x.x`.
+
+# Deployment
+
+See [this guide](./terraform/README.md) for details on how to build and deploy
+the application and its dependent infrastructure.
+
+# Obtaining SCIO examples
+
+SCIO examples are based  on examples found in the upstream SCIO [repository](https://github.com/spotify/scio). To obtain the examples, use [`fetch_scala_examples.py`](infrastructure/fetch_scala_examples.py) script:
+
+```shell
+python fetch_scala_examples.py --output_dir <output_dir>
+```
+this will download all supported SCIO examples into `<output_dir>`.
+
+To add a new example, add it to the list of examples in the script [`fetch_scala_examples.py`](infrastructure/fetch_scala_examples.py). You will need to provide the path to the example in the `scio` git repository, the name of the example, description, run options, categories and tags.
+
+# Manual Example deployment
+
+The following requirements are needed for deploying examples manually:
+
+1. GCP project with deployed Playground backend
+2. Python (3.9.x)
+3. Login into GCP (gcloud default login or using service account key)
+
+## Run example deployment script
+Example deployment scripts uses following environment variables:
+
+- GOOGLE_CLOUD_PROJECT    - GCP project id where Playground backend is deployed
+- BEAM_ROOT_DIR           - root folder to search for playground examples
+- SDK_CONFIG              - location of sdk and default example configuration file
+- BEAM_EXAMPLE_CATEGORIES - location of example category configuration file
+- BEAM_USE_WEBGRPC        - use grpc-Web instead of grpc (default)
+- GRPC_TIMEOUT            - timeout for grpc calls (defaults to 10 sec)
+- BEAM_CONCURRENCY        - number of eaxmples to run in parallel (defaults to 10)
+- SERVER_ADDRESS          - address of the backend runnner service for a particular SDK
+
+```
+usage: ci_cd.py [-h] --step {CI,CD} [--namespace NAMESPACE] --datastore-project DATASTORE_PROJECT --sdk {SDK_JAVA,SDK_GO,SDK_PYTHON,SDK_SCIO} --origin {PG_EXAMPLES,TB_EXAMPLES} --subdirs SUBDIRS [SUBDIRS ...]
+
+CI/CD Steps for Playground objects
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --step {CI,CD}        CI step to verify all beam examples/tests/katas. CD step to save all beam examples/tests/katas and their outputs on the GCD
+  --namespace NAMESPACE
+                        Datastore namespace to use when saving data (default: Playground)
+  --datastore-project DATASTORE_PROJECT
+                        Datastore project to use when saving data
+  --sdk {SDK_JAVA,SDK_GO,SDK_PYTHON,SDK_SCIO}
+                        Supported SDKs
+  --origin {PG_EXAMPLES,TB_EXAMPLES}
+                        ORIGIN field of pg_examples/pg_snippets
+  --subdirs SUBDIRS [SUBDIRS ...]
+                        limit sub directories to walk through, relative to BEAM_ROOT_DIR
+```
+
+Helper script to deploy examples for all supported sdk's:
+
+```shell
+cd playground/infrastructure
+
+export BEAM_ROOT_DIR="../../"
+export SDK_CONFIG="../../playground/sdks.yaml"
+export BEAM_EXAMPLE_CATEGORIES="../categories.yaml"
+export BEAM_USE_WEBGRPC=yes
+export BEAM_CONCURRENCY=4
+export PLAYGROUND_DNS_NAME="your registered dns name for Playground"
+
+for sdk in go java python scio; do
+    export SDK=$sdk &&
+    export SERVER_ADDRESS=https://${SDK}.$PLAYGROUND_DNS_NAME &&
+
+    python3 ci_cd.py --datastore-project $GOOGLE_CLOUD_PROJECT \
+                     --step CD --sdk SDK_${SDK^^} \
+                     --origin PG_EXAMPLES \
+                     --subdirs ./learning/katas ./examples ./sdks
+done
+```
diff --git a/playground/backend/CONTRIBUTE.md b/playground/backend/CONTRIBUTE.md
index a11873acee5..cc095c3df98 100644
--- a/playground/backend/CONTRIBUTE.md
+++ b/playground/backend/CONTRIBUTE.md
@@ -26,36 +26,47 @@ This guide consists of:
 - [Code processing pipeline](#code-processing-pipeline)
 - [How to add a new supported language](#how-to-add-a-new-supported-language)
 
+See also:
+- [Database schema](SCHEMA.md)
+
 ## Project structure
 
 ```
 backend/
-├── cmd                   # set up and start backend application
-├── configs               # config files for each SDK
-├── containers            # set up and build backend images
-├── internal              # backend business logic
-│   ├── api                   # generated grpc API files
-│   ├── cache                 # logic of work with cache
-│   ├── code_processing       # logic of processing the received code
-│   ├── components            # logic of work for more difficult processes using several packages
-│   ├── constants             # application constants to use them anywhere in the application
-│   ├── db                    # logic of work with database, e.g. the Cloud Datastore
-│   ├── environment           # backend environments e.g. SDK of the instance
-│   ├── errors                # custom errors to send them to the client
-│   ├── executors             # logic of work with code executors
-│   ├── fs_tool               # logic of work with a file system of code processing to prepare required files/folders
-│   ├── logger                # custom logger
-│   ├── preparers             # logic of preparing code before execution
-│   ├── setup_tools           # logic of set up of executors and file systems by SDK requirements
-│   ├── streaming             # logic of saving execution output as a stream
-│   ├── tests                 # logic of work with unit/integration tests in the application, e.g. testing scripts to download mock data to database
-│   ├── utils                 # different useful tools
-│   └── validators            # logic of validation code before execution
-├── go.mod                        # define backend go module and contain all project's dependencies
-├── logging.properties            # config file to set up log for Java code
-├── properties.yaml               # property file consists of application properties required for the operation logic
-├── start_datastore_emulator.sh   # shell script to run the datastore emulator for local deployment or testing
-├── stop_datastore_emulator.sh    # shell script to stop the datastore emulator
+├── cmd
+│   ├── migration_tool           # tool to apply database migrations
+│   ├── remove_unused_snippets   # tool to remove old snippets manually
+│   └── server                   # entry point to the backend application
+├── configs                      # config files for each SDK
+├── containers                   # set up and build backend docker images
+├── datasets                     # datasets for examples using Kafka emulator
+├── internal                     # backend logic
+│   ├── api                      # generated grpc API files
+│   ├── cache                    # logic for working with cache
+│   ├── code_processing          # logic for processing the received code
+│   ├── components               # backend components
+│   ├── constants                # code constants used in the application
+│   ├── db                       # logic for working with database, e.g. the Cloud Datastore
+│   ├── emulators                # logic for starting various emulators, e.g. Kafka
+│   ├── environment              # tools for working with application environment settings
+│   ├── errors                   # custom errors
+│   ├── executors                # logic used to run the user submitted code
+│   ├── external_functions       # logic for calling Google Cloud Functions
+│   ├── fs_tool                  # logic for woking with filesystem operations during run preparation
+│   ├── logger                   # cusotm logger
+│   ├── preparers                # logic for preparing the user submitted code before execution
+│   ├── setup_tools              # logic for setting up executors
+│   ├── streaming                # implementation of run output streamer
+│   ├── tasks                    # periodic tasks scheduler
+│   ├── tests                    # common testing logic
+│   ├── utils                    # miscellaneous tools
+│   └── validators               # logic for pre-execution code validation
+├── playground_functions         # Google Cloud Functions for write access to the database
+├── functions.go                 # entry point for Cloud Functions
+├── go.mod                       # Go project build configuration
+├── logging.properties           # configuration for Java runner logger
+├── new_scio_project.sh          # script for creating new SCIO project, used by SCIO runner
+└── properties.yaml              # application properties
 ...
 ```
 
@@ -148,7 +159,7 @@ enum Sdk {
           dataset: { dataset_1 }
    datasets:
      { dataset_1 }:
-          location: { local | GCS }
+          location: local
           format: { json | avro }
 ```
 5. Create a PR to the [Apache Beam Repository](https://github.com/apache/beam)
diff --git a/playground/backend/SCHEMA.md b/playground/backend/SCHEMA.md
new file mode 100644
index 00000000000..cabdbca86af
--- /dev/null
+++ b/playground/backend/SCHEMA.md
@@ -0,0 +1,185 @@
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+-->
+
+- [Database](#database)
+  - [Datastore namespaces](#datastore-namespaces)
+  - [Migrations](#migrations)
+  - [Datastore schema](#datastore-schema)
+    - [pg\_schema\_versions](#pg_schema_versions)
+    - [pg\_sdks](#pg_sdks)
+    - [pg\_examples](#pg_examples)
+    - [pg\_snippets](#pg_snippets)
+      - [DatasetNestedEntity](#datasetnestedentity)
+    - [pg\_datasets](#pg_datasets)
+    - [pg\_files](#pg_files)
+    - [pg\_pc\_objects](#pg_pc_objects)
+  - [Datastore indexes](#datastore-indexes)
+- [Redis](#redis)
+
+# Database
+
+Beam Playground uses Google Cloud Platform Datastore for storing examples and snippets. Redis is used for caching catalog reads from Datastore to avoid having to enumerate all of the exmaples on each catalog request.
+
+## Datastore namespaces
+Playground can use custom namespace to store entities in Datastore to support simultaneous deployment of several backend instances in the same GCP project. By default `Playground` namespace is used. custom namespace can be selected by setting `DATASTORE_NAMESPACE` environment variable.
+
+## Migrations
+If a breaking change to DB schema is made, it's adviced to implement a migration procedure to handle the data change in the Datastore. There are no formalized rules on when a migration should be implemented, but in general the following should be considered:
+- all examples and precompiled objects are recreated during the deployment process and are not modified during the application runtime
+- user code snippets do survive application update and may require data migration
+
+In order to implement a migration a new Go file wtih name like `migration_xxx.go` should be created under the `internal/db/schema/migration` path. The file should contain a structure which implements the following interface:
+```go
+type Version interface {
+	// GetVersion returns the version string of the schema
+	GetVersion() string
+	// GetDescription returns the description of the schema version
+	GetDescription() string
+	// InitiateData initializes the data for the schema or performs a migration
+	InitiateData(args *DBArgs) error
+}
+```
+
+After implementing the migration logic inside of the `InitiateData()` function it should be covered by tests and the new migration should be added to the list of exisitng migrations in the `cmd/server/server.go` file inside of `setupDBStructure()` function.
+
+## Datastore schema
+There are several entity kinds in Datastore:
+| Entity kind | Description | Corresponding Go struct |
+|-------------|-------------|-------------------------|
+| [`pg_schema_versions`](#pg_schema_versions) | Schema version entity | `entity.SchemaEntity` |
+| [`pg_sdks`](#pg_sdks) | SDK entity | `entity.SDKEntity` |
+| [`pg_examples`](#pg_examples) | Example entity | `entity.ExampleEntity` |
+| [`pg_snippets`](#pg_snippets) | Snippet entity | `entity.SnippetEntity` |
+| [`pg_datasets`](#pg_datasets) | Dataset entity | `entity.DatasetEntity` |
+| [`pg_files`](#pg_files) | File entity | `entity.FileEntity` |
+| [`pg_pc_objects`](#pg_pc_objects) | Precompiled object entity | `entity.PrecompiledObjectEntity` |
+
+### pg_schema_versions
+This entity kind is used to store schema version and description of changes.
+| Field name | Description | Type |
+|------------|-------------|------|
+| Name/ID | Schema version | `Key` |
+| `descr` | Description of changes | `string` |
+
+### pg_sdks
+This entity kind is used to store information about each supported SDK.
+
+| Field name | Description | Type |
+|------------|-------------|------|
+| Name/ID | SDK name | `Key` |
+| `defaultExample` | Name of the default example for this SDK | `string` |
+
+### pg_examples
+This entity kind is used to store example catalog items.
+
+| Field name | Description | Type |
+|------------|-------------|------|
+| Name/ID | Example ID. Has form of `<SDK>_<Example name>` | `Key` |
+| `name` | Example name | `string` |
+| `sdk` | SDK ID | `Key` |
+| `descr` | Example description | `string` |
+| `tags` | Example tags | `[]string` |
+| `cats` | Example categories | `[]string` |
+| `path` | Url of the example on Github | `string` |
+| `type` | Type of the example. Possible values are `PRECOMPILED_OBJECT_TYPE_UNSPECIFIED`, `PRECOMPILED_OBJECT_TYPE_EXAMPLE`, `PRECOMPILED_OBJECT_TYPE_KATA`, `PRECOMPILED_OBJECT_TYPE_UNIT_TEST` | `string` |
+| `origin` | `PG_EXAMPLES` for Playground examples, `TB_EXAMPLES` for Tour of Beam examples | `string` |
+| `schVer` | Schema version | `Key` |
+| `urlVCS` | Url of the example on Github | `string` |
+| `urlNotebook` | Url to a Collab notebook which has the example code | `string` |
+| `alwaysRun` | If true, frontend will ignore any precompiled objects assosciated with the example and run it always | `bool` |
+
+### pg_snippets
+This entity kind is used to store snippets.
+
+| Field name | Description | Type |
+|------------|-------------|------|
+| Name/ID | Snippet ID. For shared user code the ID is computed based on the snippet content hash, for the snippets containing examples code the ID is the same as for the related example | `Key` |
+| `ownerId` | ***Cannot find any usage***| `string` |
+| `sdk` | SDK ID | `Key` |
+| `pipeOpts` | Pipeline options | `string` |
+| `created` | Creation time | `time.Time` |
+| `lVisited` | Last visit time | `time.Time` |
+| `origin` | `PG_SNIPPETS` for Playground examples, `TB_SNIPPETS` for Tour of Beam examples, `PG_USER` for snippets with code shared by users, `TB_USER` for snippets created by Tour Of Beam users | `string` |
+| `visitCount` | Number of times the snippet was visited | `int` |
+| `schVer` | Schema version | `Key` |
+| `numberOfFiles` | Number of files in the snippet. Used to derive file keys. | `int` |
+| `complexity` | Complexity of the snippet. Possible values are `COMPLEXITY_UNSPECIFIED`, `COMPLEXITY_BASIC`, `COMPLEXITY_MEDIUM`, `COMPLEXITY_ADVANCED` | `string` |
+| `persistenceKey` | Used to track snippets created with Tour of Beam. When Tour of Beam user save a new snippet, all other snippets with the same persistenceKey are removed. | `string` |
+| `datasets` | Contains an array of `DatasetNestedEntity` objects which describe datasets and emulators assosciated with the snippet | `[]DatasetNestedEntity` |
+
+#### DatasetNestedEntity
+| Field name | Description | Type |
+|------------|-------------|------|
+| `config` | A JSON serialized `map[string]string` object which contains emulator configuration | `string` |
+| `dataset` | Dataset ID | `Key` |
+| `emulator` | Emulator name. Currently only `kafka` is supported | `string` |
+
+### pg_datasets
+| Field name | Description | Type |
+|------------|-------------|------|
+| Name/ID | Name of the dataset | Key |
+| `path` | Path to the dataset file on the runner filesystem under path specified in `DATASETS_PATH` environment variable (`/opt/playground/backend/datasets` by default) | `string` |
+
+### pg_files
+| Field name | Description | Type |
+|------------|-------------|------|
+| Name/ID | This field is constructed by concatenating snippet ID with an underscore (`_`) and an ordinal number of the file in the snippet. For example, if snippet `SDK_JAVA_Example` has `numberOfFiles` set to `2` then there will be two `pg_files` entities with `SDK_JAVA_Example_0` and `SDK_JAVA_Example_1` keys. | Key |
+| `content` | Content of the file | `string` |
+| `cntxLine` | Line number on which frontend will initially focus the text editor cursor when the file is being displayed | `int32` |
+| `isMain` | Whether the file is the main file in the snippet. There can only be one main file in the snippet | `bool` |
+| `name` | Name of the file which will shown to the user by the frontend | `string` |
+
+### pg_pc_objects
+These entities contain pre-compiled (cached) outputs of examples. There are three types of precompiled objects:
+- `OUTPUT`, containing example's run output
+- `LOG`, contianing example's log output
+- `GRAPH`, containing example's execution graph output
+All of these precompiled objects share the same schema
+
+| Field name | Description | Type |
+|------------|-------------|------|
+| Name/ID | Key is constructed by concatenating example's ID with precompiled object type, e.g. `SDK_GO_WordCount_OUTPUT`, `SDK_GO_WordCount_LOG`, `SDK_GO_WordCount_GRAPH` | Key |
+| `content` | Saved output of the example's run | `string` |
+
+## Datastore indexes
+Indexes are defined in [`index.yaml`](../index.yaml) file. The file is used during deployment to create indexes in the Datastore.
+
+# Redis
+
+Playground uses Redis as a cache for examples catalog to avoid having to re-enumerate all exmaples upon each request, as a temporary storage for examples output (logs, graphs, etc.) and as a message bus to relay events like a user request for pipeline cancellation.
+
+Each pipeline run uses pipleine id as a Redis key, with the following subkeys ([source](internal/cache/cache.go)):
+| Key | Subkey | Description |
+|-----|--------|-------------|
+| Pipeline Id | `STATUS` | Pipeline status. Possible values can be found in [`api.proto`](../api/v1/api.proto) in `Status` enum. |
+| Pipeline Id | `RUN_OUTPUT` | Pipeline run output. |
+| Pipeline Id | `RUN_ERROR` | Pipeline run error message. |
+| Pipeline Id | `VALIDATION_OUTPUT` | Pipeline validation step output. |
+| Pipeline Id | `PREPARATION_OUTPUT` | Pipeline preparation step output. |
+| Pipeline Id | `COMPILE_OUTPUT` | Pipeline compilation step output. |
+| Pipeline Id | `CANCELED` | Used to signal that user has requested pipeline cancellation. Runner periodically polls the cache to check if this key has been set to `true` and cancels the pipeline if it has. |
+| Pipeline Id | `RUN_OUTPUT_INDEX` | Index of the start of the run step's output. Upon each request of the pipeline execution logs this value is set to the end of the returned log and used in subsequent requests to skip already sent log fragment. |
+| Pipeline Id | `LOGS` | Pipeline execution logs. |
+
+Additionally there are keys used globally by the Playground:
+| Key | Subkey | Description |
+|-----|--------|-------------|
+| `EXAMPLES_CATALOG` | None | Used to store cached version of examples catalog. |
+| `SDKS_CATALOG` | None | Used to store cached version of supported SDKS list with list of names of default examples. |
+| `DEFAULT_PRECOMPILED_OBJECTS` | Sdk | Used to store a default example metadata in cache. |
diff --git a/playground/backend/containers/python/build.gradle b/playground/backend/containers/python/build.gradle
index 30ead18f861..9b11b19b16b 100644
--- a/playground/backend/containers/python/build.gradle
+++ b/playground/backend/containers/python/build.gradle
@@ -23,7 +23,7 @@ apply from: "$project.rootDir/playground/backend/containers/git-functions.gradle
 applyDockerNature()
 
 def playgroundJobServerProject = "${project.path.replace('-container', '')}"
-
+def default_beam_version = "2.44.0"
 description = project(playgroundJobServerProject).description + " :: Container"
 
 configurations {
@@ -77,7 +77,7 @@ docker {
                project.rootProject["go-base-image"] :
                "golang:1.18-bullseye",
           'SDK_TAG'      : project.rootProject.hasProperty(["sdk-tag"]) ?
-                project.rootProject["sdk-tag"] : project.rootProject.sdk_version,
+                project.rootProject["sdk-tag"] : default_beam_version,
           'GIT_COMMIT'   : getGitCommitHash(),
           'GIT_TIMESTAMP': getGitCommitTimestamp()])
 }
diff --git a/playground/backend/containers/scio/build.gradle b/playground/backend/containers/scio/build.gradle
index e4afb763992..affb3c778cf 100644
--- a/playground/backend/containers/scio/build.gradle
+++ b/playground/backend/containers/scio/build.gradle
@@ -71,7 +71,7 @@ docker {
    buildArgs(
          ['BASE_IMAGE'   : project.rootProject.hasProperty(["base-image"]) ?
                project.rootProject["base-image"] :
-               "openjdk:8",
+               "openjdk:11",
           'GIT_COMMIT'   : getGitCommitHash(),
           'GIT_TIMESTAMP': getGitCommitTimestamp()])
 }
diff --git a/playground/doc/load_your_code/images/share-my-code.png b/playground/doc/load_your_code/images/share-my-code.png
new file mode 100644
index 00000000000..f0cd54f4ff8
Binary files /dev/null and b/playground/doc/load_your_code/images/share-my-code.png differ
diff --git a/playground/doc/load_your_code/images/workflow.png b/playground/doc/load_your_code/images/workflow.png
new file mode 100644
index 00000000000..653bccaf5f5
Binary files /dev/null and b/playground/doc/load_your_code/images/workflow.png differ
diff --git a/playground/frontend/lib/constants/links.dart b/playground/frontend/lib/constants/links.dart
deleted file mode 100644
index b9aa3d3d170..00000000000
--- a/playground/frontend/lib/constants/links.dart
+++ /dev/null
@@ -1,19 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-const kAddExampleLink = 'https://beam.apache.org/get-started/try-beam-playground/#how-to-add-new-examples';
diff --git a/playground/frontend/lib/modules/actions/components/new_example.dart b/playground/frontend/lib/modules/actions/components/new_example.dart
index 30e44f24c73..6f1b1c9bf1e 100644
--- a/playground/frontend/lib/modules/actions/components/new_example.dart
+++ b/playground/frontend/lib/modules/actions/components/new_example.dart
@@ -42,7 +42,7 @@ class NewExampleButton extends StatelessWidget {
           PlaygroundComponents.analyticsService.sendUnawaited(
             const NewExampleAnalyticsEvent(),
           );
-          unawaited(launchUrl(Uri.parse('/')));
+          unawaited(launchUrl(Uri.parse(BeamLinks.newExample)));
         },
       ),
     );
diff --git a/playground/frontend/lib/modules/examples/examples_dropdown_content.dart b/playground/frontend/lib/modules/examples/examples_dropdown_content.dart
index df0743001c5..0ed1304d460 100644
--- a/playground/frontend/lib/modules/examples/examples_dropdown_content.dart
+++ b/playground/frontend/lib/modules/examples/examples_dropdown_content.dart
@@ -21,7 +21,6 @@ import 'package:flutter_gen/gen_l10n/app_localizations.dart';
 import 'package:playground_components/playground_components.dart';
 import 'package:url_launcher/url_launcher.dart';
 
-import '../../constants/links.dart';
 import '../../constants/sizes.dart';
 import 'components/example_list/example_list.dart';
 import 'components/filter/filter.dart';
@@ -88,7 +87,7 @@ class _ExamplesDropdownContentState extends State<ExamplesDropdownContent> {
                 ),
               ),
             ),
-            onPressed: () => launchUrl(Uri.parse(kAddExampleLink)),
+            onPressed: () => launchUrl(Uri.parse(BeamLinks.newExample)),
           ),
         ),
       ],
diff --git a/playground/frontend/playground_components/lib/src/constants/backend_urls.dart b/playground/frontend/playground_components/lib/src/constants/backend_urls.dart
index 20c2d6348d9..af9390ddc6d 100644
--- a/playground/frontend/playground_components/lib/src/constants/backend_urls.dart
+++ b/playground/frontend/playground_components/lib/src/constants/backend_urls.dart
@@ -23,11 +23,19 @@ const defaultBackendUrlTemplate = 'https://{node}.play.beam.apache.org';
 
 /// The URLs for local backend development.
 const backendUrlOverrides = <String, String>{
+  // Uncomment the following lines to use staging backend:
   // 'router': 'https://router.play-dev.beam.apache.org',
   // 'go': 'https://go.play-dev.beam.apache.org',
   // 'java': 'https://java.play-dev.beam.apache.org',
   // 'python': 'https://python.play-dev.beam.apache.org',
   // 'scio': 'https://scio.play-dev.beam.apache.org',
+
+  // Uncomment the following lines to use local backend:
+  // 'router': 'http://localhost:8081',
+  // 'go': 'http://localhost:8084',
+  // 'java': 'http://localhost:8086',
+  // 'python': 'http://localhost:8088',
+  // 'scio': 'http://localhost:8090',
 };
 
 /// The URL templates that will not be probed
diff --git a/playground/frontend/playground_components/lib/src/constants/links.dart b/playground/frontend/playground_components/lib/src/constants/links.dart
index f8505c30533..9a29bf72983 100644
--- a/playground/frontend/playground_components/lib/src/constants/links.dart
+++ b/playground/frontend/playground_components/lib/src/constants/links.dart
@@ -24,6 +24,7 @@ class BeamLinks {
 
   // GitHub
   static const github = 'https://github.com/apache/beam';
+  static const newExample = 'https://github.com/apache/beam/blob/master/playground/load_your_code.md';
   static const reportIssue = 'https://github.com/apache/beam/issues';
 
   // Projects
diff --git a/playground/infrastructure/models.py b/playground/infrastructure/models.py
index bf667eeac38..0c7311e0f77 100644
--- a/playground/infrastructure/models.py
+++ b/playground/infrastructure/models.py
@@ -38,7 +38,6 @@ class DatasetFormat(str, Enum):
 
 class DatasetLocation(str, Enum):
     LOCAL = "local"
-    GCS = "GCS"
 
 
 class Dataset(BaseModel):
@@ -72,24 +71,101 @@ class Tag(BaseModel):
     Tag represents the beam-playground embedded yaml content
     """
 
-    filepath: str = Field(..., min_length=1)
-    line_start: int
-    line_finish: int
-    context_line: int = 1
-    name: str = Field(..., min_length=1)
+    # These parameters are parsed from YAML and are required:
+
+    categories: List[str] = []
+    """
+    Titles of categories to list this snippet in. Non-existent categories will be created.
+    """
+
     complexity: ComplexityEnum
+    """
+    How hard is the snippet to understand.
+    """
+
+    context_line: int = 1
+    """
+    The line number to scroll to when the snippet is loaded.
+    This applies after the metadata block is removed from the file, so discount for those lines.
+    Integer, 1-based.
+    """
+
     description: str
-    categories: List[str] = []
-    pipeline_options: str = ""
+    """
+    Description text to show on mouse over.
+    """
+
+    name: str = Field(..., min_length=1)
+    """
+    Name to show in the dropdown and to compose snippet ID from.
+    """
+
+    # These parameters are parsed from YAML and are optional:
+
     datasets: Dict[str, Dataset] = {}
+    """
+    Datasets which will be used by emulators. Example:
+    datasets:
+      CountWords:
+        location: local
+        format: json
+    """
+
+    default_example: bool = False
+    """
+    Whether this is the default example in its SDK.
+    If multiple snippets set this to `true` the behavior is undefined.
+    """
+
     emulators: List[Emulator] = []
+    """
+    List of emulators to start during pipeline execution. Currently only `kafka` type is supported.
+    Example:
+    emulators:
+      - type: kafka
+        topic:
+          id: dataset
+          source_dataset: CountWords
+    """
+
     always_run: bool = False
+    """
+    The example output will not be cached and playground will always run its code when user clicks 'Run' if this is set to 'True'
+    """
+
     never_run: bool = False
+    """
+    If 'True', it will not be possible to run this example from Playground UI. It will be read-only.
+    """
+
+
     multifile: bool = False
-    default_example: bool = False
+    """
+    Whether this is a file of a multi-file example.
+    """
+
+    pipeline_options: str = ""
+    """
+    Pipeline options. Example:
+    pipeline_options: --name1 value1 --name2 value2
+    """
+
     tags: List[str] = []
+    """
+    Tags by which this snippet can be found in the dropdown.
+    """
+
     url_notebook: Optional[HttpUrl] = None
+    """
+    The URL of the Colab notebook that is based on this snippet.
+    """
+
+    # These parameters are NOT parsed from YAML but are added at construction:
+
+    filepath: str = Field(..., min_length=1)
     files: List[ImportFile] = []
+    line_start: int
+    line_finish: int
 
     class Config:
         supported_categories = []
diff --git a/playground/infrastructure/requirements.txt b/playground/infrastructure/requirements.txt
index a5023df8d0a..a05ab1d0792 100644
--- a/playground/infrastructure/requirements.txt
+++ b/playground/infrastructure/requirements.txt
@@ -27,4 +27,4 @@ grpcio-tools==1.51.1
 protobuf==4.21.12
 google-cloud-datastore==2.11.0
 
-requests~=2.28.2
+requests~=2.28.2
\ No newline at end of file
diff --git a/playground/load_your_code.md b/playground/load_your_code.md
new file mode 100644
index 00000000000..3d858bc3ee7
--- /dev/null
+++ b/playground/load_your_code.md
@@ -0,0 +1,600 @@
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+-->
+
+# How to Add an Example/Snippet/Learning Content into Apache Beam Playground
+
+Adding a new example, code snippet, or Tour of Beam learning unit into the Playground is a three-step process:
+
+1. Prepare a code snippet.
+2. Add the code snippet to Apache Beam and/or Playground.
+3. Create a link to view the code snippet in Playground or to embed in a website page.
+
+
+Playground sources and output presentation formats:
+
+![Workflow](doc/load_your_code/images/workflow.png)
+
+This guide will walk through all steps.
+
+
+# Table of Contents
+
+- [Step 1. Prepare a code snippet](#step-1-prepare-a-code-snippet)
+  * [Code snippet data sources and dependencies](#code-snippet-data-sources-and-dependencies)
+  * [Emphasizing parts of code](#emphasizing-parts-of-code)
+  * [Named Sections](#named-sections)
+- [Step 2. Add the code snippet to the Apache Beam repo and/or Playground](#step-2-add-the-code-snippet-to-the-apache-beam-repo-andor-playground)
+  * [Source 1. How to add an example to Playground Examples Catalog](#source-1-how-to-add-an-example-to-playground-examples-catalog)
+    + [1. Add the file with example to a directory](#1-add-the-file-with-example-to-a-directory)
+    + [2. Add metadata to describe example](#2-add-metadata-to-describe-the-example)
+      - [Kafka emulator](#kafka-emulator)
+    + [3. Submit a PR](#3-submit-a-pr)
+    + [4. Save the snippet ID](#4-save-the-snippet-id)
+  * [Source 2. How to add an unlisted example to Playground](#source-2-how-to-add-an-unlisted-example-to-playground)
+  * [Source 3. How to add a Tour of Beam unit](#source-3-how-to-add-a-tour-of-beam-unit)
+  * [Source 4. How to add a snippet with the app alone](#source-4-how-to-add-a-snippet-with-the-app-alone)
+  * [Source 5. How to load code from GitHub or other HTTPS URL](#source-5-how-to-load-code-from-github-or-other-https-url)
+- [Step 3. Create a link or embed the snippet](#step-3-create-a-link-or-embed-the-snippet)
+  * [Link to a snippet](#link-to-a-snippet)
+    + [Link to an example from the Playground Examples Catalog](#link-to-an-example-from-the-playground-examples-catalog)
+    + [Link to an unlisted example](#link-to-an-unlisted-example)
+    + [Link to a snippet from a Tour of Beam unit](#link-to-a-snippet-from-a-tour-of-beam-unit)
+    + [Link to a user-shared snippet](#link-to-a-user-shared-snippet)
+    + [Link to a GitHub or HTTPS URL snippet](#link-to-a-github-or-other-https-url-snippet)
+    + [Link to an empty editor](#link-to-an-empty-editor)
+  * [Link to multiple snippets](#link-to-multiple-snippets)
+  * [Embedding a snippet into HTML](#embedding-a-snippet-into-html)
+    + [Embedding a snippet from Playground Examples Catalog](#embedding-a-snippet-from-playground-examples-catalog)
+    + [Embedding a user-shared snippet](#embedding-a-user-shared-snippet)
+    + [Embedding a snippet from other sources](#embedding-a-snippet-from-other-sources)
+  * [Embedding a snippet into the Apache Beam website](#embedding-a-snippet-into-the-apache-beam-website)
+- [Snippet view options](#snippet-view-options)
+  * [Read-only sections](#read-only-sections)
+  * [Folding everything except named sections](#folding-everything-except-named-sections)
+  * [Hiding everything except a named section](#hiding-everything-except-a-named-section)
+
+
+## Step 1. Prepare a code snippet
+
+Playground runs example code snippets using Apache Beam Direct Runner
+and requires that a code snippet is a complete runnable code.
+
+### Code snippet data sources and dependencies
+
+Code snippets can use data sources to demonstrate transforms and concepts. Playground restricts code access to Internet for security reasons. Following are the recommend ways for code snippet's data sources and dependecies:
+
+| Source/Dependency   | Notes                                                     |
+|----------|-----------------------------------------------------------|
+| File     | Store code snippet's data file in a GCS bucket in `apache-beam-testing` project.     |
+| BigQuery | Create a BigQuery dataset/table in `apache-beam-testing` project. |
+| Python package | Python packages accessible by Playground are located in a [Beam Python SDK container](https://github.com/apache/beam/tree/master/sdks/python/container) and in [Playground Python container](https://github.com/apache/beam/tree/master/playground/backend/containers/python). Add required packages to [Playground Python container](https://github.com/apache/beam/tree/master/playground/backend/containers/python). Please submit pull request with changes to the container or conta [...]
+| GitHub repo | If your example clones or dependes on files in a GitHub repo, copy required files to a GCS bucket in `apache-beam-testing` project and use the GCS files. |
+
+
+### Emphasizing parts of code
+
+Playground provides multiple features to help focus users on certain parts of the code.
+
+Playground automatically applies the following to all snippets:
+- Folds a comment if a snippet starts with one.
+- Folds imports.
+
+### Named Sections
+
+Playground supports *Named Sections* to tag code blocks and provide the following view options:
+- Fold all blocks except tagged code blocks.
+  This can be useful to help user focus on specific code blocks and features presented in a snippet.
+- Hide all code except tagged code blocks.
+  This can be useful to create runnable snippets illustrating specific concepts or transforms,
+  and hide all non-essential code blocks.
+  Such snippet can be embedded on a website to make examples in documentation and tutorials runnable.
+- Make certain code parts read-only.
+  This feature can be useful to create learning units where user modifications are desired
+  only in certain parts of the code.
+
+Please see [Snippet View Options](#snippet-view-options) section for details how different view options can be used.
+
+If you do not need any of those view options, skip to the [next step](#step-2-add-the-code-snippet-to-the-apache-beam-repo-andor-playground).
+
+*Named Sections* are defined with the following syntax:
+
+```
+// [START section_name]
+void method() {
+...
+}
+// [END section_name]
+```
+
+Create a named section for each part of your code that you want the above features for.
+To learn more details about the syntax please see
+the [README of the editor](https://pub.dev/packages/flutter_code_editor) that Playground uses.
+
+## Step 2. Add the code snippet to the Apache Beam repo and/or Playground
+
+There are several types of code snippets in the Playground:
+
+1. Example — a code snippet displayed in the Playground Examples Catalog.
+   See [how to add a new example here](#source-1-how-to-add-an-example-to-playground-examples-catalog).
+2. Unlisted Example — the same as an example, but is not listed in the example dropdown
+   and can only be accessed by direct linking. These are typically embedded on a website.
+   See [how to add a new unlisted example here](#source-2-how-to-add-an-unlisted-example-to-playground).
+3. [Tour of Beam](https://github.com/apache/beam/tree/master/learning/tour-of-beam) learning unit.
+   See [how to add a new Tour of Beam unit here](#source-3-how-to-add-a-tour-of-beam-unit).
+4. User-shared code snippets do not require a PR and should be used for code
+   not displayed on Beam resources.
+   See [how to add a snippet with the app alone here](#source-4-how-to-add-a-snippet-with-the-app-alone).
+5. GitHub or other HTTPS URL sources.
+   See [how to load a snippet from external GitHub or other HTTPS URL here](#source-5-how-to-load-code-from-github-or-other-https-url).
+
+See the [workflow above](#how-to-add-an-examplesnippetlearning-content-into-apache-beam-playground) how artifacts map to these sources.
+
+### Source 1. How to add an example to Playground Examples Catalog
+
+Playground Examples Catalog helps users discover example snippets
+and is the recommended way to add examples. Playground automatically scans,
+verifies and deploys example snippets from the directories listed below.
+
+> **Note:** SCIO examples are stored in a separate repository. To add support for a new SCIO example, please refer to [this section of `TASKS.md`](TASKS.md#obtaining-scio-examples).
+
+#### 1. Add the file with example to a directory
+
+Playground Java, Python, and Go examples are automatically picked from these
+predefined directories by the `playground_examples_ci.yml` GitHub workflow
+after a PR is merged to Beam repo:
+- `/examples`
+- `/learning/katas`
+- `/sdks`.
+
+Adding Scala example snippets automatically is not supported,
+and Scala example snippets can be added to the catalog manually.
+
+#### 2. Add metadata to describe the example
+
+Playground relies on metadata comments block to identify and place an example into the database,
+which is required for an example to show in the Examples Catalog.
+See [this](https://github.com/apache/beam/blob/3e080ff212d8ed7208c8486b515bb73c5d294475/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java#L20-L36) for an example.
+Playground automatically removes metadata comments block before storing the example in database,
+so the metadata is not visible to end users. The block is in the format of a YAML map:
+
+```yaml
+beam-playground:
+  # Name of the Beam example that will be displayed in the Playground Examples Catalog. Required.
+  name: ""
+  # Description of the Beam example that will be displayed in the Playground Examples Catalog. Required.
+  description: ""
+  # Contains information about pipeline options of the Beam example/test/kata. Optional.
+  pipeline_options: "--name1 value1 --name2 value2"
+  # The line number to scroll to when the snippet is loaded.
+  # Note that lines of the metadata block are cut so line numbers after it are shifted.
+  # Optional, defaults to 1 (the first line).
+  context_line: 1
+  # Categories this example is included into. See below for the supported values.
+  # Optional, defaults to no categories making the example unlisted.
+  categories:
+    - "Combiners"
+    - "Core Transforms"
+  # Tags by which this snippet can be found in the Example Catalog. Optional.
+  tags:
+    - "numbers"
+    - "count"
+  # Helps user to identify example's complexity. Values: BASIC|MEDIUM|ADVANCED. Required.
+  complexity: BASIC
+  # Specifies the example to be loaded as default when its SDK selected in the Playground.
+  # See section "Default examples" below. Optional, defaults to false.
+  default_example: true
+  # If the snippet has a Colab notebook, can link the URL of the Colab notebook that is based on this snippet.
+  url_notebook: "https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb"
+  # Specifies if the given example consists of multiple files or not. Optional, defaults to false.
+  multifile: true
+  # Specifies example caching by Playground. Optional, defaults to false (the output is cached).
+  always_run: true
+  # Datasets which will be used by emulators. Optional.
+  # Please see section "Kafka emulator" for more information.
+  datasets:
+    # Dataset name
+    CountWords:
+      # Dataset location. Only "local" is supported. Required.
+      location: local
+      # Dataset format. Supported values are "avro" and "json". Required.
+      format: avro
+  # List of emulators to start during pipeline execution. Currently only `kafka` type is supported. Optional.
+  emulators:
+     - type: kafka
+       topic:
+         # Dataset id. Will be used as a topic name.
+         id: dataset
+         # Name of dataset specified in "datasets" section.
+         source_dataset: "CountWords"
+```
+
+For metadata reference, see the fields in the `Tag` class [here](infrastructure/models.py).
+
+##### Categories
+
+The list of supported categories for an example is
+[here](https://github.com/apache/beam/blob/master/playground/categories.yaml).
+To add a new category, submit a PR that adds a category to the [categories.yaml](https://github.com/apache/beam/blob/master/playground/categories.yaml).
+When it is merged, the new category can be used in an example.
+
+##### Default examples
+
+Each SDK must have a single default example.
+If there is none, the user will see an error in the app and a blank editor.
+If there are more than one, it is not defined which one will be selected.
+
+##### Kafka emulator
+
+Examples which require Kafka server emulator need to include the `emulators` tag
+and provide `dataset` in the example's tag. You can refer to an example
+[here](/examples/java/src/main/java/org/apache/beam/examples/KafkaWordCountJson.java).
+
+1. Add your dataset in either JSON or Avro format into the `playground/backend/datasets` path.
+
+2. Add the following elements to the example's metadata tag:
+    ```YAML
+    emulators:
+      - type: kafka
+        topic:
+          id: dataset
+          source_dataset: <dataset_name>
+    datasets:
+      <dataset_name>:
+        location: local
+        format: json # or 'avro'
+    ```
+    replace `<dataset_name>` with the name of your dataset file without the file name extension.
+
+3. Use the exact string `"kafka_server:9092"` as the server name in your code snippet.
+   This string will be replaced by the actual host name and port automatically
+   before the compilation step by Playground.
+
+>**Kafka emulator limitations:**
+> - Playground Kafka emulator currently supports only Beam Java SDK.
+> - The exact string `"kafka_server:9092"` should be present in the code snippet;
+    any other variation like `"kafa_server" + ":9092"` will not work.
+
+#### 3. Submit a PR
+
+Create and submit a PR with the code snippet into [the Apache Beam repository](https://github.com/apache/beam)
+following the [Contribution guide](https://beam.apache.org/contribute/).
+Verify that all pre-commit tests are passing.
+
+Playground CI will verify and deploy the example to Playground Example Catalog when the PR is merged.
+
+#### 4. Save the snippet ID
+
+The snippet will be assigned an ID.
+You can find it in the address bar of the browser when you select it in the dropdown.
+
+For example, in this URL:
+
+```
+https://play.beam.apache.org/?path=SDK_JAVA_MinimalWordCount&sdk=java
+```
+
+the ID is: `SDK_JAVA_MinimalWordCount`.
+
+You will need the snippet ID to embed the Playground with the snippet into a website page.
+
+### Source 2. How to add an unlisted example to Playground
+
+Not all examples must be visible in the example dropdown.
+Some examples are best in the context of Apache Beam documentation.
+To embed them into the documentation, use unlisted examples.
+They work and are checked and cached the same way as the examples displayed in the Playground catalog.
+
+Proceed the same way as with [Source 1. Playground Examples Catalog](#source-1-how-to-add-an-example-to-playground-examples-catalog) except:
+1. Use the directory `/learning/beamdoc`
+2. Do not use the following attributes:
+   - `categories`
+   - `default_example`
+   - `tags`
+
+The ID of the snippet is a function of the SDK and the `name` attribute from its metadata:
+
+| SDK    | ID              |
+|--------|-----------------|
+| Go     | SDK_GO_name     |
+| Java   | SDK_JAVA_name   |
+| Python | SDK_PYTHON_name |
+
+### Source 3. How to add a Tour of Beam unit
+
+"Tour of Beam" is a separate project that combines learning materials with runnable snippets
+and allows students to track their learning progress.
+It uses the Playground engine, and so its content is added in a similar way.
+
+A Tour of Beam unit consists of learning materials and an optional runnable snippet.
+See [the learning content README](../learning/tour-of-beam/learning-content/README.md) on how to add units and link snippets to them.
+
+#### Adding a snippet
+
+Tour of Beam snippets are checked and cached the same way as Playground examples.
+
+Proceed the same way as with [Source 1. Playground Examples Catalog](#source-1-how-to-add-an-example-to-playground-examples-catalog) except:
+
+1. Use the directory `/learning/tour-of-beam/learning-content`.
+   It is recommended to follow the directory hierarchy as described in
+   [the learning content README](../learning/tour-of-beam/learning-content/README.md).
+2. Do not use the following attributes:
+   - `categories`
+   - `default_example`
+   - `tags`
+
+The ID of the snippet is a function of the SDK and the `name` attribute from its metadata:
+
+| SDK    | ID                          |
+|--------|-----------------------------|
+| Go     | TB_EXAMPLES_SDK_GO_name     |
+| Java   | TB_EXAMPLES_SDK_JAVA_name   |
+| Python | TB_EXAMPLES_SDK_PYTHON_name |
+
+For instance, for the Go the example `CSV` it is `TB_EXAMPLES_SDK_GO_CSV`.
+
+### Source 4. How to add a snippet with the app alone
+
+A code snippet can be saved to the Playground using **"Share my code"** button in the Playground:
+
+![Sharing code with the app alone](doc/load_your_code/images/share-my-code.png)
+
+This is easy and fast. It does not require any interaction with the Beam team.
+
+>**Share my code** considerations:
+> - A user-shared snippet is immutable.
+    If you edit the code and re-share, a new snippet and a new link will be generated.
+> - Playground automatically applies a 3-month retention policy to shared snippets that are not used.
+    To request a deletion of a snippet, please send an email to
+    [dev@beam.apache.org](mailto:dev@beam.apache.org?subject=[Playground]%20Delete%20a%20snippet)
+    with subject: *[Playground] Delete a snippet*.
+> - Playground does not cache output or graph for user-shared snippets.
+> - Playground does not verify user-shared snippets.
+
+### Source 5. How to load code from GitHub or another HTTPS URL
+
+Playground can load a snippet stored on an HTTPS server using the provided URL,
+including GitHub direct links to raw file content.
+
+This is as easy and fast as using *Share my code* button, but also allows you to modify a snippet
+after it is published without changing a link.
+
+>Loading snippet from HTTPS URL considerations:
+> - Playground does not cache output or graph for HTTPS URL snippets.
+> - Playground does not verify HTTPS URL snippets.
+
+#### Cross-Origin Resource Sharing
+
+For Playground to be able to load the snippet over HTTPS, the HTTPS server needs to allow
+the access by sending the following header:
+
+```
+Access-Control-Allow-Origin: *
+```
+
+at least when requested with `*.beam.apache.org` as
+[`referer`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referer).
+
+This is related to Cross-Origin Resource Sharing (CORS), to read more about CORS please see
+[CORS (Cross-Origin Resource Sharing)](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS).
+
+Many prefer to host code snippets in their GitHub repositories.
+GitHub is known to allow cross-origin access on direct links to raw file content.
+An example of loading a GitHub snippet:
+
+```
+https://play.beam.apache.org/?sdk=go&url=https://raw.githubusercontent.com/apache/beam-starter-go/main/main.go
+```
+
+## Step 3. Create a link or embed the snippet
+
+The snippet can now be shown in the Playground.
+Choose any of the following ways.
+
+### Link to a snippet
+
+#### Link to an example from the Playground Examples Catalog
+
+1. Open your snippet in the dropdown menu.
+2. Without changing it, click "Share my code".
+3. Copy the link.
+
+The link contains the `path` to your snippet in the database. It is in the following format:
+```
+https://play.beam.apache.org/?path=SDK_JAVA_MinimalWordCount&sdk=java
+```
+
+A special case is the default snippet for an SDK. It can be loaded by the following link:
+
+```
+https://play.beam.apache.org/?sdk=python&default=true
+```
+
+This way if another snippet is ever made default, the links you shared will lead
+to the new snippet.
+
+#### Link to an unlisted example
+
+Link to an unlisted example can be constructed by providing your snippet ID and SDK in the following URL:
+```
+https://play.beam.apache.org/?path=<ID>&sdk=<SDK>
+```
+
+The ID of the snippet is a function of the SDK and the `name` attribute from its metadata:
+
+| SDK    | ID              |
+|--------|-----------------|
+| Go     | SDK_GO_name     |
+| Java   | SDK_JAVA_name   |
+| Python | SDK_PYTHON_name |
+
+#### Link to a snippet from a Tour of Beam unit
+
+Link to a snippet can be constructed by providing your snippet ID and SDK in the following URL:
+```
+https://play.beam.apache.org/?path=<ID>&sdk=<SDK>
+```
+
+The ID of the snippet is a function of the SDK and the `name` attribute from its metadata:
+
+| SDK    | ID                          |
+|--------|-----------------------------|
+| Go     | TB_EXAMPLES_SDK_GO_name     |
+| Java   | TB_EXAMPLES_SDK_JAVA_name   |
+| Python | TB_EXAMPLES_SDK_PYTHON_name |
+
+For instance, for the Go the example `CSV` it is `TB_EXAMPLES_SDK_GO_CSV`, and the link is
+```
+https://play.beam.apache.org/?path=TB_EXAMPLES_SDK_GO_CSV&sdk=go
+```
+
+#### Link to a user-shared snippet
+
+You get the link when you click "Share my code" button. It is in the following format:
+
+```
+https://play.beam.apache.org/?sdk=java&shared=SNIPPET_ID
+```
+
+#### Link to a GitHub or other HTTPS URL snippet
+
+Add the URL to the `url` parameter, for example:
+
+```
+https://play.beam.apache.org/?sdk=go&url=https://raw.githubusercontent.com/apache/beam-starter-go/main/main.go
+```
+
+#### Link to an empty editor
+
+You can link to an empty editor to make your users start their snippets from scratch:
+
+```
+https://play.beam.apache.org/?sdk=go&empty=true
+```
+
+### Link to multiple snippets
+
+The above URLs load snippets that you want. But what happens if the user switches SDK? Normally this will be shown:
+- The catalog default example for the new SDK.
+- The empty editor for the new SDK if the Playground is embedded.
+
+This can be changed by linking to multiple examples, up to one per SDK.
+
+For this purpose, make a JSON array with any combination of parameters that
+are allowed for loading single examples, for instance:
+
+```json
+[
+  {
+    "sdk": "java",
+    "path": "SDK_JAVA_AggregationMax"
+  },
+  {
+    "sdk": "go",
+    "url": "https://raw.githubusercontent.com/apache/beam-starter-go/main/main.go"
+  }
+]
+```
+
+Then pass it in`examples` query parameter like this:
+
+`https://play.beam.apache.org/?sdk=go&examples=[{"sdk":"java","path":"SDK_JAVA_AggregationMax"},{"sdk":"go","url":"https://raw.githubusercontent.com/apache/beam-starter-go/main/main.go"}]`
+
+This starts with the Go example loaded from the URL.
+If SDK is then switched to Java, the `AggregationMax` catalog example is loaded for it.
+If SDK is switched to any other one, the default example for that SDK is loaded, because no override was provided.
+
+
+### Embedding a snippet into HTML
+
+Embedded Playground is a simplified interface of the Playground web app designed to be embedded
+into an `<iframe>` in web pages. It supports most of the Playground web app features.
+The embedded Playground URLs start with `https://play.beam.apache.org/embedded`
+and use the same query string parameters as the Playground web app.
+Additionally, the Embedded playground supports `editable=0` parameter to make the editor read-only.
+
+#### Embedding a snippet from Playground Examples Catalog
+
+1. Open your snippet in the dropdown menu.
+2. Without changing it, click "Share my code".
+3. Go to "Embed" tab.
+4. Copy the HTML code and add to your web page.
+
+#### Embedding a user-shared snippet
+
+1. Open your code by the link that you got when you shared it.
+2. Again click "Share my code".
+3. Go to "Embed" tab.
+4. Copy the HTML code and add to your web page.
+
+#### Embedding a snippet from other sources
+
+1. Follow the instructions to [get a link](#link-to-an-example-from-the-playground-examples-catalog) to your code.
+2. Optionally make the link to the Embedded Playground by replacing `play.beam.apache.org/?...`
+   with `play.beam.apache.org/embedded?...` because the embedded interface is simpler.
+3. Insert this link into an `<iframe>` HTML element as follows:
+
+```html
+<iframe
+  src="https://play.beam.apache.org/embedded?sdk=go&url=https://raw.githubusercontent.com/apache/beam-starter-go/main/main.go"
+  width="90%"
+  height="600px"
+  allow="clipboard-write"
+/>
+```
+
+### Embedding a snippet into the Apache Beam website
+
+Apache Beam website uses [Hugo Markdown preprocessor](https://gohugo.io).
+Custom Hugo shortcodes were added to Apache Beam website to embed Playground snippets.
+Use the custom shortcodes to embed Playground into the Apache Beam website:
+
+- `playground` shortcode, see [this comment](https://github.com/apache/beam/blob/master/website/www/site/layouts/shortcodes/playground.html) for a complete example.
+- `playground_snippet` shortcode, see [this comment](https://github.com/apache/beam/blob/master/website/www/site/layouts/shortcodes/playground_snippet.html) for all supported options.
+
+These shortcodes generate an `iframe` with the URLs described above.
+
+## Snippet view options
+
+If your code contains named sections as described in the
+[Step 1. Prepare a code snippet](#step-1-prepare-a-code-snippet),
+you can apply view options to those sections. Otherwise skip this.
+
+### Read-only sections
+
+Add `readonly` parameter with comma-separated section names:
+
+`https://play.beam.apache.org/?sdk=go&url=...&readonly=section_name`
+
+### Folding everything except named sections
+
+Add `unfold` parameter with comma-separated section names:
+
+`https://play.beam.apache.org/?sdk=go&url=...&unfold=section_name`
+
+This folds all foldable blocks that do not overlap with any of the given sections.
+
+### Hiding everything except a named section
+
+Add `show` parameter with a single section name:
+
+`https://play.beam.apache.org/?sdk=go&url=...&show=section_name`
+
+It is still the whole snippet that is sent for execution, although only the given section is visible.
+
+This also makes the editor read-only so the user cannot add code that conflicts with the hidden text.
diff --git a/playground/terraform/README.md b/playground/terraform/README.md
index a02ce35f595..0d072e6f9b0 100644
--- a/playground/terraform/README.md
+++ b/playground/terraform/README.md
@@ -150,3 +150,97 @@ NAME               AGE     STATUS
 GCP Project       time     Active
 ```
 3. Open Beam Playground frontend webpage in a web browser (e.g. https://playground.zone) to ensure that Playground frontend page is available
+
+# Use HELM to update environment
+Helm is responsible for deploying Beam Playground on the GKE cluster
+Use the steps below to update HELM Chart and apply it:
+
+1. Configure authentication for the Google Cloud Platform:
+```
+gcloud init
+```
+```
+gcloud auth application-default login
+```
+2. Run the following command to authenticate in the Docker registry:
+```
+gcloud auth configure-docker <region>-docker.pkg.dev
+```
+3. Run the following command to authenticate in GKE:
+```
+gcloud container clusters get-credentials --region <zone> <gke_name> --project <project_id>
+```
+4. Clone Apache BEAM repository from Git
+5. Make required changes in playground/infrastructure/helm-playground/values.yaml file:
+```
+replicaCount: 1
+image:
+   java_image: beam_playground-backend-java
+   go_image: beam_playground-backend-go
+   router_image: beam_playground-backend-router
+   scio_image: beam_playground-backend-scio
+   python_image: beam_playground-backend-python
+   frontend_image: beam_playground-frontend
+   pullPolicy: Always
+service:
+   type: NodePort
+   targetPort: 8080
+   port: 443
+healthcheck:
+   port: 8080
+   livInitialDelaySeconds: 30
+   livPeriodSeconds: 30
+   readInitialDelaySeconds: 30
+   readPeriodSeconds: 30
+autoscaling:
+   runners:
+ 	  maxReplicas: 4
+ 	  minReplicas: 2
+   rest:
+ 	  maxReplicas: 4
+ 	  minReplicas: 1
+   utilization:
+ 	memoryUtilization: 80
+ 	cpuUtilization: 95
+static_ip: <external IP address>
+redis_ip: <REDIS IP address>:6379
+project_id: <Project ID>
+registry: <region>-docker.pkg.dev/<project_id>/<repository_id>
+static_ip_name: <ip_address_name>
+tag: <docker-tag>
+datastore_name: <datastore-namespace>
+dns_name: <dns-name>
+func_clean: playground-function-cleanup-<env>
+func_put: playground-function-delete-<env>
+func_view: playground-function-view-<env>
+```
+6. Execute following command to update HELM:
+```
+helm upgrade playground /playground/infrastructure/helm-helm-playground
+```
+## Variables description for "values.yaml"
+See Also:
+
+[Services](https://cloud.google.com/kubernetes-engine/docs/concepts/service)
+
+[Configure Liveness, Readiness and Startup Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)
+
+[Cluster autoscaling](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler#:~:text=GKE's%20cluster%20autoscaler%20automatically%20resizes,minimum%20size%20that%20you%20designate.)
+
+
+|Variable                           |Description                                        |
+|-----------------------------------|:--------------------------------------------------|
+|service/type                       |connection type for service in the GKE Cluster     |
+|service/targetPort                 |Service target port, connection from Service to POD|
+|service/port                       |Service port, connection from ingress to service   |
+|healthcheck/port                   |Internal POD port for Healthcheck                  |
+|healthcheck/livInitialDelaySeconds |pre-polling delay                                  |
+|healthcheck/livPeriodSeconds       |Poll period                                        |
+|healthcheck/readInitialDelaySeconds|Check if POD is ready                              |
+|healthcheck/readPeriodSeconds      |Poll period for checking if POD is ready           |
+|autoscaling/runners/maxReplicas    |maximum number of PODs for a runner                |
+|autoscaling/runners/minReplicas    |minimum number of PODs for a runner                |
+|autoscaling/rest/maxReplicas       |max number of PODs per computer                    |
+|autoscaling/rest/minReplicas       |minimum number of PODs for a router                |
+|autoscaling/memoryUtilization      |POD scaling activation threshold based on RAM usage|
+|autoscaling/cpuUtilization         |POD scaling activation threshold based on CPU usage|
diff --git a/website/www/site/content/en/contribute/release-guide.md b/website/www/site/content/en/contribute/release-guide.md
index 48ac902fe83..6cf6630e222 100644
--- a/website/www/site/content/en/contribute/release-guide.md
+++ b/website/www/site/content/en/contribute/release-guide.md
@@ -1263,6 +1263,10 @@ Also, update [the Wikipedia article on Apache Beam](https://en.wikipedia.org/wik
 
 **********
 
+## 13. Update Beam Playground
+
+After new Beam Release is published Beam Playgorund can be updated according to [Playground Readme](https://github.com/apache/beam/blob/master/playground/TASKS.md#referenced-beam-sdk-update) and [Playground deployment guide](https://github.com/apache/beam/blob/master/playground/terraform/README.md).
+
 ## Improve the process
 
 It is important that we improve the release processes over time.
diff --git a/website/www/site/content/en/get-started/try-beam-playground.md b/website/www/site/content/en/get-started/try-beam-playground.md
index 354f9594a61..731d54cf5d4 100644
--- a/website/www/site/content/en/get-started/try-beam-playground.md
+++ b/website/www/site/content/en/get-started/try-beam-playground.md
@@ -32,78 +32,7 @@ You can try the available Apache Beam examples at
 {{< playground_snippet language="scio" path="SDK_SCIO_MinimalWordCount" >}}
 {{< /playground >}}
 
-## How To Add New Examples
-
-To add an Apache Beam example/test/kata into Beam Playground catalog,
-add the `beam-playground` tag into the file to be added.
-`beam-playground` tag is a yaml format comment:
-
-{{< highlight java >}}
-// beam-playground:
-//   name: Name of the example/test/kata
-//   description: Description of the example/test/kata
-//   multifile: false
-//   pipeline_options: --option1 value1 --option2 value2
-//   default_example: false
-//   context_line: 10
-//   categories:
-//     - category 1
-//     - category 2
-//     - category N
-
-// example code
-{{< /highlight >}}
-{{< highlight py >}}
-# beam-playground:
-#   name: Name of the example/test/kata
-#   description: Description of the example/test/kata
-#   multifile: false
-#   pipeline_options: --option1 value1 --option2 value2
-#   default_example: false
-#   context_line: 10
-#   categories:
-#     - category 1
-#     - category 2
-#     - category N
-
-# example code
-{{< /highlight >}}
-{{< highlight go >}}
-// beam-playground:
-//   name: Name of the example/test/kata
-//   description: Description of the example/test/kata
-//   multifile: false
-//   pipeline_options: --option1 value1 --option2 value2
-//   default_example: false
-//   context_line: 10
-//   categories:
-//     - category 1
-//     - category 2
-//     - category N
-
-// example code
-{{< /highlight >}}
-
-The 'beam-playground' tag consists of the following **required** elements:
-
-- `beam-playground` - tag title.
-- `name` - string field. Name of the Beam example/test/kata that will be displayed in the Beam Playground
-examples catalog.
-- `description` - string field. Description of the Beam example/test/kata that will be displayed in Beam Playground.
-- `multifile` - boolean field. Specifies if the given example consists of multiple files or not.
-- `pipeline_options` - string field (optional). Contains information about pipeline options of the Beam example/test/kata.
-- `default_example` - boolean field (optional). Specifies if the given example is default or not. If some example is tagged
-  as default it means that this example is shown when its SDK is chosen in Beam Playground.
-  Only one example can be set as a default for each SDK.
-- `context_line` - integer field. The line where the main part of the Beam example/test/kata begins.
-- `categories` - list type field.
-Lists categories this example is included into. Available categories are listed in
-[playground/categories.yaml](https://github.com/apache/beam/blob/master/playground/categories.yaml).
-If some Beam example/kata/test needs to add a new category, then please submit PR with the changes to `categories.yaml`.
-After the category has been added, it can be used in the examples.
-
-More details on examples in Apache Beam Playground can be found
-[here](https://docs.google.com/document/d/1LBeGVTYwJHYbtmLt06OjhBR1CJ1Wgz18MEZjvNkuofc/edit?usp=sharing).
+See [here](https://github.com/apache/beam/blob/master/playground/load_your_code.md) for adding new examples.
 
 ## Next Steps