You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "TSultanov (via GitHub)" <gi...@apache.org> on 2023/04/11 15:16:13 UTC
[GitHub] [beam] TSultanov commented on a diff in pull request #25507: Overview of adding snippets to Playground (#25506)
TSultanov commented on code in PR #25507:
URL: https://github.com/apache/beam/pull/25507#discussion_r1162977014
##########
playground/load_your_code.md:
##########
@@ -0,0 +1,562 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# How to Add an Example/Snippet/Learning Content into Apache Beam Playground
+
+Adding a new example, code snippet or Tour of Beam learning unit into the Playground is a three-step process:
+
+1. Prepare a code snippet.
+2. Add the code snippet to Apache Beam and/or Playground.
+3. Create a link to view the code snippet in Playground or to embed in a website page.
+
+
+Playground sources and output presentation formats:
+
+
+
+The guide will walk through all steps.
+
+
+# Table of Contents
+
+- [Step 1. Prepare a code snippet](#step-1-prepare-a-code-snippet)
+ * [Data access limitations](#data-access-limitations)
+ * [Emphasizing parts of code](#emphasizing-parts-of-code)
+ * [Named Sections](#named-sections)
+- [Step 2. Add the code snippet to the Apache Beam repo and/or Playground](#step-2-add-the-code-snippet-to-the-apache-beam-repo-andor-playground)
+ * [Source 1. How to add an example to Playground Examples Catalog](#source-1-how-to-add-an-example-to-playground-examples-catalog)
+ + [1. Add the file with example to a directory](#1-add-the-file-with-example-to-a-directory)
+ + [2. Add metadata to describe example](#2-add-metadata-to-describe-the-example)
+ - [Kafka emulator](#kafka-emulator)
+ + [3. Make a PR](#3-make-a-pr)
+ + [4. Save the snippet ID](#4-save-the-snippet-id)
+ * [Source 2. How to add an unlisted example to Playground](#source-2-how-to-add-an-unlisted-example-to-playground)
+ * [Source 3. How to add a Tour of Beam unit](#source-3-how-to-add-a-tour-of-beam-unit)
+ * [Source 4. How to add a snippet with the app alone](#source-4-how-to-add-a-snippet-with-the-app-alone)
+ * [Source 5. How to load code from GitHub or other HTTPS URL](#source-5-how-to-load-code-from-github-or-other-https-url)
+- [Step 3. Create a link or embed snippet](#step-3-create-a-link-or-embed-snippet)
+ * [Link to a snippet](#link-to-a-snippet)
+ + [Link to an example from the Playground Examples Catalog](#link-to-an-example-from-the-playground-examples-catalog)
+ + [Link to snippet (`unlisted` Playground Examples Catalog item)](#link-to-snippet---unlisted--playground-examples-catalog-item-)
+ + [Link to a Tour of Beam unit](#link-to-a-tour-of-beam-unit)
+ + [Link to a user-shared snippet](#link-to-a-user-shared-snippet)
+ + [Link to a GitHub or HTTPS URL snippet](#link-to-a-github-or-https-url-snippet)
+ + [Link to an empty editor](#link-to-an-empty-editor)
+ * [Embedding a snippet into HTML](#embedding-a-snippet-into-html)
+ + [Embedding a snippet from Playground Examples Catalog](#embedding-a-snippet-from-playground-examples-catalog)
+ + [Embedding a user-shared snippet](#embedding-a-user-shared-snippet)
+ + [Embedding a snippet from other sources](#embedding-a-snippet-from-other-sources)
+ * [Embedding a snippet into the Apache Beam website](#embedding-a-snippet-into-the-apache-beam-website)
+- [Snippet view options](#snippet-view-options)
+ * [Read-only sections](#read-only-sections)
+ * [Folding everything except named sections](#folding-everything-except-named-sections)
+ * [Hiding everything except a named section](#hiding-everything-except-a-named-section)
+ * [Linking to multiple examples](#linking-to-multiple-examples)
+
+
+## Step 1. Prepare a code snippet
+
+Playground runs example code snippets using Apache Beam Direct Runner
+and requires that a code snippet is a complete runnable code.
+
+### Data access limitations
+
+For security reasons, the snippets in Playground cannot access arbitrary internet resources.
+Depending on your data source you should do the following to access data:
+
+| Source | Notes |
+|----------|-----------------------------------------------------------|
+| File | Put to a GCS bucket in `apache-beam-testing` project. |
+| BigQuery | The table must be added to `apache-beam-testing` project. |
+
+Machine Learning: PyTorch, SKLearn -- change the playground itself. (request for enhancement to add a dependency to the Python image).
+To request adding, please send an email to dev@...
+
+CLONING 3RD PARTY REPOS.
+IF ML MODELS -- FROM PLAYGROUND BEAM ML EXAMPLES SUPPORT
+
+### Emphasizing parts of code
+
+Playground provides multiple features to help focus users on certain parts of the code.
+
+Playground automatically applies the following to all snippets:
+- Folds a comment if a snippet starts with one.
+- Folds imports.
+
+### Named Sections
+
+Playground supports *Named Sections* to tag code blocks and provide the following view options:
+- Fold all blocks except tagged code blocks.
+ This can be useful to help user focus on specific code blocks and features presented in a snippet.
+- Hide all code except tagged code blocks.
+ This can be useful to create runnable snippets illustrating specific concepts or transforms,
+ and hide all non-essential code blocks.
+ Such snippet can be embedded on a website to make examples in documentation and tutorials runnable.
+- Make certain code parts read-only.
+ This feature can be useful to create learning units where user modifications are desired
+ only in certain parts of the code.
+Please see [Snippet View Options](#snippet-view-options) section for details how different view options can be used.
+
+If you do not need any of those view options, skip to the [next step](#step-2-add-the-code-snippet-to-the-apache-beam-repo-and-playground).
+
+*Named Sections* are defined with the following syntax:
+
+```
+// [START section_name]
+void method() {
+...
+}
+// [END section_name]
+```
+
+Create a named section for each part of your code that you want the above features for.
+To learn more details about the syntax please see
+the [README of the editor](https://pub.dev/packages/flutter_code_editor) that Playground uses.
+
+## Step 2. Add the code snippet to the Apache Beam repo and/or Playground
+
+There are several types of code snippets in the Playground:
+
+1. Public Example — a code snippet displayed in the Playground Examples Catalog.
+ See [how to add a new public example here](#source-1-how-to-add-an-example-to-playground-examples-catalog).
+2. Unlisted Example — the same as a public example, but it does not show in the example dropdown
+ and can only be accessed by direct linking. These are typically embedded on a website.
+ See [how to add a new unlisted example here](#source-2-how-to-add-an-unlisted-example-to-playground).
+3. [Tour of Beam](https://github.com/apache/beam/tree/master/learning/tour-of-beam) learning unit.
+ See [how to add a new Tour of Beam unit here](#source-3-how-to-add-a-tour-of-beam-unit).
+4. User-shared code snippets do not require a PR and should be used for code
+ not displayed on Beam resources.
+ See [how to add a snippet with the app alone here](#source-4-how-to-add-a-snippet-with-the-app-alone).
+5. GitHub or other HTTPS URL sources.
+ See [how to load a snippet from external GitHub or other HTTPS URL here](#source-5-how-to-load-code-from-github-or-other-https-url).
+
+See the [workflow above](#how-to-add-an-examplesnippetlearning-content-into-apache-beam-playground) how artifacts map to these sources.
+
+### Source 1. How to add an example to Playground Examples Catalog
+
+Playground Examples Catalog helps users discover example snippets
+and is a recommended way to add examples. Playground automatically scans,
+verifies and deploys example snippets from the directories listed below.
+
+#### 1. Add the file with example to a directory
+
+Playground Java, Python, and Go examples are automatically picked from these
+predefined directories by the `playground_examples_ci.yml` GitHub workflow
+after a PR is merged to Beam repo:
+- `/examples`
+- `/learning/katas`
+- `/sdks`.
+
+Adding Scala example snippets automatically is not supported,
+and Scala example snippets can be added to the catalog manually.
+
+#### 2. Add metadata to describe the example
+
+Playground relies on metadata comments block to identify and place an example into the database,
+and present in the Examples Catalog.
+See [this](https://github.com/apache/beam/blob/3e080ff212d8ed7208c8486b515bb73c5d294475/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java#L20-L36) for an example.
+Playground automatically removes metadata comments block before storing the example in database
+so the metadata is not visible to end users. The block is in the format of a YAML map.
+
+Example tag metadata quick reference (all elements are **required**, unless marked **optional**):
+
+| Field | Description |
+|--------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `beam-playground:` | tag title |
+| `name` | string. Name of the Beam example that will be displayed in the Playground Examples Catalog. |
+| `description` | string. Description of the Beam example that will be displayed in the Playground Examples Catalog. |
+| `pipeline_options` | string (optional). Contains information about pipeline options of the Beam example/test/kata. |
+| `context_line` | integer. The line number to scroll to when the snippet is loaded. Note that lines of the metadata block are cut so line numbers after it are shifted. |
+| `categories` | list of strings. Lists categories this example is included into. Non-existent categories will be created. |
+| `tags` | list of strings (optional). Tags by which this snippet can be found in the Example Catalog. |
+| `complexity` | BASIC/MEDIUM/ADVANCED enum. Helps user to identify example's complexity. |
+| `default_example` | boolean (optional). Specifies the example to be loaded as default when its SDK selected in the Playground. Only one example can be set as the default for each SDK. |
+| `url_notebook` | string (optional). If the snippet has a Colab notebook, can link the URL of the Colab notebook that is based on this snippet. |
+| `multifile` | boolean (optional). Specifies if the given example consists of multiple files or not. Default: `false`. |
+| `always_run` | boolean (optional). Specifies example caching by Playground. Default: `false` (the output is cached). |
+| `datasets` | enum (optional). Datasets which will be used by emulators. Please see `Dataset` in Tag class model [here](infrastructure/models.py) for reference. |
Review Comment:
I think we should refer to the "Kafka emulator" section here for clarification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org