You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by yh...@apache.org on 2026/08/19 17:25:11 UTC
(beam) branch master updated: Update SKILLs based on review practice (#39805)
This is an automated email from the ASF dual-hosted git repository.
Abacn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 94510075efe Update SKILLs based on review practice (#39805)
94510075efe is described below
commit 94510075efe1ea52c9f2a3dcc42ab8fc0a54b9e2
Author: Yi Hu <ya...@google.com>
AuthorDate: Wed Aug 19 13:25:04 2026 -0400
Update SKILLs based on review practice (#39805)
* Update SKILLs based on review practice
* Add a Protos section in Apache Beam Core Concepts. Protos need
special attention and avoid arbitrary additions
* Add a recommendation for documentation and comment additions
* Removed Java/Python stuff from proto SKILLs
---
.agent/skills/beam-concepts/SKILL.md | 31 +++++++++++++++++++++++++++++++
.agent/skills/contributing/SKILL.md | 3 +++
2 files changed, 34 insertions(+)
diff --git a/.agent/skills/beam-concepts/SKILL.md b/.agent/skills/beam-concepts/SKILL.md
index da3dd9fbf31..2c8c3995671 100644
--- a/.agent/skills/beam-concepts/SKILL.md
+++ b/.agent/skills/beam-concepts/SKILL.md
@@ -25,6 +25,37 @@ description: Explains core Apache Beam programming model concepts including PCol
## The Beam Model
Evolved from Google's MapReduce, FlumeJava, and Millwheel projects. Originally called the "Dataflow Model."
+### Protos
+
+The `/model` directory defines the official, language-agnostic Protocol Buffer (`.proto`) and gRPC service specifications that establish the **Beam Model** and the **Beam Portability Framework**.
+
+#### Why `/model` Exists (Portability & Decoupling)
+
+Without a standardized model representation, supporting $N$ SDK languages across $M$ execution runners would require $N \times M$ separate translation layers. By defining all core pipeline concepts, data encodings, metrics, and worker RPC protocols as Protobuf messages and gRPC services, `/model` acts as the universal lingua franca:
+
+* **SDKs** compile user pipelines into standardized Runner API protobuf graphs.
+
+* **Runners** inspect, optimize, and distribute these graphs without needing SDK-specific language runtimes.
+
+* **Workers (SDK Harnesses)** execute user code (`DoFn`s) and communicate with runners over standardized Fn API gRPC channels.
+
+#### Core Directories & What They Do
+
+1. **`/model/pipeline` (Runner API & Core Model)**: Defines the SDK- and runner-independent representation of pipelines (`Pipeline`, `Components`, `PTransform`, `PCollection`, `Coder`), timestamps/constants, Beam Schemas (`Row`, `Field`), and execution metrics (`MonitoringInfo`).
+2. **`/model/fn-execution` (Fn API & Provisioning)**: Defines bidirectional gRPC services between runners and worker SDK harnesses for bundle execution (`Control`), element streaming (`Data`), state/timer access (`State`), log forwarding (`Logging`), and container initialization (`Provisioning`).
+3. **`/model/job-management` (Job, Expansion, & Artifact APIs)**: Defines gRPC interfaces for submitting and monitoring jobs on remote servers (`JobService`), resolving cross-language transforms in remote SDKs (`ExpansionService`), and staging dependency artifacts or container images (`ArtifactService`).
+4. **`/model/interactive` (Interactive API)**: Defines metadata and stream headers for recording and replaying data in Interactive Beam notebooks.
+
+#### What Agents Need to Pay Special Attention To
+
+* **Conservative Proto Changes**: Proto changes are generally conservative and accepted only when there is a compelling reason and community consensus. Every addition introduces a new obligation that each SDK and runner must support; adding new Beam model elements (portable types, capabilities) increases the compatibility gap between SDK and runner capabilities.
+* **URNs Are the API Contract**: Transforms, coders, windowing strategies, environments, and metrics are bound together by standardized string URNs (e.g., `beam:transform:pardo:v1`, `beam:coder:bytes:v1`). When inspecting or creating transforms across languages, always verify URN mappings and registry handlers in both the SDK and Runner runtimes.
+* **Strict Backward & Wire Compatibility**:
+ * Never renumber, delete, or modify existing field IDs or URN strings in `.proto` files, as they are used across distributed RPC boundaries and persisted checkpoints.
+* **Build System & Naming Collisions**:
+ * Modifying files in `/model` requires re-generating language bindings (e.g., `./gradlew :model:pipeline:generateProto`).
+ * Avoid protobuf field names that conflict with reserved keywords in target languages (e.g., `class` in Java or `output` in Python, as noted in `beam_fn_api.proto` comments).
+
## Key Abstractions
### Pipeline
diff --git a/.agent/skills/contributing/SKILL.md b/.agent/skills/contributing/SKILL.md
index bac50c5d0cd..2a60c5a3cd9 100644
--- a/.agent/skills/contributing/SKILL.md
+++ b/.agent/skills/contributing/SKILL.md
@@ -66,6 +66,9 @@ description: Guides the contribution workflow for Apache Beam, including creatin
- Every source file needs Apache license header
- New dependencies must have Apache-compatible open source licenses
- Add unit tests for your changes
+- Document public-facing APIs in a standard way (Javadoc, docstrings/pydoc).
+ - Documentation should be user-friendly, informative.
+ - Implementation details belong in inline code comments.
- Use descriptive commit messages
### 5. Create Pull Request