You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/05/03 04:08:01 UTC
[PR] docs: rewrite AGENTS.md as agent-facing index [texera]
Yicong-Huang opened a new pull request, #4825:
URL: https://github.com/apache/texera/pull/4825
## What changes were proposed in this PR?
Rewrite `AGENTS.md` so it works as a slim, index-style guide for coding agents instead of duplicating contributor docs.
- Architecture map: services + Amber breakdown.
- Topic redirects: contribution / lint / format / testing / license header now point to `CONTRIBUTING.md`, `SECURITY.md`, `.github/PULL_REQUEST_TEMPLATE`, the issue templates, `.licenserc.yaml`, and per-module READMEs.
- Agent-specific rules consolidated: scope/safety, worktree-based dev, short Conventional-Commits branch and commit naming, issue-first workflow, brevity for issues and PRs (incl. required frontend before/after screenshots), test-driven development, and CI label gating (label → stacks via `LABEL_STACKS` in `required-checks.yml`).
## Any related issues, documentation, discussions?
Closes #4823
## How was this PR tested?
Documentation-only change. Reviewed the rendered Markdown locally; verified all in-repo links resolve.
## Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: rewrite AGENTS.md as agent-facing index [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang merged PR #4825:
URL: https://github.com/apache/texera/pull/4825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: rewrite AGENTS.md as agent-facing index [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #4825:
URL: https://github.com/apache/texera/pull/4825#discussion_r3177644777
##########
AGENTS.md:
##########
@@ -1,235 +1,187 @@
# AGENTS.md
-Guidance for coding agents working in the Apache Texera repository.
-
-## Project Overview
-
-Apache Texera is a collaborative data science and AI/ML workflow system. The
-repo is a multi-language monorepo with Scala/sbt backend services, Python worker
-runtime code, an Angular frontend, and a TypeScript/Bun agent service.
-
-Major areas:
-
-- `amber/`: workflow execution engine, Scala tests, Python worker runtime, and
- Python operator dependencies.
-- `common/`: shared Scala modules for auth, config, DAO, workflow core,
- workflow operators, and Python-template building.
-- `config-service/`, `access-control-service/`, `file-service/`,
- `computing-unit-managing-service/`, `workflow-compiling-service/`: backend
- services wired through `build.sbt`.
-- `frontend/`: Angular application. Uses Yarn 4.14.1, Node >= 20.19.0, Nx,
- Prettier, ESLint, Karma/Jasmine, and ng-zorro.
-- `agent-service/`: TypeScript Elysia service for Texera LLM agents. CI uses
- Bun 1.3.3.
-- `pyright-language-service/`: TypeScript service for Python language support.
-- `sql/`: database DDL used by local runs and CI.
-- `bin/`: shell scripts and Dockerfiles for services, local deployment, and
- generated protobuf assets.
-
-## Ground Rules
-
-- Keep changes narrowly scoped. Do not rewrite unrelated files or move code
- between services unless the task explicitly requires it.
-- Preserve local user changes. Check `git status --short` before editing and do
- not revert unrelated dirty files.
-- Follow existing module boundaries and naming patterns. Prefer local helpers
- and service abstractions over introducing new framework-level utilities.
-- Add or update tests when behavior changes. For small UI-only fixes where unit
- tests are not practical, document the manual test steps.
-- Never commit secrets, local config, generated build output, caches, or binary
- artifacts. Examples to avoid include `python_udf.conf`, `.env` files, `target/`,
- `dist/`, `.pytest_cache/`, `.ruff_cache/`, and local logs.
-
-## Licensing
-
-- New source/config files should include the Apache 2.0 ASF license header unless
- `.licenserc.yaml` excludes that file type or path.
-- Markdown files are excluded from the license-header check.
-- Keep third-party/vendored-code attribution intact. `common/workflow-operator`
- has special license handling in `project/AddMetaInfLicenseFiles.scala`.
-- GitHub Actions in ASF repositories should use approved actions and preferably
- pinned SHAs, matching the existing workflow style.
-
-## Scala / Backend
-
-- Scala version: 2.13.18.
-- Java in CI: Temurin JDK 11.
-- Formatting: `.scalafmt.conf` uses scalafmt 2.6.4 with `maxColumn = 100`.
-- Lint rules live in `.scalafix.conf` and include `ProcedureSyntax` and
- `RemoveUnused`.
-
-Useful root commands:
+## Architecture Map
-```bash
-sbt scalafmtCheckAll
-sbt scalafmtAll
-sbt "scalafixAll --check"
-sbt scalafixAll
-sbt clean package
-sbt test
-```
+Apache Texera: Scala/sbt backend services + the Amber workflow execution
+engine, an Angular UI, and helper TypeScript services. JVM modules wired in
Review Comment:
Done in 09cd7ec665 — intro now reads "…an Angular UI, and the agent service."
##########
AGENTS.md:
##########
@@ -1,235 +1,187 @@
# AGENTS.md
-Guidance for coding agents working in the Apache Texera repository.
-
-## Project Overview
-
-Apache Texera is a collaborative data science and AI/ML workflow system. The
-repo is a multi-language monorepo with Scala/sbt backend services, Python worker
-runtime code, an Angular frontend, and a TypeScript/Bun agent service.
-
-Major areas:
-
-- `amber/`: workflow execution engine, Scala tests, Python worker runtime, and
- Python operator dependencies.
-- `common/`: shared Scala modules for auth, config, DAO, workflow core,
- workflow operators, and Python-template building.
-- `config-service/`, `access-control-service/`, `file-service/`,
- `computing-unit-managing-service/`, `workflow-compiling-service/`: backend
- services wired through `build.sbt`.
-- `frontend/`: Angular application. Uses Yarn 4.14.1, Node >= 20.19.0, Nx,
- Prettier, ESLint, Karma/Jasmine, and ng-zorro.
-- `agent-service/`: TypeScript Elysia service for Texera LLM agents. CI uses
- Bun 1.3.3.
-- `pyright-language-service/`: TypeScript service for Python language support.
-- `sql/`: database DDL used by local runs and CI.
-- `bin/`: shell scripts and Dockerfiles for services, local deployment, and
- generated protobuf assets.
-
-## Ground Rules
-
-- Keep changes narrowly scoped. Do not rewrite unrelated files or move code
- between services unless the task explicitly requires it.
-- Preserve local user changes. Check `git status --short` before editing and do
- not revert unrelated dirty files.
-- Follow existing module boundaries and naming patterns. Prefer local helpers
- and service abstractions over introducing new framework-level utilities.
-- Add or update tests when behavior changes. For small UI-only fixes where unit
- tests are not practical, document the manual test steps.
-- Never commit secrets, local config, generated build output, caches, or binary
- artifacts. Examples to avoid include `python_udf.conf`, `.env` files, `target/`,
- `dist/`, `.pytest_cache/`, `.ruff_cache/`, and local logs.
-
-## Licensing
-
-- New source/config files should include the Apache 2.0 ASF license header unless
- `.licenserc.yaml` excludes that file type or path.
-- Markdown files are excluded from the license-header check.
-- Keep third-party/vendored-code attribution intact. `common/workflow-operator`
- has special license handling in `project/AddMetaInfLicenseFiles.scala`.
-- GitHub Actions in ASF repositories should use approved actions and preferably
- pinned SHAs, matching the existing workflow style.
-
-## Scala / Backend
-
-- Scala version: 2.13.18.
-- Java in CI: Temurin JDK 11.
-- Formatting: `.scalafmt.conf` uses scalafmt 2.6.4 with `maxColumn = 100`.
-- Lint rules live in `.scalafix.conf` and include `ProcedureSyntax` and
- `RemoveUnused`.
-
-Useful root commands:
+## Architecture Map
-```bash
-sbt scalafmtCheckAll
-sbt scalafmtAll
-sbt "scalafixAll --check"
-sbt scalafixAll
-sbt clean package
-sbt test
-```
+Apache Texera: Scala/sbt backend services + the Amber workflow execution
+engine, an Angular UI, and helper TypeScript services. JVM modules wired in
+[`build.sbt`](build.sbt).
-Targeted tests are preferred while iterating. Examples:
+| Area | Path | Detail |
+| --- | --- | --- |
+| Workflow execution engine (Amber) | `amber/` | [amber/README.md](amber/README.md) |
+| Backend services | `config-service/`, `access-control-service/`, `file-service/`, `computing-unit-managing-service/`, `workflow-compiling-service/` | `build.sbt` |
+| Shared Scala libs | `common/` (`auth`, `config`, `dao`, `workflow-core`, `workflow-operator`, `pybuilder`) | `build.sbt` |
+| Frontend (Angular) | `frontend/` | [frontend/README.md](frontend/README.md) |
+| Agent service (Bun/TS, LLM agents) | `agent-service/` | `agent-service/package.json` |
+| Pyright language service | `pyright-language-service/` | [pyright-language-service/README.md](pyright-language-service/README.md) |
+| Deploy scripts / Dockerfiles | `bin/` | [README](bin/README.md) / [k8s](bin/k8s/README.md) / [single-node](bin/single-node/README.md) |
+| DDL, sbt plugins | `sql/`, `project/` | files therein |
-```bash
-sbt "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.e2e.ReconfigurationSpec"
-sbt "WorkflowCompilingService/testOnly *SomeSpec"
-```
+### Amber breakdown
-CI creates PostgreSQL databases from:
+| Path | Role |
+| --- | --- |
+| `amber/src/main/scala` | Pekko actors, scheduler, reconfiguration, fault tolerance, gRPC/proto |
+| `amber/src/main/python/pyamber` | Python engine (`pyamber`) — bridge to the Scala engine |
+| `amber/src/main/python/pytexera` | Python operator SDK exposed to UDFs |
-```bash
-psql -h localhost -U postgres -f sql/texera_ddl.sql
-psql -h localhost -U postgres -f sql/iceberg_postgres_catalog.sql
-psql -h localhost -U postgres -f sql/texera_lakefs.sql
-psql -h localhost -U postgres -v DB_NAME=texera_db_for_test_cases -f sql/texera_ddl.sql
+## Where Things Live
+
+| Topic | Source of truth |
+| --- | --- |
+| Contribution / PR / lint / format / testing / license header | [CONTRIBUTING.md](CONTRIBUTING.md) |
+| Reporting security issues | [SECURITY.md](SECURITY.md) |
+| PR template | [.github/PULL_REQUEST_TEMPLATE](.github/PULL_REQUEST_TEMPLATE) |
+| Issue templates | [bug](.github/ISSUE_TEMPLATE/bug-template.yaml) / [task](.github/ISSUE_TEMPLATE/task-template.yaml) / [feature](.github/ISSUE_TEMPLATE/feature-template.yaml) |
+| License-header coverage; vendored `workflow-operator` | [.licenserc.yaml](.licenserc.yaml); [project/AddMetaInfLicenseFiles.scala](project/AddMetaInfLicenseFiles.scala) |
+| Local single-node / k8s deploy | [single-node](bin/single-node/README.md), [k8s](bin/k8s/README.md) |
+
+If a topic is above, **read that file** instead of asking here.
+
+## Agent-Specific Rules
+
+### Scope and safety
+
+- Narrowly scoped changes. No unrelated rewrites or cross-service moves.
+- `git status --short` before editing; don't revert unrelated dirty files.
+- Never commit secrets / local config / build output / caches / binaries
+ (`python_udf.conf`, `.env`, `target/`, `dist/`, `.pytest_cache/`,
+ `.ruff_cache/`, logs).
+
+### Develop in a worktree
+
+Leave `texera/` on `main`. One worktree per PR, branched off a freshly
+fetched `upstream/main`.
+
+```
+texera/ # stays on main, never dirty
+texera-worktrees/<branch>/ # one worktree per PR
```
-## Python Runtime
+Reset to `upstream/main` at start; `git log upstream/main..HEAD` should
+contain only this PR's commits before pushing; remove the worktree after
+merge.
-Python worker code lives primarily under `amber/src/main/python`.
+### Environment
-- Supported CI Python versions: 3.10, 3.11, 3.12, 3.13.
-- Ruff config is in `amber/src/main/python/pyproject.toml`.
-- Ruff line length is 88 and target version is `py310`.
-- Generated protobuf code under `amber/src/main/python/proto` is excluded from
- Ruff.
+| Component | Version |
+| --- | --- |
+| Java | JDK 11 |
+| Scala | 2.13 |
+| Python | 3.12 |
+| Node | 24 |
-Useful commands:
+One Python venv shared across worktrees, sibling of the texera checkout:
+
+```
+<workspace>/
+├── texera/ # main checkout
+├── texera-worktrees/<br>/ # per-PR worktrees
+└── venv312/ # shared Python 3.12 venv
+```
```bash
-cd amber/src/main/python
-ruff check .
-ruff format --check .
-pytest -sv
-python -m pytest core/runnables/test_main_loop.py -v
+python3.12 -m venv ../venv312 && source ../venv312/bin/activate
+pip install -r amber/requirements.txt -r amber/operator-requirements.txt
```
-Install dependencies from `amber/requirements.txt` and
-`amber/operator-requirements.txt` when running the Python runtime or tests
-outside CI.
+Tests that spawn Python workers need an interpreter path. Edit `python.path`
+in [`udf.conf`](common/config/src/main/resources/udf.conf) or
+`export UDF_PYTHON_PATH="$(pwd)/../venv312/bin/python"` (env var overrides).
+Without it, `sbt` Python-integration tests fail to launch a worker.
-## Frontend
+### Branch and commit naming
-The Angular frontend lives in `frontend/`.
+Short, **Conventional Commits**, same shape for branch and commit subject.
-- Node engine: `>=20.19.0`.
-- Package manager: Yarn 4.14.1 via Corepack.
-- Formatting is Prettier plus prettier-eslint. Prettier uses 2 spaces,
- semicolons, double quotes, `printWidth: 120`, and LF endings.
-- Unit tests are Karma/Jasmine. Specs should live next to frontend code as
- `.spec.ts` files.
+| Kind | Branch | Commit |
+| --- | --- | --- |
+| Feature | `feat/agent-workflow-edit` | `feat(agent-service): enable workflow edit` |
+| Bug fix | `fix/marker-replay` | `fix(amber): marker replay during reconfiguration` |
+| Tests | `test/pyamber-handlers` | `test(pyamber): add handler unit tests` |
+| Chore | `chore/angular-21` | `chore(deps): upgrade frontend to Angular 21` |
+| CI | `ci/cache-action-bump` | `ci: bump coursier/cache-action to v8.1.0` |
-Useful commands:
+Both ≤ ~60 chars; scope is a module (`amber`, `pyamber`, `frontend`,
+`agent-service`, `file-service`, …) — not `amber-python`. No
+`Co-authored-by:` trailer for the repo owner.
-```bash
-cd frontend
-corepack enable
-corepack prepare yarn@4.14.1 --activate
-yarn install --immutable --inline-builds --network-timeout=100000
-yarn format:ci
-yarn format:fix
-yarn lint
-yarn test --watch=false
-yarn test:ci
-yarn build:ci
-yarn start
+### Issues and PRs
+
+Issue-first; both stay short.
+
+```
+issue (template + Type) -> PR (Closes #N, template) -> review -> merge
```
-For UI changes, include screenshots/GIFs or clear manual verification steps in
-the PR description when the behavior is visual or interactive.
+- Every change starts as an issue (minor typo / docs excepted). File against
+ `apache/texera`, never a fork.
+- Pick the right template **and** set the GitHub Issue **Type** explicitly
+ (`Bug` / `Task` / `Feature`); the template's `type:` frontmatter doesn't
+ always apply on creation.
+- Reference the issue: `Closes #N` (or `Fixes` / `Resolves`, or "related to").
+- Issue titles are **plain prose** — Conventional Commits is for commits and
+ PR titles only.
+- Task issues match `task-template.yaml` exactly: Task Summary + Task Type.
+ No priority / proposed step / code blocks.
Review Comment:
Done in 09cd7ec665. Bullet now just says `Task issues match `task-template.yaml` exactly.` — no field list, no prohibitions.
##########
AGENTS.md:
##########
@@ -1,235 +1,187 @@
# AGENTS.md
-Guidance for coding agents working in the Apache Texera repository.
-
-## Project Overview
-
-Apache Texera is a collaborative data science and AI/ML workflow system. The
-repo is a multi-language monorepo with Scala/sbt backend services, Python worker
-runtime code, an Angular frontend, and a TypeScript/Bun agent service.
-
-Major areas:
-
-- `amber/`: workflow execution engine, Scala tests, Python worker runtime, and
- Python operator dependencies.
-- `common/`: shared Scala modules for auth, config, DAO, workflow core,
- workflow operators, and Python-template building.
-- `config-service/`, `access-control-service/`, `file-service/`,
- `computing-unit-managing-service/`, `workflow-compiling-service/`: backend
- services wired through `build.sbt`.
-- `frontend/`: Angular application. Uses Yarn 4.14.1, Node >= 20.19.0, Nx,
- Prettier, ESLint, Karma/Jasmine, and ng-zorro.
-- `agent-service/`: TypeScript Elysia service for Texera LLM agents. CI uses
- Bun 1.3.3.
-- `pyright-language-service/`: TypeScript service for Python language support.
-- `sql/`: database DDL used by local runs and CI.
-- `bin/`: shell scripts and Dockerfiles for services, local deployment, and
- generated protobuf assets.
-
-## Ground Rules
-
-- Keep changes narrowly scoped. Do not rewrite unrelated files or move code
- between services unless the task explicitly requires it.
-- Preserve local user changes. Check `git status --short` before editing and do
- not revert unrelated dirty files.
-- Follow existing module boundaries and naming patterns. Prefer local helpers
- and service abstractions over introducing new framework-level utilities.
-- Add or update tests when behavior changes. For small UI-only fixes where unit
- tests are not practical, document the manual test steps.
-- Never commit secrets, local config, generated build output, caches, or binary
- artifacts. Examples to avoid include `python_udf.conf`, `.env` files, `target/`,
- `dist/`, `.pytest_cache/`, `.ruff_cache/`, and local logs.
-
-## Licensing
-
-- New source/config files should include the Apache 2.0 ASF license header unless
- `.licenserc.yaml` excludes that file type or path.
-- Markdown files are excluded from the license-header check.
-- Keep third-party/vendored-code attribution intact. `common/workflow-operator`
- has special license handling in `project/AddMetaInfLicenseFiles.scala`.
-- GitHub Actions in ASF repositories should use approved actions and preferably
- pinned SHAs, matching the existing workflow style.
-
-## Scala / Backend
-
-- Scala version: 2.13.18.
-- Java in CI: Temurin JDK 11.
-- Formatting: `.scalafmt.conf` uses scalafmt 2.6.4 with `maxColumn = 100`.
-- Lint rules live in `.scalafix.conf` and include `ProcedureSyntax` and
- `RemoveUnused`.
-
-Useful root commands:
+## Architecture Map
-```bash
-sbt scalafmtCheckAll
-sbt scalafmtAll
-sbt "scalafixAll --check"
-sbt scalafixAll
-sbt clean package
-sbt test
-```
+Apache Texera: Scala/sbt backend services + the Amber workflow execution
+engine, an Angular UI, and helper TypeScript services. JVM modules wired in
+[`build.sbt`](build.sbt).
-Targeted tests are preferred while iterating. Examples:
+| Area | Path | Detail |
+| --- | --- | --- |
+| Workflow execution engine (Amber) | `amber/` | [amber/README.md](amber/README.md) |
+| Backend services | `config-service/`, `access-control-service/`, `file-service/`, `computing-unit-managing-service/`, `workflow-compiling-service/` | `build.sbt` |
+| Shared Scala libs | `common/` (`auth`, `config`, `dao`, `workflow-core`, `workflow-operator`, `pybuilder`) | `build.sbt` |
+| Frontend (Angular) | `frontend/` | [frontend/README.md](frontend/README.md) |
+| Agent service (Bun/TS, LLM agents) | `agent-service/` | `agent-service/package.json` |
+| Pyright language service | `pyright-language-service/` | [pyright-language-service/README.md](pyright-language-service/README.md) |
+| Deploy scripts / Dockerfiles | `bin/` | [README](bin/README.md) / [k8s](bin/k8s/README.md) / [single-node](bin/single-node/README.md) |
+| DDL, sbt plugins | `sql/`, `project/` | files therein |
-```bash
-sbt "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.e2e.ReconfigurationSpec"
-sbt "WorkflowCompilingService/testOnly *SomeSpec"
-```
+### Amber breakdown
-CI creates PostgreSQL databases from:
+| Path | Role |
+| --- | --- |
+| `amber/src/main/scala` | Pekko actors, scheduler, reconfiguration, fault tolerance, gRPC/proto |
+| `amber/src/main/python/pyamber` | Python engine (`pyamber`) — bridge to the Scala engine |
+| `amber/src/main/python/pytexera` | Python operator SDK exposed to UDFs |
-```bash
-psql -h localhost -U postgres -f sql/texera_ddl.sql
-psql -h localhost -U postgres -f sql/iceberg_postgres_catalog.sql
-psql -h localhost -U postgres -f sql/texera_lakefs.sql
-psql -h localhost -U postgres -v DB_NAME=texera_db_for_test_cases -f sql/texera_ddl.sql
+## Where Things Live
+
+| Topic | Source of truth |
+| --- | --- |
+| Contribution / PR / lint / format / testing / license header | [CONTRIBUTING.md](CONTRIBUTING.md) |
+| Reporting security issues | [SECURITY.md](SECURITY.md) |
+| PR template | [.github/PULL_REQUEST_TEMPLATE](.github/PULL_REQUEST_TEMPLATE) |
+| Issue templates | [bug](.github/ISSUE_TEMPLATE/bug-template.yaml) / [task](.github/ISSUE_TEMPLATE/task-template.yaml) / [feature](.github/ISSUE_TEMPLATE/feature-template.yaml) |
+| License-header coverage; vendored `workflow-operator` | [.licenserc.yaml](.licenserc.yaml); [project/AddMetaInfLicenseFiles.scala](project/AddMetaInfLicenseFiles.scala) |
+| Local single-node / k8s deploy | [single-node](bin/single-node/README.md), [k8s](bin/k8s/README.md) |
+
+If a topic is above, **read that file** instead of asking here.
+
+## Agent-Specific Rules
+
+### Scope and safety
+
+- Narrowly scoped changes. No unrelated rewrites or cross-service moves.
+- `git status --short` before editing; don't revert unrelated dirty files.
+- Never commit secrets / local config / build output / caches / binaries
+ (`python_udf.conf`, `.env`, `target/`, `dist/`, `.pytest_cache/`,
+ `.ruff_cache/`, logs).
+
+### Develop in a worktree
+
+Leave `texera/` on `main`. One worktree per PR, branched off a freshly
+fetched `upstream/main`.
+
+```
+texera/ # stays on main, never dirty
+texera-worktrees/<branch>/ # one worktree per PR
```
-## Python Runtime
+Reset to `upstream/main` at start; `git log upstream/main..HEAD` should
+contain only this PR's commits before pushing; remove the worktree after
+merge.
-Python worker code lives primarily under `amber/src/main/python`.
+### Environment
-- Supported CI Python versions: 3.10, 3.11, 3.12, 3.13.
-- Ruff config is in `amber/src/main/python/pyproject.toml`.
-- Ruff line length is 88 and target version is `py310`.
-- Generated protobuf code under `amber/src/main/python/proto` is excluded from
- Ruff.
+| Component | Version |
+| --- | --- |
+| Java | JDK 11 |
+| Scala | 2.13 |
+| Python | 3.12 |
+| Node | 24 |
-Useful commands:
+One Python venv shared across worktrees, sibling of the texera checkout:
+
+```
+<workspace>/
+├── texera/ # main checkout
+├── texera-worktrees/<br>/ # per-PR worktrees
+└── venv312/ # shared Python 3.12 venv
+```
```bash
-cd amber/src/main/python
-ruff check .
-ruff format --check .
-pytest -sv
-python -m pytest core/runnables/test_main_loop.py -v
+python3.12 -m venv ../venv312 && source ../venv312/bin/activate
+pip install -r amber/requirements.txt -r amber/operator-requirements.txt
```
-Install dependencies from `amber/requirements.txt` and
-`amber/operator-requirements.txt` when running the Python runtime or tests
-outside CI.
+Tests that spawn Python workers need an interpreter path. Edit `python.path`
+in [`udf.conf`](common/config/src/main/resources/udf.conf) or
+`export UDF_PYTHON_PATH="$(pwd)/../venv312/bin/python"` (env var overrides).
+Without it, `sbt` Python-integration tests fail to launch a worker.
-## Frontend
+### Branch and commit naming
-The Angular frontend lives in `frontend/`.
+Short, **Conventional Commits**, same shape for branch and commit subject.
-- Node engine: `>=20.19.0`.
-- Package manager: Yarn 4.14.1 via Corepack.
-- Formatting is Prettier plus prettier-eslint. Prettier uses 2 spaces,
- semicolons, double quotes, `printWidth: 120`, and LF endings.
-- Unit tests are Karma/Jasmine. Specs should live next to frontend code as
- `.spec.ts` files.
+| Kind | Branch | Commit |
+| --- | --- | --- |
+| Feature | `feat/agent-workflow-edit` | `feat(agent-service): enable workflow edit` |
+| Bug fix | `fix/marker-replay` | `fix(amber): marker replay during reconfiguration` |
+| Tests | `test/pyamber-handlers` | `test(pyamber): add handler unit tests` |
+| Chore | `chore/angular-21` | `chore(deps): upgrade frontend to Angular 21` |
+| CI | `ci/cache-action-bump` | `ci: bump coursier/cache-action to v8.1.0` |
-Useful commands:
+Both ≤ ~60 chars; scope is a module (`amber`, `pyamber`, `frontend`,
+`agent-service`, `file-service`, …) — not `amber-python`. No
+`Co-authored-by:` trailer for the repo owner.
-```bash
-cd frontend
-corepack enable
-corepack prepare yarn@4.14.1 --activate
-yarn install --immutable --inline-builds --network-timeout=100000
-yarn format:ci
-yarn format:fix
-yarn lint
-yarn test --watch=false
-yarn test:ci
-yarn build:ci
-yarn start
+### Issues and PRs
+
+Issue-first; both stay short.
+
+```
+issue (template + Type) -> PR (Closes #N, template) -> review -> merge
```
-For UI changes, include screenshots/GIFs or clear manual verification steps in
-the PR description when the behavior is visual or interactive.
+- Every change starts as an issue (minor typo / docs excepted). File against
+ `apache/texera`, never a fork.
+- Pick the right template **and** set the GitHub Issue **Type** explicitly
+ (`Bug` / `Task` / `Feature`); the template's `type:` frontmatter doesn't
+ always apply on creation.
+- Reference the issue: `Closes #N` (or `Fixes` / `Resolves`, or "related to").
+- Issue titles are **plain prose** — Conventional Commits is for commits and
+ PR titles only.
+- Task issues match `task-template.yaml` exactly: Task Summary + Task Type.
+ No priority / proposed step / code blocks.
+- Prefer **tables** and small **ASCII diagrams** over long bullets. Don't
+ restate the diff or the template.
+- For bugs, lead with **root cause** and a **before -> after** sketch:
+ ```
+ Before: reconfiguration -> replay marker -> worker hangs
+ After: reconfiguration -> replay marker -> resume from checkpoint
+ ```
+- **Frontend PRs**: any visible UI change requires screenshots / GIF,
+ **before / after** side by side. For purely visual fixes that's the
+ primary verification under "How was this PR tested?"; interactive flows
+ also list manual steps (click path, browser, viewport).
+
+### Tests come first
+
+TDD. Write the test before the source change.
-## Agent Service
+```
+write/adjust test (red) -> edit source (green) -> refactor
+```
-The standalone LLM agent service lives in `agent-service/`.
+| Situation | Order |
+| --- | --- |
+| New feature / behavior change | Failing test, then implement. |
+| Bug fix | Regression test reproducing the bug, then fix. |
+| Code with **no tests** | **Characterization tests** pin current behavior first; only then change source. |
+| Refactor (no behavior change) | Tests stay green throughout — no assertion edits. |
-- Runtime/package tool in CI: Bun 1.3.3.
-- Source is TypeScript ESM.
+Every test must cover:
-Useful commands:
+- **Both directions**: positive (valid → expected) **and** negative (invalid
+ / error → specific failure mode).
+- **Edge cases**: empty / null / zero / max / boundary, unicode,
+ concurrency/order, missing or malformed config.
+- **Don't assume valid.** External input (user / API / file / message) must
+ be tested with bad input.
+- Known bugs not fixed here: pin the buggy behavior with a comment **plus**
+ an `@pytest.mark.xfail(strict=True)` (or equivalent) test for the intended
+ contract.
Review Comment:
Removed in 09cd7ec665.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: rewrite AGENTS.md as agent-facing index [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #4825:
URL: https://github.com/apache/texera/pull/4825#discussion_r3177623485
##########
AGENTS.md:
##########
@@ -1,235 +1,187 @@
# AGENTS.md
-Guidance for coding agents working in the Apache Texera repository.
-
-## Project Overview
-
-Apache Texera is a collaborative data science and AI/ML workflow system. The
-repo is a multi-language monorepo with Scala/sbt backend services, Python worker
-runtime code, an Angular frontend, and a TypeScript/Bun agent service.
-
-Major areas:
-
-- `amber/`: workflow execution engine, Scala tests, Python worker runtime, and
- Python operator dependencies.
-- `common/`: shared Scala modules for auth, config, DAO, workflow core,
- workflow operators, and Python-template building.
-- `config-service/`, `access-control-service/`, `file-service/`,
- `computing-unit-managing-service/`, `workflow-compiling-service/`: backend
- services wired through `build.sbt`.
-- `frontend/`: Angular application. Uses Yarn 4.14.1, Node >= 20.19.0, Nx,
- Prettier, ESLint, Karma/Jasmine, and ng-zorro.
-- `agent-service/`: TypeScript Elysia service for Texera LLM agents. CI uses
- Bun 1.3.3.
-- `pyright-language-service/`: TypeScript service for Python language support.
-- `sql/`: database DDL used by local runs and CI.
-- `bin/`: shell scripts and Dockerfiles for services, local deployment, and
- generated protobuf assets.
-
-## Ground Rules
-
-- Keep changes narrowly scoped. Do not rewrite unrelated files or move code
- between services unless the task explicitly requires it.
-- Preserve local user changes. Check `git status --short` before editing and do
- not revert unrelated dirty files.
-- Follow existing module boundaries and naming patterns. Prefer local helpers
- and service abstractions over introducing new framework-level utilities.
-- Add or update tests when behavior changes. For small UI-only fixes where unit
- tests are not practical, document the manual test steps.
-- Never commit secrets, local config, generated build output, caches, or binary
- artifacts. Examples to avoid include `python_udf.conf`, `.env` files, `target/`,
- `dist/`, `.pytest_cache/`, `.ruff_cache/`, and local logs.
-
-## Licensing
-
-- New source/config files should include the Apache 2.0 ASF license header unless
- `.licenserc.yaml` excludes that file type or path.
-- Markdown files are excluded from the license-header check.
-- Keep third-party/vendored-code attribution intact. `common/workflow-operator`
- has special license handling in `project/AddMetaInfLicenseFiles.scala`.
-- GitHub Actions in ASF repositories should use approved actions and preferably
- pinned SHAs, matching the existing workflow style.
-
-## Scala / Backend
-
-- Scala version: 2.13.18.
-- Java in CI: Temurin JDK 11.
-- Formatting: `.scalafmt.conf` uses scalafmt 2.6.4 with `maxColumn = 100`.
-- Lint rules live in `.scalafix.conf` and include `ProcedureSyntax` and
- `RemoveUnused`.
-
-Useful root commands:
+## Architecture Map
-```bash
-sbt scalafmtCheckAll
-sbt scalafmtAll
-sbt "scalafixAll --check"
-sbt scalafixAll
-sbt clean package
-sbt test
-```
+Apache Texera: Scala/sbt backend services + the Amber workflow execution
+engine, an Angular UI, and helper TypeScript services. JVM modules wired in
Review Comment:
helper TypeScript services -> agent service
##########
AGENTS.md:
##########
@@ -1,235 +1,187 @@
# AGENTS.md
-Guidance for coding agents working in the Apache Texera repository.
-
-## Project Overview
-
-Apache Texera is a collaborative data science and AI/ML workflow system. The
-repo is a multi-language monorepo with Scala/sbt backend services, Python worker
-runtime code, an Angular frontend, and a TypeScript/Bun agent service.
-
-Major areas:
-
-- `amber/`: workflow execution engine, Scala tests, Python worker runtime, and
- Python operator dependencies.
-- `common/`: shared Scala modules for auth, config, DAO, workflow core,
- workflow operators, and Python-template building.
-- `config-service/`, `access-control-service/`, `file-service/`,
- `computing-unit-managing-service/`, `workflow-compiling-service/`: backend
- services wired through `build.sbt`.
-- `frontend/`: Angular application. Uses Yarn 4.14.1, Node >= 20.19.0, Nx,
- Prettier, ESLint, Karma/Jasmine, and ng-zorro.
-- `agent-service/`: TypeScript Elysia service for Texera LLM agents. CI uses
- Bun 1.3.3.
-- `pyright-language-service/`: TypeScript service for Python language support.
-- `sql/`: database DDL used by local runs and CI.
-- `bin/`: shell scripts and Dockerfiles for services, local deployment, and
- generated protobuf assets.
-
-## Ground Rules
-
-- Keep changes narrowly scoped. Do not rewrite unrelated files or move code
- between services unless the task explicitly requires it.
-- Preserve local user changes. Check `git status --short` before editing and do
- not revert unrelated dirty files.
-- Follow existing module boundaries and naming patterns. Prefer local helpers
- and service abstractions over introducing new framework-level utilities.
-- Add or update tests when behavior changes. For small UI-only fixes where unit
- tests are not practical, document the manual test steps.
-- Never commit secrets, local config, generated build output, caches, or binary
- artifacts. Examples to avoid include `python_udf.conf`, `.env` files, `target/`,
- `dist/`, `.pytest_cache/`, `.ruff_cache/`, and local logs.
-
-## Licensing
-
-- New source/config files should include the Apache 2.0 ASF license header unless
- `.licenserc.yaml` excludes that file type or path.
-- Markdown files are excluded from the license-header check.
-- Keep third-party/vendored-code attribution intact. `common/workflow-operator`
- has special license handling in `project/AddMetaInfLicenseFiles.scala`.
-- GitHub Actions in ASF repositories should use approved actions and preferably
- pinned SHAs, matching the existing workflow style.
-
-## Scala / Backend
-
-- Scala version: 2.13.18.
-- Java in CI: Temurin JDK 11.
-- Formatting: `.scalafmt.conf` uses scalafmt 2.6.4 with `maxColumn = 100`.
-- Lint rules live in `.scalafix.conf` and include `ProcedureSyntax` and
- `RemoveUnused`.
-
-Useful root commands:
+## Architecture Map
-```bash
-sbt scalafmtCheckAll
-sbt scalafmtAll
-sbt "scalafixAll --check"
-sbt scalafixAll
-sbt clean package
-sbt test
-```
+Apache Texera: Scala/sbt backend services + the Amber workflow execution
+engine, an Angular UI, and helper TypeScript services. JVM modules wired in
+[`build.sbt`](build.sbt).
-Targeted tests are preferred while iterating. Examples:
+| Area | Path | Detail |
+| --- | --- | --- |
+| Workflow execution engine (Amber) | `amber/` | [amber/README.md](amber/README.md) |
+| Backend services | `config-service/`, `access-control-service/`, `file-service/`, `computing-unit-managing-service/`, `workflow-compiling-service/` | `build.sbt` |
+| Shared Scala libs | `common/` (`auth`, `config`, `dao`, `workflow-core`, `workflow-operator`, `pybuilder`) | `build.sbt` |
+| Frontend (Angular) | `frontend/` | [frontend/README.md](frontend/README.md) |
+| Agent service (Bun/TS, LLM agents) | `agent-service/` | `agent-service/package.json` |
+| Pyright language service | `pyright-language-service/` | [pyright-language-service/README.md](pyright-language-service/README.md) |
+| Deploy scripts / Dockerfiles | `bin/` | [README](bin/README.md) / [k8s](bin/k8s/README.md) / [single-node](bin/single-node/README.md) |
+| DDL, sbt plugins | `sql/`, `project/` | files therein |
-```bash
-sbt "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.e2e.ReconfigurationSpec"
-sbt "WorkflowCompilingService/testOnly *SomeSpec"
-```
+### Amber breakdown
-CI creates PostgreSQL databases from:
+| Path | Role |
+| --- | --- |
+| `amber/src/main/scala` | Pekko actors, scheduler, reconfiguration, fault tolerance, gRPC/proto |
+| `amber/src/main/python/pyamber` | Python engine (`pyamber`) — bridge to the Scala engine |
+| `amber/src/main/python/pytexera` | Python operator SDK exposed to UDFs |
-```bash
-psql -h localhost -U postgres -f sql/texera_ddl.sql
-psql -h localhost -U postgres -f sql/iceberg_postgres_catalog.sql
-psql -h localhost -U postgres -f sql/texera_lakefs.sql
-psql -h localhost -U postgres -v DB_NAME=texera_db_for_test_cases -f sql/texera_ddl.sql
+## Where Things Live
+
+| Topic | Source of truth |
+| --- | --- |
+| Contribution / PR / lint / format / testing / license header | [CONTRIBUTING.md](CONTRIBUTING.md) |
+| Reporting security issues | [SECURITY.md](SECURITY.md) |
+| PR template | [.github/PULL_REQUEST_TEMPLATE](.github/PULL_REQUEST_TEMPLATE) |
+| Issue templates | [bug](.github/ISSUE_TEMPLATE/bug-template.yaml) / [task](.github/ISSUE_TEMPLATE/task-template.yaml) / [feature](.github/ISSUE_TEMPLATE/feature-template.yaml) |
+| License-header coverage; vendored `workflow-operator` | [.licenserc.yaml](.licenserc.yaml); [project/AddMetaInfLicenseFiles.scala](project/AddMetaInfLicenseFiles.scala) |
+| Local single-node / k8s deploy | [single-node](bin/single-node/README.md), [k8s](bin/k8s/README.md) |
+
+If a topic is above, **read that file** instead of asking here.
+
+## Agent-Specific Rules
+
+### Scope and safety
+
+- Narrowly scoped changes. No unrelated rewrites or cross-service moves.
+- `git status --short` before editing; don't revert unrelated dirty files.
+- Never commit secrets / local config / build output / caches / binaries
+ (`python_udf.conf`, `.env`, `target/`, `dist/`, `.pytest_cache/`,
+ `.ruff_cache/`, logs).
+
+### Develop in a worktree
+
+Leave `texera/` on `main`. One worktree per PR, branched off a freshly
+fetched `upstream/main`.
+
+```
+texera/ # stays on main, never dirty
+texera-worktrees/<branch>/ # one worktree per PR
```
-## Python Runtime
+Reset to `upstream/main` at start; `git log upstream/main..HEAD` should
+contain only this PR's commits before pushing; remove the worktree after
+merge.
-Python worker code lives primarily under `amber/src/main/python`.
+### Environment
-- Supported CI Python versions: 3.10, 3.11, 3.12, 3.13.
-- Ruff config is in `amber/src/main/python/pyproject.toml`.
-- Ruff line length is 88 and target version is `py310`.
-- Generated protobuf code under `amber/src/main/python/proto` is excluded from
- Ruff.
+| Component | Version |
+| --- | --- |
+| Java | JDK 11 |
+| Scala | 2.13 |
+| Python | 3.12 |
+| Node | 24 |
-Useful commands:
+One Python venv shared across worktrees, sibling of the texera checkout:
+
+```
+<workspace>/
+├── texera/ # main checkout
+├── texera-worktrees/<br>/ # per-PR worktrees
+└── venv312/ # shared Python 3.12 venv
+```
```bash
-cd amber/src/main/python
-ruff check .
-ruff format --check .
-pytest -sv
-python -m pytest core/runnables/test_main_loop.py -v
+python3.12 -m venv ../venv312 && source ../venv312/bin/activate
+pip install -r amber/requirements.txt -r amber/operator-requirements.txt
```
-Install dependencies from `amber/requirements.txt` and
-`amber/operator-requirements.txt` when running the Python runtime or tests
-outside CI.
+Tests that spawn Python workers need an interpreter path. Edit `python.path`
+in [`udf.conf`](common/config/src/main/resources/udf.conf) or
+`export UDF_PYTHON_PATH="$(pwd)/../venv312/bin/python"` (env var overrides).
+Without it, `sbt` Python-integration tests fail to launch a worker.
-## Frontend
+### Branch and commit naming
-The Angular frontend lives in `frontend/`.
+Short, **Conventional Commits**, same shape for branch and commit subject.
-- Node engine: `>=20.19.0`.
-- Package manager: Yarn 4.14.1 via Corepack.
-- Formatting is Prettier plus prettier-eslint. Prettier uses 2 spaces,
- semicolons, double quotes, `printWidth: 120`, and LF endings.
-- Unit tests are Karma/Jasmine. Specs should live next to frontend code as
- `.spec.ts` files.
+| Kind | Branch | Commit |
+| --- | --- | --- |
+| Feature | `feat/agent-workflow-edit` | `feat(agent-service): enable workflow edit` |
+| Bug fix | `fix/marker-replay` | `fix(amber): marker replay during reconfiguration` |
+| Tests | `test/pyamber-handlers` | `test(pyamber): add handler unit tests` |
+| Chore | `chore/angular-21` | `chore(deps): upgrade frontend to Angular 21` |
+| CI | `ci/cache-action-bump` | `ci: bump coursier/cache-action to v8.1.0` |
-Useful commands:
+Both ≤ ~60 chars; scope is a module (`amber`, `pyamber`, `frontend`,
+`agent-service`, `file-service`, …) — not `amber-python`. No
+`Co-authored-by:` trailer for the repo owner.
-```bash
-cd frontend
-corepack enable
-corepack prepare yarn@4.14.1 --activate
-yarn install --immutable --inline-builds --network-timeout=100000
-yarn format:ci
-yarn format:fix
-yarn lint
-yarn test --watch=false
-yarn test:ci
-yarn build:ci
-yarn start
+### Issues and PRs
+
+Issue-first; both stay short.
+
+```
+issue (template + Type) -> PR (Closes #N, template) -> review -> merge
```
-For UI changes, include screenshots/GIFs or clear manual verification steps in
-the PR description when the behavior is visual or interactive.
+- Every change starts as an issue (minor typo / docs excepted). File against
+ `apache/texera`, never a fork.
+- Pick the right template **and** set the GitHub Issue **Type** explicitly
+ (`Bug` / `Task` / `Feature`); the template's `type:` frontmatter doesn't
+ always apply on creation.
+- Reference the issue: `Closes #N` (or `Fixes` / `Resolves`, or "related to").
+- Issue titles are **plain prose** — Conventional Commits is for commits and
+ PR titles only.
+- Task issues match `task-template.yaml` exactly: Task Summary + Task Type.
+ No priority / proposed step / code blocks.
+- Prefer **tables** and small **ASCII diagrams** over long bullets. Don't
+ restate the diff or the template.
+- For bugs, lead with **root cause** and a **before -> after** sketch:
+ ```
+ Before: reconfiguration -> replay marker -> worker hangs
+ After: reconfiguration -> replay marker -> resume from checkpoint
+ ```
+- **Frontend PRs**: any visible UI change requires screenshots / GIF,
+ **before / after** side by side. For purely visual fixes that's the
+ primary verification under "How was this PR tested?"; interactive flows
+ also list manual steps (click path, browser, viewport).
+
+### Tests come first
+
+TDD. Write the test before the source change.
-## Agent Service
+```
+write/adjust test (red) -> edit source (green) -> refactor
+```
-The standalone LLM agent service lives in `agent-service/`.
+| Situation | Order |
+| --- | --- |
+| New feature / behavior change | Failing test, then implement. |
+| Bug fix | Regression test reproducing the bug, then fix. |
+| Code with **no tests** | **Characterization tests** pin current behavior first; only then change source. |
+| Refactor (no behavior change) | Tests stay green throughout — no assertion edits. |
-- Runtime/package tool in CI: Bun 1.3.3.
-- Source is TypeScript ESM.
+Every test must cover:
-Useful commands:
+- **Both directions**: positive (valid → expected) **and** negative (invalid
+ / error → specific failure mode).
+- **Edge cases**: empty / null / zero / max / boundary, unicode,
+ concurrency/order, missing or malformed config.
+- **Don't assume valid.** External input (user / API / file / message) must
+ be tested with bad input.
+- Known bugs not fixed here: pin the buggy behavior with a comment **plus**
+ an `@pytest.mark.xfail(strict=True)` (or equivalent) test for the intended
+ contract.
Review Comment:
remove
##########
AGENTS.md:
##########
@@ -1,235 +1,187 @@
# AGENTS.md
-Guidance for coding agents working in the Apache Texera repository.
-
-## Project Overview
-
-Apache Texera is a collaborative data science and AI/ML workflow system. The
-repo is a multi-language monorepo with Scala/sbt backend services, Python worker
-runtime code, an Angular frontend, and a TypeScript/Bun agent service.
-
-Major areas:
-
-- `amber/`: workflow execution engine, Scala tests, Python worker runtime, and
- Python operator dependencies.
-- `common/`: shared Scala modules for auth, config, DAO, workflow core,
- workflow operators, and Python-template building.
-- `config-service/`, `access-control-service/`, `file-service/`,
- `computing-unit-managing-service/`, `workflow-compiling-service/`: backend
- services wired through `build.sbt`.
-- `frontend/`: Angular application. Uses Yarn 4.14.1, Node >= 20.19.0, Nx,
- Prettier, ESLint, Karma/Jasmine, and ng-zorro.
-- `agent-service/`: TypeScript Elysia service for Texera LLM agents. CI uses
- Bun 1.3.3.
-- `pyright-language-service/`: TypeScript service for Python language support.
-- `sql/`: database DDL used by local runs and CI.
-- `bin/`: shell scripts and Dockerfiles for services, local deployment, and
- generated protobuf assets.
-
-## Ground Rules
-
-- Keep changes narrowly scoped. Do not rewrite unrelated files or move code
- between services unless the task explicitly requires it.
-- Preserve local user changes. Check `git status --short` before editing and do
- not revert unrelated dirty files.
-- Follow existing module boundaries and naming patterns. Prefer local helpers
- and service abstractions over introducing new framework-level utilities.
-- Add or update tests when behavior changes. For small UI-only fixes where unit
- tests are not practical, document the manual test steps.
-- Never commit secrets, local config, generated build output, caches, or binary
- artifacts. Examples to avoid include `python_udf.conf`, `.env` files, `target/`,
- `dist/`, `.pytest_cache/`, `.ruff_cache/`, and local logs.
-
-## Licensing
-
-- New source/config files should include the Apache 2.0 ASF license header unless
- `.licenserc.yaml` excludes that file type or path.
-- Markdown files are excluded from the license-header check.
-- Keep third-party/vendored-code attribution intact. `common/workflow-operator`
- has special license handling in `project/AddMetaInfLicenseFiles.scala`.
-- GitHub Actions in ASF repositories should use approved actions and preferably
- pinned SHAs, matching the existing workflow style.
-
-## Scala / Backend
-
-- Scala version: 2.13.18.
-- Java in CI: Temurin JDK 11.
-- Formatting: `.scalafmt.conf` uses scalafmt 2.6.4 with `maxColumn = 100`.
-- Lint rules live in `.scalafix.conf` and include `ProcedureSyntax` and
- `RemoveUnused`.
-
-Useful root commands:
+## Architecture Map
-```bash
-sbt scalafmtCheckAll
-sbt scalafmtAll
-sbt "scalafixAll --check"
-sbt scalafixAll
-sbt clean package
-sbt test
-```
+Apache Texera: Scala/sbt backend services + the Amber workflow execution
+engine, an Angular UI, and helper TypeScript services. JVM modules wired in
+[`build.sbt`](build.sbt).
-Targeted tests are preferred while iterating. Examples:
+| Area | Path | Detail |
+| --- | --- | --- |
+| Workflow execution engine (Amber) | `amber/` | [amber/README.md](amber/README.md) |
+| Backend services | `config-service/`, `access-control-service/`, `file-service/`, `computing-unit-managing-service/`, `workflow-compiling-service/` | `build.sbt` |
+| Shared Scala libs | `common/` (`auth`, `config`, `dao`, `workflow-core`, `workflow-operator`, `pybuilder`) | `build.sbt` |
+| Frontend (Angular) | `frontend/` | [frontend/README.md](frontend/README.md) |
+| Agent service (Bun/TS, LLM agents) | `agent-service/` | `agent-service/package.json` |
+| Pyright language service | `pyright-language-service/` | [pyright-language-service/README.md](pyright-language-service/README.md) |
+| Deploy scripts / Dockerfiles | `bin/` | [README](bin/README.md) / [k8s](bin/k8s/README.md) / [single-node](bin/single-node/README.md) |
+| DDL, sbt plugins | `sql/`, `project/` | files therein |
-```bash
-sbt "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.e2e.ReconfigurationSpec"
-sbt "WorkflowCompilingService/testOnly *SomeSpec"
-```
+### Amber breakdown
-CI creates PostgreSQL databases from:
+| Path | Role |
+| --- | --- |
+| `amber/src/main/scala` | Pekko actors, scheduler, reconfiguration, fault tolerance, gRPC/proto |
+| `amber/src/main/python/pyamber` | Python engine (`pyamber`) — bridge to the Scala engine |
+| `amber/src/main/python/pytexera` | Python operator SDK exposed to UDFs |
-```bash
-psql -h localhost -U postgres -f sql/texera_ddl.sql
-psql -h localhost -U postgres -f sql/iceberg_postgres_catalog.sql
-psql -h localhost -U postgres -f sql/texera_lakefs.sql
-psql -h localhost -U postgres -v DB_NAME=texera_db_for_test_cases -f sql/texera_ddl.sql
+## Where Things Live
+
+| Topic | Source of truth |
+| --- | --- |
+| Contribution / PR / lint / format / testing / license header | [CONTRIBUTING.md](CONTRIBUTING.md) |
+| Reporting security issues | [SECURITY.md](SECURITY.md) |
+| PR template | [.github/PULL_REQUEST_TEMPLATE](.github/PULL_REQUEST_TEMPLATE) |
+| Issue templates | [bug](.github/ISSUE_TEMPLATE/bug-template.yaml) / [task](.github/ISSUE_TEMPLATE/task-template.yaml) / [feature](.github/ISSUE_TEMPLATE/feature-template.yaml) |
+| License-header coverage; vendored `workflow-operator` | [.licenserc.yaml](.licenserc.yaml); [project/AddMetaInfLicenseFiles.scala](project/AddMetaInfLicenseFiles.scala) |
+| Local single-node / k8s deploy | [single-node](bin/single-node/README.md), [k8s](bin/k8s/README.md) |
+
+If a topic is above, **read that file** instead of asking here.
+
+## Agent-Specific Rules
+
+### Scope and safety
+
+- Narrowly scoped changes. No unrelated rewrites or cross-service moves.
+- `git status --short` before editing; don't revert unrelated dirty files.
+- Never commit secrets / local config / build output / caches / binaries
+ (`python_udf.conf`, `.env`, `target/`, `dist/`, `.pytest_cache/`,
+ `.ruff_cache/`, logs).
+
+### Develop in a worktree
+
+Leave `texera/` on `main`. One worktree per PR, branched off a freshly
+fetched `upstream/main`.
+
+```
+texera/ # stays on main, never dirty
+texera-worktrees/<branch>/ # one worktree per PR
```
-## Python Runtime
+Reset to `upstream/main` at start; `git log upstream/main..HEAD` should
+contain only this PR's commits before pushing; remove the worktree after
+merge.
-Python worker code lives primarily under `amber/src/main/python`.
+### Environment
-- Supported CI Python versions: 3.10, 3.11, 3.12, 3.13.
-- Ruff config is in `amber/src/main/python/pyproject.toml`.
-- Ruff line length is 88 and target version is `py310`.
-- Generated protobuf code under `amber/src/main/python/proto` is excluded from
- Ruff.
+| Component | Version |
+| --- | --- |
+| Java | JDK 11 |
+| Scala | 2.13 |
+| Python | 3.12 |
+| Node | 24 |
-Useful commands:
+One Python venv shared across worktrees, sibling of the texera checkout:
+
+```
+<workspace>/
+├── texera/ # main checkout
+├── texera-worktrees/<br>/ # per-PR worktrees
+└── venv312/ # shared Python 3.12 venv
+```
```bash
-cd amber/src/main/python
-ruff check .
-ruff format --check .
-pytest -sv
-python -m pytest core/runnables/test_main_loop.py -v
+python3.12 -m venv ../venv312 && source ../venv312/bin/activate
+pip install -r amber/requirements.txt -r amber/operator-requirements.txt
```
-Install dependencies from `amber/requirements.txt` and
-`amber/operator-requirements.txt` when running the Python runtime or tests
-outside CI.
+Tests that spawn Python workers need an interpreter path. Edit `python.path`
+in [`udf.conf`](common/config/src/main/resources/udf.conf) or
+`export UDF_PYTHON_PATH="$(pwd)/../venv312/bin/python"` (env var overrides).
+Without it, `sbt` Python-integration tests fail to launch a worker.
-## Frontend
+### Branch and commit naming
-The Angular frontend lives in `frontend/`.
+Short, **Conventional Commits**, same shape for branch and commit subject.
-- Node engine: `>=20.19.0`.
-- Package manager: Yarn 4.14.1 via Corepack.
-- Formatting is Prettier plus prettier-eslint. Prettier uses 2 spaces,
- semicolons, double quotes, `printWidth: 120`, and LF endings.
-- Unit tests are Karma/Jasmine. Specs should live next to frontend code as
- `.spec.ts` files.
+| Kind | Branch | Commit |
+| --- | --- | --- |
+| Feature | `feat/agent-workflow-edit` | `feat(agent-service): enable workflow edit` |
+| Bug fix | `fix/marker-replay` | `fix(amber): marker replay during reconfiguration` |
+| Tests | `test/pyamber-handlers` | `test(pyamber): add handler unit tests` |
+| Chore | `chore/angular-21` | `chore(deps): upgrade frontend to Angular 21` |
+| CI | `ci/cache-action-bump` | `ci: bump coursier/cache-action to v8.1.0` |
-Useful commands:
+Both ≤ ~60 chars; scope is a module (`amber`, `pyamber`, `frontend`,
+`agent-service`, `file-service`, …) — not `amber-python`. No
+`Co-authored-by:` trailer for the repo owner.
-```bash
-cd frontend
-corepack enable
-corepack prepare yarn@4.14.1 --activate
-yarn install --immutable --inline-builds --network-timeout=100000
-yarn format:ci
-yarn format:fix
-yarn lint
-yarn test --watch=false
-yarn test:ci
-yarn build:ci
-yarn start
+### Issues and PRs
+
+Issue-first; both stay short.
+
+```
+issue (template + Type) -> PR (Closes #N, template) -> review -> merge
```
-For UI changes, include screenshots/GIFs or clear manual verification steps in
-the PR description when the behavior is visual or interactive.
+- Every change starts as an issue (minor typo / docs excepted). File against
+ `apache/texera`, never a fork.
+- Pick the right template **and** set the GitHub Issue **Type** explicitly
+ (`Bug` / `Task` / `Feature`); the template's `type:` frontmatter doesn't
+ always apply on creation.
+- Reference the issue: `Closes #N` (or `Fixes` / `Resolves`, or "related to").
+- Issue titles are **plain prose** — Conventional Commits is for commits and
+ PR titles only.
+- Task issues match `task-template.yaml` exactly: Task Summary + Task Type.
+ No priority / proposed step / code blocks.
Review Comment:
don't mention what's in the template. don't talk about negation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: rewrite AGENTS.md as agent-facing index [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #4825:
URL: https://github.com/apache/texera/pull/4825#discussion_r3177646059
##########
AGENTS.md:
##########
@@ -1,235 +1,185 @@
# AGENTS.md
-Guidance for coding agents working in the Apache Texera repository.
-
-## Project Overview
-
-Apache Texera is a collaborative data science and AI/ML workflow system. The
-repo is a multi-language monorepo with Scala/sbt backend services, Python worker
-runtime code, an Angular frontend, and a TypeScript/Bun agent service.
-
-Major areas:
-
-- `amber/`: workflow execution engine, Scala tests, Python worker runtime, and
- Python operator dependencies.
-- `common/`: shared Scala modules for auth, config, DAO, workflow core,
- workflow operators, and Python-template building.
-- `config-service/`, `access-control-service/`, `file-service/`,
- `computing-unit-managing-service/`, `workflow-compiling-service/`: backend
- services wired through `build.sbt`.
-- `frontend/`: Angular application. Uses Yarn 4.14.1, Node >= 20.19.0, Nx,
- Prettier, ESLint, Karma/Jasmine, and ng-zorro.
-- `agent-service/`: TypeScript Elysia service for Texera LLM agents. CI uses
- Bun 1.3.3.
-- `pyright-language-service/`: TypeScript service for Python language support.
-- `sql/`: database DDL used by local runs and CI.
-- `bin/`: shell scripts and Dockerfiles for services, local deployment, and
- generated protobuf assets.
-
-## Ground Rules
-
-- Keep changes narrowly scoped. Do not rewrite unrelated files or move code
- between services unless the task explicitly requires it.
-- Preserve local user changes. Check `git status --short` before editing and do
- not revert unrelated dirty files.
-- Follow existing module boundaries and naming patterns. Prefer local helpers
- and service abstractions over introducing new framework-level utilities.
-- Add or update tests when behavior changes. For small UI-only fixes where unit
- tests are not practical, document the manual test steps.
-- Never commit secrets, local config, generated build output, caches, or binary
- artifacts. Examples to avoid include `python_udf.conf`, `.env` files, `target/`,
- `dist/`, `.pytest_cache/`, `.ruff_cache/`, and local logs.
-
-## Licensing
-
-- New source/config files should include the Apache 2.0 ASF license header unless
- `.licenserc.yaml` excludes that file type or path.
-- Markdown files are excluded from the license-header check.
-- Keep third-party/vendored-code attribution intact. `common/workflow-operator`
- has special license handling in `project/AddMetaInfLicenseFiles.scala`.
-- GitHub Actions in ASF repositories should use approved actions and preferably
- pinned SHAs, matching the existing workflow style.
-
-## Scala / Backend
-
-- Scala version: 2.13.18.
-- Java in CI: Temurin JDK 11.
-- Formatting: `.scalafmt.conf` uses scalafmt 2.6.4 with `maxColumn = 100`.
-- Lint rules live in `.scalafix.conf` and include `ProcedureSyntax` and
- `RemoveUnused`.
-
-Useful root commands:
+## Architecture Map
-```bash
-sbt scalafmtCheckAll
-sbt scalafmtAll
-sbt "scalafixAll --check"
-sbt scalafixAll
-sbt clean package
-sbt test
-```
+Apache Texera: Scala/sbt backend services + the Amber workflow execution
+engine, an Angular UI, and the agent service. JVM modules wired in
+[`build.sbt`](build.sbt).
-Targeted tests are preferred while iterating. Examples:
+| Area | Path | Detail |
+| --- | --- | --- |
+| Workflow execution engine (Amber) | `amber/` | [amber/README.md](amber/README.md) |
+| Backend services | `config-service/`, `access-control-service/`, `file-service/`, `computing-unit-managing-service/`, `workflow-compiling-service/` | `build.sbt` |
+| Shared Scala libs | `common/` (`auth`, `config`, `dao`, `workflow-core`, `workflow-operator`, `pybuilder`) | `build.sbt` |
+| Frontend (Angular) | `frontend/` | [frontend/README.md](frontend/README.md) |
+| Agent service (Bun/TS, LLM agents) | `agent-service/` | `agent-service/package.json` |
+| Pyright language service | `pyright-language-service/` | [pyright-language-service/README.md](pyright-language-service/README.md) |
+| Deploy scripts / Dockerfiles | `bin/` | [README](bin/README.md) / [k8s](bin/k8s/README.md) / [single-node](bin/single-node/README.md) |
+| DDL, sbt plugins | `sql/`, `project/` | files therein |
-```bash
-sbt "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.e2e.ReconfigurationSpec"
-sbt "WorkflowCompilingService/testOnly *SomeSpec"
-```
+### Amber breakdown
-CI creates PostgreSQL databases from:
+| Path | Role |
+| --- | --- |
+| `amber/src/main/scala` | Pekko actors, scheduler, reconfiguration, fault tolerance, gRPC/proto |
+| `amber/src/main/python/pyamber` | Python engine (`pyamber`) — bridge to the Scala engine |
+| `amber/src/main/python/pytexera` | Python operator SDK exposed to UDFs |
-```bash
-psql -h localhost -U postgres -f sql/texera_ddl.sql
-psql -h localhost -U postgres -f sql/iceberg_postgres_catalog.sql
-psql -h localhost -U postgres -f sql/texera_lakefs.sql
-psql -h localhost -U postgres -v DB_NAME=texera_db_for_test_cases -f sql/texera_ddl.sql
+## Where Things Live
+
+| Topic | Source of truth |
+| --- | --- |
+| Contribution / PR / lint / format / testing / license header | [CONTRIBUTING.md](CONTRIBUTING.md) |
+| Reporting security issues | [SECURITY.md](SECURITY.md) |
+| PR template | [.github/PULL_REQUEST_TEMPLATE](.github/PULL_REQUEST_TEMPLATE) |
+| Issue templates | [bug](.github/ISSUE_TEMPLATE/bug-template.yaml) / [task](.github/ISSUE_TEMPLATE/task-template.yaml) / [feature](.github/ISSUE_TEMPLATE/feature-template.yaml) |
+| License-header coverage; vendored `workflow-operator` | [.licenserc.yaml](.licenserc.yaml); [project/AddMetaInfLicenseFiles.scala](project/AddMetaInfLicenseFiles.scala) |
+| Local single-node / k8s deploy | [single-node](bin/single-node/README.md), [k8s](bin/k8s/README.md) |
+
+If a topic is above, **read that file** instead of asking here.
+
+## Agent-Specific Rules
+
+### Scope and safety
+
+- Narrowly scoped changes. No unrelated rewrites or cross-service moves.
+- `git status --short` before editing; don't revert unrelated dirty files.
+- Never commit secrets / local config / build output / caches / binaries
+ (`python_udf.conf`, `.env`, `target/`, `dist/`, `.pytest_cache/`,
+ `.ruff_cache/`, logs).
+
+### Develop in a worktree
+
+Leave `texera/` on `main`. One worktree per PR, branched off a freshly
+fetched `upstream/main`.
+
+```
+texera/ # stays on main, never dirty
+texera-worktrees/<branch>/ # one worktree per PR
```
-## Python Runtime
+Reset to `upstream/main` at start; `git log upstream/main..HEAD` should
+contain only this PR's commits before pushing; remove the worktree after
+merge.
-Python worker code lives primarily under `amber/src/main/python`.
+### Environment
-- Supported CI Python versions: 3.10, 3.11, 3.12, 3.13.
-- Ruff config is in `amber/src/main/python/pyproject.toml`.
-- Ruff line length is 88 and target version is `py310`.
-- Generated protobuf code under `amber/src/main/python/proto` is excluded from
- Ruff.
+| Component | Version |
+| --- | --- |
+| Java | JDK 11 |
+| Scala | 2.13 |
+| Python | 3.12 |
+| Node | 24 |
-Useful commands:
+One Python venv shared across worktrees, sibling of the texera checkout:
+
+```
+<workspace>/
+├── texera/ # main checkout
+├── texera-worktrees/<br>/ # per-PR worktrees
+└── venv312/ # shared Python 3.12 venv
+```
```bash
-cd amber/src/main/python
-ruff check .
-ruff format --check .
-pytest -sv
-python -m pytest core/runnables/test_main_loop.py -v
+python3.12 -m venv ../venv312 && source ../venv312/bin/activate
+pip install -r amber/requirements.txt -r amber/operator-requirements.txt
```
-Install dependencies from `amber/requirements.txt` and
-`amber/operator-requirements.txt` when running the Python runtime or tests
-outside CI.
+Tests that spawn Python workers need an interpreter path. Edit `python.path`
+in [`udf.conf`](common/config/src/main/resources/udf.conf) or
+`export UDF_PYTHON_PATH="$(pwd)/../venv312/bin/python"` (env var overrides).
+Without it, `sbt` Python-integration tests fail to launch a worker.
-## Frontend
+### Branch and commit naming
-The Angular frontend lives in `frontend/`.
+Short, **Conventional Commits**, same shape for branch and commit subject.
-- Node engine: `>=20.19.0`.
-- Package manager: Yarn 4.14.1 via Corepack.
-- Formatting is Prettier plus prettier-eslint. Prettier uses 2 spaces,
- semicolons, double quotes, `printWidth: 120`, and LF endings.
-- Unit tests are Karma/Jasmine. Specs should live next to frontend code as
- `.spec.ts` files.
+| Kind | Branch | Commit |
+| --- | --- | --- |
+| Feature | `feat/agent-workflow-edit` | `feat(agent-service): enable workflow edit` |
+| Bug fix | `fix/marker-replay` | `fix(amber): marker replay during reconfiguration` |
+| Tests | `test/pyamber-handlers` | `test(pyamber): add handler unit tests` |
+| Chore | `chore/angular-21` | `chore(deps): upgrade frontend to Angular 21` |
+| CI | `ci/cache-action-bump` | `ci: bump coursier/cache-action to v8.1.0` |
-Useful commands:
+Both ≤ ~60 chars. For code changes, if you use a scope, use the module name
+(`amber`, `pyamber`, `frontend`, `agent-service`, `file-service`, …) — not
+`amber-python`. Use `chore(deps): ...` for dependency-only updates, and
+`ci: ...` for CI-only changes. No `Co-authored-by:` trailer for the repo
+owner.
-```bash
-cd frontend
-corepack enable
-corepack prepare yarn@4.14.1 --activate
-yarn install --immutable --inline-builds --network-timeout=100000
-yarn format:ci
-yarn format:fix
-yarn lint
-yarn test --watch=false
-yarn test:ci
-yarn build:ci
-yarn start
+### Issues and PRs
+
+Issue-first; both stay short.
+
+```
+issue (template + Type) -> PR (Closes #N, template) -> review -> merge
```
-For UI changes, include screenshots/GIFs or clear manual verification steps in
-the PR description when the behavior is visual or interactive.
+- Every change starts as an issue (minor typo / docs excepted). File against
+ `apache/texera`, never a fork.
+- Pick the right template **and** set the GitHub Issue **Type** explicitly
+ (`Bug` / `Task` / `Feature`); the template's `type:` frontmatter doesn't
+ always apply on creation.
+- Reference the issue: `Closes #N` (or `Fixes` / `Resolves`, or "related to").
+- Issue titles are **plain prose**; never use the Conventional Commits
+ format (`type(scope): ...`) — that prefix is for commit and PR titles only.
+- Task issues match `task-template.yaml` exactly.
Review Comment:
remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: rewrite AGENTS.md as agent-facing index [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on code in PR #4825:
URL: https://github.com/apache/texera/pull/4825#discussion_r3177641376
##########
AGENTS.md:
##########
@@ -1,235 +1,187 @@
# AGENTS.md
-Guidance for coding agents working in the Apache Texera repository.
-
-## Project Overview
-
-Apache Texera is a collaborative data science and AI/ML workflow system. The
-repo is a multi-language monorepo with Scala/sbt backend services, Python worker
-runtime code, an Angular frontend, and a TypeScript/Bun agent service.
-
-Major areas:
-
-- `amber/`: workflow execution engine, Scala tests, Python worker runtime, and
- Python operator dependencies.
-- `common/`: shared Scala modules for auth, config, DAO, workflow core,
- workflow operators, and Python-template building.
-- `config-service/`, `access-control-service/`, `file-service/`,
- `computing-unit-managing-service/`, `workflow-compiling-service/`: backend
- services wired through `build.sbt`.
-- `frontend/`: Angular application. Uses Yarn 4.14.1, Node >= 20.19.0, Nx,
- Prettier, ESLint, Karma/Jasmine, and ng-zorro.
-- `agent-service/`: TypeScript Elysia service for Texera LLM agents. CI uses
- Bun 1.3.3.
-- `pyright-language-service/`: TypeScript service for Python language support.
-- `sql/`: database DDL used by local runs and CI.
-- `bin/`: shell scripts and Dockerfiles for services, local deployment, and
- generated protobuf assets.
-
-## Ground Rules
-
-- Keep changes narrowly scoped. Do not rewrite unrelated files or move code
- between services unless the task explicitly requires it.
-- Preserve local user changes. Check `git status --short` before editing and do
- not revert unrelated dirty files.
-- Follow existing module boundaries and naming patterns. Prefer local helpers
- and service abstractions over introducing new framework-level utilities.
-- Add or update tests when behavior changes. For small UI-only fixes where unit
- tests are not practical, document the manual test steps.
-- Never commit secrets, local config, generated build output, caches, or binary
- artifacts. Examples to avoid include `python_udf.conf`, `.env` files, `target/`,
- `dist/`, `.pytest_cache/`, `.ruff_cache/`, and local logs.
-
-## Licensing
-
-- New source/config files should include the Apache 2.0 ASF license header unless
- `.licenserc.yaml` excludes that file type or path.
-- Markdown files are excluded from the license-header check.
-- Keep third-party/vendored-code attribution intact. `common/workflow-operator`
- has special license handling in `project/AddMetaInfLicenseFiles.scala`.
-- GitHub Actions in ASF repositories should use approved actions and preferably
- pinned SHAs, matching the existing workflow style.
-
-## Scala / Backend
-
-- Scala version: 2.13.18.
-- Java in CI: Temurin JDK 11.
-- Formatting: `.scalafmt.conf` uses scalafmt 2.6.4 with `maxColumn = 100`.
-- Lint rules live in `.scalafix.conf` and include `ProcedureSyntax` and
- `RemoveUnused`.
-
-Useful root commands:
+## Architecture Map
-```bash
-sbt scalafmtCheckAll
-sbt scalafmtAll
-sbt "scalafixAll --check"
-sbt scalafixAll
-sbt clean package
-sbt test
-```
+Apache Texera: Scala/sbt backend services + the Amber workflow execution
+engine, an Angular UI, and helper TypeScript services. JVM modules wired in
+[`build.sbt`](build.sbt).
-Targeted tests are preferred while iterating. Examples:
+| Area | Path | Detail |
+| --- | --- | --- |
+| Workflow execution engine (Amber) | `amber/` | [amber/README.md](amber/README.md) |
+| Backend services | `config-service/`, `access-control-service/`, `file-service/`, `computing-unit-managing-service/`, `workflow-compiling-service/` | `build.sbt` |
+| Shared Scala libs | `common/` (`auth`, `config`, `dao`, `workflow-core`, `workflow-operator`, `pybuilder`) | `build.sbt` |
+| Frontend (Angular) | `frontend/` | [frontend/README.md](frontend/README.md) |
+| Agent service (Bun/TS, LLM agents) | `agent-service/` | `agent-service/package.json` |
+| Pyright language service | `pyright-language-service/` | [pyright-language-service/README.md](pyright-language-service/README.md) |
+| Deploy scripts / Dockerfiles | `bin/` | [README](bin/README.md) / [k8s](bin/k8s/README.md) / [single-node](bin/single-node/README.md) |
+| DDL, sbt plugins | `sql/`, `project/` | files therein |
-```bash
-sbt "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.e2e.ReconfigurationSpec"
-sbt "WorkflowCompilingService/testOnly *SomeSpec"
-```
+### Amber breakdown
-CI creates PostgreSQL databases from:
+| Path | Role |
+| --- | --- |
+| `amber/src/main/scala` | Pekko actors, scheduler, reconfiguration, fault tolerance, gRPC/proto |
+| `amber/src/main/python/pyamber` | Python engine (`pyamber`) — bridge to the Scala engine |
+| `amber/src/main/python/pytexera` | Python operator SDK exposed to UDFs |
-```bash
-psql -h localhost -U postgres -f sql/texera_ddl.sql
-psql -h localhost -U postgres -f sql/iceberg_postgres_catalog.sql
-psql -h localhost -U postgres -f sql/texera_lakefs.sql
-psql -h localhost -U postgres -v DB_NAME=texera_db_for_test_cases -f sql/texera_ddl.sql
+## Where Things Live
+
+| Topic | Source of truth |
+| --- | --- |
+| Contribution / PR / lint / format / testing / license header | [CONTRIBUTING.md](CONTRIBUTING.md) |
+| Reporting security issues | [SECURITY.md](SECURITY.md) |
+| PR template | [.github/PULL_REQUEST_TEMPLATE](.github/PULL_REQUEST_TEMPLATE) |
+| Issue templates | [bug](.github/ISSUE_TEMPLATE/bug-template.yaml) / [task](.github/ISSUE_TEMPLATE/task-template.yaml) / [feature](.github/ISSUE_TEMPLATE/feature-template.yaml) |
+| License-header coverage; vendored `workflow-operator` | [.licenserc.yaml](.licenserc.yaml); [project/AddMetaInfLicenseFiles.scala](project/AddMetaInfLicenseFiles.scala) |
+| Local single-node / k8s deploy | [single-node](bin/single-node/README.md), [k8s](bin/k8s/README.md) |
+
+If a topic is above, **read that file** instead of asking here.
+
+## Agent-Specific Rules
+
+### Scope and safety
+
+- Narrowly scoped changes. No unrelated rewrites or cross-service moves.
+- `git status --short` before editing; don't revert unrelated dirty files.
+- Never commit secrets / local config / build output / caches / binaries
+ (`python_udf.conf`, `.env`, `target/`, `dist/`, `.pytest_cache/`,
+ `.ruff_cache/`, logs).
+
+### Develop in a worktree
+
+Leave `texera/` on `main`. One worktree per PR, branched off a freshly
+fetched `upstream/main`.
+
+```
+texera/ # stays on main, never dirty
+texera-worktrees/<branch>/ # one worktree per PR
```
-## Python Runtime
+Reset to `upstream/main` at start; `git log upstream/main..HEAD` should
+contain only this PR's commits before pushing; remove the worktree after
+merge.
-Python worker code lives primarily under `amber/src/main/python`.
+### Environment
-- Supported CI Python versions: 3.10, 3.11, 3.12, 3.13.
-- Ruff config is in `amber/src/main/python/pyproject.toml`.
-- Ruff line length is 88 and target version is `py310`.
-- Generated protobuf code under `amber/src/main/python/proto` is excluded from
- Ruff.
+| Component | Version |
+| --- | --- |
+| Java | JDK 11 |
+| Scala | 2.13 |
+| Python | 3.12 |
+| Node | 24 |
Review Comment:
Upgraded to 24 in #4658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: rewrite AGENTS.md as agent-facing index [texera]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #4825:
URL: https://github.com/apache/texera/pull/4825#discussion_r3177624097
##########
AGENTS.md:
##########
@@ -1,235 +1,187 @@
# AGENTS.md
-Guidance for coding agents working in the Apache Texera repository.
-
-## Project Overview
-
-Apache Texera is a collaborative data science and AI/ML workflow system. The
-repo is a multi-language monorepo with Scala/sbt backend services, Python worker
-runtime code, an Angular frontend, and a TypeScript/Bun agent service.
-
-Major areas:
-
-- `amber/`: workflow execution engine, Scala tests, Python worker runtime, and
- Python operator dependencies.
-- `common/`: shared Scala modules for auth, config, DAO, workflow core,
- workflow operators, and Python-template building.
-- `config-service/`, `access-control-service/`, `file-service/`,
- `computing-unit-managing-service/`, `workflow-compiling-service/`: backend
- services wired through `build.sbt`.
-- `frontend/`: Angular application. Uses Yarn 4.14.1, Node >= 20.19.0, Nx,
- Prettier, ESLint, Karma/Jasmine, and ng-zorro.
-- `agent-service/`: TypeScript Elysia service for Texera LLM agents. CI uses
- Bun 1.3.3.
-- `pyright-language-service/`: TypeScript service for Python language support.
-- `sql/`: database DDL used by local runs and CI.
-- `bin/`: shell scripts and Dockerfiles for services, local deployment, and
- generated protobuf assets.
-
-## Ground Rules
-
-- Keep changes narrowly scoped. Do not rewrite unrelated files or move code
- between services unless the task explicitly requires it.
-- Preserve local user changes. Check `git status --short` before editing and do
- not revert unrelated dirty files.
-- Follow existing module boundaries and naming patterns. Prefer local helpers
- and service abstractions over introducing new framework-level utilities.
-- Add or update tests when behavior changes. For small UI-only fixes where unit
- tests are not practical, document the manual test steps.
-- Never commit secrets, local config, generated build output, caches, or binary
- artifacts. Examples to avoid include `python_udf.conf`, `.env` files, `target/`,
- `dist/`, `.pytest_cache/`, `.ruff_cache/`, and local logs.
-
-## Licensing
-
-- New source/config files should include the Apache 2.0 ASF license header unless
- `.licenserc.yaml` excludes that file type or path.
-- Markdown files are excluded from the license-header check.
-- Keep third-party/vendored-code attribution intact. `common/workflow-operator`
- has special license handling in `project/AddMetaInfLicenseFiles.scala`.
-- GitHub Actions in ASF repositories should use approved actions and preferably
- pinned SHAs, matching the existing workflow style.
-
-## Scala / Backend
-
-- Scala version: 2.13.18.
-- Java in CI: Temurin JDK 11.
-- Formatting: `.scalafmt.conf` uses scalafmt 2.6.4 with `maxColumn = 100`.
-- Lint rules live in `.scalafix.conf` and include `ProcedureSyntax` and
- `RemoveUnused`.
-
-Useful root commands:
+## Architecture Map
-```bash
-sbt scalafmtCheckAll
-sbt scalafmtAll
-sbt "scalafixAll --check"
-sbt scalafixAll
-sbt clean package
-sbt test
-```
+Apache Texera: Scala/sbt backend services + the Amber workflow execution
+engine, an Angular UI, and helper TypeScript services. JVM modules wired in
+[`build.sbt`](build.sbt).
-Targeted tests are preferred while iterating. Examples:
+| Area | Path | Detail |
+| --- | --- | --- |
+| Workflow execution engine (Amber) | `amber/` | [amber/README.md](amber/README.md) |
+| Backend services | `config-service/`, `access-control-service/`, `file-service/`, `computing-unit-managing-service/`, `workflow-compiling-service/` | `build.sbt` |
+| Shared Scala libs | `common/` (`auth`, `config`, `dao`, `workflow-core`, `workflow-operator`, `pybuilder`) | `build.sbt` |
+| Frontend (Angular) | `frontend/` | [frontend/README.md](frontend/README.md) |
+| Agent service (Bun/TS, LLM agents) | `agent-service/` | `agent-service/package.json` |
+| Pyright language service | `pyright-language-service/` | [pyright-language-service/README.md](pyright-language-service/README.md) |
+| Deploy scripts / Dockerfiles | `bin/` | [README](bin/README.md) / [k8s](bin/k8s/README.md) / [single-node](bin/single-node/README.md) |
+| DDL, sbt plugins | `sql/`, `project/` | files therein |
-```bash
-sbt "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.e2e.ReconfigurationSpec"
-sbt "WorkflowCompilingService/testOnly *SomeSpec"
-```
+### Amber breakdown
-CI creates PostgreSQL databases from:
+| Path | Role |
+| --- | --- |
+| `amber/src/main/scala` | Pekko actors, scheduler, reconfiguration, fault tolerance, gRPC/proto |
+| `amber/src/main/python/pyamber` | Python engine (`pyamber`) — bridge to the Scala engine |
+| `amber/src/main/python/pytexera` | Python operator SDK exposed to UDFs |
-```bash
-psql -h localhost -U postgres -f sql/texera_ddl.sql
-psql -h localhost -U postgres -f sql/iceberg_postgres_catalog.sql
-psql -h localhost -U postgres -f sql/texera_lakefs.sql
-psql -h localhost -U postgres -v DB_NAME=texera_db_for_test_cases -f sql/texera_ddl.sql
+## Where Things Live
+
+| Topic | Source of truth |
+| --- | --- |
+| Contribution / PR / lint / format / testing / license header | [CONTRIBUTING.md](CONTRIBUTING.md) |
+| Reporting security issues | [SECURITY.md](SECURITY.md) |
+| PR template | [.github/PULL_REQUEST_TEMPLATE](.github/PULL_REQUEST_TEMPLATE) |
+| Issue templates | [bug](.github/ISSUE_TEMPLATE/bug-template.yaml) / [task](.github/ISSUE_TEMPLATE/task-template.yaml) / [feature](.github/ISSUE_TEMPLATE/feature-template.yaml) |
+| License-header coverage; vendored `workflow-operator` | [.licenserc.yaml](.licenserc.yaml); [project/AddMetaInfLicenseFiles.scala](project/AddMetaInfLicenseFiles.scala) |
+| Local single-node / k8s deploy | [single-node](bin/single-node/README.md), [k8s](bin/k8s/README.md) |
+
+If a topic is above, **read that file** instead of asking here.
+
+## Agent-Specific Rules
+
+### Scope and safety
+
+- Narrowly scoped changes. No unrelated rewrites or cross-service moves.
+- `git status --short` before editing; don't revert unrelated dirty files.
+- Never commit secrets / local config / build output / caches / binaries
+ (`python_udf.conf`, `.env`, `target/`, `dist/`, `.pytest_cache/`,
+ `.ruff_cache/`, logs).
+
+### Develop in a worktree
+
+Leave `texera/` on `main`. One worktree per PR, branched off a freshly
+fetched `upstream/main`.
+
+```
+texera/ # stays on main, never dirty
+texera-worktrees/<branch>/ # one worktree per PR
```
-## Python Runtime
+Reset to `upstream/main` at start; `git log upstream/main..HEAD` should
+contain only this PR's commits before pushing; remove the worktree after
+merge.
-Python worker code lives primarily under `amber/src/main/python`.
+### Environment
-- Supported CI Python versions: 3.10, 3.11, 3.12, 3.13.
-- Ruff config is in `amber/src/main/python/pyproject.toml`.
-- Ruff line length is 88 and target version is `py310`.
-- Generated protobuf code under `amber/src/main/python/proto` is excluded from
- Ruff.
+| Component | Version |
+| --- | --- |
+| Java | JDK 11 |
+| Scala | 2.13 |
+| Python | 3.12 |
+| Node | 24 |
Review Comment:
The environment table now points agents at Node 24, but the repo's actual source-of-truth still pins/tests Node 20.19.0 for the frontend (`frontend/package.json` and `.github/workflows/build.yml`). Since this file is supposed to be an index into current project guidance, documenting a different major version here is likely to send contributors onto a toolchain that CI does not validate.
##########
AGENTS.md:
##########
@@ -1,235 +1,187 @@
# AGENTS.md
-Guidance for coding agents working in the Apache Texera repository.
-
-## Project Overview
-
-Apache Texera is a collaborative data science and AI/ML workflow system. The
-repo is a multi-language monorepo with Scala/sbt backend services, Python worker
-runtime code, an Angular frontend, and a TypeScript/Bun agent service.
-
-Major areas:
-
-- `amber/`: workflow execution engine, Scala tests, Python worker runtime, and
- Python operator dependencies.
-- `common/`: shared Scala modules for auth, config, DAO, workflow core,
- workflow operators, and Python-template building.
-- `config-service/`, `access-control-service/`, `file-service/`,
- `computing-unit-managing-service/`, `workflow-compiling-service/`: backend
- services wired through `build.sbt`.
-- `frontend/`: Angular application. Uses Yarn 4.14.1, Node >= 20.19.0, Nx,
- Prettier, ESLint, Karma/Jasmine, and ng-zorro.
-- `agent-service/`: TypeScript Elysia service for Texera LLM agents. CI uses
- Bun 1.3.3.
-- `pyright-language-service/`: TypeScript service for Python language support.
-- `sql/`: database DDL used by local runs and CI.
-- `bin/`: shell scripts and Dockerfiles for services, local deployment, and
- generated protobuf assets.
-
-## Ground Rules
-
-- Keep changes narrowly scoped. Do not rewrite unrelated files or move code
- between services unless the task explicitly requires it.
-- Preserve local user changes. Check `git status --short` before editing and do
- not revert unrelated dirty files.
-- Follow existing module boundaries and naming patterns. Prefer local helpers
- and service abstractions over introducing new framework-level utilities.
-- Add or update tests when behavior changes. For small UI-only fixes where unit
- tests are not practical, document the manual test steps.
-- Never commit secrets, local config, generated build output, caches, or binary
- artifacts. Examples to avoid include `python_udf.conf`, `.env` files, `target/`,
- `dist/`, `.pytest_cache/`, `.ruff_cache/`, and local logs.
-
-## Licensing
-
-- New source/config files should include the Apache 2.0 ASF license header unless
- `.licenserc.yaml` excludes that file type or path.
-- Markdown files are excluded from the license-header check.
-- Keep third-party/vendored-code attribution intact. `common/workflow-operator`
- has special license handling in `project/AddMetaInfLicenseFiles.scala`.
-- GitHub Actions in ASF repositories should use approved actions and preferably
- pinned SHAs, matching the existing workflow style.
-
-## Scala / Backend
-
-- Scala version: 2.13.18.
-- Java in CI: Temurin JDK 11.
-- Formatting: `.scalafmt.conf` uses scalafmt 2.6.4 with `maxColumn = 100`.
-- Lint rules live in `.scalafix.conf` and include `ProcedureSyntax` and
- `RemoveUnused`.
-
-Useful root commands:
+## Architecture Map
-```bash
-sbt scalafmtCheckAll
-sbt scalafmtAll
-sbt "scalafixAll --check"
-sbt scalafixAll
-sbt clean package
-sbt test
-```
+Apache Texera: Scala/sbt backend services + the Amber workflow execution
+engine, an Angular UI, and helper TypeScript services. JVM modules wired in
+[`build.sbt`](build.sbt).
-Targeted tests are preferred while iterating. Examples:
+| Area | Path | Detail |
+| --- | --- | --- |
+| Workflow execution engine (Amber) | `amber/` | [amber/README.md](amber/README.md) |
+| Backend services | `config-service/`, `access-control-service/`, `file-service/`, `computing-unit-managing-service/`, `workflow-compiling-service/` | `build.sbt` |
+| Shared Scala libs | `common/` (`auth`, `config`, `dao`, `workflow-core`, `workflow-operator`, `pybuilder`) | `build.sbt` |
+| Frontend (Angular) | `frontend/` | [frontend/README.md](frontend/README.md) |
+| Agent service (Bun/TS, LLM agents) | `agent-service/` | `agent-service/package.json` |
+| Pyright language service | `pyright-language-service/` | [pyright-language-service/README.md](pyright-language-service/README.md) |
+| Deploy scripts / Dockerfiles | `bin/` | [README](bin/README.md) / [k8s](bin/k8s/README.md) / [single-node](bin/single-node/README.md) |
+| DDL, sbt plugins | `sql/`, `project/` | files therein |
-```bash
-sbt "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.e2e.ReconfigurationSpec"
-sbt "WorkflowCompilingService/testOnly *SomeSpec"
-```
+### Amber breakdown
-CI creates PostgreSQL databases from:
+| Path | Role |
+| --- | --- |
+| `amber/src/main/scala` | Pekko actors, scheduler, reconfiguration, fault tolerance, gRPC/proto |
+| `amber/src/main/python/pyamber` | Python engine (`pyamber`) — bridge to the Scala engine |
+| `amber/src/main/python/pytexera` | Python operator SDK exposed to UDFs |
-```bash
-psql -h localhost -U postgres -f sql/texera_ddl.sql
-psql -h localhost -U postgres -f sql/iceberg_postgres_catalog.sql
-psql -h localhost -U postgres -f sql/texera_lakefs.sql
-psql -h localhost -U postgres -v DB_NAME=texera_db_for_test_cases -f sql/texera_ddl.sql
+## Where Things Live
+
+| Topic | Source of truth |
+| --- | --- |
+| Contribution / PR / lint / format / testing / license header | [CONTRIBUTING.md](CONTRIBUTING.md) |
+| Reporting security issues | [SECURITY.md](SECURITY.md) |
+| PR template | [.github/PULL_REQUEST_TEMPLATE](.github/PULL_REQUEST_TEMPLATE) |
+| Issue templates | [bug](.github/ISSUE_TEMPLATE/bug-template.yaml) / [task](.github/ISSUE_TEMPLATE/task-template.yaml) / [feature](.github/ISSUE_TEMPLATE/feature-template.yaml) |
+| License-header coverage; vendored `workflow-operator` | [.licenserc.yaml](.licenserc.yaml); [project/AddMetaInfLicenseFiles.scala](project/AddMetaInfLicenseFiles.scala) |
+| Local single-node / k8s deploy | [single-node](bin/single-node/README.md), [k8s](bin/k8s/README.md) |
+
+If a topic is above, **read that file** instead of asking here.
+
+## Agent-Specific Rules
+
+### Scope and safety
+
+- Narrowly scoped changes. No unrelated rewrites or cross-service moves.
+- `git status --short` before editing; don't revert unrelated dirty files.
+- Never commit secrets / local config / build output / caches / binaries
+ (`python_udf.conf`, `.env`, `target/`, `dist/`, `.pytest_cache/`,
+ `.ruff_cache/`, logs).
+
+### Develop in a worktree
+
+Leave `texera/` on `main`. One worktree per PR, branched off a freshly
+fetched `upstream/main`.
+
+```
+texera/ # stays on main, never dirty
+texera-worktrees/<branch>/ # one worktree per PR
```
-## Python Runtime
+Reset to `upstream/main` at start; `git log upstream/main..HEAD` should
+contain only this PR's commits before pushing; remove the worktree after
+merge.
-Python worker code lives primarily under `amber/src/main/python`.
+### Environment
-- Supported CI Python versions: 3.10, 3.11, 3.12, 3.13.
-- Ruff config is in `amber/src/main/python/pyproject.toml`.
-- Ruff line length is 88 and target version is `py310`.
-- Generated protobuf code under `amber/src/main/python/proto` is excluded from
- Ruff.
+| Component | Version |
+| --- | --- |
+| Java | JDK 11 |
+| Scala | 2.13 |
+| Python | 3.12 |
+| Node | 24 |
-Useful commands:
+One Python venv shared across worktrees, sibling of the texera checkout:
+
+```
+<workspace>/
+├── texera/ # main checkout
+├── texera-worktrees/<br>/ # per-PR worktrees
+└── venv312/ # shared Python 3.12 venv
+```
```bash
-cd amber/src/main/python
-ruff check .
-ruff format --check .
-pytest -sv
-python -m pytest core/runnables/test_main_loop.py -v
+python3.12 -m venv ../venv312 && source ../venv312/bin/activate
+pip install -r amber/requirements.txt -r amber/operator-requirements.txt
```
-Install dependencies from `amber/requirements.txt` and
-`amber/operator-requirements.txt` when running the Python runtime or tests
-outside CI.
+Tests that spawn Python workers need an interpreter path. Edit `python.path`
+in [`udf.conf`](common/config/src/main/resources/udf.conf) or
+`export UDF_PYTHON_PATH="$(pwd)/../venv312/bin/python"` (env var overrides).
+Without it, `sbt` Python-integration tests fail to launch a worker.
-## Frontend
+### Branch and commit naming
-The Angular frontend lives in `frontend/`.
+Short, **Conventional Commits**, same shape for branch and commit subject.
-- Node engine: `>=20.19.0`.
-- Package manager: Yarn 4.14.1 via Corepack.
-- Formatting is Prettier plus prettier-eslint. Prettier uses 2 spaces,
- semicolons, double quotes, `printWidth: 120`, and LF endings.
-- Unit tests are Karma/Jasmine. Specs should live next to frontend code as
- `.spec.ts` files.
+| Kind | Branch | Commit |
+| --- | --- | --- |
+| Feature | `feat/agent-workflow-edit` | `feat(agent-service): enable workflow edit` |
+| Bug fix | `fix/marker-replay` | `fix(amber): marker replay during reconfiguration` |
+| Tests | `test/pyamber-handlers` | `test(pyamber): add handler unit tests` |
+| Chore | `chore/angular-21` | `chore(deps): upgrade frontend to Angular 21` |
+| CI | `ci/cache-action-bump` | `ci: bump coursier/cache-action to v8.1.0` |
-Useful commands:
+Both ≤ ~60 chars; scope is a module (`amber`, `pyamber`, `frontend`,
+`agent-service`, `file-service`, …) — not `amber-python`. No
+`Co-authored-by:` trailer for the repo owner.
Review Comment:
The rule here says commit scopes should be module names, but the examples immediately above use `chore(deps): ...` and `ci: ...`, which are not module scopes. Because this file is meant to be an agent-facing source of truth, that contradiction leaves it unclear which forms are actually allowed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org