You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/05/29 17:35:46 UTC
(spark-kubernetes-operator) branch main updated: [SPARK-57147] Add `AGENTS.md` and symlink `CLAUDE.md` to it
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 40eea13 [SPARK-57147] Add `AGENTS.md` and symlink `CLAUDE.md` to it
40eea13 is described below
commit 40eea1318a0459730ffcbe5bd1aa9ccb4c0c45a5
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Fri May 29 10:35:39 2026 -0700
[SPARK-57147] Add `AGENTS.md` and symlink `CLAUDE.md` to it
### What changes were proposed in this pull request?
This PR adds `AGENTS.md`, a contributor/agent guide for the repository, and `CLAUDE.md` as a
symlink to it.
### Why are the changes needed?
To give contributors and AI agents one accurate reference for the build, test, style, and
contribution conventions. The symlink keeps `CLAUDE.md` and `AGENTS.md` in sync without duplication.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Documentation-only change. Verified that `CLAUDE.md` resolves to `AGENTS.md`.
### Was this patch authored or co-authored using generative AI tooling?
Yes. Generated-by: Claude Code (Claude Opus 4.8)
Closes #696 from dongjoon-hyun/SPARK-57147.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
AGENTS.md | 138 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
CLAUDE.md | 1 +
2 files changed, 139 insertions(+)
diff --git a/AGENTS.md b/AGENTS.md
new file mode 100644
index 0000000..94e0818
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1,138 @@
+# Agent Guide — Apache Spark K8s Operator
+
+Guidance for AI agents and contributors working in this repository
+(Apache Spark Kubernetes Operator).
+
+## Project Overview
+
+Apache Spark K8s Operator is a subproject of Apache Spark that extends Kubernetes to manage the
+full lifecycle of Spark applications and clusters via the
+[Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/). It watches
+`SparkApplication` and `SparkCluster` custom resources and reconciles them — launching the
+driver/master pods, observing executors/workers, and releasing resources on termination.
+
+It is built on the [Java Operator SDK](https://javaoperatorsdk.io/) and the
+[fabric8](https://fabric8.io/) Kubernetes client. The runtime entry point is
+`org.apache.spark.k8s.operator.SparkOperator` in the `spark-operator` module.
+
+See [docs/architecture.md](docs/architecture.md) for the reconciliation model and state machines,
+and the rest of [docs/](docs/) for configuration and operations.
+
+## Tech Stack & Requirements
+
+The operator is implemented in **Java** and built with **Gradle** (multi-project,
+configuration-cache enabled). It links against Apache Spark's Scala 2.13 artifacts but contains no
+Scala source. Key runtime libraries:
+
+- [Apache Spark](https://spark.apache.org/) `spark-core` / `spark-kubernetes` (Scala 2.13) and
+ Hadoop — used to build driver/submission specs (compile-only in `spark-operator`, bundled into
+ `spark-submission-worker`).
+- [Java Operator SDK](https://javaoperatorsdk.io/) (`operator-framework`) — reconciliation runtime.
+- [fabric8](https://fabric8.io/) `kubernetes-client` — Kubernetes API access and CRD generation.
+- Log4j 2 + SLF4J for logging, Dropwizard Metrics for metrics, and Lombok for code generation.
+
+Packaging and tooling: the Shadow plugin (operator fat jar), CycloneDX (SBOM), and the
+Spotless / Checkstyle / PMD / SpotBugs / JaCoCo quality gate (see [Code Style](#code-style)).
+**All dependency versions are pinned centrally in
+[gradle/libs.versions.toml](gradle/libs.versions.toml)** — the single source of truth; bump versions
+there, never inline in a module.
+
+**Requirements**
+
+- **JDK 21 or newer** — the Gradle toolchain auto-provisions JDK 26 and targets Java 21 bytecode;
+ CI builds on Java 21 / 25 / 26.
+- **Docker** — required for `./gradlew buildDockerImage`.
+- **Kubernetes cluster** (e.g. minikube), **Helm 3**, and **kubectl** — to deploy and run the
+ operator.
+- **Chainsaw** (and **Go** to install it) — to run the `tests/e2e` suites.
+
+## Module Layout
+
+Gradle multi-project build (see [settings.gradle](settings.gradle)); root group is
+`org.apache.spark.k8s.operator`, version `1.0.0-SNAPSHOT`.
+
+- `spark-operator-api/` — CRD model: `SparkApplication` / `SparkCluster` spec, status, and diff
+ types. Minimal dependencies (fabric8 `kubernetes-client` only); CRDs are generated here.
+- `spark-submission-worker/` — Bridges to Spark's `spark-kubernetes` to build driver/submission
+ specs. Depends on `spark-operator-api`.
+- `spark-operator/` — The operator runtime: reconcilers, observers, metrics, health probes, config,
+ and listeners. Shaded into a single fat jar via the Shadow plugin. Depends on the other two
+ modules.
+- `build-tools/docs-utils/` — Gradle subproject (`build-tools-docs-utils`) that generates the
+ config and CRD documentation.
+- `tests/e2e/` — [Chainsaw](https://kyverno.github.io/chainsaw/) end-to-end tests, grouped by
+ scenario (e.g. `state-transition`, `python`, `watched-namespaces`); shared assertions live in
+ `tests/e2e/assertions/`.
+- `build-tools/`, `config/`, `examples/`, `docs/`, `dev/` — packaging (Docker / Helm), linter
+ configs, example custom resources, documentation, and developer scripts.
+
+## Build, Test & Dev Commands
+
+Use `./gradlew` (the bundled Gradle wrapper). The build targets Java 21 bytecode while the
+toolchain provisions JDK 26, so a JDK 21+ is required.
+
+```bash
+./gradlew build -x test # Compile + run linters, skip tests (fast)
+./gradlew build # Full build: compile, linters, and all unit tests
+./gradlew :spark-operator:test # Run a single module's tests
+./gradlew :spark-operator:test --tests "org.apache.spark.k8s.operator.SparkOperatorTest" # one class
+./gradlew spotlessApply # Auto-format sources (run before committing)
+./gradlew spotlessCheck # Verify formatting only
+./gradlew javadoc # Generate Javadoc (CI gate)
+./gradlew buildDockerImage # Build the apache/spark-kubernetes-operator:<version> image
+./gradlew dependencyUpdates # Report newer dependency versions
+```
+
+Helm chart (the lint mirrors CI):
+
+```bash
+helm lint --strict build-tools/helm/spark-kubernetes-operator
+helm install spark -f build-tools/helm/spark-kubernetes-operator/values.yaml \
+ build-tools/helm/spark-kubernetes-operator/
+```
+
+End-to-end tests need a running cluster (e.g. minikube) plus a freshly built image and Chainsaw:
+
+```bash
+chainsaw test --test-dir ./tests/e2e/state-transition --parallel 1
+```
+
+`./gradlew build` runs the full quality gate locally — Checkstyle, PMD, SpotBugs, Spotless, and
+JaCoCo. Run it before pushing; CI runs the same matrix on Java 21 / 25 / 26 (x86 and arm).
+
+## Code Style
+
+- Java with 2-space indentation; max line length 100 (Checkstyle `severity=error`).
+- Linter configs live under [config/](config/): Checkstyle (`config/checkstyle/checkstyle.xml`),
+ PMD (`config/pmd/ruleset.xml`), SpotBugs (`config/spotbugs/spotbugs_exclude.xml`). Each fails the
+ build on violation.
+- Formatting is enforced by Spotless — run `./gradlew spotlessApply`. It trims trailing whitespace,
+ removes unused imports, and ensures a trailing newline. Wrap a block in `// spotless:off` /
+ `// spotless:on` to protect it.
+- Import order, with a blank line between groups: `java`, `javax`, `scala`, everything else, then
+ `org.apache.spark` last. Static imports come first.
+- [Lombok](https://projectlombok.org/) is available (`@Slf4j`, `@Getter`, …) — prefer it over
+ boilerplate, matching existing usage.
+- Every non-Markdown source file needs the ASF Apache-2.0 license header (copy it from any existing
+ `.java` / `.gradle` file). Markdown, `dev/`, and `.github/` are exempt — see
+ [.github/.licenserc.yaml](.github/.licenserc.yaml). Markdown follows
+ [.markdownlint.yaml](.markdownlint.yaml) (the MD013 line-length rule is disabled).
+
+## Testing
+
+- Unit tests use JUnit 5 (Jupiter) and Mockito; Kubernetes interactions are exercised with the
+ fabric8 `kubernetes-server-mock` and the Java Operator SDK JUnit extension.
+- Put tests in `src/test/java` mirroring the main package, and name classes `*Test`.
+- Behavioral coverage lives in the Chainsaw specs under `tests/e2e/<group>/`.
+
+## Commit & PR Guidelines
+
+- Issues are tracked in the ASF JIRA project `SPARK`. Prefix every commit and PR title with the
+ ticket: `[SPARK-XXXXX] Short imperative summary`
+ (e.g. `[SPARK-57114] Upgrade spotless-plugin-gradle to 8.6.0`).
+- PRs are merged via **squash** or **rebase** — no merge commits (see [.asf.yaml](.asf.yaml)). Keep
+ changes surgical and scoped to the ticket.
+- Run `./gradlew spotlessApply` and `./gradlew build` before opening a PR.
+- Write PR descriptions in English following
+ [.github/PULL_REQUEST_TEMPLATE](.github/PULL_REQUEST_TEMPLATE), and follow the
+ [Spark contributing guide](https://spark.apache.org/contributing.html).
diff --git a/CLAUDE.md b/CLAUDE.md
new file mode 120000
index 0000000..47dc3e3
--- /dev/null
+++ b/CLAUDE.md
@@ -0,0 +1 @@
+AGENTS.md
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org