You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2026/01/21 02:09:57 UTC
(skywalking) branch docs/claude-md-e2e-testing created (now 0d67d7266a)
This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a change to branch docs/claude-md-e2e-testing
in repository https://gitbox.apache.org/repos/asf/skywalking.git
at 0d67d7266a Add E2E testing documentation to CLAUDE.md
This branch includes the following new commits:
new 0d67d7266a Add E2E testing documentation to CLAUDE.md
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(skywalking) 01/01: Add E2E testing documentation to CLAUDE.md
Posted by wu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a commit to branch docs/claude-md-e2e-testing
in repository https://gitbox.apache.org/repos/asf/skywalking.git
commit 0d67d7266ad9105d9425a897aff662e7111c1dc5
Author: Wu Sheng <wu...@foxmail.com>
AuthorDate: Wed Jan 21 10:09:32 2026 +0800
Add E2E testing documentation to CLAUDE.md
- Add E2E tool installation instructions with pinned CI version
- Document test structure, configuration, and available test cases
- Add quick start guide for running e2e tests locally
- Include debugging workflow with rebuild detection logic
- Document environment variables and swctl verification commands
---
CLAUDE.md | 238 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 238 insertions(+)
diff --git a/CLAUDE.md b/CLAUDE.md
index 19704b8c23..8ef8f92288 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -206,6 +206,244 @@ JSON and Markdown files are excluded (JSON doesn't support comments, see `.licen
./mvnw package -Dmaven.test.skip
```
+## E2E Testing
+
+SkyWalking uses [Apache SkyWalking Infra E2E](https://github.com/apache/skywalking-infra-e2e) for end-to-end testing. E2E tests validate the entire system including OAP server, storage backends, agents, and integrations.
+
+### E2E Tool Installation
+
+```bash
+# Install the same version used in CI (recommended)
+go install github.com/apache/skywalking-infra-e2e/cmd/e2e@e7138da4f9b7a25a169c9f8d995795d4d2e34bde
+
+# Verify installation
+e2e --help
+```
+
+### E2E Test Structure
+
+```
+test/e2e-v2/
+├── cases/ # 50+ test case directories
+│ ├── simple/jdk/ # Basic Java agent test
+│ ├── storage/ # Storage backend tests (BanyanDB, ES, MySQL, PostgreSQL)
+│ ├── alarm/ # Alerting tests
+│ ├── profiling/ # Profiling tests (trace, eBPF, async)
+│ ├── kafka/ # Kafka integration
+│ ├── istio/ # Service mesh tests
+│ └── ...
+├── script/
+│ ├── env # Environment variables (agent commits, versions)
+│ ├── docker-compose/
+│ │ └── base-compose.yml # Base service definitions (oap, banyandb, provider, consumer)
+│ └── prepare/
+│ └── setup-e2e-shell/ # Tool installers (swctl, yq, kubectl, helm)
+└── java-test-service/ # Test service implementations
+ ├── e2e-service-provider/
+ ├── e2e-service-consumer/
+ └── ...
+```
+
+### E2E Configuration (e2e.yaml)
+
+Each test case has an `e2e.yaml` with four sections:
+
+```yaml
+setup:
+ env: compose # Environment: compose or kind (Kubernetes)
+ file: docker-compose.yml # Docker compose file
+ timeout: 20m # Setup timeout
+ init-system-environment: ../../../script/env # Shared env variables
+ steps: # Initialization steps
+ - name: install swctl
+ command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
+
+trigger:
+ action: http # Generate test traffic
+ interval: 3s
+ times: -1 # -1 = run until verify succeeds
+ url: http://${consumer_host}:${consumer_9092}/users
+ method: POST
+ body: '{"id":"123","name":"skywalking"}'
+
+verify:
+ retry:
+ count: 20
+ interval: 10s
+ cases:
+ - includes:
+ - ../simple-cases.yaml # Reusable verification cases
+ - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec ...
+ expected: expected/metrics.yml
+
+cleanup:
+ on: always # always|success|failure|never
+```
+
+### Running E2E Tests Locally
+
+**Prerequisites:**
+- Docker and Docker Compose
+- Go (for e2e tool installation)
+
+**Quick Start (run simple/jdk test):**
+```bash
+# 1. Build distribution and Docker image
+./mvnw clean package -Pall -Dmaven.test.skip
+make docker
+
+# 2. Build test services
+./mvnw -f test/e2e-v2/java-test-service/pom.xml clean package
+
+# 3. Run e2e test (SW_AGENT_JDK_VERSION is required)
+SW_AGENT_JDK_VERSION=8 e2e run -c test/e2e-v2/cases/simple/jdk/e2e.yaml
+```
+
+**Step-by-step debugging:**
+```bash
+# Set required environment variable
+export SW_AGENT_JDK_VERSION=8
+
+# Run individual steps instead of full test
+e2e setup -c test/e2e-v2/cases/simple/jdk/e2e.yaml # Start containers
+e2e trigger -c test/e2e-v2/cases/simple/jdk/e2e.yaml # Generate traffic
+e2e verify -c test/e2e-v2/cases/simple/jdk/e2e.yaml # Validate results
+e2e cleanup -c test/e2e-v2/cases/simple/jdk/e2e.yaml # Stop containers
+```
+
+### E2E CLI Commands
+
+| Command | Description |
+|---------|-------------|
+| `e2e run -c <path>` | Run complete test (setup → trigger → verify → cleanup) |
+| `e2e setup -c <path>` | Start containers and initialize environment |
+| `e2e trigger -c <path>` | Generate test traffic |
+| `e2e verify -c <path>` | Validate results against expected output |
+| `e2e cleanup -c <path>` | Stop and remove containers |
+
+### Common Test Cases
+
+| Category | Path | Description |
+|----------|------|-------------|
+| `simple/jdk` | `test/e2e-v2/cases/simple/jdk/` | Basic Java agent with BanyanDB |
+| `storage/banyandb` | `test/e2e-v2/cases/storage/banyandb/` | BanyanDB storage backend |
+| `storage/elasticsearch` | `test/e2e-v2/cases/storage/elasticsearch/` | Elasticsearch storage |
+| `alarm/` | `test/e2e-v2/cases/alarm/` | Alerting functionality |
+| `profiling/trace` | `test/e2e-v2/cases/profiling/trace/` | Trace profiling |
+| `log/` | `test/e2e-v2/cases/log/` | Log analysis (LAL) |
+
+### Writing E2E Tests
+
+1. **Create test directory** under `test/e2e-v2/cases/<category>/<name>/`
+
+2. **Create docker-compose.yml** extending base services:
+ ```yaml
+ version: '2.1'
+ services:
+ oap:
+ extends:
+ file: ../../../script/docker-compose/base-compose.yml
+ service: oap
+ banyandb:
+ extends:
+ file: ../../../script/docker-compose/base-compose.yml
+ service: banyandb
+ ```
+
+3. **Create e2e.yaml** with setup, trigger, verify sections
+
+4. **Create expected/ directory** with expected YAML outputs for verification
+
+5. **Create verification cases** (e.g., `simple-cases.yaml`) with swctl queries
+
+### Verification with swctl
+
+The `swctl` CLI queries OAP's GraphQL API:
+
+```bash
+# Query service metrics
+swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql \
+ metrics exec --expression=service_resp_time --service-name=e2e-service-provider
+
+# List services
+swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql \
+ service ls
+
+# Query traces
+swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql \
+ trace ls --service-name=e2e-service-provider
+```
+
+### Environment Variables
+
+Key version commits in `test/e2e-v2/script/env`:
+- `SW_AGENT_JAVA_COMMIT` - Java agent version
+- `SW_BANYANDB_COMMIT` - BanyanDB version
+- `SW_CTL_COMMIT` - swctl CLI version
+- `SW_AGENT_*_COMMIT` - Other agent versions (Go, Python, NodeJS, PHP)
+
+### Debugging E2E Tests
+
+**If a test fails, do NOT run cleanup immediately.** Keep containers running to debug:
+
+```bash
+# 1. Setup containers (only once)
+e2e setup -c test/e2e-v2/cases/simple/jdk/e2e.yaml
+
+# 2. Generate traffic
+e2e trigger -c test/e2e-v2/cases/simple/jdk/e2e.yaml
+
+# 3. Verify (can re-run multiple times after fixing issues)
+e2e verify -c test/e2e-v2/cases/simple/jdk/e2e.yaml
+
+# Check container logs to debug failures
+docker compose -f test/e2e-v2/cases/simple/jdk/docker-compose.yml logs oap
+docker compose -f test/e2e-v2/cases/simple/jdk/docker-compose.yml logs provider
+
+# Only cleanup when done debugging
+e2e cleanup -c test/e2e-v2/cases/simple/jdk/e2e.yaml
+```
+
+**Determining if rebuild is needed:**
+
+Compare file timestamps against last package build. If any files changed after package, rebuild is needed:
+```bash
+# Find runtime-related files modified after package was built
+find oap-server apm-protocol -type f \( \
+ -name "*.java" -o -name "*.yaml" -o -name "*.yml" -o \
+ -name "*.json" -o -name "*.xml" -o -name "*.properties" -o \
+ -name "*.proto" \
+\) -newer dist/apache-skywalking-apm-bin.tar.gz 2>/dev/null
+
+# Find test case files modified after package was built
+find test/e2e-v2 -type f \( \
+ -name "*.yaml" -o -name "*.yml" -o -name "*.java" -o -name "*.json" \
+\) -newer dist/apache-skywalking-apm-bin.tar.gz 2>/dev/null
+```
+
+Also compare git commit ID in binary vs current HEAD:
+```bash
+# Commit ID in packaged binary
+unzip -p dist/apache-skywalking-apm-bin/oap-libs/server-starter-*.jar version.properties | grep git.commit.id
+
+# Current HEAD
+git rev-parse HEAD
+```
+
+**If rebuild is needed, stop e2e first:**
+```bash
+# 1. Cleanup running containers
+e2e cleanup -c test/e2e-v2/cases/simple/jdk/e2e.yaml
+
+# 2. Rebuild
+./mvnw clean package -Pall -Dmaven.test.skip && make docker
+
+# 3. Restart e2e
+e2e setup -c test/e2e-v2/cases/simple/jdk/e2e.yaml
+e2e trigger -c test/e2e-v2/cases/simple/jdk/e2e.yaml
+e2e verify -c test/e2e-v2/cases/simple/jdk/e2e.yaml
+```
+
## Git Submodules
The project uses submodules for protocol definitions and UI: