You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hugegraph.apache.org by ji...@apache.org on 2026/01/29 07:39:33 UTC
(incubator-hugegraph-computer) 01/01: docs: add AGENTS.md and update docs & gitignore
This is an automated email from the ASF dual-hosted git repository.
jin pushed a commit to branch docs
in repository https://gitbox.apache.org/repos/asf/incubator-hugegraph-computer.git
commit 5a6a5fe7374f831603ffdbe6efd76132b8a34da1
Author: imbajin <ji...@apache.org>
AuthorDate: Thu Jan 29 15:39:19 2026 +0800
docs: add AGENTS.md and update docs & gitignore
Add AI-assistant guidance files (AGENTS.md) at repository root and under vermeer, and expand documentation across the project: significantly update top-level README.md, computer/README.md, and vermeer/README.md with architecture, quick-starts, build/test instructions, and examples. Also update CI badge link in README and add AI-assistant-specific ignore patterns to .gitignore and vermeer/.gitignore to avoid tracking assistant artifacts.
---
.gitignore | 9 +
AGENTS.md | 237 +++++++++++++++++++++++
README.md | 234 ++++++++++++++++++++---
computer/README.md | 523 ++++++++++++++++++++++++++++++++++++++++++++++++++-
vermeer/.gitignore | 10 +
vermeer/AGENTS.md | 207 +++++++++++++++++++++
vermeer/README.md | 536 +++++++++++++++++++++++++++++++++++++++++++++++------
7 files changed, 1664 insertions(+), 92 deletions(-)
diff --git a/.gitignore b/.gitignore
index e909d27e..99283ffa 100644
--- a/.gitignore
+++ b/.gitignore
@@ -55,6 +55,15 @@ build/
*.log
*.pyc
+# AI assistant specific files (we only maintain AGENTS.md)
+CLAUDE.md
+GEMINI.md
+CURSOR.md
+COPILOT.md
+.cursorrules
+.cursor/
+.github/copilot-instructions.md
+
# maven ignore
apache-hugegraph-*-incubating-*/
diff --git a/AGENTS.md b/AGENTS.md
new file mode 100644
index 00000000..b3afd40e
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1,237 @@
+# AGENTS.md
+
+This file provides guidance to AI coding assistants when working with code in this repository.
+
+## Repository Overview
+
+This is the Apache HugeGraph-Computer repository containing two distinct graph computing systems:
+
+1. **computer** (Java/Maven): A distributed BSP/Pregel-style graph processing framework that runs on Kubernetes or YARN
+2. **vermeer** (Go): A high-performance in-memory graph computing platform with master-worker architecture
+
+Both integrate with HugeGraph for graph data input/output.
+
+## Build & Test Commands
+
+### Computer (Java)
+
+**Prerequisites:**
+- JDK 11 for building/running
+- JDK 8 for HDFS dependencies
+- Maven 3.5+
+- For K8s module: run `mvn clean install` first to generate CRD classes under computer-k8s
+
+**Build:**
+```bash
+cd computer
+mvn clean compile -Dmaven.javadoc.skip=true
+```
+
+**Tests:**
+```bash
+# Unit tests
+mvn test -P unit-test
+
+# Integration tests
+mvn test -P integrate-test
+```
+
+**Run single test:**
+```bash
+# Run specific test class
+mvn test -P unit-test -Dtest=ClassName
+
+# Run specific test method
+mvn test -P unit-test -Dtest=ClassName#methodName
+```
+
+**License check:**
+```bash
+mvn apache-rat:check
+```
+
+**Package:**
+```bash
+mvn clean package -DskipTests
+```
+
+### Vermeer (Go)
+
+**Prerequisites:**
+- Go 1.23+
+- `curl` and `unzip` (for downloading binary dependencies)
+
+**First-time setup:**
+```bash
+cd vermeer
+make init # Downloads supervisord and protoc binaries, installs Go deps
+```
+
+**Build:**
+```bash
+make # Build for current platform
+make build-linux-amd64
+make build-linux-arm64
+```
+
+**Development build with hot-reload UI:**
+```bash
+go build -tags=dev
+```
+
+**Clean:**
+```bash
+make clean # Remove built binaries and generated assets
+make clean-all # Also remove downloaded tools
+```
+
+**Run:**
+```bash
+# Using binary directly
+./vermeer --env=master
+./vermeer --env=worker
+
+# Using script (configure in vermeer.sh)
+./vermeer.sh start master
+./vermeer.sh start worker
+```
+
+**Regenerate protobuf (if proto files changed):**
+```bash
+go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28.0
+go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2.0
+tools/protoc/osxm1/protoc *.proto --go-grpc_out=. --go_out=.
+```
+
+## Architecture
+
+### Computer (Java) - BSP/Pregel Framework
+
+**Module Structure:**
+- `computer-api`: Public interfaces for graph processing (Computation, Vertex, Edge, Aggregator, Combiner, GraphFactory)
+- `computer-core`: Runtime implementation (WorkerService, MasterService, messaging, BSP coordination, managers)
+- `computer-algorithm`: Built-in algorithms (PageRank, LPA, WCC, SSSP, TriangleCount, etc.)
+- `computer-driver`: Job submission and driver-side coordination
+- `computer-k8s`: Kubernetes deployment integration
+- `computer-yarn`: YARN deployment integration
+- `computer-k8s-operator`: Kubernetes operator for job management
+- `computer-dist`: Distribution packaging
+- `computer-test`: Integration and unit tests
+
+**Key Design Patterns:**
+
+1. **API/Implementation Separation**: Algorithms depend only on `computer-api` interfaces; `computer-core` provides runtime implementation. Algorithms are dynamically loaded via config.
+
+2. **Manager Pattern**: `WorkerService` composes multiple managers (MessageSendManager, MessageRecvManager, WorkerAggrManager, DataServerManager, SortManagers, SnapshotManager, etc.) with lifecycle hooks: `initAll()`, `beforeSuperstep()`, `afterSuperstep()`, `closeAll()`.
+
+3. **BSP Coordination**: Explicit barrier synchronization via etcd (EtcdBspClient). Each superstep follows:
+ - `workerStepPrepareDone` β `waitMasterStepPrepareDone`
+ - Local compute (vertices process messages)
+ - `workerStepComputeDone` β `waitMasterStepComputeDone`
+ - Aggregators/snapshots
+ - `workerStepDone` β `waitMasterStepDone` (master returns SuperstepStat)
+
+4. **Computation Contract**: Algorithms implement `Computation<M extends Value>`:
+ - `compute0(context, vertex)`: Initialize at superstep 0
+ - `compute(context, vertex, messages)`: Process messages in subsequent supersteps
+ - Access to aggregators, combiners, and message sending via `ComputationContext`
+
+**Important Files:**
+- Algorithm contract: `computer/computer-api/src/main/java/org/apache/hugegraph/computer/core/worker/Computation.java`
+- Runtime orchestration: `computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java`
+- BSP coordination: `computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/bsp/Bsp4Worker.java`
+- Example algorithm: `computer/computer-algorithm/src/main/java/org/apache/hugegraph/computer/algorithm/centrality/pagerank/PageRank.java`
+
+### Vermeer (Go) - In-Memory Computing Engine
+
+**Directory Structure:**
+- `algorithms/`: Go algorithm implementations (pagerank.go, sssp.go, louvain.go, etc.)
+- `apps/`:
+ - `bsp/`: BSP coordination helpers
+ - `graphio/`: HugeGraph I/O adapters (reads via gRPC to store/pd, writes via HTTP REST)
+ - `master/`: Master scheduling, HTTP endpoints, worker management
+ - `compute/`: Worker-side compute logic
+ - `protos/`: Generated protobuf/gRPC definitions
+ - `common/`: Utilities, logging, metrics
+- `client/`: Client libraries
+- `tools/`: Binary dependencies (supervisord, protoc)
+- `ui/`: Web UI assets
+
+**Key Patterns:**
+
+1. **Maker/Registry Pattern**: Graph loaders/writers register themselves via init() (e.g., `LoadMakers[LoadTypeHugegraph] = &HugegraphMaker{}`). Master selects loader by type.
+
+2. **HugeGraph Integration**:
+ - `hugegraph.go` implements HugegraphMaker, HugegraphLoader, HugegraphWriter
+ - Queries PD via gRPC for partition metadata
+ - Streams vertex/edge data via gRPC from store (ScanPartition)
+ - Writes results back via HugeGraph HTTP REST API
+
+3. **Master-Worker**: Master schedules LoadPartition tasks to workers, manages worker lifecycle via WorkerManager/WorkerClient, exposes HTTP admin endpoints.
+
+**Important Files:**
+- HugeGraph integration: `vermeer/apps/graphio/hugegraph.go`
+- Master scheduling: `vermeer/apps/master/tasks/tasks.go`
+- Worker management: `vermeer/apps/master/workers/workers.go`
+- HTTP endpoints: `vermeer/apps/master/services/http_master.go`
+
+## Integration with HugeGraph
+
+**Computer (Java):**
+- `WorkerInputManager` reads vertices/edges from HugeGraph via `GraphFactory` abstraction
+- Graph data is partitioned and distributed to workers via input splits
+
+**Vermeer (Go):**
+- Directly queries HugeGraph PD (metadata service) for partition information
+- Uses gRPC to stream graph data from HugeGraph store
+- Writes computed results back via HugeGraph HTTP REST API (adds properties to vertices)
+
+## Development Workflow
+
+**Adding a New Algorithm (Computer):**
+1. Create class in `computer-algorithm` implementing `Computation<MessageType>`
+2. Implement `compute0()` for initialization and `compute()` for message processing
+3. Use `context.sendMessage()` or `context.sendMessageToAllEdges()` for message passing
+4. Register aggregators in `beforeSuperstep()`, read/write in `compute()`
+5. Configure algorithm class name in job config
+
+**K8s-Operator Development:**
+- CRD classes are auto-generated; run `mvn clean install` in `computer-k8s-operator` first
+- Generated classes appear in `computer-k8s/target/generated-sources/`
+- CRD generation script: `computer-k8s-operator/crd-generate/Makefile`
+
+**Vermeer Asset Updates:**
+- Web UI assets must be regenerated after changes: `cd asset && go generate`
+- Or use `make generate-assets` from vermeer root
+- For dev mode with hot-reload: `go build -tags=dev`
+
+## Testing Notes
+
+**Computer:**
+- Integration tests require etcd, HDFS, HugeGraph, and Kubernetes (see `.github/workflows/computer-ci.yml`)
+- Test environment setup scripts in `computer-dist/src/assembly/travis/`
+- Unit tests run in isolation without external dependencies
+
+**Vermeer:**
+- Test scripts in `vermeer/test/`
+- Configuration files in `vermeer/config/` (master.ini, worker.ini templates)
+
+## CI/CD
+
+CI pipeline (`.github/workflows/computer-ci.yml`) runs:
+1. License check (Apache RAT)
+2. Setup HDFS (Hadoop 3.3.2)
+3. Setup Minikube/Kubernetes
+4. Load test data into HugeGraph
+5. Compile with Java 11
+6. Run integration tests (`-P integrate-test`)
+7. Run unit tests (`-P unit-test`)
+8. Upload coverage to Codecov
+
+## Important Notes
+
+- **Computer K8s module**: Must run `mvn clean install` before editing to generate CRD classes
+- **Java version**: Build requires JDK 11; HDFS dependencies require JDK 8
+- **Vermeer binary deps**: First-time builds need `make init` to download supervisord/protoc
+- **BSP coordination**: Computer uses etcd for barrier synchronization (configure via `BSP_ETCD_URL`)
+- **Memory management**: Both systems auto-manage memory by spilling to disk when needed
diff --git a/README.md b/README.md
index f557fc63..eee4c05e 100644
--- a/README.md
+++ b/README.md
@@ -1,50 +1,234 @@
# Apache HugeGraph-Computer
[](https://www.apache.org/licenses/LICENSE-2.0.html)
-[](https://github.com/apache/hugegraph-computer/actions/workflows/ci.yml)
+[](https://github.com/apache/hugegraph-computer/actions/workflows/computer-ci.yml)
[](https://codecov.io/gh/apache/incubator-hugegraph-computer)
[](https://hub.docker.com/repository/docker/hugegraph/hugegraph-computer)
[](https://deepwiki.com/apache/hugegraph-computer)
-The [hugegraph-computer](./computer/README.md) is a distributed graph processing system for hugegraph.
-(Also, the in-memory computing engine(vermeer) is on the way π§)
+Apache HugeGraph-Computer is a comprehensive graph computing solution providing two complementary systems for different deployment scenarios:
+
+- **[Vermeer](./vermeer/README.md)** (Go): High-performance in-memory computing engine for single-machine deployments
+- **[Computer](./computer/README.md)** (Java): Distributed BSP/Pregel framework for large-scale cluster computing
+
+## Quick Comparison
+
+| Feature | Vermeer (Go) | Computer (Java) |
+|---------|--------------|-----------------|
+| **Best for** | Single machine, quick start | Large-scale distributed computing |
+| **Deployment** | Single binary | Kubernetes or YARN cluster |
+| **Memory model** | In-memory first | Auto spill to disk |
+| **Setup time** | Minutes | Hours (requires K8s/YARN) |
+| **Algorithms** | 20+ algorithms | 45+ algorithms |
+| **Architecture** | Master-Worker | BSP (Bulk Synchronous Parallel) |
+| **API** | REST + gRPC | Java API |
+| **Web UI** | Built-in dashboard | N/A |
+| **Data sources** | HugeGraph, CSV, HDFS | HugeGraph, HDFS |
+
+## Architecture Overview
+
+```mermaid
+graph TB
+ subgraph HugeGraph-Computer
+ subgraph Vermeer["Vermeer (Go) - In-Memory Engine"]
+ VM[Master :6688] --> VW1[Worker 1 :6789]
+ VM --> VW2[Worker 2 :6789]
+ VM --> VW3[Worker N :6789]
+ end
+ subgraph Computer["Computer (Java) - Distributed BSP"]
+ CM[Master Service] --> CW1[Worker Pod 1]
+ CM --> CW2[Worker Pod 2]
+ CM --> CW3[Worker Pod N]
+ end
+ end
+
+ HG[(HugeGraph Server)] <--> Vermeer
+ HG <--> Computer
+
+ style Vermeer fill:#e1f5fe
+ style Computer fill:#fff3e0
+```
+
+## HugeGraph Ecosystem Integration
+
+```
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+β HugeGraph Ecosystem β
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
+β βββββββββββββββ βββββββββββββββ βββββββββββββββββββ β
+β β Hubble β β Toolchain β β HugeGraph-AI β β
+β β (UI) β β (Tools) β β (LLM/RAG) β β
+β ββββββββ¬βββββββ ββββββββ¬βββββββ ββββββββββ¬βββββββββ β
+β β β β β
+β ββββββββββββββββββββΌβββββββββββββββββββββ β
+β β β
+β βββββββββΌββββββββ β
+β β HugeGraph β β
+β β Server β β
+β βββββββββ¬ββββββββ β
+β β β
+β ββββββββββββββββββββΌβββββββββββββββββββ β
+β β β β β
+β ββββββββΌβββββββ ββββββββΌβββββββ βββββββΌββββββ β
+β β Vermeer β β Computer β β Store β β
+β β (Memory) β β (BSP/K8s) β β (PD) β β
+β βββββββββββββββ βββββββββββββββ βββββββββββββ β
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+```
-## Learn More
+## Getting Started with Vermeer (Recommended)
-The [project homepage](https://hugegraph.apache.org/docs/quickstart/hugegraph-computer/) contains more information about hugegraph-computer.
+For quick start and single-machine deployments, we recommend **Vermeer**:
-And here are links of other repositories:
+### Docker Quick Start
-1. [hugegraph](https://github.com/apache/hugegraph) (graph's core component - Graph server + PD + Store)
-2. [hugegraph-toolchain](https://github.com/apache/hugegraph-toolchain) (graph tools **[loader](https://github.com/apache/incubator-hugegraph-toolchain/tree/master/hugegraph-loader)/[dashboard](https://github.com/apache/incubator-hugegraph-toolchain/tree/master/hugegraph-hubble)/[tool](https://github.com/apache/incubator-hugegraph-toolchain/tree/master/hugegraph-tools)/[client](https://github.com/apache/incubator-hugegraph-toolchain/tree/master/hugegraph-client)**)
-3. [hugegraph-ai](https://github.com/apache/incubator-hugegraph-ai) (integrated **Graph AI/LLM/KG** system)
-4. [hugegraph-website](https://github.com/apache/hugegraph-doc) (**doc & website** code)
+```bash
+# Pull the image
+docker pull hugegraph/vermeer:latest
+# Run with docker-compose
+docker-compose up -d
+```
-## Note
+### Binary Quick Start
-- If some classes under computer-k8s cannot be found, you need to execute `mvn clean install` in advance to generate corresponding classes.
+```bash
+# Download and extract (example for Linux AMD64)
+wget https://github.com/apache/hugegraph-computer/releases/download/vX.X.X/vermeer-linux-amd64.tar.gz
+tar -xzf vermeer-linux-amd64.tar.gz
+cd vermeer
+
+# Run master and worker
+./vermeer --env=master &
+./vermeer --env=worker &
+```
+
+See the **[Vermeer README](./vermeer/README.md)** for detailed configuration and usage.
+
+## Getting Started with Computer (Distributed)
+
+For large-scale distributed graph processing across clusters:
+
+### Prerequisites
+
+- JDK 11 or later
+- Maven 3.5+
+- Kubernetes cluster or YARN cluster
+
+### Build from Source
+
+```bash
+cd computer
+mvn clean package -DskipTests
+```
+
+### Deploy on Kubernetes
+
+```bash
+# Configure your K8s cluster in computer-k8s module
+# Submit a graph computing job
+java -jar computer-driver.jar --config job-config.properties
+```
+
+See the **[Computer README](./computer/README.md)** for detailed deployment and development guide.
+
+## Supported Algorithms
+
+### Common Algorithms (Both Systems)
+
+| Category | Algorithms |
+|----------|-----------|
+| **Centrality** | PageRank, Personalized PageRank, Betweenness Centrality, Closeness Centrality, Degree Centrality |
+| **Community Detection** | Louvain, LPA (Label Propagation), SLPA, WCC (Weakly Connected Components) |
+| **Path Finding** | SSSP (Single Source Shortest Path), BFS (Breadth-First Search) |
+| **Graph Structure** | Triangle Count, K-Core, Clustering Coefficient, Cycle Detection |
+| **Similarity** | Jaccard Similarity |
+
+### Vermeer-Specific Features
+
+- In-memory optimized implementations
+- Weighted Louvain variant
+- REST API for algorithm execution
+
+### Computer-Specific Algorithms
+
+- Count Triangle (distributed implementation)
+- Rings detection
+- ClusteringCoefficient variations
+- Custom algorithm development framework
+
+See individual README files for complete algorithm lists and usage examples.
+
+## Performance Characteristics
+
+### Vermeer (In-Memory)
+
+- **Throughput**: Optimized for fast iteration on medium-sized graphs (millions of vertices/edges)
+- **Latency**: Sub-second query response via REST API
+- **Memory**: Requires graph to fit in total worker memory
+- **Scalability**: Horizontal scaling by adding worker nodes
+
+### Computer (Distributed BSP)
+
+- **Throughput**: Handles billions of vertices/edges via distributed processing
+- **Latency**: Batch-oriented with superstep barriers
+- **Memory**: Auto spill to disk when memory is insufficient
+- **Scalability**: Elastic scaling on K8s with pod autoscaling
+
+## Use Cases
+
+### When to Use Vermeer
+
+- Quick prototyping and experimentation
+- Interactive graph analytics with Web UI
+- Medium-scale graphs (up to hundreds of millions of edges)
+- Single-machine or small cluster deployments
+- REST API integration requirements
+
+### When to Use Computer
+
+- Large-scale batch processing (billions of vertices)
+- Existing Kubernetes or YARN infrastructure
+- Custom algorithm development with Java
+- Memory-constrained environments (auto spill to disk)
+- Integration with Hadoop ecosystem
+
+## Documentation
+
+- [Project Homepage](https://hugegraph.apache.org/docs/quickstart/hugegraph-computer/)
+- [Vermeer Documentation](./vermeer/README.md)
+- [Computer Documentation](./computer/README.md)
+- [HugeGraph Documentation](https://hugegraph.apache.org/docs/)
+
+## Related Projects
+
+1. [hugegraph](https://github.com/apache/hugegraph) - Graph database core (Server + PD + Store)
+2. [hugegraph-toolchain](https://github.com/apache/hugegraph-toolchain) - Graph tools (Loader/Hubble/Tools/Client)
+3. [hugegraph-ai](https://github.com/apache/incubator-hugegraph-ai) - Graph AI/LLM/Knowledge Graph system
+4. [hugegraph-website](https://github.com/apache/hugegraph-doc) - Documentation and website
## Contributing
-- Welcome to contribute to HugeGraph, please see [How to Contribute](https://hugegraph.apache.org/docs/contribution-guidelines/contribute/) for more information.
-- Note: It's recommended to use [GitHub Desktop](https://desktop.github.com/) to greatly simplify the PR and commit process.
-- Thank you to all the people who already contributed to HugeGraph!
+Welcome to contribute to HugeGraph-Computer! Please see:
-[](https://github.com/apache/incubator-hugegraph-computer/graphs/contributors)
+- [How to Contribute](https://hugegraph.apache.org/docs/contribution-guidelines/contribute/) for guidelines
+- [GitHub Issues](https://github.com/apache/hugegraph-computer/issues) for bug reports and feature requests
-## License
+We recommend using [GitHub Desktop](https://desktop.github.com/) to simplify the PR process.
+
+Thank you to all contributors!
-hugegraph-computer is licensed under [Apache 2.0](https://github.com/apache/incubator-hugegraph-computer/blob/master/LICENSE) License.
+[](https://github.com/apache/incubator-hugegraph-computer/graphs/contributors)
-### Contact Us
+## License
----
+HugeGraph-Computer is licensed under [Apache 2.0 License](https://github.com/apache/incubator-hugegraph-computer/blob/master/LICENSE).
- - [GitHub Issues](https://github.com/apache/incubator-hugegraph-computer/issues): Feedback on usage issues and functional requirements (quick response)
- - Feedback Email: [dev@hugegraph.apache.org](mailto:dev@hugegraph.apache.org) ([subscriber](https://hugegraph.apache.org/docs/contribution-guidelines/subscribe/) only)
- - Slack: [join the ASF HugeGraph channel](https://the-asf.slack.com/archives/C059UU2FJ23)
- - WeChat public account: Apache HugeGraph, welcome to scan this QR code to follow us.
+## Contact Us
- <img src="https://github.com/apache/hugegraph-doc/blob/master/assets/images/wechat.png?raw=true" alt="QR png" width="350"/>
+- **GitHub Issues**: [Report bugs or request features](https://github.com/apache/incubator-hugegraph-computer/issues)
+- **Email**: [dev@hugegraph.apache.org](mailto:dev@hugegraph.apache.org) ([subscribe first](https://hugegraph.apache.org/docs/contribution-guidelines/subscribe/))
+- **Slack**: [Join ASF HugeGraph channel](https://the-asf.slack.com/archives/C059UU2FJ23)
+- **WeChat**: Scan QR code to follow Apache HugeGraph official account
+<img src="https://github.com/apache/hugegraph-doc/blob/master/assets/images/wechat.png?raw=true" alt="WeChat QR Code" width="350"/>
diff --git a/computer/README.md b/computer/README.md
index 7368f088..173c576e 100644
--- a/computer/README.md
+++ b/computer/README.md
@@ -1,12 +1,519 @@
-# Apache HugeGraph-Computer
+# Apache HugeGraph-Computer (Java)
-The hugegraph-computer is a distributed graph processing system for hugegraph. It is an implementation of [Pregel](https://kowshik.github.io/JPregel/pregel_paper.pdf). It runs on Kubernetes or YARN framework.
+HugeGraph-Computer is a distributed graph processing framework implementing the [Pregel](https://kowshik.github.io/JPregel/pregel_paper.pdf) model (BSP - Bulk Synchronous Parallel). It runs on Kubernetes or YARN clusters and integrates with HugeGraph for graph input/output.
## Features
-- Support distributed MPP graph computing, and integrates with HugeGraph as graph input/output storage.
-- Based on BSP (Bulk Synchronous Parallel) model, an algorithm performs computing through multiple parallel iterations, every iteration is a superstep.
-- Auto memory management. The framework will never be OOM(Out of Memory) since it will split some data to disk if it doesn't have enough memory to hold all the data.
-- The part of edges or the messages of supernode can be in memory, so you will never lose it.
-- You can load the data from HDFS or HugeGraph, output the results to HDFS or HugeGraph, or adapt any other systems manually as needed.
-- Easy to develop a new algorithm. You need to focus on a vertex only processing just like as in a single server, without worrying about message transfer and memory/storage management.
+- **Distributed MPP Computing**: Massively parallel graph processing across cluster nodes
+- **BSP Model**: Algorithm execution through iterative supersteps with global synchronization
+- **Auto Memory Management**: Automatic spill to disk when memory is insufficient - never OOM
+- **Flexible Data Sources**: Load from HugeGraph or HDFS, output to HugeGraph or HDFS
+- **Easy Algorithm Development**: Focus on single-vertex logic without worrying about distribution
+- **Production-Ready**: Battle-tested on billion-scale graphs with Kubernetes integration
+
+## Architecture
+
+### Module Structure
+
+```
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+β HugeGraph-Computer β
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
+β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
+β β computer-driver β β
+β β (Job Submission & Coordination) β β
+β βββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββ β
+β β β
+β βββββββββββββββββββββββββββΌββββββββββββββββββββββββββββ β
+β β Deployment Layer (choose one) β β
+β β ββββββββββββββββ ββββββββββββββββ β β
+β β β computer-k8s β β computer-yarnβ β β
+β β ββββββββββββββββ ββββββββββββββββ β β
+β βββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββ β
+β β β
+β βββββββββββββββββββββββββββΌββββββββββββββββββββββββββββ β
+β β computer-core β β
+β β (WorkerService, MasterService, BSP) β β
+β β ββββββββββββββββββββββββββββββββββββββββββββββββ β β
+β β β Managers: Message, Aggregation, Snapshot... β β β
+β β ββββββββββββββββββββββββββββββββββββββββββββββββ β β
+β βββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββ β
+β β β
+β βββββββββββββββββββββββββββΌββββββββββββββββββββββββββββ β
+β β computer-algorithm β β
+β β (PageRank, LPA, WCC, SSSP, TriangleCount...) β β
+β βββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββ β
+β β β
+β βββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββ β
+β β computer-api β β
+β β (Computation, Vertex, Edge, Aggregator, Value) β β
+β βββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+```
+
+### Module Descriptions
+
+| Module | Description |
+|--------|-------------|
+| **computer-api** | Public interfaces for algorithm development (`Computation`, `Vertex`, `Edge`, `Aggregator`, `Combiner`) |
+| **computer-core** | Runtime implementation (WorkerService, MasterService, messaging, BSP coordination, memory management) |
+| **computer-algorithm** | Built-in graph algorithms (45+ implementations) |
+| **computer-driver** | Job submission and driver-side coordination |
+| **computer-k8s** | Kubernetes deployment integration |
+| **computer-yarn** | YARN deployment integration |
+| **computer-k8s-operator** | Kubernetes operator for job lifecycle management |
+| **computer-dist** | Distribution packaging and assembly |
+| **computer-test** | Integration tests and unit tests |
+
+## Prerequisites
+
+- **JDK 11** or later (for building and running)
+- **Maven 3.5+** for building
+- **Kubernetes cluster** or **YARN cluster** for deployment
+- **etcd** for BSP coordination (configured via `BSP_ETCD_URL`)
+
+**Note**: For K8s-operator module development, run `mvn clean install` in `computer-k8s-operator` first to generate CRD classes.
+
+## Quick Start
+
+### Build from Source
+
+```bash
+cd computer
+
+# Compile (skip javadoc for faster builds)
+mvn clean compile -Dmaven.javadoc.skip=true
+
+# Package (skip tests for faster packaging)
+mvn clean package -DskipTests
+```
+
+### Run Tests
+
+```bash
+# Unit tests
+mvn test -P unit-test
+
+# Integration tests (requires etcd, K8s, HugeGraph)
+mvn test -P integrate-test
+
+# Run specific test class
+mvn test -P unit-test -Dtest=ClassName
+
+# Run specific test method
+mvn test -P unit-test -Dtest=ClassName#methodName
+```
+
+### License Check
+
+```bash
+mvn apache-rat:check
+```
+
+### Deploy on Kubernetes
+
+#### 1. Configure Job
+
+Create `job-config.properties`:
+
+```properties
+# Algorithm class
+algorithm.class=org.apache.hugegraph.computer.algorithm.centrality.pagerank.PageRank
+
+# HugeGraph connection
+hugegraph.url=http://hugegraph-server:8080
+hugegraph.graph=hugegraph
+
+# K8s configuration
+k8s.namespace=default
+k8s.image=hugegraph/hugegraph-computer:latest
+k8s.master.cpu=2
+k8s.master.memory=4Gi
+k8s.worker.replicas=3
+k8s.worker.cpu=4
+k8s.worker.memory=8Gi
+
+# BSP coordination (etcd)
+bsp.etcd.url=http://etcd-cluster:2379
+
+# Algorithm parameters (PageRank example)
+pagerank.damping_factor=0.85
+pagerank.max_iterations=20
+pagerank.convergence_tolerance=0.0001
+```
+
+#### 2. Submit Job
+
+```bash
+java -jar computer-driver/target/computer-driver-${VERSION}.jar \
+ --config job-config.properties
+```
+
+#### 3. Monitor Job
+
+```bash
+# Check pod status
+kubectl get pods -n default
+
+# View master logs
+kubectl logs hugegraph-computer-master-xxx -n default
+
+# View worker logs
+kubectl logs hugegraph-computer-worker-0 -n default
+```
+
+### Deploy on YARN
+
+**Note**: YARN deployment support is under development. Use Kubernetes for production deployments.
+
+## Available Algorithms
+
+### Centrality Algorithms
+
+| Algorithm | Class | Description |
+|-----------|-------|-------------|
+| PageRank | `algorithm.centrality.pagerank.PageRank` | Standard PageRank |
+| Personalized PageRank | `algorithm.centrality.pagerank.PersonalizedPageRank` | Source-specific PageRank |
+| Betweenness Centrality | `algorithm.centrality.betweenness.BetweennessCentrality` | Shortest-path-based centrality |
+| Closeness Centrality | `algorithm.centrality.closeness.ClosenessCentrality` | Average distance centrality |
+| Degree Centrality | `algorithm.centrality.degree.DegreeCentrality` | In/out degree counting |
+
+### Community Detection
+
+| Algorithm | Class | Description |
+|-----------|-------|-------------|
+| LPA | `algorithm.community.lpa.Lpa` | Label Propagation Algorithm |
+| WCC | `algorithm.community.wcc.Wcc` | Weakly Connected Components |
+| Louvain | `algorithm.community.louvain.Louvain` | Modularity-based community detection |
+| K-Core | `algorithm.community.kcore.KCore` | K-core decomposition |
+
+### Path Finding
+
+| Algorithm | Class | Description |
+|-----------|-------|-------------|
+| SSSP | `algorithm.path.sssp.Sssp` | Single Source Shortest Path |
+| BFS | `algorithm.traversal.bfs.Bfs` | Breadth-First Search |
+| Rings | `algorithm.path.rings.Rings` | Cycle/ring detection |
+
+### Graph Structure
+
+| Algorithm | Class | Description |
+|-----------|-------|-------------|
+| Triangle Count | `algorithm.trianglecount.TriangleCount` | Count triangles |
+| Clustering Coefficient | `algorithm.clusteringcoefficient.ClusteringCoefficient` | Local clustering measure |
+
+**Full algorithm list**: See `computer-algorithm/src/main/java/org/apache/hugegraph/computer/algorithm/`
+
+## Developing Custom Algorithms
+
+### Algorithm Contract
+
+Algorithms implement the `Computation` interface from `computer-api`:
+
+```java
+package org.apache.hugegraph.computer.core.worker;
+
+public interface Computation<M extends Value> {
+ /**
+ * Initialization at superstep 0
+ */
+ void compute0(ComputationContext context, Vertex vertex);
+
+ /**
+ * Message processing in subsequent supersteps
+ */
+ void compute(ComputationContext context, Vertex vertex, Iterator<M> messages);
+}
+```
+
+### Example: Simple PageRank
+
+```java
+package org.apache.hugegraph.computer.algorithm.centrality.pagerank;
+
+import org.apache.hugegraph.computer.core.worker.Computation;
+import org.apache.hugegraph.computer.core.worker.ComputationContext;
+
+public class PageRank implements Computation<DoubleValue> {
+
+ public static final String OPTION_ALPHA = "pagerank.alpha";
+ public static final String OPTION_MAX_ITERATIONS = "pagerank.max_iterations";
+
+ private double alpha;
+ private int maxIterations;
+
+ @Override
+ public void init(Config config) {
+ this.alpha = config.getDouble(OPTION_ALPHA, 0.85);
+ this.maxIterations = config.getInt(OPTION_MAX_ITERATIONS, 20);
+ }
+
+ @Override
+ public void compute0(ComputationContext context, Vertex vertex) {
+ // Initialize: set initial PR value
+ vertex.value(new DoubleValue(1.0));
+
+ // Send PR to neighbors
+ int edgeCount = vertex.numEdges();
+ if (edgeCount > 0) {
+ double contribution = 1.0 / edgeCount;
+ context.sendMessageToAllEdges(vertex, new DoubleValue(contribution));
+ }
+ }
+
+ @Override
+ public void compute(ComputationContext context, Vertex vertex, Iterator<DoubleValue> messages) {
+ // Sum incoming PR contributions
+ double sum = 0.0;
+ while (messages.hasNext()) {
+ sum += messages.next().value();
+ }
+
+ // Calculate new PR value
+ double newPR = (1.0 - alpha) + alpha * sum;
+ vertex.value(new DoubleValue(newPR));
+
+ // Send to neighbors if not converged
+ if (context.superstep() < maxIterations) {
+ int edgeCount = vertex.numEdges();
+ if (edgeCount > 0) {
+ double contribution = newPR / edgeCount;
+ context.sendMessageToAllEdges(vertex, new DoubleValue(contribution));
+ }
+ } else {
+ vertex.inactivate();
+ }
+ }
+}
+```
+
+### Key Concepts
+
+#### 1. Supersteps
+
+- **Superstep 0**: Initialization via `compute0()`
+- **Superstep 1+**: Message processing via `compute()`
+- **Barrier Synchronization**: All workers complete superstep N before starting N+1
+
+#### 2. Message Passing
+
+```java
+// Send to specific vertex
+context.sendMessage(targetId, new DoubleValue(1.0));
+
+// Send to all outgoing edges
+context.sendMessageToAllEdges(vertex, new DoubleValue(1.0));
+```
+
+#### 3. Aggregators
+
+Global state shared across all workers:
+
+```java
+// Register aggregator in compute0()
+context.registerAggregator("sum", new DoubleValue(0.0), SumAggregator.class);
+
+// Write to aggregator
+context.aggregateValue("sum", new DoubleValue(vertex.value()));
+
+// Read aggregator value (available in next superstep)
+DoubleValue total = context.aggregatedValue("sum");
+```
+
+#### 4. Combiners
+
+Reduce message volume by combining messages at sender:
+
+```java
+public class SumCombiner implements Combiner<DoubleValue> {
+ @Override
+ public void combine(DoubleValue v1, DoubleValue v2, DoubleValue result) {
+ result.value(v1.value() + v2.value());
+ }
+}
+```
+
+### Algorithm Development Workflow
+
+1. **Implement `Computation` interface** in `computer-algorithm`
+2. **Add configuration options** with `OPTION_*` constants
+3. **Implement `compute0()` for initialization**
+4. **Implement `compute()` for message processing**
+5. **Configure in job properties**:
+ ```properties
+ algorithm.class=com.example.MyAlgorithm
+ myalgorithm.param1=value1
+ ```
+6. **Build and test**:
+ ```bash
+ mvn clean package -DskipTests
+ ```
+
+## BSP Coordination
+
+HugeGraph-Computer uses etcd for BSP barrier synchronization:
+
+### BSP Lifecycle (per superstep)
+
+1. **Worker Prepare**: `workerStepPrepareDone` β `waitMasterStepPrepareDone`
+2. **Compute Phase**: Workers process vertices and messages locally
+3. **Worker Compute Done**: `workerStepComputeDone` β `waitMasterStepComputeDone`
+4. **Aggregation**: Aggregators combine global state
+5. **Worker Step Done**: `workerStepDone` β `waitMasterStepDone` (master returns `SuperstepStat`)
+
+### Manager Pattern
+
+`WorkerService` composes multiple managers with lifecycle hooks:
+
+- `MessageSendManager`: Outgoing message buffering and sending
+- `MessageRecvManager`: Incoming message receiving and sorting
+- `WorkerAggrManager`: Aggregator value collection
+- `DataServerManager`: Inter-worker data transfer
+- `SortManagers`: Message and edge sorting
+- `SnapshotManager`: Checkpoint creation
+
+All managers implement:
+- `initAll()`: Initialize before first superstep
+- `beforeSuperstep()`: Prepare for superstep
+- `afterSuperstep()`: Cleanup after superstep
+- `closeAll()`: Shutdown cleanup
+
+## Configuration Reference
+
+### Job Configuration
+
+```properties
+# === Algorithm ===
+algorithm.class=<fully qualified class name>
+algorithm.message_class=<message value class>
+algorithm.result_class=<result value class>
+
+# === HugeGraph Input ===
+hugegraph.url=http://localhost:8080
+hugegraph.graph=hugegraph
+hugegraph.input.vertex_label=person
+hugegraph.input.edge_label=knows
+hugegraph.input.filter=<gremlin filter>
+
+# === HugeGraph Output ===
+hugegraph.output.vertex_property=pagerank_value
+hugegraph.output.edge_property=<property name>
+
+# === HDFS Input ===
+input.hdfs.path=/graph/input
+input.hdfs.format=json
+
+# === HDFS Output ===
+output.hdfs.path=/graph/output
+output.hdfs.format=json
+
+# === Worker Resources ===
+worker.count=3
+worker.memory=8Gi
+worker.cpu=4
+worker.thread_count=<cpu cores>
+
+# === BSP Coordination ===
+bsp.etcd.url=http://etcd:2379
+bsp.max_superstep=100
+bsp.log_interval=10
+
+# === Memory Management ===
+worker.data.dirs=/data1,/data2
+worker.write_buffer_size=134217728
+worker.max_spill_size=1073741824
+```
+
+## Memory Management
+
+Computer auto-manages memory to prevent OOM:
+
+1. **In-Memory Buffering**: Vertices, edges, messages buffered in memory
+2. **Spill Threshold**: When memory usage exceeds threshold, spill to disk
+3. **Disk Storage**: Configurable data directories (`worker.data.dirs`)
+4. **Automatic Cleanup**: Spilled data cleaned after superstep completion
+
+**Best Practice**: Allocate worker memory β₯ 2x graph size for optimal performance.
+
+## Troubleshooting
+
+### K8s CRD Classes Not Found
+
+```bash
+# Generate CRD classes first
+cd computer-k8s-operator
+mvn clean install
+```
+
+Generated classes appear in `computer-k8s/target/generated-sources/`.
+
+### etcd Connection Errors
+
+- Verify `bsp.etcd.url` is reachable from all pods
+- Check etcd cluster health: `etcdctl endpoint health`
+- Ensure firewall allows port 2379
+
+### Out of Memory Errors
+
+- Increase `worker.memory` in job config
+- Reduce `worker.write_buffer_size` to trigger earlier spilling
+- Increase `worker.count` to distribute graph across more workers
+
+### Slow Convergence
+
+- Check algorithm parameters (e.g., `pagerank.convergence_tolerance`)
+- Monitor superstep logs for progress
+- Consider using combiners to reduce message volume
+
+## Important Files
+
+| File | Description |
+|------|-------------|
+| `computer-api/.../Computation.java` | Algorithm interface contract (computer/computer-api/src/main/java/org/apache/hugegraph/computer/core/worker/Computation.java:25) |
+| `computer-core/.../WorkerService.java` | Worker runtime orchestration (computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java:1) |
+| `computer-core/.../Bsp4Worker.java` | BSP coordination logic (computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/bsp/Bsp4Worker.java:1) |
+| `computer-algorithm/.../PageRank.java` | Example algorithm implementation (computer/computer-algorithm/src/main/java/org/apache/hugegraph/computer/algorithm/centrality/pagerank/PageRank.java:1) |
+
+## Testing
+
+### CI/CD Pipeline
+
+The CI pipeline (`.github/workflows/computer-ci.yml`) runs:
+
+1. License check (Apache RAT)
+2. Setup HDFS (Hadoop 3.3.2)
+3. Setup Minikube/Kubernetes
+4. Load test data into HugeGraph
+5. Compile with JDK 11
+6. Run integration tests (`-P integrate-test`)
+7. Run unit tests (`-P unit-test`)
+8. Upload coverage to Codecov
+
+### Local Testing
+
+```bash
+# Setup test environment (etcd, HDFS, K8s)
+cd computer-dist/src/assembly/travis
+./start-etcd.sh
+./start-hdfs.sh
+./start-minikube.sh
+
+# Run tests
+cd ../../../../
+mvn test -P integrate-test
+```
+
+## Links
+
+- [Project Homepage](https://hugegraph.apache.org/docs/quickstart/hugegraph-computer/)
+- [Main README](../README.md)
+- [Vermeer (Go) README](../vermeer/README.md)
+- [GitHub Issues](https://github.com/apache/hugegraph-computer/issues)
+
+## Contributing
+
+See the main [Contributing Guide](../README.md#contributing) for how to contribute.
+
+## License
+
+HugeGraph-Computer is licensed under [Apache 2.0 License](https://github.com/apache/incubator-hugegraph-computer/blob/master/LICENSE).
diff --git a/vermeer/.gitignore b/vermeer/.gitignore
index b9d714e5..8fb775a2 100644
--- a/vermeer/.gitignore
+++ b/vermeer/.gitignore
@@ -98,3 +98,13 @@ tools/protoc/*/include/
# Generated files (should be generated via go generate) #
######################
asset/assets_vfsdata.go
+
+# AI assistant specific files (we only maintain AGENTS.md) #
+######################
+CLAUDE.md
+GEMINI.md
+CURSOR.md
+COPILOT.md
+.cursorrules
+.cursor/
+.github/copilot-instructions.md
diff --git a/vermeer/AGENTS.md b/vermeer/AGENTS.md
new file mode 100644
index 00000000..2fb2df04
--- /dev/null
+++ b/vermeer/AGENTS.md
@@ -0,0 +1,207 @@
+# AGENTS.md
+
+This file provides guidance to AI coding assistants when working with code in this repository.
+
+## Repository Overview
+
+Vermeer is a high-performance in-memory graph computing platform written in Go. It features a single-binary deployment model with master-worker architecture, supporting 20+ graph algorithms and seamless HugeGraph integration.
+
+## Build & Test Commands
+
+**Prerequisites:**
+- Go 1.23+
+- `curl` and `unzip` (for downloading binary dependencies)
+
+**First-time setup:**
+```bash
+make init # Downloads supervisord and protoc binaries, installs Go deps
+```
+
+**Build:**
+```bash
+make # Build for current platform
+make build-linux-amd64
+make build-linux-arm64
+```
+
+**Development build with hot-reload UI:**
+```bash
+go build -tags=dev
+```
+
+**Clean:**
+```bash
+make clean # Remove built binaries and generated assets
+make clean-all # Also remove downloaded tools
+```
+
+**Run:**
+```bash
+# Using binary directly
+./vermeer --env=master
+./vermeer --env=worker
+
+# Using script (configure in vermeer.sh)
+./vermeer.sh start master
+./vermeer.sh start worker
+```
+
+**Tests:**
+```bash
+# Run with build tag vermeer_test
+go test -tags=vermeer_test -v
+
+# Specific test modes
+go test -tags=vermeer_test -v -mode=algorithms
+go test -tags=vermeer_test -v -mode=function
+go test -tags=vermeer_test -v -mode=scheduler
+```
+
+**Regenerate protobuf (if proto files changed):**
+```bash
+go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28.0
+go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2.0
+tools/protoc/osxm1/protoc *.proto --go-grpc_out=. --go_out=.
+```
+
+## Architecture
+
+### Directory Structure
+
+```
+vermeer/
+βββ main.go # Single binary entry point
+βββ algorithms/ # Algorithm implementations
+β βββ algorithms.go # AlgorithmMaker registry
+β βββ pagerank.go
+β βββ louvain.go
+β βββ ...
+βββ apps/
+β βββ master/ # Master service
+β β βββ services/ # HTTP handlers
+β β βββ workers/ # Worker management (WorkerManager, WorkerClient)
+β β βββ tasks/ # Task scheduling
+β β βββ graphs/ # Graph metadata management
+β βββ worker/ # Worker service entry
+β βββ compute/ # Worker-side compute logic
+β β βββ api.go # Algorithm interface definition
+β β βββ task.go # Compute task execution
+β β βββ ...
+β βββ graphio/ # Graph I/O (HugeGraph, CSV, HDFS)
+β β βββ hugegraph.go # HugeGraph integration
+β βββ protos/ # gRPC definitions
+β βββ common/ # Utilities, logging, metrics
+β βββ structure/ # Graph data structures
+β βββ storage/ # Persistence layer
+β βββ bsp/ # BSP coordination helpers
+βββ config/ # Configuration templates
+βββ tools/ # Binary dependencies (supervisord, protoc)
+βββ ui/ # Web dashboard
+```
+
+### Key Design Patterns
+
+**1. Maker/Registry Pattern**
+
+Graph loaders and writers register themselves via `init()`:
+
+```go
+func init() {
+ LoadMakers[LoadTypeHugegraph] = &HugegraphMaker{}
+}
+```
+
+Master selects loader by type from the registry. Algorithms follow the same pattern in `algorithms/algorithms.go`.
+
+**2. Master-Worker Architecture**
+
+- **Master**: Schedules LoadPartition tasks to workers, manages worker lifecycle via WorkerManager/WorkerClient, exposes HTTP endpoints for graph/task management
+- **Worker**: Executes compute tasks, reports status back to master via gRPC
+- Communication: Master uses gRPC clients to workers (apps/master/workers/); workers connect to master on startup
+
+**3. HugeGraph Integration**
+
+Implementation in `apps/graphio/hugegraph.go`:
+
+1. **Metadata Query**: Queries HugeGraph PD (metadata service) via gRPC for partition information
+2. **Data Loading**: Streams vertices/edges from HugeGraph Store via gRPC (`ScanPartition`)
+3. **Result Writing**: Writes computed results back via HugeGraph HTTP REST API (adds vertex properties)
+
+The loader queries PD first (`QueryPartitions`), then creates LoadPartition tasks for each partition, which workers execute by calling `ScanPartition` on store nodes.
+
+**4. Algorithm Interface**
+
+Algorithms implement the interface defined in `apps/compute/api.go`. Each algorithm must register itself in `algorithms/algorithms.go` by appending to the `Algorithms` slice.
+
+**5. Single Binary Entry Point**
+
+`main.go` loads config from `config/{env}.ini`, then starts either master or worker based on `run_mode` parameter. The `--env` flag specifies which config file to use (e.g., `--env=master` loads `config/master.ini`).
+
+## Important Files
+
+- Entry point: `main.go`
+- Algorithm interface: `apps/compute/api.go`
+- Algorithm registry: `algorithms/algorithms.go`
+- HugeGraph integration: `apps/graphio/hugegraph.go`
+- Master scheduling: `apps/master/tasks/tasks.go`
+- Worker management: `apps/master/workers/workers.go`
+- HTTP endpoints: `apps/master/services/http_master.go`
+
+## Development Workflow
+
+**Adding a New Algorithm:**
+
+1. Create file in `algorithms/` implementing the interface from `apps/compute/api.go`
+2. Register in `algorithms/algorithms.go` by appending to `Algorithms` slice
+3. Implement required methods: `Init()`, `Compute()`, `Aggregate()`, `Terminate()`
+4. Rebuild: `make`
+
+**Modifying Web UI:**
+
+1. Edit files in `ui/`
+2. Regenerate assets: `cd asset && go generate`
+3. Or use dev build: `go build -tags=dev` (hot-reload enabled)
+
+**Modifying Protobuf Definitions:**
+
+1. Edit `.proto` files in `apps/protos/`
+2. Regenerate Go code using protoc (adjust path for platform):
+ ```bash
+ tools/protoc/osxm1/protoc apps/protos/*.proto --go-grpc_out=. --go_out=.
+ ```
+
+## Configuration
+
+**Master (`config/master.ini`):**
+- `http_peer`: Master HTTP listen address (default: 0.0.0.0:6688)
+- `grpc_peer`: Master gRPC listen address (default: 0.0.0.0:6689)
+- `run_mode`: Must be "master"
+- `task_parallel_num`: Number of parallel tasks
+
+**Worker (`config/worker.ini`):**
+- `http_peer`: Worker HTTP listen address (default: 0.0.0.0:6788)
+- `grpc_peer`: Worker gRPC listen address (default: 0.0.0.0:6789)
+- `master_peer`: Master gRPC address to connect (must match master's `grpc_peer`)
+- `run_mode`: Must be "worker"
+
+## Memory Management
+
+Vermeer uses an in-memory-first approach. Graphs are distributed across workers and stored in memory. Ensure total worker memory exceeds graph size by 2-3x for algorithm workspace.
+
+## Testing Notes
+
+Tests require the build tag `vermeer_test`:
+
+```bash
+go test -tags=vermeer_test -v
+```
+
+Test modes (set via `-mode` flag):
+- `algorithms`: Algorithm correctness tests
+- `function`: Functional integration tests
+- `scheduler`: Scheduler behavior tests
+
+Test configuration via flags:
+- `-master`: Master HTTP address
+- `-worker01/02/03`: Worker HTTP addresses
+- `-auth`: Authentication type
diff --git a/vermeer/README.md b/vermeer/README.md
index 2d9e8207..f1e71ab5 100644
--- a/vermeer/README.md
+++ b/vermeer/README.md
@@ -1,125 +1,543 @@
-# Vermeer Graph Compute Engine
+# Vermeer - High-Performance In-Memory Graph Computing
+
+Vermeer is a high-performance in-memory graph computing platform with a single-binary deployment model. It provides 20+ graph algorithms, custom algorithm extensions, and seamless integration with HugeGraph.
+
+## Key Features
+
+- **Single Binary Deployment**: Zero external dependencies, run anywhere
+- **In-Memory Performance**: Optimized for fast iteration on medium to large graphs
+- **Master-Worker Architecture**: Horizontal scalability by adding worker nodes
+- **REST API + gRPC**: Easy integration with existing systems
+- **Web UI Dashboard**: Built-in monitoring and job management
+- **Multi-Source Support**: HugeGraph, local CSV, HDFS
+- **20+ Graph Algorithms**: Production-ready implementations
+
+## Architecture
+
+```mermaid
+graph TB
+ subgraph Client["Client Layer"]
+ API[REST API Client]
+ UI[Web UI Dashboard]
+ end
+
+ subgraph Master["Master Node"]
+ HTTP[HTTP Server :6688]
+ GRPC_M[gRPC Server :6689]
+ GM[Graph Manager]
+ TM[Task Manager]
+ WM[Worker Manager]
+ SCH[Scheduler]
+ end
+
+ subgraph Workers["Worker Nodes"]
+ W1[Worker 1 :6789]
+ W2[Worker 2 :6789]
+ W3[Worker N :6789]
+ end
+
+ subgraph DataSources["Data Sources"]
+ HG[(HugeGraph)]
+ CSV[Local CSV]
+ HDFS[HDFS]
+ end
+
+ API --> HTTP
+ UI --> HTTP
+ HTTP --> GM
+ HTTP --> TM
+ GRPC_M <--> W1
+ GRPC_M <--> W2
+ GRPC_M <--> W3
+
+ W1 <--> HG
+ W2 <--> HG
+ W3 <--> HG
+ W1 <--> CSV
+ W1 <--> HDFS
+
+ style Master fill:#e1f5fe
+ style Workers fill:#fff3e0
+ style DataSources fill:#f1f8e9
+```
+
+### Directory Structure
-## Introduction
-Vermeer is a high-performance distributed graph computing platform based on memory, supporting more than 15 graph algorithms, custom algorithm extensions, and custom data source access.
+```
+vermeer/
+βββ main.go # Single binary entry point
+βββ Makefile # Build automation
+βββ algorithms/ # 20+ algorithm implementations
+β βββ pagerank.go
+β βββ louvain.go
+β βββ sssp.go
+β βββ ...
+βββ apps/
+β βββ master/ # Master service
+β β βββ services/ # HTTP handlers
+β β βββ workers/ # Worker management
+β β βββ tasks/ # Task scheduling
+β βββ compute/ # Worker-side compute logic
+β βββ graphio/ # Graph I/O (HugeGraph, CSV, HDFS)
+β β βββ hugegraph.go # HugeGraph integration
+β βββ protos/ # gRPC definitions
+β βββ common/ # Utilities, logging, metrics
+βββ config/ # Configuration templates
+β βββ master.ini
+β βββ worker.ini
+βββ tools/ # Binary dependencies (supervisord, protoc)
+βββ ui/ # Web dashboard
+```
-## Run with Docker
+## Quick Start
+
+### Option 1: Docker (Recommended)
Pull the image:
-```
+
+```bash
docker pull hugegraph/vermeer:latest
```
-Create local configuration files, for example, `~/master.ini` and `~/worker.ini`.
+Create local configuration files `~/master.ini` and `~/worker.ini` (see [Configuration](#configuration) section).
-Run with Docker. The `--env` flag specifies the file name.
+Run with Docker:
-```
-master: docker run -v ~/:/go/bin/config hugegraph/vermeer --env=master
-worker: docker run -v ~/:/go/bin/config hugegraph/vermeer --env=worker
+```bash
+# Master node
+docker run -v ~/:/go/bin/config hugegraph/vermeer --env=master
+
+# Worker node
+docker run -v ~/:/go/bin/config hugegraph/vermeer --env=worker
```
-We've also provided a `docker-compose` file. Once you've created `~/master.ini` and `~/worker.ini`, and updated the `master_peer` in `worker.ini` to `172.20.0.10:6689`, you can run it using the following command:
+#### Docker Compose
-```
+Update `master_peer` in `~/worker.ini` to `172.20.0.10:6689`, then:
+
+```bash
docker-compose up -d
```
-## Start
+### Option 2: Binary Download
+```bash
+# Download binary (replace version and platform)
+wget https://github.com/apache/hugegraph-computer/releases/download/vX.X.X/vermeer-linux-amd64.tar.gz
+tar -xzf vermeer-linux-amd64.tar.gz
+cd vermeer
+
+# Run master and worker
+./vermeer --env=master &
+./vermeer --env=worker &
```
-master: ./vermeer --env=master
-worker: ./vermeer --env=worker01
-```
-The parameter env specifies the name of the configuration file in the useconfig folder.
-```
+The `--env` parameter specifies the configuration file name in the `config/` folder (e.g., `master.ini`, `worker.ini`).
+
+#### Using the Shell Script
+
+Configure parameters in `vermeer.sh`, then:
+
+```bash
./vermeer.sh start master
./vermeer.sh start worker
```
-Configuration items are specified in vermeer.sh
-## supervisord
-Can be used with supervisord to start and stop services, automatically start applications, rotate logs, and more; for the configuration file, refer to config/supervisor.conf;
-Configuration file reference config/supervisor.conf
+### Option 3: Build from Source
-````
-# run as daemon
-./supervisord -c supervisor.conf -d
-````
+#### Prerequisites
-## Build from Source
+- Go 1.23 or later
+- `curl` and `unzip` utilities (for downloading dependencies)
+- Internet connection (for first-time setup)
-### Requirements
-* Go 1.23 or later
-* `curl` and `unzip` utilities (for downloading dependencies)
-* Internet connection (for first-time setup)
+#### Build Steps
-### Quick Start
-
-**Recommended**: Use Makefile for building:
+**Recommended**: Use Makefile:
```bash
-# First time setup (downloads binary dependencies)
+# First-time setup (downloads supervisord and protoc binaries)
make init
-# Build vermeer
+# Build for current platform
make
+
+# Or build for specific platform
+make build-linux-amd64
+make build-linux-arm64
```
-**Alternative**: Use the build script:
+**Alternative**: Use build script:
```bash
-# For AMD64
-./build.sh amd64
+# Auto-detect platform
+./build.sh
-# For ARM64
+# Or specify architecture
+./build.sh amd64
./build.sh arm64
```
-# The script will:
-# - Auto-detect your OS and architecture if no parameter is provided
-# - Download required tools if not present
-# - Generate assets and build the binary
-# - Exit with error message if any step fails
+#### Development Build
-### Build Targets
+For development with hot-reload of web UI:
```bash
-make build # Build for current platform
-make build-linux-amd64 # Build for Linux AMD64
-make build-linux-arm64 # Build for Linux ARM64
-make clean # Clean generated files
-make help # Show all available targets
+go build -tags=dev
```
-### Development Build
+#### Clean Build Artifacts
-For development with hot-reload of web UI:
+```bash
+make clean # Remove binaries and generated assets
+make clean-all # Also remove downloaded tools (supervisord, protoc)
+```
+
+## Configuration
+
+### Master Configuration (`master.ini`)
+
+```ini
+[master]
+# Master server listen address
+listen_addr = :6688
+
+# Master gRPC server address
+grpc_addr = :6689
+
+# Worker heartbeat timeout (seconds)
+worker_timeout = 30
+
+# Task execution timeout (seconds)
+task_timeout = 3600
+
+[hugegraph]
+# HugeGraph PD address for metadata
+pd_peers = 127.0.0.1:8686
+
+# HugeGraph HTTP endpoint for result writing
+server = http://127.0.0.1:8080
+
+# Graph space name
+graph = hugegraph
+```
+
+### Worker Configuration (`worker.ini`)
+
+```ini
+[worker]
+# Worker listen address
+listen_addr = :6789
+
+# Master gRPC address to connect
+master_peer = 127.0.0.1:6689
+
+# Worker ID (unique)
+worker_id = worker01
+
+# Number of compute threads
+compute_threads = 4
+
+# Memory limit (GB)
+memory_limit = 8
+
+[storage]
+# Local disk path for spilling
+data_path = ./data
+```
+
+## Available Algorithms
+
+| Algorithm | Category | Description |
+|-----------|----------|-------------|
+| **PageRank** | Centrality | Measures vertex importance via link structure |
+| **Personalized PageRank** | Centrality | PageRank from specific source vertices |
+| **Betweenness Centrality** | Centrality | Measures vertex importance via shortest paths |
+| **Closeness Centrality** | Centrality | Measures average distance to all other vertices |
+| **Degree Centrality** | Centrality | Simple in/out degree calculation |
+| **Louvain** | Community Detection | Modularity-based community detection |
+| **Louvain (Weighted)** | Community Detection | Weighted variant for edge-weighted graphs |
+| **LPA** | Community Detection | Label Propagation Algorithm |
+| **SLPA** | Community Detection | Speaker-Listener Label Propagation |
+| **WCC** | Community Detection | Weakly Connected Components |
+| **SCC** | Community Detection | Strongly Connected Components |
+| **SSSP** | Path Finding | Single Source Shortest Path (Dijkstra) |
+| **Triangle Count** | Graph Structure | Counts triangles in the graph |
+| **K-Core** | Graph Structure | Finds k-core subgraphs |
+| **K-Out** | Graph Structure | K-degree filtering |
+| **Clustering Coefficient** | Graph Structure | Measures local clustering |
+| **Cycle Detection** | Graph Structure | Detects cycles in directed graphs |
+| **Jaccard Similarity** | Similarity | Computes neighbor-based similarity |
+| **Depth (BFS)** | Traversal | Breadth-First Search depth assignment |
+
+## API Overview
+
+Vermeer exposes a REST API on port `6688` (configurable in `master.ini`).
+
+### Key Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/api/v1/graphs` | POST | Load graph from data source |
+| `/api/v1/graphs/{graph_id}` | GET | Get graph metadata |
+| `/api/v1/graphs/{graph_id}` | DELETE | Unload graph from memory |
+| `/api/v1/compute` | POST | Execute algorithm on loaded graph |
+| `/api/v1/tasks/{task_id}` | GET | Get task status and results |
+| `/api/v1/workers` | GET | List connected workers |
+| `/ui/` | GET | Web UI dashboard |
+
+### Example: Run PageRank
```bash
-go build -tags=dev
+# 1. Load graph from HugeGraph
+curl -X POST http://localhost:6688/api/v1/graphs \
+ -H "Content-Type: application/json" \
+ -d '{
+ "graph_name": "my_graph",
+ "load_type": "hugegraph",
+ "hugegraph": {
+ "pd_peers": ["127.0.0.1:8686"],
+ "graph_name": "hugegraph"
+ }
+ }'
+
+# 2. Run PageRank
+curl -X POST http://localhost:6688/api/v1/compute \
+ -H "Content-Type: application/json" \
+ -d '{
+ "graph_name": "my_graph",
+ "algorithm": "pagerank",
+ "params": {
+ "max_iterations": 20,
+ "damping_factor": 0.85
+ },
+ "output": {
+ "type": "hugegraph",
+ "property_name": "pagerank_value"
+ }
+ }'
+
+# 3. Check task status
+curl http://localhost:6688/api/v1/tasks/{task_id}
+```
+
+### OLAP vs OLTP Modes
+
+- **OLAP Mode**: Load entire graph into memory, run multiple algorithms
+- **OLTP Mode**: Query-driven, load subgraphs on demand (planned feature)
+
+## Data Sources
+
+### HugeGraph Integration
+
+Vermeer integrates with HugeGraph via:
+
+1. **Metadata Query**: Queries HugeGraph PD (metadata service) via gRPC for partition information
+2. **Data Loading**: Streams vertices/edges from HugeGraph Store via gRPC (`ScanPartition`)
+3. **Result Writing**: Writes computed results back via HugeGraph REST API (adds vertex properties)
+
+Configuration in graph load request:
+
+```json
+{
+ "load_type": "hugegraph",
+ "hugegraph": {
+ "pd_peers": ["127.0.0.1:8686"],
+ "graph_name": "hugegraph",
+ "vertex_label": "person",
+ "edge_label": "knows"
+ }
+}
+```
+
+### Local CSV Files
+
+Load graphs from local CSV files:
+
+```json
+{
+ "load_type": "csv",
+ "csv": {
+ "vertex_file": "/path/to/vertices.csv",
+ "edge_file": "/path/to/edges.csv",
+ "delimiter": ","
+ }
+}
+```
+
+### HDFS
+
+Load from Hadoop Distributed File System:
+
+```json
+{
+ "load_type": "hdfs",
+ "hdfs": {
+ "namenode": "hdfs://namenode:9000",
+ "vertex_path": "/graph/vertices",
+ "edge_path": "/graph/edges"
+ }
+}
+```
+
+## Developing Custom Algorithms
+
+Custom algorithms implement the `Algorithm` interface in `algorithms/algorithms.go`:
+
+```go
+type Algorithm interface {
+ // Initialize the algorithm
+ Init(params map[string]interface{}) error
+
+ // Compute one iteration for a vertex
+ Compute(vertex *Vertex, messages []Message) (halt bool, outMessages []Message)
+
+ // Aggregate global state (optional)
+ Aggregate() interface{}
+
+ // Check termination condition
+ Terminate(iteration int) bool
+}
+```
+
+### Example: Simple Degree Count
+
+```go
+package algorithms
+
+type DegreeCount struct {
+ maxIter int
+}
+
+func (dc *DegreeCount) Init(params map[string]interface{}) error {
+ dc.maxIter = params["max_iterations"].(int)
+ return nil
+}
+
+func (dc *DegreeCount) Compute(vertex *Vertex, messages []Message) (bool, []Message) {
+ // Store degree as vertex value
+ vertex.SetValue(float64(len(vertex.OutEdges)))
+
+ // Halt after first iteration
+ return true, nil
+}
+
+func (dc *DegreeCount) Terminate(iteration int) bool {
+ return iteration >= dc.maxIter
+}
+```
+
+Register the algorithm in `algorithms/algorithms.go`:
+
+```go
+func init() {
+ RegisterAlgorithm("degree_count", &DegreeCount{})
+}
+```
+
+## Memory Management
+
+Vermeer uses an in-memory-first approach:
+
+1. **Graph Loading**: Vertices and edges are distributed across workers and stored in memory
+2. **Automatic Partitioning**: Master assigns partitions to workers based on capacity
+3. **Memory Monitoring**: Workers report memory usage to master
+4. **Graceful Degradation**: If memory is insufficient, algorithms may fail (disk spilling not yet implemented)
+
+**Best Practice**: Ensure total worker memory exceeds graph size by 2-3x for algorithm workspace.
+
+## Supervisord Integration
+
+Run Vermeer as a daemon with automatic restarts and log rotation:
+
+```bash
+# Configuration in config/supervisor.conf
+./tools/supervisord -c config/supervisor.conf -d
```
----
+Sample supervisor configuration:
-### Protobuf Development
+```ini
+[program:vermeer-master]
+command=/path/to/vermeer --env=master
+autostart=true
+autorestart=true
+stdout_logfile=/var/log/vermeer-master.log
+```
-If you need to regenerate protobuf files:
+## Protobuf Development
+
+If you modify `.proto` files, regenerate Go code:
```bash
# Install protobuf Go plugins
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28.0
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2.0
-# Generate protobuf files
+# Generate (adjust protoc path for your platform)
tools/protoc/osxm1/protoc *.proto --go-grpc_out=. --go_out=.
```
----
+## Performance Tuning
+
+### Worker Configuration
+
+- **compute_threads**: Set to number of CPU cores for CPU-bound algorithms
+- **memory_limit**: Set to 70-80% of available RAM
+- **partition_count**: Increase for better parallelism (default: auto-calculated)
+
+### Master Configuration
+
+- **worker_timeout**: Increase for slow networks or heavily loaded workers
+- **task_timeout**: Increase for long-running algorithms (e.g., Louvain on large graphs)
+
+### Algorithm-Specific
+
+- **PageRank**: Use `damping_factor=0.85`, `tolerance=0.0001` for faster convergence
+- **Louvain**: Enable `weighted=true` only if edge weights are meaningful
+- **SSSP**: Provide source vertex ID for single-source queries
+
+## Monitoring
+
+Access the Web UI dashboard at `http://master-ip:6688/ui/` for:
+
+- Worker status and resource usage
+- Active and completed tasks
+- Graph metadata and statistics
+- Real-time logs
+
+## Troubleshooting
+
+### Workers Not Connecting
+
+- Verify `master_peer` in `worker.ini` matches master's gRPC address
+- Check firewall rules for port `6689` (gRPC)
+- Ensure master is running before starting workers
+
+### Out of Memory Errors
+
+- Reduce graph size or increase worker memory
+- Distribute graph across more workers
+- Use algorithms with lower memory footprint (e.g., degree centrality vs. betweenness)
+
+### Slow Algorithm Execution
+
+- Increase `compute_threads` in worker config
+- Check network latency between master and workers
+- Profile algorithm with built-in metrics (access via API)
+## Links
+- [Project Homepage](https://hugegraph.apache.org/docs/quickstart/hugegraph-computer/)
+- [Main README](../README.md)
+- [Computer (Java) README](../computer/README.md)
+- [GitHub Issues](https://github.com/apache/hugegraph-computer/issues)
+- [Docker Hub](https://hub.docker.com/r/hugegraph/vermeer)
+## Contributing
+See the main [Contributing Guide](../README.md#contributing) for how to contribute to Vermeer.
+## License
+Vermeer is part of Apache HugeGraph-Computer, licensed under [Apache 2.0 License](https://github.com/apache/incubator-hugegraph-computer/blob/master/LICENSE).