[I] [server] ZooKeeper Removal and Embedded Raft Metadata Backend [fluss]

["XuQianJin-Stars (via GitHub)" <gi...@apache.org>]

XuQianJin-Stars opened a new issue, #4014:
URL: https://github.com/apache/fluss/issues/4014

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
   
   
   ### Motivation
   
   Fluss currently depends on an external Apache ZooKeeper ensemble for cluster coordination, metadata persistence, sequence IDs, TabletServer liveness, dynamic config, and ACLs. Clients never talk to ZooKeeper (`bootstrap.servers` + Admin RPC only), but **every server deployment** still needs a healthy ZK quorum.
   
   This extra stateful system is a major operational burden:
   
   - A production Fluss cluster cannot start without ZooKeeper (quorum, tick time, session timeout, SASL/ACL, backups).
   - ZK failures cascade into coordinator leadership, TabletServer registration, and metadata availability.
   - Container, Kubernetes, edge, and single-node developer setups pay for a second service that exists only for coordination.
   - Extra network hops and a second failure domain sit on the metadata path.
   
   This matches the published architecture direction: ZooKeeper is transitional and should be replaced by **KvStore for metadata** and **Raft for coordination**. Kafka KRaft and similar embedded-journal designs show the same class of change is viable when the consensus group stays small and the data plane stays out of the quorum.
   
   **Goals**
   
   1. Default deploy is Fluss-only: CoordinatorServers form a Raft group; no ZooKeeper process is required.
   2. A supported dual-backend window so existing ZK clusters can keep running and migrate with tools, not a flag day.
   3. No change to client APIs, log/KV format, or `bootstrap.servers`.
   4. A later major (Phase 6) removes ZK mode after soak. That deletion is **not** part of the first release of this work.
   
   Tracking FIP: **FIP-52**.
   
   ### Solution
   
   Replace ZooKeeper with an **embedded Apache Ratis** Raft quorum on CoordinatorServers, plus a RocksDB-backed metadata KvStore.
   
   - Only coordinators are Raft voters. TabletServers are observers: they discover a coordinator from `coordinator.seed.nodes`, register/heartbeat over RPC, and consume a pull-based `FetchMetadata` stream.
   - Consensus is KRaft-style commit-apply: the store is mutated only after a log entry is committed. The Raft term is the coordinator epoch.
   - Backend selection is exclusive:
     - `zookeeper.address` only → ZooKeeper (deprecated)
     - `coordinator.seed.nodes` only → Raft
     - both keys → fail fast
     - neither key → **Raft** (default)
   - Public client APIs, binary log/KV format, and `bootstrap.servers` stay unchanged. New operator CLI: `fluss raft *` and `fluss zk-to-raft-*`.
   - This issue / FIP covers Phases 1–5 (dual-support, testing, migration tools, docs, Raft as default). **Phase 6 (delete ZooKeeper mode and the shaded ZK dependency) is out of the first merge.**
   
   **Related materials**
   
   - FIP-52: `docs/FIP-52-zookeeper-removal.md`
   - Implementation branch: `feature/zookeeper-removal-raft`
   - User docs: `website/docs/install-deploy/deploying-with-raft.md`, `website/docs/install-deploy/migrate-zk-to-raft.md`
   
   ### Anything else?
   
   _No response_
   
   ### Willingness to contribute
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org