[D] What is the current status and roadmap for Gravitino Server High Availability (HA)? [gravitino]

["zhengsg (via GitHub)" <gi...@apache.org>]

GitHub user zhengsg created a discussion: What is the current status and roadmap for Gravitino Server High Availability (HA)?

I am evaluating Apache Gravitino for our enterprise production data platform. As we plan to manage critical metadata and credentials at scale using Gravitino, High Availability (HA) and disaster recovery have become our top priorities.

I would like to consult the community on the following points:

1. **Current HA Capabilities:** Does the current Gravitino Server support a multi-node active-passive or active-active deployment? If we deploy multiple Gravitino server instances backed by a shared relational database (e.g., MySQL or PostgreSQL), are there any known limitations regarding distributed locks, event synchronization, or internal caches (like the cache consistency improvements mentioned in version 1.3.0)?
2. **Client-Side Failover:** Are there built-in mechanisms in the Gravitino Java/Python clients or the gcli tool to accept multiple server URIs and perform automatic polling/failover?
3. **Roadmap & Plans:** What is the community's official roadmap or ongoing plan for formal HA architecture in 2026?

GitHub link: https://github.com/apache/gravitino/discussions/13144

----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org

Re: [D] What is the current status and roadmap for Gravitino Server High Availability (HA)? [gravitino]

["zhengsg (via GitHub)" <gi...@apache.org>]

GitHub user zhengsg added a comment to the discussion: What is the current status and roadmap for Gravitino Server High Availability (HA)?

Thank you so much for the detailed explanation and the clear roadmap! It is exciting to hear that the multi-node concurrency and cache consistency issues have been addressed in the main branch and will be officially released in 2.0.0, especially with the upcoming Redis shared entity cache (#13106). This provides a very solid foundation for the server-side HA.

Here is more context on our requirements and what we hope to achieve:

### 1. Server-Side HA & Deployment Modes
While adding a Load Balancer (LB) in front of multiple Gravitino instances works perfectly for stateless web services, we want to clarify how Gravitino handles internal state and background tasks in a distributed environment:
* **Active-Active vs. Active-Passive:** With the 2.0.0 cache consistency fixes, can we safely run Gravitino in a true **Active-Active** mode where any instance can handle read/write requests simultaneously behind an LB?
* **Background/Asynchronous Tasks:** Does Gravitino Server run any background cron jobs, event listeners, or asynchronous tasks (e.g., catalog syncing, credential refreshing, or metrics collection)? If so, how does the community prevent duplicate execution across multiple nodes? Is there a leader election mechanism (like via ZooKeeper/etcd or DB locks) planned for Gravitino servers?

### 2. Client-Side Automatic Failover & Routing
While a centralized Load Balancer (like Nginx, AWS ALB, or Kubernetes Ingress) is a standard solution, enterprise platforms often prefer client-side failover for critical data infrastructure (similar to HDFS Nameservice or Hive Metastore URI lists) due to the following reasons:
* **Avoiding Single Point of Failure (SPOF):** An LB introduces another component to maintain. If the LB goes down, the entire data platform is blocked.
* **Latency & Performance:** Direct client-to-server routing can reduce network hops for high-throughput metadata operations.
* **Proposal:** We would love to know if the community is open to accepting a feature where Gravitino clients (Java/Python) can accept a comma-separated list of server URIs (e.g., `http://host1:8090,http://host2:8090`) and automatically handle connection polling and failover when one node becomes unresponsive.

Best regards

GitHub link: https://github.com/apache/gravitino/discussions/13144#discussioncomment-18442971

----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org

Re: [D] What is the current status and roadmap for Gravitino Server High Availability (HA)? [gravitino]

["yuqi1129 (via GitHub)" <gi...@apache.org>]

GitHub user yuqi1129 added a comment to the discussion: What is the current status and roadmap for Gravitino Server High Availability (HA)?

Thanks for providing the additional context.

1. All Gravitino Server instances in a multi-node deployment run in active mode. When they are placed behind a load balancer, any instance can handle read and write requests. For Gravitino-managed metadata, we generally rely on the shared relational database, including its transactions and concurrency-control mechanisms, to coordinate writes across instances. For external catalogs, such as Hive, Iceberg, or JDBC catalogs, concurrency and consistency are the responsibility of the external catalog or metastore.

2. Gravitino Server does run background jobs, event listeners, and asynchronous tasks. So far, we have not found duplicate execution across multiple nodes to cause harmful side effects. If you have encountered a specific problem, please share more details about the task, deployment topology, and observed behavior so that we can investigate it. We currently do not plan to introduce a leader-election mechanism, because doing so would add substantial architectural and operational complexity. We would prefer to evaluate such a mechanism based on concrete use cases and issues.

3. We would welcome support for configuring multiple server URIs in the Java and Python clients and performing automatic failover when an endpoint becomes unavailable. This would improve client-side reliability. The retry semantics, especially for writes whose outcome may be unknown after a connection failure, would need to be designed carefully. Would you be interested in contributing this feature? We would be happy to discuss the design with you.

GitHub link: https://github.com/apache/gravitino/discussions/13144#discussioncomment-18448615

----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org

Re: [D] What is the current status and roadmap for Gravitino Server High Availability (HA)? [gravitino]

["yuqi1129 (via GitHub)" <gi...@apache.org>]

GitHub user yuqi1129 added a comment to the discussion: What is the current status and roadmap for Gravitino Server High Availability (HA)?

@zhengsg 
Thanks for your attention.

1. For the first question, I would like to say that in branch-1.3,  for a multiple-node deployment environment, there still exists the following challenge.
- Cache consistency. As Gravitino uses Caffeine cache as the default implementation, there would be a consistency problem in a multiple-node deployment; this problem has been fixed in the main branch, but it's eventual consistency.  For strong consistency, there exists a Redis implementation on the way, and you can see: https://github.com/apache/gravitino/pull/13106
- About the lock, we haven't seen and met concurrent problem for Gravitino-managed catalogs, but for external catalogs like, Hive, Iceberg, and JDBC, It's out of Gravitino's scope of responsibility to manage it. 

In all, you can try the main branch, but for 1.3, you need to disable the entity cache to make it work well in a distributed environment. 

2. There is no special configuration for the client side when deploying a distributed environment; you may need to add a balancer that's backed by multiple Gravitno instances, which is a common behaviour for any web service. 

3. The multiple-node concurrent and cache consistency problem has been resolved in the main branch and will be included in the next release (2.0.0), so what do you mean by `HA architecture` here? Could you provide more informations?

GitHub link: https://github.com/apache/gravitino/discussions/13144#discussioncomment-18431809

----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org