You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/07/12 16:32:47 UTC
[GitHub] [pulsar] rdhabalia opened a new issue, #16551: PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar
rdhabalia opened a new issue, #16551:
URL: https://github.com/apache/pulsar/issues/16551
# Motivation
Cluster migration or Blue-Green cluster deployment is one of the proven solutions to migrate live traffic from one cluster to another. One of the examples is applications running on Kubernetes sometimes require a Kubernetes cluster upgrade which can cause downtime for the entire application during a Kubernetes cluster upgrade. Blue-green deployment is an application release model that gradually transfers user traffic from a previous version of an app or microservice to a nearly identical new release—both of which are running in production.
The old version can be called the blue environment while the new version can be known as the green environment. Once production traffic is fully transferred from blue to green, blue can standby in case of rollback or be pulled from production and updated to become the template upon which the next update is made.
We need such capability in Apache pulsar to migrate live traffic from the blue cluster to the green cluster so, eventually, the entire traffic moves from the blue cluster to the green cluster without causing downtime for the topics.
# Goal
This PIP adds support to migrate and redirect the blue cluster’s traffic to the green cluster. Therefore, the Broker will support admin-API using which admin-user can mark migrate cluster along with redirection URLs where traffic should be redirected. Broker persists migration state and new redirected cluster’s URL as part of cluster metadata.
Once the cluster is marked as migrating, the broker asynchronously marks each topic owned by that broker as migrated by calling the new managed-ledger API `asyncMigrate()`. Once, the topic is marked as migrated, broker notifies all the producers and consumers (which have drained the backlog for their subscriptions) with a new client-protocol command called “Migrated-Topic” which has redirection URLs to the green cluster. Producers and consumers for those topics cache the redirection URLs and retry to connect to the broker with that URL which redirects them to the green cluster.
Broker can redirect only those consumers which have reached to end of a terminated topic or create a new subscription in the blue cluster. Therefore, the broker can determine the redirection of consumers in the consumer-creation phase and the pulsar client has to handle redirection after sending producer/consumer creation requests.
Broker will unsubscribe the subscription once that subscription reaches end of topic and broker will also not allow creation of any new producer or subscription for the topics. Therefore, eventually, all the topics in the blue cluster will not have a subscription or producer attached and eventually, those topics will be deleted by the garbage collector.
Broker marks the cluster state as migration-completed once all the topics are deleted and that cluster will not allow any new topic creation.
# Broker/Client changes
## Managed-Ledger Changes
This PIP will add API to managed-ledger to change ManagedLedger state as migrated. This API will terminate the topic and persist the status of managed-ledger as migrated. Broker triggers migrate API of managed-ledger once cluster becomes the blue cluster and traffic should redirect to the green cluster.
```
1. ManagedLedger.java
CompletableFuture<Position> asyncMigrate();
2. MLDataFormats.proto
message ManagedLedgerInfo {
// Flag to check if topic is terminated and migrated to different cluster
optional bool migrated = 4;
}
```
## Broker changes
1. Broker provides pulsar-admin API to mark the cluster in migration state.
**Pulsar Admin Change**
```
pulsar-admin clusters set-cluster-migrated \
–brokerServiceUrl <> \
–brokerServiceUrlTls <>
```
2. Broker stores cluster migration state and redirection urls in cluster-metadata.
**Cluster-metadata change**
```
ClusterData.java
boolean migrated;
ClusterUrl migratedClusterUrl;
class ClusterUrl {
String serviceUrl;
String serviceUrlTls;
String brokerServiceUrl;
String brokerServiceUrlTls;
}
```
3. Broker runs periodic task to check cluster migration state and moves topic into migration state by calling managed-ledger’s migration-api.
```
ServiceConfiguration.java
private long clusterMigrationCheckDurationSeconds = 0; //disable task with default value=0
```
4. Broker sends topic migration message to client so, producer/consumer at client side can handle redirection accordingly.
```
PulsarApi.proto
message CommandMigratedTopic {
required uint64 consumer_id = 1;
optional string brokerServiceUrl = 2;
optional string brokerServiceUrlTls = 3;
}
```
## Client Changes
Once producers and consumers receive the Migrated-Topic command with a list of redirect URLs, they will cache those URLs and try to reconnect with a broker by using those URLs. The client will add handling of the `CommandMigratedTopic` protocol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PIP-188: Cluster migration or Blue-Green cluster deployment support in Pulsar [pulsar]
Posted by "asafm (via GitHub)" <gi...@apache.org>.
asafm commented on issue #16551:
URL: https://github.com/apache/pulsar/issues/16551#issuecomment-2020222613
@rdhabalia Did you guys documented this feature in the docs section?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #16551: PIP-188: Cluster migration or Blue-Green cluster deployment support in Pulsar
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #16551:
URL: https://github.com/apache/pulsar/issues/16551#issuecomment-1242850099
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] horizonzy commented on issue #16551: PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar
Posted by GitBox <gi...@apache.org>.
horizonzy commented on issue #16551:
URL: https://github.com/apache/pulsar/issues/16551#issuecomment-1182880031
Hi, PIP-184 already exist. https://github.com/apache/pulsar/issues/16481, should increate the number to 185
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] rdhabalia closed issue #16551: PIP-188: Cluster migration or Blue-Green cluster deployment support in Pulsar
Posted by GitBox <gi...@apache.org>.
rdhabalia closed issue #16551: PIP-188: Cluster migration or Blue-Green cluster deployment support in Pulsar
URL: https://github.com/apache/pulsar/issues/16551
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PIP-188: Cluster migration or Blue-Green cluster deployment support in Pulsar [pulsar]
Posted by "asafm (via GitHub)" <gi...@apache.org>.
asafm commented on issue #16551:
URL: https://github.com/apache/pulsar/issues/16551#issuecomment-2020224064
@rdhabalia I was thinking of adding this features to the new Features page (https://pulsar.apache.org/features/) but a prerequisite is documentation :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org