You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Zixuan Liu <no...@gmail.com> on 2024/03/22 08:44:28 UTC
[DISCUSS] GEO-replication issues on topic level
Hi all,
The GEO-replication can be enabled on the Namespace and topic levels. When
GEO-replication is enabled on the namespace level, it automatically creates
the topic for the remote cluster, but the topic level misses this feature,
which can cause unexpected problems.
When two clusters use different global configuration metadatastore, the
local cluster has a partitioned topic, and then we enable the
GEO-replication on the topic level, I expect a partitioned topic will be
created on the remote cluster, not a non-partitioned topic. BTW,
`allowAutoTopicCreation` was enabled.
There are two options:
1. When the GEO replication is enabled on the topic level , we can create
the topic for the remote cluster(usually, we have superuser permissions):
https://github.com/apache/pulsar/pull/22203
2. When the remote cluster has no topic, stop GEO-replication and throw an
error.
We also need to consider resource/permission issues, so like the remote
clusters disables topic creation or exceed the maximum number of topics.
Please let me know your thoughts.
More contexts: https://github.com/apache/pulsar/pull/21679 (This is an
incorrect implement, but there is more context here.)
Thanks,
Zixuan
Re: [DISCUSS] GEO-replication issues on topic level
Posted by Zixuan Liu <no...@gmail.com>.
> Therefore, creating topics at the remote clusters when the admin
uploads the replication policies is an acceptable solution. That is
following the current behavior of the topic creating for the
geo-replication enabling in namespace level and no-partition topic at
the topic level. That is not a break-changing process that does not
require a proposal or a simple proposal to record this improvement.
+1, sounds good
Xiangying Meng <xi...@apache.org> 于2024年3月29日周五 10:32写道：
> Hi zixuan,
>
> Thanks for your work in improving this geo-replication issue.
> In my opinion, this should be a mistake when implementing pulsar
> geo-replication at the topic level.
> As we know, after a user configures replication policies at the
> namespace level, the topics under the namespace will be created at the
> remote clusters when the topic is created in the local cluster. If the
> geo-replication policies are enabled at the topic level for a
> non-partition topic, the topic will be created automatically in the
> remote cluster when building the replicator producer.
> However, for a partitioned topic, the topics created automatically are
> no-partition topics in the remote cluster.
>
> Therefore, creating topics at the remote clusters when the admin
> uploads the replication policies is an acceptable solution. That is
> following the current behavior of the topic creating for the
> geo-replication enabling in namespace level and no-partition topic at
> the topic level. That is not a break-changing process that does not
> require a proposal or a simple proposal to record this improvement.
>
> Thanks,
> Xiangying
>
> On Fri, Mar 22, 2024 at 4:44 PM Zixuan Liu <no...@gmail.com> wrote:
> >
> > Hi all,
> >
> > The GEO-replication can be enabled on the Namespace and topic levels.
> When
> > GEO-replication is enabled on the namespace level, it automatically
> creates
> > the topic for the remote cluster, but the topic level misses this
> feature,
> > which can cause unexpected problems.
> >
> > When two clusters use different global configuration metadatastore, the
> > local cluster has a partitioned topic, and then we enable the
> > GEO-replication on the topic level, I expect a partitioned topic will be
> > created on the remote cluster, not a non-partitioned topic. BTW,
> > `allowAutoTopicCreation` was enabled.
> >
> > There are two options:
> >
> > 1. When the GEO replication is enabled on the topic level , we can create
> > the topic for the remote cluster(usually, we have superuser permissions):
> > https://github.com/apache/pulsar/pull/22203
> > 2. When the remote cluster has no topic, stop GEO-replication and throw
> an
> > error.
> >
> > We also need to consider resource/permission issues, so like the remote
> > clusters disables topic creation or exceed the maximum number of topics.
> >
> > Please let me know your thoughts.
> >
> > More contexts: https://github.com/apache/pulsar/pull/21679 (This is an
> > incorrect implement, but there is more context here.)
> >
> > Thanks,
> > Zixuan
>
Re: [DISCUSS] GEO-replication issues on topic level
Posted by Xiangying Meng <xi...@apache.org>.
Hi zixuan,
Thanks for your work in improving this geo-replication issue.
In my opinion, this should be a mistake when implementing pulsar
geo-replication at the topic level.
As we know, after a user configures replication policies at the
namespace level, the topics under the namespace will be created at the
remote clusters when the topic is created in the local cluster. If the
geo-replication policies are enabled at the topic level for a
non-partition topic, the topic will be created automatically in the
remote cluster when building the replicator producer.
However, for a partitioned topic, the topics created automatically are
no-partition topics in the remote cluster.
Therefore, creating topics at the remote clusters when the admin
uploads the replication policies is an acceptable solution. That is
following the current behavior of the topic creating for the
geo-replication enabling in namespace level and no-partition topic at
the topic level. That is not a break-changing process that does not
require a proposal or a simple proposal to record this improvement.
Thanks,
Xiangying
On Fri, Mar 22, 2024 at 4:44 PM Zixuan Liu <no...@gmail.com> wrote:
>
> Hi all,
>
> The GEO-replication can be enabled on the Namespace and topic levels. When
> GEO-replication is enabled on the namespace level, it automatically creates
> the topic for the remote cluster, but the topic level misses this feature,
> which can cause unexpected problems.
>
> When two clusters use different global configuration metadatastore, the
> local cluster has a partitioned topic, and then we enable the
> GEO-replication on the topic level, I expect a partitioned topic will be
> created on the remote cluster, not a non-partitioned topic. BTW,
> `allowAutoTopicCreation` was enabled.
>
> There are two options:
>
> 1. When the GEO replication is enabled on the topic level , we can create
> the topic for the remote cluster(usually, we have superuser permissions):
> https://github.com/apache/pulsar/pull/22203
> 2. When the remote cluster has no topic, stop GEO-replication and throw an
> error.
>
> We also need to consider resource/permission issues, so like the remote
> clusters disables topic creation or exceed the maximum number of topics.
>
> Please let me know your thoughts.
>
> More contexts: https://github.com/apache/pulsar/pull/21679 (This is an
> incorrect implement, but there is more context here.)
>
> Thanks,
> Zixuan