You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "nodece (via GitHub)" <gi...@apache.org> on 2025/04/25 10:16:36 UTC
[PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
nodece opened a new pull request, #24213:
URL: https://github.com/apache/pulsar/pull/24213
### Motivation
The changes from #24118 have been merged and should be documented in a Pulsar Improvement Proposal (PIP) to record these important updates.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#issuecomment-2830214416
Another approach is to create the missed partitioned metadata automatically. For example, with this PIP implemented, when the broker detects `my-topic-partition-0` is accessed by a client but there is no partition metadata for `my-topic`. The broker should try creating the partition metadata to unblock the client. The number of partitions are determined by the largest partition index `N` in `my-topic-partition-N`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "crossoverJie (via GitHub)" <gi...@apache.org>.
crossoverJie commented on PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#issuecomment-2833899278
> Another approach is to create the missed partitioned metadata automatically. For example, with this PIP implemented, when the broker detects `my-topic-partition-0` is accessed by a client but there is no partition metadata for `my-topic`. The broker should try creating the partition metadata to unblock the client. The number of partitions are determined by the largest partition index `N` in `my-topic-partition-N`.
I agree with this plan. Our cluster has been running for many years, so there are likely quite a few topics with metadata errors.
Perhaps we should provide an additional admin interface to query topics with metadata errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on code in PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#discussion_r2151204060
##########
pip/pip-414.md:
##########
@@ -0,0 +1,101 @@
+# PIP-414: Enforce topic consistency check
+
+## Background Knowledge
+
+In Apache Pulsar, topics can be either non-partitioned or partitioned. A non-partitioned topic handles all messages on a
+single topic instance, while a partitioned topic is logically split across multiple partitions for parallelism and
+scalability. Partitioned topic is backed by multiple internal non-partitioned topics, one per partition (e.g.,
+`topic-partition-0`, `topic-partition-1`, etc.).
+
+## Motivation
+
+Apache Pulsar allows clients to produce to or consume from a fully qualified partitioned topic name (e.g.,
+`public/default/test-partitioned-topic-partition-N`), even if the topic metadata does not yet exist. If topic
+auto-creation is enabled and `allowAutoTopicCreationType=non-partitioned` is set, the broker will automatically create a
+non-partitioned topic for that partition name.
+
+This behavior can lead to inconsistencies: users expecting to interact with a partitioned topic may unknowingly create
+non-partitioned topics for individual partitions. As a result, listing topics may yield unexpected structures, with
+partitioned and non-partitioned topics mixed under similar names.
+
+While this issue can also occur in GEO-replicated environments due to asynchronous metadata propagation, the core
+problem is the lack of topic validation. Disabling this behavior and enforcing
+strict topic-type consistency is essential to ensure clarity, prevent accidental misconfigurations, and maintain the
+integrity of topic structures in Pulsar.
+
+## Goals
+
+### In Scope
+
+- Introduce validation to enforce consistency between topic metadata and topic type.
+- Prevent producing or consuming from a topic when metadata indicates a mismatch.
+
+## High Level Design
+
+The proposed solution adds server-side validation to ensure that the topic type matches the metadata.
+
+- When accessing a non-partitioned topic, the broker checks that no partition metadata exists(which shouldn't be there).
+- When accessing a partitioned topic (via a `-partition-N` name), the broker ensures that valid partition metadata
+ exists and that the topic is indeed part of a partitioned group.
+
+- If these checks fail, the broker rejects the client request with an appropriate error.
+
+## Detailed Design
+
+### Design & Implementation Details
+
+- Validation is added when loading the topic.
+- If a mismatch is detected:
+ - Non-partitioned topic with partition metadata → reject request.
Review Comment:
> From the broker’s perspective, it always fetches the partitioned topic metadata first. So it’s unclear what “non-partitioned topic with partition metadata” actually refers to here. Could you clarify this?
When the client provides a topic name that does not include the `-partition-N` suffix, but the topic does have partitioned metadata, it means the client is referring to the base topic of a partitioned topic. While this scenario is uncommon in normal operation, it's still important to handle it properly by checking for the presence of partitioned metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on code in PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#discussion_r2150665439
##########
pip/pip-414.md:
##########
@@ -0,0 +1,101 @@
+# PIP-414: Enforce topic consistency check
+
+## Background Knowledge
+
+In Apache Pulsar, topics can be either non-partitioned or partitioned. A non-partitioned topic handles all messages on a
+single topic instance, while a partitioned topic is logically split across multiple partitions for parallelism and
+scalability. Partitioned topic is backed by multiple internal non-partitioned topics, one per partition (e.g.,
+`topic-partition-0`, `topic-partition-1`, etc.).
+
+## Motivation
+
+Apache Pulsar allows clients to produce to or consume from a fully qualified partitioned topic name (e.g.,
+`public/default/test-partitioned-topic-partition-N`), even if the topic metadata does not yet exist. If topic
Review Comment:
Is it correct?
##########
pip/pip-414.md:
##########
@@ -0,0 +1,101 @@
+# PIP-414: Enforce topic consistency check
+
+## Background Knowledge
+
+In Apache Pulsar, topics can be either non-partitioned or partitioned. A non-partitioned topic handles all messages on a
+single topic instance, while a partitioned topic is logically split across multiple partitions for parallelism and
+scalability. Partitioned topic is backed by multiple internal non-partitioned topics, one per partition (e.g.,
+`topic-partition-0`, `topic-partition-1`, etc.).
+
+## Motivation
+
+Apache Pulsar allows clients to produce to or consume from a fully qualified partitioned topic name (e.g.,
+`public/default/test-partitioned-topic-partition-N`), even if the topic metadata does not yet exist. If topic
Review Comment:
```suggestion
`public/default/test-partitioned-topic-partition-N`), even if the partitioned topic metadata does not yet exist. If topic
```
##########
pip/pip-414.md:
##########
@@ -0,0 +1,101 @@
+# PIP-414: Enforce topic consistency check
+
+## Background Knowledge
+
+In Apache Pulsar, topics can be either non-partitioned or partitioned. A non-partitioned topic handles all messages on a
+single topic instance, while a partitioned topic is logically split across multiple partitions for parallelism and
+scalability. Partitioned topic is backed by multiple internal non-partitioned topics, one per partition (e.g.,
+`topic-partition-0`, `topic-partition-1`, etc.).
+
+## Motivation
+
+Apache Pulsar allows clients to produce to or consume from a fully qualified partitioned topic name (e.g.,
+`public/default/test-partitioned-topic-partition-N`), even if the topic metadata does not yet exist. If topic
+auto-creation is enabled and `allowAutoTopicCreationType=non-partitioned` is set, the broker will automatically create a
+non-partitioned topic for that partition name.
+
+This behavior can lead to inconsistencies: users expecting to interact with a partitioned topic may unknowingly create
+non-partitioned topics for individual partitions. As a result, listing topics may yield unexpected structures, with
+partitioned and non-partitioned topics mixed under similar names.
+
+While this issue can also occur in GEO-replicated environments due to asynchronous metadata propagation, the core
+problem is the lack of topic validation. Disabling this behavior and enforcing
+strict topic-type consistency is essential to ensure clarity, prevent accidental misconfigurations, and maintain the
+integrity of topic structures in Pulsar.
+
+## Goals
+
+### In Scope
+
+- Introduce validation to enforce consistency between topic metadata and topic type.
+- Prevent producing or consuming from a topic when metadata indicates a mismatch.
+
+## High Level Design
+
+The proposed solution adds server-side validation to ensure that the topic type matches the metadata.
+
+- When accessing a non-partitioned topic, the broker checks that no partition metadata exists(which shouldn't be there).
+- When accessing a partitioned topic (via a `-partition-N` name), the broker ensures that valid partition metadata
+ exists and that the topic is indeed part of a partitioned group.
+
+- If these checks fail, the broker rejects the client request with an appropriate error.
+
+## Detailed Design
+
+### Design & Implementation Details
+
+- Validation is added when loading the topic.
+- If a mismatch is detected:
+ - Non-partitioned topic with partition metadata → reject request.
Review Comment:
How do we know the requested topic is non-partitioned? Is there a concrete example demonstrating this case?
From the broker’s perspective, it always fetches the partitioned topic metadata first. So it’s unclear what “non-partitioned topic with partition metadata” actually refers to here. Could you clarify this?
##########
pip/pip-414.md:
##########
@@ -0,0 +1,101 @@
+# PIP-414: Enforce topic consistency check
+
+## Background Knowledge
+
+In Apache Pulsar, topics can be either non-partitioned or partitioned. A non-partitioned topic handles all messages on a
+single topic instance, while a partitioned topic is logically split across multiple partitions for parallelism and
+scalability. Partitioned topic is backed by multiple internal non-partitioned topics, one per partition (e.g.,
+`topic-partition-0`, `topic-partition-1`, etc.).
+
+## Motivation
+
+Apache Pulsar allows clients to produce to or consume from a fully qualified partitioned topic name (e.g.,
+`public/default/test-partitioned-topic-partition-N`), even if the topic metadata does not yet exist. If topic
+auto-creation is enabled and `allowAutoTopicCreationType=non-partitioned` is set, the broker will automatically create a
+non-partitioned topic for that partition name.
+
+This behavior can lead to inconsistencies: users expecting to interact with a partitioned topic may unknowingly create
+non-partitioned topics for individual partitions. As a result, listing topics may yield unexpected structures, with
+partitioned and non-partitioned topics mixed under similar names.
+
+While this issue can also occur in GEO-replicated environments due to asynchronous metadata propagation, the core
+problem is the lack of topic validation. Disabling this behavior and enforcing
+strict topic-type consistency is essential to ensure clarity, prevent accidental misconfigurations, and maintain the
+integrity of topic structures in Pulsar.
+
+## Goals
+
+### In Scope
+
+- Introduce validation to enforce consistency between topic metadata and topic type.
+- Prevent producing or consuming from a topic when metadata indicates a mismatch.
+
+## High Level Design
+
+The proposed solution adds server-side validation to ensure that the topic type matches the metadata.
+
+- When accessing a non-partitioned topic, the broker checks that no partition metadata exists(which shouldn't be there).
+- When accessing a partitioned topic (via a `-partition-N` name), the broker ensures that valid partition metadata
+ exists and that the topic is indeed part of a partitioned group.
+
+- If these checks fail, the broker rejects the client request with an appropriate error.
+
+## Detailed Design
+
+### Design & Implementation Details
+
+- Validation is added when loading the topic.
+- If a mismatch is detected:
+ - Non-partitioned topic with partition metadata → reject request.
+ - Partitioned topic (e.g., `-partition-0`) with missing metadata → reject request.
Review Comment:
I believe this validation is already added.
I tried to build from the master branch and start a standalone.
Then, using the CLI try to publish messages
```
bin/pulsar-client produce -m "{}" test-partition-0
```
The error messages generated:
```
2025-06-16T12:16:25,670-0700 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Error while producing messages
2025-06-16T12:16:25,671-0700 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - {"errorMsg":"Partition metadata not found for the partitioned topic: persistent://public/default/test-partition-0","reqId":1347957860605533513, "remote":"localhost/127.0.0.1:6650", "local":"/127.0.0.1:57439"}
org.apache.pulsar.client.api.PulsarClientException$NotAllowedException: {"errorMsg":"Partition metadata not found for the partitioned topic: persistent://public/default/test-partition-0","reqId":1347957860605533513, "remote":"localhost/127.0.0.1:6650", "local":"/127.0.0.1:57439"}
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1112) ~[org.apache.pulsar-pulsar-client-api-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:89) ~[org.apache.pulsar-pulsar-client-original-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.pulsar.client.cli.CmdProduce.publish(CmdProduce.java:308) [org.apache.pulsar-pulsar-client-tools-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.pulsar.client.cli.CmdProduce.run(CmdProduce.java:287) [org.apache.pulsar-pulsar-client-tools-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.pulsar.client.cli.AbstractCmd.call(AbstractCmd.java:27) [org.apache.pulsar-pulsar-client-tools-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.pulsar.client.cli.AbstractCmd.call(AbstractCmd.java:23) [org.apache.pulsar-pulsar-client-tools-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at picocli.CommandLine.executeUserObject(CommandLine.java:2041) [info.picocli-picocli-4.7.5.jar:4.7.5]
at picocli.CommandLine.access$1500(CommandLine.java:148) [info.picocli-picocli-4.7.5.jar:4.7.5]
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) [info.picocli-picocli-4.7.5.jar:4.7.5]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) [info.picocli-picocli-4.7.5.jar:4.7.5]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) [info.picocli-picocli-4.7.5.jar:4.7.5]
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) [info.picocli-picocli-4.7.5.jar:4.7.5]
at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) [info.picocli-picocli-4.7.5.jar:4.7.5]
at org.apache.pulsar.internal.CommanderFactory$CommandExecutionStrategy.execute(CommanderFactory.java:57) [org.apache.pulsar-pulsar-client-tools-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at picocli.CommandLine.execute(CommandLine.java:2170) [info.picocli-picocli-4.7.5.jar:4.7.5]
at org.apache.pulsar.client.cli.PulsarClientTool.run(PulsarClientTool.java:216) [org.apache.pulsar-pulsar-client-tools-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.pulsar.client.cli.PulsarClientTool.main(PulsarClientTool.java:234) [org.apache.pulsar-pulsar-client-tools-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
2025-06-16T12:16:25,678-0700 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 0 messages successfully produced
```
Could you please share the steps for reproducing this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on code in PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#discussion_r2151289645
##########
pip/pip-414.md:
##########
@@ -0,0 +1,101 @@
+# PIP-414: Enforce topic consistency check
+
+## Background Knowledge
+
+In Apache Pulsar, topics can be either non-partitioned or partitioned. A non-partitioned topic handles all messages on a
+single topic instance, while a partitioned topic is logically split across multiple partitions for parallelism and
+scalability. Partitioned topic is backed by multiple internal non-partitioned topics, one per partition (e.g.,
+`topic-partition-0`, `topic-partition-1`, etc.).
+
+## Motivation
+
+Apache Pulsar allows clients to produce to or consume from a fully qualified partitioned topic name (e.g.,
+`public/default/test-partitioned-topic-partition-N`), even if the topic metadata does not yet exist. If topic
+auto-creation is enabled and `allowAutoTopicCreationType=non-partitioned` is set, the broker will automatically create a
+non-partitioned topic for that partition name.
+
+This behavior can lead to inconsistencies: users expecting to interact with a partitioned topic may unknowingly create
+non-partitioned topics for individual partitions. As a result, listing topics may yield unexpected structures, with
+partitioned and non-partitioned topics mixed under similar names.
+
+While this issue can also occur in GEO-replicated environments due to asynchronous metadata propagation, the core
+problem is the lack of topic validation. Disabling this behavior and enforcing
+strict topic-type consistency is essential to ensure clarity, prevent accidental misconfigurations, and maintain the
+integrity of topic structures in Pulsar.
+
+## Goals
+
+### In Scope
+
+- Introduce validation to enforce consistency between topic metadata and topic type.
+- Prevent producing or consuming from a topic when metadata indicates a mismatch.
+
+## High Level Design
+
+The proposed solution adds server-side validation to ensure that the topic type matches the metadata.
+
+- When accessing a non-partitioned topic, the broker checks that no partition metadata exists(which shouldn't be there).
+- When accessing a partitioned topic (via a `-partition-N` name), the broker ensures that valid partition metadata
+ exists and that the topic is indeed part of a partitioned group.
+
+- If these checks fail, the broker rejects the client request with an appropriate error.
+
+## Detailed Design
+
+### Design & Implementation Details
+
+- Validation is added when loading the topic.
+- If a mismatch is detected:
+ - Non-partitioned topic with partition metadata → reject request.
+ - Partitioned topic (e.g., `-partition-0`) with missing metadata → reject request.
Review Comment:
Oh I see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#issuecomment-2848657154
Then support creating missed partitions via admin API for such cases could be the solution. Currently the create-missed-partitions API does not work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#issuecomment-2834091641
> I agree with this plan. Our cluster has been running for many years, so there are likely quite a few topics with metadata errors.
@crossoverJie If the broker automatically creates the partitioned metadata, this may break the user behavior.
> Perhaps we should provide an additional admin interface to query topics with metadata errors.
If the topic type is incorrect, the user can use the create partitioned topic command to fix the metadata error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#issuecomment-2848558168
I don't think create missing topic partition metadata in runtime is OK, as @nodece says, it would introduce unpredictable consequences. Actually, I think add a new config is the relatively good way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#issuecomment-2830885230
> This is difficult because we can not obtain the maximum partition index.
We can list all non-partitioned topics of the `<topic>-partition-<index>` pattern and find the maximum `index`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui merged PR #24213:
URL: https://github.com/apache/pulsar/pull/24213
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#issuecomment-2833253312
> > This is difficult because we can not obtain the maximum partition index.
>
> We can list all non-partitioned topics of the `<topic>-partition-<index>` pattern and find the maximum `index`.
@BewareMyPower Do this during topic creation (e.g., via admin topics create-partitioned-topic) or at runtime?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#issuecomment-2834452409
> Do this during topic creation (e.g., via admin topics create-partitioned-topic) or at runtime?
Topic creation, or `BrokerService#getTopic` on a non-partitioned topic whose name ends with `-partition-<index>`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#issuecomment-2893727386
> Then support creating missed partitions via admin API for such cases could be the solution. Currently the create-missed-partitions API does not work.
@BewareMyPower `admin topics create-partitioned-topic my-topic -p 1` has been fixed by #24225
Could you review this PIP again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#issuecomment-2830545092
> At least we need to recover the partition metadata for such orphan partitions. Currently it's impossible:
>
> ```shell
> > admin topics create-partitioned-topic my-topic -p 1
> 2025-04-25T19:45:14,373+0800 [AsyncHttpClient-7-2] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://localhost:8080/admin/v2/persistent/public/default/my-topic/partitions?createLocalTopicOnly=false] Failed to perform http put request: javax.ws.rs.ClientErrorException: HTTP 409 {"reason":"This topic already exists"}
> This topic already exists
> ```
Good catch, I can fix this bug. I think this command helps the user to recover the partitioned metadata.
> Another approach is to create the missed partitioned metadata automatically.
This is difficult because we can not obtain the maximum partition index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#issuecomment-2834495620
> > Do this during topic creation (e.g., via admin topics create-partitioned-topic) or at runtime?
>
> Topic creation, or `BrokerService#getTopic` on a non-partitioned topic whose name ends with `-partition-<index>`.
@BewareMyPower Topic creation is the best place to handle this. Doing it at runtime (e.g., in BrokerService#getTopic) would be too late and would introduce black-box behavior that's harder to reason about and debug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on code in PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#discussion_r2151205507
##########
pip/pip-414.md:
##########
@@ -0,0 +1,101 @@
+# PIP-414: Enforce topic consistency check
+
+## Background Knowledge
+
+In Apache Pulsar, topics can be either non-partitioned or partitioned. A non-partitioned topic handles all messages on a
+single topic instance, while a partitioned topic is logically split across multiple partitions for parallelism and
+scalability. Partitioned topic is backed by multiple internal non-partitioned topics, one per partition (e.g.,
+`topic-partition-0`, `topic-partition-1`, etc.).
+
+## Motivation
+
+Apache Pulsar allows clients to produce to or consume from a fully qualified partitioned topic name (e.g.,
+`public/default/test-partitioned-topic-partition-N`), even if the topic metadata does not yet exist. If topic
+auto-creation is enabled and `allowAutoTopicCreationType=non-partitioned` is set, the broker will automatically create a
+non-partitioned topic for that partition name.
+
+This behavior can lead to inconsistencies: users expecting to interact with a partitioned topic may unknowingly create
+non-partitioned topics for individual partitions. As a result, listing topics may yield unexpected structures, with
+partitioned and non-partitioned topics mixed under similar names.
+
+While this issue can also occur in GEO-replicated environments due to asynchronous metadata propagation, the core
+problem is the lack of topic validation. Disabling this behavior and enforcing
+strict topic-type consistency is essential to ensure clarity, prevent accidental misconfigurations, and maintain the
+integrity of topic structures in Pulsar.
+
+## Goals
+
+### In Scope
+
+- Introduce validation to enforce consistency between topic metadata and topic type.
+- Prevent producing or consuming from a topic when metadata indicates a mismatch.
+
+## High Level Design
+
+The proposed solution adds server-side validation to ensure that the topic type matches the metadata.
+
+- When accessing a non-partitioned topic, the broker checks that no partition metadata exists(which shouldn't be there).
+- When accessing a partitioned topic (via a `-partition-N` name), the broker ensures that valid partition metadata
+ exists and that the topic is indeed part of a partitioned group.
+
+- If these checks fail, the broker rejects the client request with an appropriate error.
+
+## Detailed Design
+
+### Design & Implementation Details
+
+- Validation is added when loading the topic.
+- If a mismatch is detected:
+ - Non-partitioned topic with partition metadata → reject request.
+ - Partitioned topic (e.g., `-partition-0`) with missing metadata → reject request.
Review Comment:
This PIP has already been implemented in Pulsar, but it's important to formally document these changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-414: Enforce topic consistency check [pulsar]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on PR #24213:
URL: https://github.com/apache/pulsar/pull/24213#issuecomment-2893724750
> I don't think create missing topic partition metadata in runtime is OK, as @nodece says, it would introduce unpredictable consequences. Actually, I think add a new config is the relatively good way.
@dao-jun Instead of introducing a flag to enable or disable the creation of missing topic partition metadata at runtime, this feature will always be enabled. Since disabling this feature can lead to unexpected behavior and complexity, enabling it unconditionally simplifies the design and avoids confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org