You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "mattisonchao (via GitHub)" <gi...@apache.org> on 2023/08/20 02:09:00 UTC
[GitHub] [pulsar] mattisonchao opened a new pull request, #21033: [improve][pip] Replace reader with table view in the topic policy service
mattisonchao opened a new pull request, #21033:
URL: https://github.com/apache/pulsar/pull/21033
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #21033: [improve][pip] Replace reader with table view in the topic policy service
Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on code in PR #21033:
URL: https://github.com/apache/pulsar/pull/21033#discussion_r1314259139
##########
pip/pip-284.md:
##########
@@ -0,0 +1,114 @@
+# Background knowledge
+
+Apache Pulsar introduced the topic-level policies by [PIP-39](https://github.com/apache/pulsar/wiki/PIP-39%3A-Namespace-Change-Events).
+It uses the reader API to read messages and relies on compaction to implement table-like logic in the memory.
+
+[PIP-104](https://github.com/apache/pulsar/issues/12356) Introduced new consumer type `TableView` to support the same logic by
+Public Stable API.
+
+# Motivation
+
+Due to a lot of problems caused by the complex topic policy logic is as follows. we can use a new stable `TableView` API instead of
+previous one.
+
+- https://github.com/apache/pulsar/pull/20763
+- https://github.com/apache/pulsar/pull/20613
+- https://github.com/apache/pulsar/pull/19746
+- etc...
+
+# Goals
+
+## In Scope
+
+- Reduce complex logic.
+- Reuse the public stable API to improve the project quality.
+
+## Out of Scope
+
+none.
+
+# High Level Design
+
+```mermaid
+sequenceDiagram
+ participant client
+ participant topic_policy_service
+ participant system_event_topic
+ client ->> topic_policy_service: Hi, what's the topic policy of `persistent://public/default/test`
+ loop policy update event fetching
+ topic_policy_service ->> system_event_topic: Do you have new events?;
+ alt new policy event
+ system_event_topic ->> topic_policy_service: Yes, Sure. here you are.
+ else no policy event
+ system_event_topic ->> topic_policy_service: No, Please wait for a moment.
+ end;
+ end
+ Note right of topic_policy_service: Update the local cache table by keeping the latest one;
+ topic_policy_service -->> client: Here you are.
+```
+
+# Detailed Design
+
+## Design & Implementation Details
+
+The new implementation will continue reuse the previous system topic `namespace/__change_event` to store the topic policy data.
+and it will change the reading logic from the raw reader to TableView to enhance the robustness.
+
+## Public-facing Changes
+
+none.
+
+### Public API
+
+none.
+
+### Binary protocol
+
+none.
+
+### Configuration
+
+none.
+
+### CLI
+
+none.
+
+### Metrics
+
+none.
+
+# Monitoring
+
+none.
+
+# Security Considerations
+
+none.
+
+# Backward & Forward Compatibility
+
+Without backward compatibility since we will reuse the same system topic `namesapce/__change_event`.
+
+## Revert
+
+none.
+
+## Upgrade
+
+none.
+
+# Alternatives
+
+none.
+
+# General Notes
+
+none.
+
+# Links
+
+none.
+
+* Mailing List discussion thread: https://lists.apache.org/thread/9v00sfpfxjpm775vgltkjoxwnllsgskg
+* Mailing List voting thread:
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on a diff in pull request #21033: [improve][pip] Replace reader with table view in the topic policy service
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on code in PR #21033:
URL: https://github.com/apache/pulsar/pull/21033#discussion_r1314252440
##########
pip/pip-284.md:
##########
@@ -0,0 +1,114 @@
+# Background knowledge
+
+Apache Pulsar introduced the topic-level policies by [PIP-39](https://github.com/apache/pulsar/wiki/PIP-39%3A-Namespace-Change-Events).
+It uses the reader API to read messages and relies on compaction to implement table-like logic in the memory.
+
+[PIP-104](https://github.com/apache/pulsar/issues/12356) Introduced new consumer type `TableView` to support the same logic by
+Public Stable API.
+
+# Motivation
+
+Due to a lot of problems caused by the complex topic policy logic is as follows. we can use a new stable `TableView` API instead of
+previous one.
+
+- https://github.com/apache/pulsar/pull/20763
+- https://github.com/apache/pulsar/pull/20613
+- https://github.com/apache/pulsar/pull/19746
+- etc...
+
+# Goals
+
+## In Scope
+
+- Reduce complex logic.
+- Reuse the public stable API to improve the project quality.
+
+## Out of Scope
+
+none.
+
+# High Level Design
+
+```mermaid
+sequenceDiagram
+ participant client
+ participant topic_policy_service
+ participant system_event_topic
+ client ->> topic_policy_service: Hi, what's the topic policy of `persistent://public/default/test`
+ loop policy update event fetching
+ topic_policy_service ->> system_event_topic: Do you have new events?;
+ alt new policy event
+ system_event_topic ->> topic_policy_service: Yes, Sure. here you are.
+ else no policy event
+ system_event_topic ->> topic_policy_service: No, Please wait for a moment.
+ end;
+ end
+ Note right of topic_policy_service: Update the local cache table by keeping the latest one;
+ topic_policy_service -->> client: Here you are.
+```
+
+# Detailed Design
+
+## Design & Implementation Details
+
+The new implementation will continue reuse the previous system topic `namespace/__change_event` to store the topic policy data.
+and it will change the reading logic from the raw reader to TableView to enhance the robustness.
+
+## Public-facing Changes
+
+none.
+
+### Public API
+
+none.
+
+### Binary protocol
+
+none.
+
+### Configuration
+
+none.
+
+### CLI
+
+none.
+
+### Metrics
+
+none.
+
+# Monitoring
+
+none.
+
+# Security Considerations
+
+none.
+
+# Backward & Forward Compatibility
+
+Without backward compatibility since we will reuse the same system topic `namesapce/__change_event`.
+
+## Revert
+
+none.
+
+## Upgrade
+
+none.
+
+# Alternatives
+
+none.
+
+# General Notes
+
+none.
+
+# Links
+
+none.
+
+* Mailing List discussion thread: https://lists.apache.org/thread/9v00sfpfxjpm775vgltkjoxwnllsgskg
+* Mailing List voting thread:
Review Comment:
Need to update vote thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao merged pull request #21033: [improve][pip] Replace reader with table view in the topic policy service
Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao merged PR #21033:
URL: https://github.com/apache/pulsar/pull/21033
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org