You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "namest504 (via GitHub)" <gi...@apache.org> on 2025/10/14 07:47:27 UTC
[PR] [improve][pip] PIP-444: Expose Raw Message Access in TableView API [pulsar]
namest504 opened a new pull request, #24842:
URL: https://github.com/apache/pulsar/pull/24842
This PR introduces PIP-444, which proposes adding a `getRawMessage()` method to the TableView API.
This allows users to access the full message metadata (e.g., properties, event time), not just the payload.
The implementation for this proposal can be found in PR #24809.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#issuecomment-3407413899
Here's the MessageMapperTableViewImpl experiment:
https://github.com/lhotari/pulsar/blob/lh-tableview-refactoring-to-support-message-values/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageMapperTableViewImpl.java
for the builder, there would be these methods (instead of the createForMessages)
```java
<V> TableView<V> createMapped(Function<Message<T>, V> mapper, boolean releasePooledMessage) throws PulsarClientException;
<V> CompletableFuture<TableView<V>> createMappedAsync(Function<Message<T>, V> mapper, boolean releasePooledMessage) throws PulsarClientException;
```
with this, you could create a `TableView<Message<T>>` by calling the createMapped/createMappedAsync method with parameters `java.util.function.Function.identity(), false`.
@namest504 I think that this mapper function based approach would provide more flexibility for different use cases where message properties would need to be taken into account. Do you see any problems with this approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#issuecomment-3407339060
When I was working on the experiment, https://github.com/lhotari/pulsar/commit/c5a4991, one idea that came into mind is that it would be possible to allow creating a view based on a Function that maps the message to some other object. This would allow using a custom object that takes whatever properties are needed from the message properties and there wouldn't be a need to keep a reference to the complete message instance.
If someone would like to use a message based TableView, passing `java.util.function.Function.identity()` as the function would be a way to use the Message as the value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#issuecomment-3416606971
> I do have two main concerns regarding potential edge cases, and I would appreciate your guidance on them.
>
> 1. If a mapper function returns null: What would be the most appropriate way for the TableView to handle this?
In that case, the key is removed from the map since null is considered to be a tombstone value.
> 2. If the mapper throws an exception: What is the desired behavior in this scenario?
Good question. I don't think that this PIP impacts this. If the mapper throws an exception, the same message will get retried, so the table view will get stuck. The table view doesn't have any way to handle "poison pill" messages which fail every time.
I think addressing this is out of context for this PIP.
> Are these concerns that I should address in the implementation, or are they handled by other parts of the system and not something I need to consider?
>
> If my understanding of the direction is correct, I'd like to update the PIP document and the implementation PR accordingly. Please let me know if this is the right way to proceed.
Makes sense, please go ahead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Expose Raw Message Access in TableView API [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#issuecomment-3401417456
@namest504 Please subscribe to the dev mailing list using the instructions available at https://pulsar.apache.org/contact/#mailing-lists unless you already have completed the steps. (I released your previous email from the moderation queue and emails from out-of-list addresses usually end up there.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#discussion_r2468286626
##########
pip/pip-445.md:
##########
@@ -0,0 +1,137 @@
+# PIP-445: Add Builder Methods to Create Message-based TableView
+
+# Background knowledge
+
+* **TableView**: In Pulsar, a `TableView` is a client-side abstraction that provides a key-value map interface over a Pulsar topic. It consumes messages from the topic (typically a compacted one) and maintains an in-memory view of the latest value for each key. This allows applications to easily query the current state of a key without managing a consumer manually.
+
+* **Pulsar `Message<T>`**: A Pulsar message is not just its data payload. The `Message<T>` object is a container that includes the deserialized **payload** (`T`) as well as important **metadata**, such as a message key, user-defined properties (a key-value map), event time, publish time, and more.
+
+# Motivation
+
+The current `TableView` API provides a `get(String key)` method that only returns the deserialized **value** (`T`) of the latest message for a given key. This limits its usefulness for applications that need access to the message's metadata.
+
+For instance, a user might need to inspect the message **properties** to get a trace-id or check the **event time** to determine if the data is recent. Currently, the only way to access this metadata is to create a separate, redundant `Consumer` on the same topic, which is inefficient and undermines the convenience of using a `TableView`.
+
+This proposal aims to solve this problem by providing a way to create a `TableView` that exposes the entire `Message<T>` object.
+
+# Goals
+
+## In Scope
+
+* Add new generic methods, `createMapped()` and `createMappedAsync()`, to the `TableViewBuilder<T>` interface, which accept a mapping function.
+* Allow users to create a `TableView<V>` instance by providing a function that transforms a `Message<T>` into a custom object `V`. This includes the ability to create a `TableView<Message<T>>` by passing an identity function.
+* Ensure the change is fully backward-compatible and does not impact the performance of existing `TableView` users.
+
+## Out of Scope
+
+* Modifying the behavior of the existing `create()` and `createAsync()` methods in the builder.
+* Changing the underlying topic compaction logic or any broker-side functionality.
+* Handling exceptions thrown by the user-provided mapper function within the `TableView` (e.g., "poison pill" message handling).
+
+# High Level Design
+
+The proposed solution is a simple and non-breaking addition to the public client API. Instead of adding a specific method for retrieving messages, we will introduce a more flexible, generic mapping mechanism.
+
+1. New generic methods, `<V> TableView<V> createMapped(...)` and `<V> CompletableFuture<TableView<V>> createMappedAsync(...)`, will be added to the `TableViewBuilder<T>` interface.
+2. These methods will accept a `java.util.function.Function<Message<T>, V>` as a parameter. This `mapper` function defines how to transform an incoming raw `Message<T>` into a value `V` to be stored in the `TableView`.
+3. This approach provides maximum flexibility. Users who need the entire `Message<T>` object can simply pass `Function.identity()` as the mapper. Other users can create custom, memory-efficient objects containing only the necessary data from the message payload and metadata.
+4. The existing `create()` and `createAsync()` methods will remain unchanged, preserving behavior for all existing use cases.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+The changes will be confined to the Pulsar client library.
+
+* **New Abstract Class `org.apache.pulsar.client.impl.AbstractTableView<V>`**:
+ * An abstract base class will be created to contain the common logic for `TableView` implementations, such as managing the underlying consumer and handling topic events. This prevents code duplication.
+
+* **Class `org.apache.pulsar.client.impl.TableViewImpl<T>`**:
+ * This class will be refactored to extend the new `AbstractTableView<T>`. While its internal implementation will change to use the base class, its public-facing functionality and behavior will remain exactly the same. This ensures that the change has no impact on existing users.
+
+* **New Class `org.apache.pulsar.client.impl.MessageTableViewImpl<T>`**:
+ * A new class will be created that extends `AbstractTableView<Message<T>>` and implements the `TableView<Message<T>>` interface. It will be responsible for storing the full `Message<T>` object for each key, which its `get(key)` method will return.
+
+* **Class `org.apache.pulsar.client.impl.TableViewBuilderImpl<T>`**:
+ * The new `createForMessages` methods will be implemented to instantiate a new `MessageTableViewImpl`.
Review Comment:
Please update these details to match the latest plan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "namest504 (via GitHub)" <gi...@apache.org>.
namest504 commented on PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#issuecomment-3466051090
> > This PR introduces PIP-445, which proposes adding a `getRawMessage()` method to the TableView API.
> > This allows users to access the full message metadata (e.g., properties, event time), not just the payload.
>
> Please update the summary in the description of the PR
Sure, i'll change it now. Thanks your checking! 😊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "namest504 (via GitHub)" <gi...@apache.org>.
namest504 commented on PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#issuecomment-3420593332
@lhotari
I've updated the PIP based on your latest feedback. The document now reflects the `createMapped` function approach, replacing the previous `createForMessages` methods.
The corresponding implementation in #24809 has been updated as well. Ready for your review when you are. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-444: Expose Raw Message Access in TableView API [pulsar]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#issuecomment-3400532697
@namest504 Please add the following content to your PR description and select a checkbox:
```
- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "namest504 (via GitHub)" <gi...@apache.org>.
namest504 commented on PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#issuecomment-3408924943
@lhotari
Thank you for the continued guidance. I want to confirm my understanding and share my thoughts before proceeding with the implementation.
My understanding is that the goal here is to provide maximum flexibility—evolving beyond a binary choice of payload vs raw message and instead empowering users to transform a Message<T> into any custom object V that fits their specific needs. Is this correct?
I agree this is a very appropriate and powerful extension for this feature.
For example
```java
public class OriginalData {
String a;
double b;
}
public class Foo {
String a;
double b;
long eventMetaA; // from message.getEventTime()
String eventMetaB; // from message.getProperty()
}
TableView<Foo> view = builder
.createMapped(message -> {
OriginalData original = message.getValue();
return new Foo(
original.a,
original.b,
message.getEventMetaA(),
message.getEventMetaB()
);
});
```
---
I do have two main concerns regarding potential edge cases, and I would appreciate your guidance on them.
1. If a mapper function returns null: What would be the most appropriate way for the TableView to handle this?
2. If the mapper throws an exception: What is the desired behavior in this scenario?
Are these concerns that I should address in the implementation, or are they handled by other parts of the system and not something I need to consider?
---
If my understanding of the direction is correct, I'd like to update the PIP document and the implementation PR accordingly. Please let me know if this is the right way to proceed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "namest504 (via GitHub)" <gi...@apache.org>.
namest504 commented on PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#issuecomment-3402496019
Also, I've subscribed to the developer mailing list as you recommended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "namest504 (via GitHub)" <gi...@apache.org>.
namest504 closed pull request #24842: [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView
URL: https://github.com/apache/pulsar/pull/24842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "namest504 (via GitHub)" <gi...@apache.org>.
namest504 commented on code in PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#discussion_r2471849692
##########
pip/pip-445.md:
##########
@@ -0,0 +1,137 @@
+# PIP-445: Add Builder Methods to Create Message-based TableView
+
+# Background knowledge
+
+* **TableView**: In Pulsar, a `TableView` is a client-side abstraction that provides a key-value map interface over a Pulsar topic. It consumes messages from the topic (typically a compacted one) and maintains an in-memory view of the latest value for each key. This allows applications to easily query the current state of a key without managing a consumer manually.
+
+* **Pulsar `Message<T>`**: A Pulsar message is not just its data payload. The `Message<T>` object is a container that includes the deserialized **payload** (`T`) as well as important **metadata**, such as a message key, user-defined properties (a key-value map), event time, publish time, and more.
+
+# Motivation
+
+The current `TableView` API provides a `get(String key)` method that only returns the deserialized **value** (`T`) of the latest message for a given key. This limits its usefulness for applications that need access to the message's metadata.
+
+For instance, a user might need to inspect the message **properties** to get a trace-id or check the **event time** to determine if the data is recent. Currently, the only way to access this metadata is to create a separate, redundant `Consumer` on the same topic, which is inefficient and undermines the convenience of using a `TableView`.
+
+This proposal aims to solve this problem by providing a way to create a `TableView` that exposes the entire `Message<T>` object.
+
+# Goals
+
+## In Scope
+
+* Add new generic methods, `createMapped()` and `createMappedAsync()`, to the `TableViewBuilder<T>` interface, which accept a mapping function.
+* Allow users to create a `TableView<V>` instance by providing a function that transforms a `Message<T>` into a custom object `V`. This includes the ability to create a `TableView<Message<T>>` by passing an identity function.
+* Ensure the change is fully backward-compatible and does not impact the performance of existing `TableView` users.
+
+## Out of Scope
+
+* Modifying the behavior of the existing `create()` and `createAsync()` methods in the builder.
+* Changing the underlying topic compaction logic or any broker-side functionality.
+* Handling exceptions thrown by the user-provided mapper function within the `TableView` (e.g., "poison pill" message handling).
+
+# High Level Design
+
+The proposed solution is a simple and non-breaking addition to the public client API. Instead of adding a specific method for retrieving messages, we will introduce a more flexible, generic mapping mechanism.
+
+1. New generic methods, `<V> TableView<V> createMapped(...)` and `<V> CompletableFuture<TableView<V>> createMappedAsync(...)`, will be added to the `TableViewBuilder<T>` interface.
+2. These methods will accept a `java.util.function.Function<Message<T>, V>` as a parameter. This `mapper` function defines how to transform an incoming raw `Message<T>` into a value `V` to be stored in the `TableView`.
+3. This approach provides maximum flexibility. Users who need the entire `Message<T>` object can simply pass `Function.identity()` as the mapper. Other users can create custom, memory-efficient objects containing only the necessary data from the message payload and metadata.
+4. The existing `create()` and `createAsync()` methods will remain unchanged, preserving behavior for all existing use cases.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+The changes will be confined to the Pulsar client library.
+
+* **New Abstract Class `org.apache.pulsar.client.impl.AbstractTableView<V>`**:
+ * An abstract base class will be created to contain the common logic for `TableView` implementations, such as managing the underlying consumer and handling topic events. This prevents code duplication.
+
+* **Class `org.apache.pulsar.client.impl.TableViewImpl<T>`**:
+ * This class will be refactored to extend the new `AbstractTableView<T>`. While its internal implementation will change to use the base class, its public-facing functionality and behavior will remain exactly the same. This ensures that the change has no impact on existing users.
+
+* **New Class `org.apache.pulsar.client.impl.MessageTableViewImpl<T>`**:
+ * A new class will be created that extends `AbstractTableView<Message<T>>` and implements the `TableView<Message<T>>` interface. It will be responsible for storing the full `Message<T>` object for each key, which its `get(key)` method will return.
+
+* **Class `org.apache.pulsar.client.impl.TableViewBuilderImpl<T>`**:
+ * The new `createForMessages` methods will be implemented to instantiate a new `MessageTableViewImpl`.
Review Comment:
I have updated the PIP to reflect the current design. 🥹
* **Refactoring of `org.apache.pulsar.client.impl.TableViewImpl<T>`**:
* The existing `TableViewImpl<T>` class is refactored into a new inheritance structure.
* The core logic is moved to a new abstract base class, `org.apache.pulsar.client.impl.AbstractTableView<T, V>`.
* To preserve the original functionality, a new simple class `org.apache.pulsar.client.impl.TableView<T>` is introduced, which extends `AbstractTableView<T, T>`.
* **New `MessageMapperTableView` implementation**:
* `org.apache.pulsar.client.impl.MessageMapperTableView<T, V>`: A new class that extends `AbstractTableView<T, V>`. It implements the logic for the new `createMapped` methods, using a supplied `Function<Message<T>, V>` to transform messages into the values stored in the `TableView`.
* **Class `org.apache.pulsar.client.impl.TableViewBuilderImpl<T>`**:
* The builder's implementation will be updated to use the new classes.
* `create()` and `createAsync()` will now instantiate the new `org.apache.pulsar.client.impl.TableView<T>` to provide the classic `TableView` behavior.
* The new `createMapped()` and `createMappedAsync()` methods will instantiate `MessageMapperTableView<T, V>` with the user-provided mapper function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#discussion_r2431615285
##########
pip/pip-445.md:
##########
@@ -0,0 +1,126 @@
+# PIP-445: Add Builder Methods to Create Message-based TableView
+
+# Background knowledge
+
+* **TableView**: In Pulsar, a `TableView` is a client-side abstraction that provides a key-value map interface over a Pulsar topic. It consumes messages from the topic (typically a compacted one) and maintains an in-memory view of the latest value for each key. This allows applications to easily query the current state of a key without managing a consumer manually.
+
+* **Pulsar `Message<T>`**: A Pulsar message is not just its data payload. The `Message<T>` object is a container that includes the deserialized **payload** (`T`) as well as important **metadata**, such as a message key, user-defined properties (a key-value map), event time, publish time, and more.
+
+# Motivation
+
+The current `TableView` API provides a `get(String key)` method that only returns the deserialized **value** (`T`) of the latest message for a given key. This limits its usefulness for applications that need access to the message's metadata.
+
+For instance, a user might need to inspect the message **properties** to get a trace-id or check the **event time** to determine if the data is recent. Currently, the only way to access this metadata is to create a separate, redundant `Consumer` on the same topic, which is inefficient and undermines the convenience of using a `TableView`.
+
+This proposal aims to solve this problem by providing a way to create a `TableView` that exposes the entire `Message<T>` object.
+
+# Goals
+
+## In Scope
+
+* Add new methods, `createForMessages()` and `createForMessagesAsync()`, to the `TableViewBuilder<T>` interface.
+* Allow users to create a `TableView<Message<T>>` instance, which provides access to the complete `Message<T>` object for each key, including its payload, properties, and all other metadata.
+* Ensure the change is fully backward-compatible and does not impact the performance of existing `TableView` users.
+
+## Out of Scope
+
+* Modifying the behavior of the existing `create()` and `createAsync()` methods in the builder.
+* Changing the underlying topic compaction logic or any broker-side functionality.
+
+# High Level Design
+
+The proposed solution is a simple and non-breaking addition to the public client API. Instead of modifying the existing `TableView` implementation, we will introduce new methods to the `TableViewBuilder`.
+
+1. New methods, `TableView<Message<T>> createForMessages()` and `CompletableFuture<TableView<Message<T>>> createForMessagesAsync()`, will be added to the `TableViewBuilder<T>` interface.
+2. These methods will create a new, specialized `TableView` implementation (`MessageTableViewImpl`) that stores the entire `Message<T>` object for each key.
+3. The existing `create()` and `createAsync()` methods will continue to create the standard `TableView` that stores only the message value (`T`).
+
+This opt-in design provides the new functionality efficiently without impacting the performance or behavior of existing `TableView` use cases.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+The changes will be confined to the Pulsar client library.
+
+* **Interface `org.apache.pulsar.client.api.TableViewBuilder<T>`**:
+ New methods will be added to this interface to create a `TableView` for messages.
+
+* **Class `org.apache.pulsar.client.impl.TableViewBuilderImpl<T>`**:
+ The new `createForMessages` methods will be implemented to instantiate a new `MessageTableViewImpl`.
+
+* **New Class `org.apache.pulsar.client.impl.MessageTableViewImpl<T>`**:
+ A new class will be created that implements `TableView<Message<T>>`. It will be based on the existing `TableViewImpl` but its internal map will store `Message<T>` objects instead of just `T` values. Its `get(key)` method will return the full `Message<T>` object.
+
+* **Class `org.apache.pulsar.client.impl.TableViewImpl<T>`**:
+ This class will remain unchanged, ensuring no impact on existing users.
Review Comment:
TableViewImpl would be changed to use the abstract base class (as seen in experiment https://github.com/lhotari/pulsar/commit/c5a4991), but the functionality won't change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#discussion_r2431615285
##########
pip/pip-445.md:
##########
@@ -0,0 +1,126 @@
+# PIP-445: Add Builder Methods to Create Message-based TableView
+
+# Background knowledge
+
+* **TableView**: In Pulsar, a `TableView` is a client-side abstraction that provides a key-value map interface over a Pulsar topic. It consumes messages from the topic (typically a compacted one) and maintains an in-memory view of the latest value for each key. This allows applications to easily query the current state of a key without managing a consumer manually.
+
+* **Pulsar `Message<T>`**: A Pulsar message is not just its data payload. The `Message<T>` object is a container that includes the deserialized **payload** (`T`) as well as important **metadata**, such as a message key, user-defined properties (a key-value map), event time, publish time, and more.
+
+# Motivation
+
+The current `TableView` API provides a `get(String key)` method that only returns the deserialized **value** (`T`) of the latest message for a given key. This limits its usefulness for applications that need access to the message's metadata.
+
+For instance, a user might need to inspect the message **properties** to get a trace-id or check the **event time** to determine if the data is recent. Currently, the only way to access this metadata is to create a separate, redundant `Consumer` on the same topic, which is inefficient and undermines the convenience of using a `TableView`.
+
+This proposal aims to solve this problem by providing a way to create a `TableView` that exposes the entire `Message<T>` object.
+
+# Goals
+
+## In Scope
+
+* Add new methods, `createForMessages()` and `createForMessagesAsync()`, to the `TableViewBuilder<T>` interface.
+* Allow users to create a `TableView<Message<T>>` instance, which provides access to the complete `Message<T>` object for each key, including its payload, properties, and all other metadata.
+* Ensure the change is fully backward-compatible and does not impact the performance of existing `TableView` users.
+
+## Out of Scope
+
+* Modifying the behavior of the existing `create()` and `createAsync()` methods in the builder.
+* Changing the underlying topic compaction logic or any broker-side functionality.
+
+# High Level Design
+
+The proposed solution is a simple and non-breaking addition to the public client API. Instead of modifying the existing `TableView` implementation, we will introduce new methods to the `TableViewBuilder`.
+
+1. New methods, `TableView<Message<T>> createForMessages()` and `CompletableFuture<TableView<Message<T>>> createForMessagesAsync()`, will be added to the `TableViewBuilder<T>` interface.
+2. These methods will create a new, specialized `TableView` implementation (`MessageTableViewImpl`) that stores the entire `Message<T>` object for each key.
+3. The existing `create()` and `createAsync()` methods will continue to create the standard `TableView` that stores only the message value (`T`).
+
+This opt-in design provides the new functionality efficiently without impacting the performance or behavior of existing `TableView` use cases.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+The changes will be confined to the Pulsar client library.
+
+* **Interface `org.apache.pulsar.client.api.TableViewBuilder<T>`**:
+ New methods will be added to this interface to create a `TableView` for messages.
+
+* **Class `org.apache.pulsar.client.impl.TableViewBuilderImpl<T>`**:
+ The new `createForMessages` methods will be implemented to instantiate a new `MessageTableViewImpl`.
+
+* **New Class `org.apache.pulsar.client.impl.MessageTableViewImpl<T>`**:
+ A new class will be created that implements `TableView<Message<T>>`. It will be based on the existing `TableViewImpl` but its internal map will store `Message<T>` objects instead of just `T` values. Its `get(key)` method will return the full `Message<T>` object.
+
+* **Class `org.apache.pulsar.client.impl.TableViewImpl<T>`**:
+ This class will remain unchanged, ensuring no impact on existing users.
Review Comment:
This would change to use the abstract base class, but the functionality won't change.
##########
pip/pip-445.md:
##########
@@ -0,0 +1,126 @@
+# PIP-445: Add Builder Methods to Create Message-based TableView
+
+# Background knowledge
+
+* **TableView**: In Pulsar, a `TableView` is a client-side abstraction that provides a key-value map interface over a Pulsar topic. It consumes messages from the topic (typically a compacted one) and maintains an in-memory view of the latest value for each key. This allows applications to easily query the current state of a key without managing a consumer manually.
+
+* **Pulsar `Message<T>`**: A Pulsar message is not just its data payload. The `Message<T>` object is a container that includes the deserialized **payload** (`T`) as well as important **metadata**, such as a message key, user-defined properties (a key-value map), event time, publish time, and more.
+
+# Motivation
+
+The current `TableView` API provides a `get(String key)` method that only returns the deserialized **value** (`T`) of the latest message for a given key. This limits its usefulness for applications that need access to the message's metadata.
+
+For instance, a user might need to inspect the message **properties** to get a trace-id or check the **event time** to determine if the data is recent. Currently, the only way to access this metadata is to create a separate, redundant `Consumer` on the same topic, which is inefficient and undermines the convenience of using a `TableView`.
+
+This proposal aims to solve this problem by providing a way to create a `TableView` that exposes the entire `Message<T>` object.
+
+# Goals
+
+## In Scope
+
+* Add new methods, `createForMessages()` and `createForMessagesAsync()`, to the `TableViewBuilder<T>` interface.
+* Allow users to create a `TableView<Message<T>>` instance, which provides access to the complete `Message<T>` object for each key, including its payload, properties, and all other metadata.
+* Ensure the change is fully backward-compatible and does not impact the performance of existing `TableView` users.
+
+## Out of Scope
+
+* Modifying the behavior of the existing `create()` and `createAsync()` methods in the builder.
+* Changing the underlying topic compaction logic or any broker-side functionality.
+
+# High Level Design
+
+The proposed solution is a simple and non-breaking addition to the public client API. Instead of modifying the existing `TableView` implementation, we will introduce new methods to the `TableViewBuilder`.
+
+1. New methods, `TableView<Message<T>> createForMessages()` and `CompletableFuture<TableView<Message<T>>> createForMessagesAsync()`, will be added to the `TableViewBuilder<T>` interface.
+2. These methods will create a new, specialized `TableView` implementation (`MessageTableViewImpl`) that stores the entire `Message<T>` object for each key.
+3. The existing `create()` and `createAsync()` methods will continue to create the standard `TableView` that stores only the message value (`T`).
+
+This opt-in design provides the new functionality efficiently without impacting the performance or behavior of existing `TableView` use cases.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+The changes will be confined to the Pulsar client library.
+
+* **Interface `org.apache.pulsar.client.api.TableViewBuilder<T>`**:
+ New methods will be added to this interface to create a `TableView` for messages.
+
+* **Class `org.apache.pulsar.client.impl.TableViewBuilderImpl<T>`**:
+ The new `createForMessages` methods will be implemented to instantiate a new `MessageTableViewImpl`.
+
+* **New Class `org.apache.pulsar.client.impl.MessageTableViewImpl<T>`**:
+ A new class will be created that implements `TableView<Message<T>>`. It will be based on the existing `TableViewImpl` but its internal map will store `Message<T>` objects instead of just `T` values. Its `get(key)` method will return the full `Message<T>` object.
Review Comment:
There's no need to duplicate code. An abstract base class is a better solution. I did a quick refactoring to show how this can be accomplished: https://github.com/lhotari/pulsar/commit/c5a4991 .
This can be used as the basis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#issuecomment-3498413066
@namest504 Please go ahead and start a vote thread on the dev mailing list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari merged PR #24842:
URL: https://github.com/apache/pulsar/pull/24842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "namest504 (via GitHub)" <gi...@apache.org>.
namest504 commented on PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#issuecomment-3402494293
> A different approach should be taken. Please check the comment [#24809 (review)](https://github.com/apache/pulsar/pull/24809#pullrequestreview-3334058939) .
>
> The current getRawMessage solution would be a breaking change for existing TableView API users since it would unnecessarily consume more memory and CPU. That's why a different approach must be taken. I've suggested to resolve the issue by adding new methods to the API for returning a TableView with Message as the type parameter.
>
> ```java
> TableView<Message<T>> createForMessages() throws PulsarClientException;
> CompletableFuture<TableView<Message<T>>> createForMessagesAsync() throws PulsarClientException;
> ```
>
> Would that approach solve your use case?
@lhotari
Thank you for your detailed feedback and guidance.
I've revised the PR title.
I've updated the PR to adopt the builder-based approach you suggested. This new commit implements the `createForMessages()` and `createForMessagesAsync()` methods to avoid the performance impact on existing `TableView` users.
Please let me know if this new direction looks good. Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#issuecomment-3460120993
> This PR introduces PIP-445, which proposes adding a `getRawMessage()` method to the TableView API.
>
> This allows users to access the full message metadata (e.g., properties, event time), not just the payload.
Please update the summary in the description of the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "namest504 (via GitHub)" <gi...@apache.org>.
namest504 commented on code in PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#discussion_r2434397402
##########
pip/pip-445.md:
##########
@@ -0,0 +1,126 @@
+# PIP-445: Add Builder Methods to Create Message-based TableView
+
+# Background knowledge
+
+* **TableView**: In Pulsar, a `TableView` is a client-side abstraction that provides a key-value map interface over a Pulsar topic. It consumes messages from the topic (typically a compacted one) and maintains an in-memory view of the latest value for each key. This allows applications to easily query the current state of a key without managing a consumer manually.
+
+* **Pulsar `Message<T>`**: A Pulsar message is not just its data payload. The `Message<T>` object is a container that includes the deserialized **payload** (`T`) as well as important **metadata**, such as a message key, user-defined properties (a key-value map), event time, publish time, and more.
+
+# Motivation
+
+The current `TableView` API provides a `get(String key)` method that only returns the deserialized **value** (`T`) of the latest message for a given key. This limits its usefulness for applications that need access to the message's metadata.
+
+For instance, a user might need to inspect the message **properties** to get a trace-id or check the **event time** to determine if the data is recent. Currently, the only way to access this metadata is to create a separate, redundant `Consumer` on the same topic, which is inefficient and undermines the convenience of using a `TableView`.
+
+This proposal aims to solve this problem by providing a way to create a `TableView` that exposes the entire `Message<T>` object.
+
+# Goals
+
+## In Scope
+
+* Add new methods, `createForMessages()` and `createForMessagesAsync()`, to the `TableViewBuilder<T>` interface.
+* Allow users to create a `TableView<Message<T>>` instance, which provides access to the complete `Message<T>` object for each key, including its payload, properties, and all other metadata.
+* Ensure the change is fully backward-compatible and does not impact the performance of existing `TableView` users.
+
+## Out of Scope
+
+* Modifying the behavior of the existing `create()` and `createAsync()` methods in the builder.
+* Changing the underlying topic compaction logic or any broker-side functionality.
+
+# High Level Design
+
+The proposed solution is a simple and non-breaking addition to the public client API. Instead of modifying the existing `TableView` implementation, we will introduce new methods to the `TableViewBuilder`.
+
+1. New methods, `TableView<Message<T>> createForMessages()` and `CompletableFuture<TableView<Message<T>>> createForMessagesAsync()`, will be added to the `TableViewBuilder<T>` interface.
+2. These methods will create a new, specialized `TableView` implementation (`MessageTableViewImpl`) that stores the entire `Message<T>` object for each key.
+3. The existing `create()` and `createAsync()` methods will continue to create the standard `TableView` that stores only the message value (`T`).
+
+This opt-in design provides the new functionality efficiently without impacting the performance or behavior of existing `TableView` use cases.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+The changes will be confined to the Pulsar client library.
+
+* **Interface `org.apache.pulsar.client.api.TableViewBuilder<T>`**:
+ New methods will be added to this interface to create a `TableView` for messages.
+
+* **Class `org.apache.pulsar.client.impl.TableViewBuilderImpl<T>`**:
+ The new `createForMessages` methods will be implemented to instantiate a new `MessageTableViewImpl`.
+
+* **New Class `org.apache.pulsar.client.impl.MessageTableViewImpl<T>`**:
+ A new class will be created that implements `TableView<Message<T>>`. It will be based on the existing `TableViewImpl` but its internal map will store `Message<T>` objects instead of just `T` values. Its `get(key)` method will return the full `Message<T>` object.
Review Comment:
I agree. This approach seems much more maintainable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView [pulsar]
Posted by "namest504 (via GitHub)" <gi...@apache.org>.
namest504 commented on code in PR #24842:
URL: https://github.com/apache/pulsar/pull/24842#discussion_r2434398382
##########
pip/pip-445.md:
##########
@@ -0,0 +1,126 @@
+# PIP-445: Add Builder Methods to Create Message-based TableView
+
+# Background knowledge
+
+* **TableView**: In Pulsar, a `TableView` is a client-side abstraction that provides a key-value map interface over a Pulsar topic. It consumes messages from the topic (typically a compacted one) and maintains an in-memory view of the latest value for each key. This allows applications to easily query the current state of a key without managing a consumer manually.
+
+* **Pulsar `Message<T>`**: A Pulsar message is not just its data payload. The `Message<T>` object is a container that includes the deserialized **payload** (`T`) as well as important **metadata**, such as a message key, user-defined properties (a key-value map), event time, publish time, and more.
+
+# Motivation
+
+The current `TableView` API provides a `get(String key)` method that only returns the deserialized **value** (`T`) of the latest message for a given key. This limits its usefulness for applications that need access to the message's metadata.
+
+For instance, a user might need to inspect the message **properties** to get a trace-id or check the **event time** to determine if the data is recent. Currently, the only way to access this metadata is to create a separate, redundant `Consumer` on the same topic, which is inefficient and undermines the convenience of using a `TableView`.
+
+This proposal aims to solve this problem by providing a way to create a `TableView` that exposes the entire `Message<T>` object.
+
+# Goals
+
+## In Scope
+
+* Add new methods, `createForMessages()` and `createForMessagesAsync()`, to the `TableViewBuilder<T>` interface.
+* Allow users to create a `TableView<Message<T>>` instance, which provides access to the complete `Message<T>` object for each key, including its payload, properties, and all other metadata.
+* Ensure the change is fully backward-compatible and does not impact the performance of existing `TableView` users.
+
+## Out of Scope
+
+* Modifying the behavior of the existing `create()` and `createAsync()` methods in the builder.
+* Changing the underlying topic compaction logic or any broker-side functionality.
+
+# High Level Design
+
+The proposed solution is a simple and non-breaking addition to the public client API. Instead of modifying the existing `TableView` implementation, we will introduce new methods to the `TableViewBuilder`.
+
+1. New methods, `TableView<Message<T>> createForMessages()` and `CompletableFuture<TableView<Message<T>>> createForMessagesAsync()`, will be added to the `TableViewBuilder<T>` interface.
+2. These methods will create a new, specialized `TableView` implementation (`MessageTableViewImpl`) that stores the entire `Message<T>` object for each key.
+3. The existing `create()` and `createAsync()` methods will continue to create the standard `TableView` that stores only the message value (`T`).
+
+This opt-in design provides the new functionality efficiently without impacting the performance or behavior of existing `TableView` use cases.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+The changes will be confined to the Pulsar client library.
+
+* **Interface `org.apache.pulsar.client.api.TableViewBuilder<T>`**:
+ New methods will be added to this interface to create a `TableView` for messages.
+
+* **Class `org.apache.pulsar.client.impl.TableViewBuilderImpl<T>`**:
+ The new `createForMessages` methods will be implemented to instantiate a new `MessageTableViewImpl`.
+
+* **New Class `org.apache.pulsar.client.impl.MessageTableViewImpl<T>`**:
+ A new class will be created that implements `TableView<Message<T>>`. It will be based on the existing `TableViewImpl` but its internal map will store `Message<T>` objects instead of just `T` values. Its `get(key)` method will return the full `Message<T>` object.
+
+* **Class `org.apache.pulsar.client.impl.TableViewImpl<T>`**:
+ This class will remain unchanged, ensuring no impact on existing users.
Review Comment:
I agree with the review above. 😊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org