You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/12/25 05:14:21 UTC
[GitHub] [pulsar] duli559 opened a new issue #5934: support read/write
properties from/to Message in flink pulsar consumer/producer
duli559 opened a new issue #5934: support read/write properties from/to Message in flink pulsar consumer/producer
URL: https://github.com/apache/pulsar/issues/5934
**Is your feature request related to a problem? Please describe.**
Currently, flink pulsar consumer/producer can not read/write properties from/to Message, we hope to support it.
**Describe the solution you'd like**
A) For consumer,
change method `deserialize(Message message)’` access level from private to proteced in class 'PulsarConsumerSource', and we can overrided it in derived class.
B) For producer,
1. add a method like following code, and we can overrided it in derived class.
`protected Map<String, String> generateProperties(IN value) {
return new HashMap<>();
}`
2. invoke TypedMessageBuilder.properties() method addtional in `invoke(IN value, Context context)` method at class 'FlinkPulsarProducer'.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] duli559 edited a comment on issue #5934: Can support
read/write properties from/to Message in flink pulsar consumer/producer
Posted by GitBox <gi...@apache.org>.
duli559 edited a comment on issue #5934: Can support read/write properties from/to Message in flink pulsar consumer/producer
URL: https://github.com/apache/pulsar/issues/5934#issuecomment-568865055
@yjshen , hi, can you have a look, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] duli559 removed a comment on issue #5934: Can support
read/write properties from/to Message in flink pulsar consumer/producer
Posted by GitBox <gi...@apache.org>.
duli559 removed a comment on issue #5934: Can support read/write properties from/to Message in flink pulsar consumer/producer
URL: https://github.com/apache/pulsar/issues/5934#issuecomment-569187441
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] duli559 commented on issue #5934: Can support read/write
properties from/to Message in flink pulsar consumer/producer
Posted by GitBox <gi...@apache.org>.
duli559 commented on issue #5934: Can support read/write properties from/to Message in flink pulsar consumer/producer
URL: https://github.com/apache/pulsar/issues/5934#issuecomment-568865055
@yjshen , hi, can you have a lool, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] yjshen commented on issue #5934: Can support read/write
properties from/to Message in flink pulsar consumer/producer
Posted by GitBox <gi...@apache.org>.
yjshen commented on issue #5934: Can support read/write properties from/to Message in flink pulsar consumer/producer
URL: https://github.com/apache/pulsar/issues/5934#issuecomment-568889578
Hi @duli559 , thanks for bringing this up.
For the consumer, I think the proposed way is possible, your derived class could return a Tuple (Map, T) for each message, and use the tuple afterward.
For the producer, the `invoke` method is invoked for each `value` in the stream, therefore, you should use this `value` and tell which part goes into properties and which part went goes into Message body. I think you could extend the existing `FlinkPulsarProducer` directly and just overwrite the `invoke` method?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] yjshen commented on issue #5934: Can support read/write
properties from/to Message in flink pulsar consumer/producer
Posted by GitBox <gi...@apache.org>.
yjshen commented on issue #5934: Can support read/write properties from/to Message in flink pulsar consumer/producer
URL: https://github.com/apache/pulsar/issues/5934#issuecomment-568902649
@duli559 that makes sense to me. Please create a PR for this :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] jiazhai closed issue #5934: Can support read/write
properties from/to Message in flink pulsar consumer/producer
Posted by GitBox <gi...@apache.org>.
jiazhai closed issue #5934: Can support read/write properties from/to Message in flink pulsar consumer/producer
URL: https://github.com/apache/pulsar/issues/5934
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] duli559 commented on issue #5934: Can support read/write
properties from/to Message in flink pulsar consumer/producer
Posted by GitBox <gi...@apache.org>.
duli559 commented on issue #5934: Can support read/write properties from/to Message in flink pulsar consumer/producer
URL: https://github.com/apache/pulsar/issues/5934#issuecomment-569187441
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] duli559 commented on issue #5934: Can support read/write
properties from/to Message in flink pulsar consumer/producer
Posted by GitBox <gi...@apache.org>.
duli559 commented on issue #5934: Can support read/write properties from/to Message in flink pulsar consumer/producer
URL: https://github.com/apache/pulsar/issues/5934#issuecomment-568893645
Hi @yjshen , thanks for your reply.
I think Message properties not only created by `value`, but also other ways, eg: some constants, values that are updated periodically, etc. so i think if we extract a method in super class and override in derived class, it is a better way, and we don't need care other details in derived class.
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services