[jira] [Comment Edited] (FLINK-31428) Add user callbacks to PulsarSource and PulsarSink

["Yufan Sheng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-31428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700319#comment-17700319 ] 

Yufan Sheng edited comment on FLINK-31428 at 3/14/23 5:36 PM:
--------------------------------------------------------------

I don't think it's ok to expose the deserialized/serialized values. You can get and modify it only in PulsarXXXXXSchema. The callbacks can use the {{ConsumerInterceptor}} and {{ProducerInterceptor}}, add a trace id in interceptor and access it in Pulsar schema for serialized values. The context of the trace can be stored in {{ThreadLocal}}, this is common used in the trace related monitor systems.

We can add this support and that's enough, I think.


was (Author: syhily):
I don't think it's ok to expose the deserialized/serialized values. You can get and modify it only in PulsarXXXXXSchema. The callbacks can use the {{ConsumerInterceptor}} and {{ProducerInterceptor}}, add a trace id in interceptor and access it in Pulsar schema for serialized values. We can add this support and that's enough, I think.

> Add user callbacks to  PulsarSource and PulsarSink
> --------------------------------------------------
>
>                 Key: FLINK-31428
>                 URL: https://issues.apache.org/jira/browse/FLINK-31428
>             Project: Flink
>          Issue Type: New Feature
>          Components: Connectors / Pulsar
>            Reporter: Alpha Diallo
>            Priority: Major
>
> We'd like to add support for user callbacks in {{PulsarSource}} and {{{}PulsarSink{}}}. This enables specific use cases such as event tracing which requires access to low level message properties such as message IDs after an event is produced, topic partitions, etc...
> The functionality required is similar to {{ConsumerInterceptor}} and {{ProducerInterceptor}} in the Pulsar Client. However, there is a case to be made for adding new APIs that would help avoid the extra cost of ser/deser when getting the message body through the {{Message}} interface in the interceptors.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)