You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "chenyahui (via GitHub)" <gi...@apache.org> on 2023/12/29 14:57:36 UTC
[D] Does pulsar support server-side payload processor? [pulsar]
GitHub user chenyahui created a discussion: Does pulsar support server-side payload processor?
For example, is there a plug-in that supports rewriting or pruning the message payload on the broker side?
If there is such a function, we can implement data reduction on the server side to reduce the bandwidth.
GitHub link: https://github.com/apache/pulsar/discussions/21827
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Does pulsar support modify payload on broker [pulsar]
Posted by "chenyahui (via GitHub)" <gi...@apache.org>.
GitHub user chenyahui edited a discussion: Does pulsar support modify payload on broker
For example, is there a plug-in that supports rewriting or pruning the message payload on the broker side?
If there is such a function, we can implement data reduction on the server side to reduce the bandwidth.
GitHub link: https://github.com/apache/pulsar/discussions/21827
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Can pulsar provide a broker plug-in that can be used to rewrite the payload? [pulsar]
Posted by "chenyahui (via GitHub)" <gi...@apache.org>.
GitHub user chenyahui edited a discussion: Can pulsar provide a broker plug-in that can be used to rewrite the payload?
Now Pulsar has a broker plug-in named EntryFilter, which allows users to customize filtering logic and filter data that consumers do not need on the broker side.
But this is not completely sufficient: in many scenarios, we also need to streamline the payload on the broker side. The broker directly cuts out the parts of the payload that the Consumer does not need.
For example: Payload is `{"a": 1, "b": 2, "c": 3}`, and Consumer only needs `{"a": 1}`. We can directly cut the data to `{"a": 1}` on the broker side. {"a": 1}`, and then send it to Consumer, This can reduce a lot of IO bandwidth.
Therefore, can pulsar provide a broker plug-in that can be used to rewrite the payload?
GitHub link: https://github.com/apache/pulsar/discussions/21827
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Can pulsar provide a broker plug-in to rewrite the payload? [pulsar]
Posted by "chenyahui (via GitHub)" <gi...@apache.org>.
GitHub user chenyahui edited a discussion: Can pulsar provide a broker plug-in to rewrite the payload?
Now Pulsar has a broker plug-in named EntryFilter, which allows users to customize filtering logic and filter data that consumers do not need on the broker side.
But this is not completely sufficient: in many scenarios, we also need to streamline the payload on the broker side. The broker directly cuts out the parts of the payload that the Consumer does not need.
For example: Payload is `{"a": 1, "b": 2, "c": 3}`, and Consumer only needs `{"a": 1}`. We can directly cut the data to `{"a": 1}` on the broker side. {"a": 1}`, and then send it to Consumer, This can reduce a lot of IO bandwidth.
Therefore, can pulsar provide a broker plug-in that can be used to rewrite the payload?
GitHub link: https://github.com/apache/pulsar/discussions/21827
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Does pulsar support custom payload modification on broker [pulsar]
Posted by "chenyahui (via GitHub)" <gi...@apache.org>.
GitHub user chenyahui edited a discussion: Does pulsar support custom payload modification on broker
Now Pulsar has a broker plug-in named EntryFilter, which allows users to customize filtering logic and filter data that consumers do not need on the broker side.
But this is not completely sufficient: in many scenarios, we also need to streamline the payload on the broker side. The broker directly cuts out the parts of the payload that the Consumer does not need.
For example: Payload is `{"a": 1, "b": 2, "c": 3}`, and Consumer only needs `{"a": 1}`. We can directly cut the data to `{"a": 1}` on the broker side. {"a": 1}`, and then send it to Consumer, This can reduce a lot of IO bandwidth.
Therefore, can pulsar provide a broker plug-in that can be used to rewrite the payload?
GitHub link: https://github.com/apache/pulsar/discussions/21827
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Can pulsar provide a broker plug-in to rewrite the payload? [pulsar]
Posted by "chenyahui (via GitHub)" <gi...@apache.org>.
GitHub user chenyahui edited a discussion: Can pulsar provide a broker plug-in to rewrite the payload?
Now Pulsar has a broker plug-in named EntryFilter, which allows users to customize filtering logic and filter data that consumers do not need on the broker side.
But this is not completely sufficient: in many scenarios, we also need to streamline the payload on the broker side. The broker directly cuts out the parts of the payload that the Consumer does not need.
For example: Payload is `{"a": 1, "b": 2, "c": 3}`, and Consumer only needs `{"a": 1}`. We can directly cut the data to `{"a": 1}` on the broker side. {"a": 1}`, and then send it to Consumer, This can reduce a lot of IO bandwidth.
Therefore, can pulsar provide a broker plug-in to rewrite the payload?
GitHub link: https://github.com/apache/pulsar/discussions/21827
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Does pulsar support custom payload modification on broker [pulsar]
Posted by "chenyahui (via GitHub)" <gi...@apache.org>.
GitHub user chenyahui edited a discussion: Does pulsar support custom payload modification on broker
For example, is there a plug-in that supports rewriting or pruning the message payload on the broker side?
If there is such a function, we can implement data reduction on the server side to reduce the bandwidth.
GitHub link: https://github.com/apache/pulsar/discussions/21827
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Can pulsar provide a broker plug-in to rewrite the payload? [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
GitHub user dao-jun added a comment to the discussion: Can pulsar provide a broker plug-in to rewrite the payload?
It doesn't make sense.
If broker provides such a plugin, which means broker will decode the payload, it's not recommended.
It will lead to broker CPU and Memory usage increasing.
GitHub link: https://github.com/apache/pulsar/discussions/21827#discussioncomment-8008820
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Can pulsar provide a broker plugin that can be used to rewrite the payload? [pulsar]
Posted by "chenyahui (via GitHub)" <gi...@apache.org>.
GitHub user chenyahui edited a discussion: Can pulsar provide a broker plugin that can be used to rewrite the payload?
Now Pulsar has a broker plug-in named EntryFilter, which allows users to customize filtering logic and filter data that consumers do not need on the broker side.
But this is not completely sufficient: in many scenarios, we also need to streamline the payload on the broker side. The broker directly cuts out the parts of the payload that the Consumer does not need.
For example: Payload is `{"a": 1, "b": 2, "c": 3}`, and Consumer only needs `{"a": 1}`. We can directly cut the data to `{"a": 1}` on the broker side. {"a": 1}`, and then send it to Consumer, This can reduce a lot of IO bandwidth.
Therefore, can pulsar provide a broker plug-in that can be used to rewrite the payload?
GitHub link: https://github.com/apache/pulsar/discussions/21827
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Can pulsar provide a broker plug-in to rewrite the payload? [pulsar]
Posted by "chenyahui (via GitHub)" <gi...@apache.org>.
GitHub user chenyahui edited a discussion: Can pulsar provide a broker plug-in to rewrite the payload?
Now Pulsar has a broker plug-in named EntryFilter, which allows users to customize filtering logic and filter data that consumers do not need on the broker side.
But this is not completely sufficient: in many scenarios, we also need to streamline the payload on the broker side. The broker directly cuts out the parts of the payload that the Consumer does not need.
For example: Payload is `{"a": 1, "b": 2, "c": 3}`, and Consumer only needs `{"a": 1}`. We can directly cut the data to `{"a": 1}` on the broker side. {"a": 1}`, and then send it to Consumer, This can reduce a lot of IO bandwidth.
Therefore, can pulsar provide a broker plug-in that can be used to rewrite the payload?
GitHub link: https://github.com/apache/pulsar/discussions/21827
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Does pulsar support custom payload modification on broker [pulsar]
Posted by "chenyahui (via GitHub)" <gi...@apache.org>.
GitHub user chenyahui edited a discussion: Does pulsar support custom payload modification on broker
For example, is there a plug-in that supports rewriting or pruning the message payload on the broker side?
If there is such a feature, we can implement data reduction on the server side to reduce the bandwidth.
GitHub link: https://github.com/apache/pulsar/discussions/21827
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Can pulsar provide a broker plugin that can be used to rewrite the payload? [pulsar]
Posted by "chenyahui (via GitHub)" <gi...@apache.org>.
GitHub user chenyahui edited a discussion: Can pulsar provide a broker plugin that can be used to rewrite the payload?
Now Pulsar has a broker plug-in named EntryFilter, which allows users to customize filtering logic and filter data that consumers do not need on the broker side.
But this is not completely sufficient: in many scenarios, we also need to streamline the payload on the broker side. The broker directly cuts out the parts of the payload that the Consumer does not need.
For example: Payload is `{"a": 1, "b": 2, "c": 3}`, and Consumer only needs `{"a": 1}`. We can directly cut the data to `{"a": 1}` on the broker side. {"a": 1}`, and then send it to Consumer, This can reduce a lot of IO bandwidth.
Therefore, can pulsar provide a broker plug-in that can be used to rewrite the payload?
GitHub link: https://github.com/apache/pulsar/discussions/21827
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] can pulsar provide a broker plug-in to rewrite the payload? [pulsar]
Posted by "chenyahui (via GitHub)" <gi...@apache.org>.
GitHub user chenyahui edited a discussion: can pulsar provide a broker plug-in to rewrite the payload?
Now Pulsar has a broker plug-in named EntryFilter, which allows users to customize filtering logic and filter data that consumers do not need on the broker side.
But this is not completely sufficient: in many scenarios, we also need to streamline the payload on the broker side. The broker directly cuts out the parts of the payload that the Consumer does not need.
For example: Payload is `{"a": 1, "b": 2, "c": 3}`, and Consumer only needs `{"a": 1}`. We can directly cut the data to `{"a": 1}` on the broker side. {"a": 1}`, and then send it to Consumer, This can reduce a lot of IO bandwidth.
Therefore, can pulsar provide a broker plug-in that can be used to rewrite the payload?
GitHub link: https://github.com/apache/pulsar/discussions/21827
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org