You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by sun yao <fo...@yahoo.com.INVALID> on 2019/01/27 03:39:44 UTC
PIP 28: Improve pulsar proxy log messages
Hi folks,
Pulsar Proxy is almost a gateway for all pulsar requests, it would be helpful if it can record more details for the traffic, like source, target, session id, response time(different stage) for each request, even for message body. I am proposing an improve for pulsar proxy, for more information : PIP: https://github.com/apache/pulsar/wiki/PIP-28%3A-Improve-pulsar-proxy-log-messages Feel free to let me know any ideas or suggestions for this feature. Thanks.
Re: PIP 28: Improve pulsar proxy log messages
Posted by sun yao <fo...@yahoo.com.INVALID>.
Hi Dave,
Noted, thanks for providing this so good name .
On Thursday, January 31, 2019, 11:40:01 PM PST, Dave Fisher <da...@comcast.net> wrote:
Hi Sun Yao,
When you started this thread writing that the proxy is almost like a gateway. You were correct. I think that what is wanted is for Proxies to usually be lightweight / highly performant, and optionally become a highly resilient Gateway.
Let’s call this new feature Gateway mode.
Regards,
Dave
Sent from my iPhone
> On Jan 31, 2019, at 8:34 PM, sun yao <fo...@yahoo.com.INVALID> wrote:
>
>
>
> Hi Matteo,
> You are right, will give it a shot by using "netty".
> Thanks again.
>
> Sam
>
> On Thursday, January 31, 2019, 6:53:03 AM PST, Matteo Merli <ma...@gmail.com> wrote:
>
> Sure, it would be good to have that as an option.
>
> It's just that it would need to be a completely different "operating
> mode" for the proxy,
> since more than the logging, is the parsing of each frame / command
> that will be
> "expensive".
>
> It should be possible to do that by controlling the handlers in the
> netty channel pipeline,
> by having different handlers based on the config.
>
>
> --
> Matteo Merli
> <ma...@gmail.com>
>
>> On Thu, Jan 31, 2019 at 6:38 AM Sijie Guo <gu...@gmail.com> wrote:
>>
>> I think if this feature is controlled under some flags/settings. It should be a good tradeoff between performance concerns and debuggability/visibility.
>>
>> As what Yao said, people can choose what level of details that they would like to see. It is similar to log4j's debug level.
>>
>> Thanks,
>> Sijie
>>
>>> On Thu, Jan 31, 2019 at 2:30 PM sun yao <fo...@yahoo.com> wrote:
>>>
>>> Hi Sijie, Matteo and Joe ,
>>>
>>> Thanks for your kind and professional opinions .
>>>
>>> For performance concern from Joe, what I think is like this, different log levels for different detail outputs and let dev decide which is good for them, and I also mentioned in PIP about we can disable this feature by parameter in some critical scenarios.
>>>
>>> For protocol implement from Matteo and Joe, we could try to implement it for we need to get from the proxy, like what it did in other proxies, proxysql and twemproxy , or any others.
>>>
>>> The point for this feature is we could have more details for our traffic, not only for performances, and also resources usage and msg details for debug and audit, like client issues, proxy issue or broker issues or ...
>>>
>>> Thanks.
>>>
>>>
>>>
>>>
>>>
>>> On Wednesday, January 30, 2019, 11:26:07 AM PST, Joe F <jo...@gmail.com> wrote:
>>>
>>>
>>> I run Pulsar proxy in production, and the same concern here.
>>>
>>> I don't think we can get any of these metrics unless we start parsing
>>> protocol, and definitely its going to make everything slower, and create
>>> additional memory and GC pressures.
>>>
>>> Joe
>>>
>>>
>>> On Wed, Jan 30, 2019 at 11:08 AM Matteo Merli <ma...@gmail.com>
>>> wrote:
>>>
>>>> Missed to comment on this :)
>>>>
>>>> One issue might arise from the fact that proxy is not actually parsing
>>>> each and every request.
>>>>
>>>> The proxy only "speaks" Pulsar protocol initial Connect/Connected
>>>> handshake, in which the proxy forwards the client credentials and
>>>> route it through the appropriate broker.
>>>>
>>>> After the initial handshake, the proxy is essentially degrading itself
>>>> into a 1-1 TCP proxy, patching 2 TCP connections through
>>>> without checking the commands anymore. That's the reason we only have
>>>> metrics around bytes/s and "operation/s" where
>>>> operation maps to a "buffer" we're getting from a socket (with no
>>>> direct relation to IP packets).
>>>>
>>>>
>>>> --
>>>> Matteo Merli
>>>> <ma...@gmail.com>
>>>>
>>>>> On Wed, Jan 30, 2019 at 9:19 AM Sijie Guo <gu...@gmail.com> wrote:
>>>>>
>>>>> Yao,
>>>>>
>>>>> Thank you for your proposal! The proposal looks good to me +1.
>>>>>
>>>>> In general, the ASF is using lazy consensus for a lot of things, like
>>>>> adopting PIPs. basically, if there is no objection coming up within a
>>>>> period (typically 1~2 days), you are good to pull the trigger and send
>>>> PRs
>>>>> :-)
>>>>>
>>>>> - Sijie
>>>>>
>>>>> On Sun, Jan 27, 2019 at 11:39 AM sun yao <foreversunyao@yahoo.com
>>>> .invalid>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> Hi folks,
>>>>>> Pulsar Proxy is almost a gateway for all pulsar requests, it
>>>> would
>>>>>> be helpful if it can record more details for the traffic, like source,
>>>>>> target, session id, response time(different stage) for each request,
>>>> even
>>>>>> for message body. I am proposing an improve for pulsar proxy, for
>>>>>> more information : PIP:
>>>>>>
>>>> https://github.com/apache/pulsar/wiki/PIP-28%3A-Improve-pulsar-proxy-log-messages
>>>>>> Feel free to let me know any ideas or suggestions for
>>>> this
>>>>>> feature. Thanks.
>>>>>>
Re: PIP 28: Improve pulsar proxy log messages
Posted by Dave Fisher <da...@comcast.net>.
Hi Sun Yao,
When you started this thread writing that the proxy is almost like a gateway. You were correct. I think that what is wanted is for Proxies to usually be lightweight / highly performant, and optionally become a highly resilient Gateway.
Let’s call this new feature Gateway mode.
Regards,
Dave
Sent from my iPhone
> On Jan 31, 2019, at 8:34 PM, sun yao <fo...@yahoo.com.INVALID> wrote:
>
>
>
> Hi Matteo,
> You are right, will give it a shot by using "netty".
> Thanks again.
>
> Sam
>
> On Thursday, January 31, 2019, 6:53:03 AM PST, Matteo Merli <ma...@gmail.com> wrote:
>
> Sure, it would be good to have that as an option.
>
> It's just that it would need to be a completely different "operating
> mode" for the proxy,
> since more than the logging, is the parsing of each frame / command
> that will be
> "expensive".
>
> It should be possible to do that by controlling the handlers in the
> netty channel pipeline,
> by having different handlers based on the config.
>
>
> --
> Matteo Merli
> <ma...@gmail.com>
>
>> On Thu, Jan 31, 2019 at 6:38 AM Sijie Guo <gu...@gmail.com> wrote:
>>
>> I think if this feature is controlled under some flags/settings. It should be a good tradeoff between performance concerns and debuggability/visibility.
>>
>> As what Yao said, people can choose what level of details that they would like to see. It is similar to log4j's debug level.
>>
>> Thanks,
>> Sijie
>>
>>> On Thu, Jan 31, 2019 at 2:30 PM sun yao <fo...@yahoo.com> wrote:
>>>
>>> Hi Sijie, Matteo and Joe ,
>>>
>>> Thanks for your kind and professional opinions .
>>>
>>> For performance concern from Joe, what I think is like this, different log levels for different detail outputs and let dev decide which is good for them, and I also mentioned in PIP about we can disable this feature by parameter in some critical scenarios.
>>>
>>> For protocol implement from Matteo and Joe, we could try to implement it for we need to get from the proxy, like what it did in other proxies, proxysql and twemproxy , or any others.
>>>
>>> The point for this feature is we could have more details for our traffic, not only for performances, and also resources usage and msg details for debug and audit, like client issues, proxy issue or broker issues or ...
>>>
>>> Thanks.
>>>
>>>
>>>
>>>
>>>
>>> On Wednesday, January 30, 2019, 11:26:07 AM PST, Joe F <jo...@gmail.com> wrote:
>>>
>>>
>>> I run Pulsar proxy in production, and the same concern here.
>>>
>>> I don't think we can get any of these metrics unless we start parsing
>>> protocol, and definitely its going to make everything slower, and create
>>> additional memory and GC pressures.
>>>
>>> Joe
>>>
>>>
>>> On Wed, Jan 30, 2019 at 11:08 AM Matteo Merli <ma...@gmail.com>
>>> wrote:
>>>
>>>> Missed to comment on this :)
>>>>
>>>> One issue might arise from the fact that proxy is not actually parsing
>>>> each and every request.
>>>>
>>>> The proxy only "speaks" Pulsar protocol initial Connect/Connected
>>>> handshake, in which the proxy forwards the client credentials and
>>>> route it through the appropriate broker.
>>>>
>>>> After the initial handshake, the proxy is essentially degrading itself
>>>> into a 1-1 TCP proxy, patching 2 TCP connections through
>>>> without checking the commands anymore. That's the reason we only have
>>>> metrics around bytes/s and "operation/s" where
>>>> operation maps to a "buffer" we're getting from a socket (with no
>>>> direct relation to IP packets).
>>>>
>>>>
>>>> --
>>>> Matteo Merli
>>>> <ma...@gmail.com>
>>>>
>>>>> On Wed, Jan 30, 2019 at 9:19 AM Sijie Guo <gu...@gmail.com> wrote:
>>>>>
>>>>> Yao,
>>>>>
>>>>> Thank you for your proposal! The proposal looks good to me +1.
>>>>>
>>>>> In general, the ASF is using lazy consensus for a lot of things, like
>>>>> adopting PIPs. basically, if there is no objection coming up within a
>>>>> period (typically 1~2 days), you are good to pull the trigger and send
>>>> PRs
>>>>> :-)
>>>>>
>>>>> - Sijie
>>>>>
>>>>> On Sun, Jan 27, 2019 at 11:39 AM sun yao <foreversunyao@yahoo.com
>>>> .invalid>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> Hi folks,
>>>>>> Pulsar Proxy is almost a gateway for all pulsar requests, it
>>>> would
>>>>>> be helpful if it can record more details for the traffic, like source,
>>>>>> target, session id, response time(different stage) for each request,
>>>> even
>>>>>> for message body. I am proposing an improve for pulsar proxy, for
>>>>>> more information : PIP:
>>>>>>
>>>> https://github.com/apache/pulsar/wiki/PIP-28%3A-Improve-pulsar-proxy-log-messages
>>>>>> Feel free to let me know any ideas or suggestions for
>>>> this
>>>>>> feature. Thanks.
>>>>>>
Re: PIP 28: Improve pulsar proxy log messages
Posted by sun yao <fo...@yahoo.com.INVALID>.
Hi Matteo,
You are right, will give it a shot by using "netty".
Thanks again.
Sam
On Thursday, January 31, 2019, 6:53:03 AM PST, Matteo Merli <ma...@gmail.com> wrote:
Sure, it would be good to have that as an option.
It's just that it would need to be a completely different "operating
mode" for the proxy,
since more than the logging, is the parsing of each frame / command
that will be
"expensive".
It should be possible to do that by controlling the handlers in the
netty channel pipeline,
by having different handlers based on the config.
--
Matteo Merli
<ma...@gmail.com>
On Thu, Jan 31, 2019 at 6:38 AM Sijie Guo <gu...@gmail.com> wrote:
>
> I think if this feature is controlled under some flags/settings. It should be a good tradeoff between performance concerns and debuggability/visibility.
>
> As what Yao said, people can choose what level of details that they would like to see. It is similar to log4j's debug level.
>
> Thanks,
> Sijie
>
> On Thu, Jan 31, 2019 at 2:30 PM sun yao <fo...@yahoo.com> wrote:
>>
>> Hi Sijie, Matteo and Joe ,
>>
>> Thanks for your kind and professional opinions .
>>
>> For performance concern from Joe, what I think is like this, different log levels for different detail outputs and let dev decide which is good for them, and I also mentioned in PIP about we can disable this feature by parameter in some critical scenarios.
>>
>> For protocol implement from Matteo and Joe, we could try to implement it for we need to get from the proxy, like what it did in other proxies, proxysql and twemproxy , or any others.
>>
>> The point for this feature is we could have more details for our traffic, not only for performances, and also resources usage and msg details for debug and audit, like client issues, proxy issue or broker issues or ...
>>
>> Thanks.
>>
>>
>>
>>
>>
>> On Wednesday, January 30, 2019, 11:26:07 AM PST, Joe F <jo...@gmail.com> wrote:
>>
>>
>> I run Pulsar proxy in production, and the same concern here.
>>
>> I don't think we can get any of these metrics unless we start parsing
>> protocol, and definitely its going to make everything slower, and create
>> additional memory and GC pressures.
>>
>> Joe
>>
>>
>> On Wed, Jan 30, 2019 at 11:08 AM Matteo Merli <ma...@gmail.com>
>> wrote:
>>
>> > Missed to comment on this :)
>> >
>> > One issue might arise from the fact that proxy is not actually parsing
>> > each and every request.
>> >
>> > The proxy only "speaks" Pulsar protocol initial Connect/Connected
>> > handshake, in which the proxy forwards the client credentials and
>> > route it through the appropriate broker.
>> >
>> > After the initial handshake, the proxy is essentially degrading itself
>> > into a 1-1 TCP proxy, patching 2 TCP connections through
>> > without checking the commands anymore. That's the reason we only have
>> > metrics around bytes/s and "operation/s" where
>> > operation maps to a "buffer" we're getting from a socket (with no
>> > direct relation to IP packets).
>> >
>> >
>> > --
>> > Matteo Merli
>> > <ma...@gmail.com>
>> >
>> > On Wed, Jan 30, 2019 at 9:19 AM Sijie Guo <gu...@gmail.com> wrote:
>> > >
>> > > Yao,
>> > >
>> > > Thank you for your proposal! The proposal looks good to me +1.
>> > >
>> > > In general, the ASF is using lazy consensus for a lot of things, like
>> > > adopting PIPs. basically, if there is no objection coming up within a
>> > > period (typically 1~2 days), you are good to pull the trigger and send
>> > PRs
>> > > :-)
>> > >
>> > > - Sijie
>> > >
>> > > On Sun, Jan 27, 2019 at 11:39 AM sun yao <foreversunyao@yahoo.com
>> > .invalid>
>> > > wrote:
>> > >
>> > > >
>> > > >
>> > > > Hi folks,
>> > > > Pulsar Proxy is almost a gateway for all pulsar requests, it
>> > would
>> > > > be helpful if it can record more details for the traffic, like source,
>> > > > target, session id, response time(different stage) for each request,
>> > even
>> > > > for message body. I am proposing an improve for pulsar proxy, for
>> > > > more information : PIP:
>> > > >
>> > https://github.com/apache/pulsar/wiki/PIP-28%3A-Improve-pulsar-proxy-log-messages
>> > > > Feel free to let me know any ideas or suggestions for
>> > this
>> > > > feature. Thanks.
>> > > >
>> >
Re: PIP 28: Improve pulsar proxy log messages
Posted by Matteo Merli <ma...@gmail.com>.
Sure, it would be good to have that as an option.
It's just that it would need to be a completely different "operating
mode" for the proxy,
since more than the logging, is the parsing of each frame / command
that will be
"expensive".
It should be possible to do that by controlling the handlers in the
netty channel pipeline,
by having different handlers based on the config.
--
Matteo Merli
<ma...@gmail.com>
On Thu, Jan 31, 2019 at 6:38 AM Sijie Guo <gu...@gmail.com> wrote:
>
> I think if this feature is controlled under some flags/settings. It should be a good tradeoff between performance concerns and debuggability/visibility.
>
> As what Yao said, people can choose what level of details that they would like to see. It is similar to log4j's debug level.
>
> Thanks,
> Sijie
>
> On Thu, Jan 31, 2019 at 2:30 PM sun yao <fo...@yahoo.com> wrote:
>>
>> Hi Sijie, Matteo and Joe ,
>>
>> Thanks for your kind and professional opinions .
>>
>> For performance concern from Joe, what I think is like this, different log levels for different detail outputs and let dev decide which is good for them, and I also mentioned in PIP about we can disable this feature by parameter in some critical scenarios.
>>
>> For protocol implement from Matteo and Joe, we could try to implement it for we need to get from the proxy, like what it did in other proxies, proxysql and twemproxy , or any others.
>>
>> The point for this feature is we could have more details for our traffic, not only for performances, and also resources usage and msg details for debug and audit, like client issues, proxy issue or broker issues or ...
>>
>> Thanks.
>>
>>
>>
>>
>>
>> On Wednesday, January 30, 2019, 11:26:07 AM PST, Joe F <jo...@gmail.com> wrote:
>>
>>
>> I run Pulsar proxy in production, and the same concern here.
>>
>> I don't think we can get any of these metrics unless we start parsing
>> protocol, and definitely its going to make everything slower, and create
>> additional memory and GC pressures.
>>
>> Joe
>>
>>
>> On Wed, Jan 30, 2019 at 11:08 AM Matteo Merli <ma...@gmail.com>
>> wrote:
>>
>> > Missed to comment on this :)
>> >
>> > One issue might arise from the fact that proxy is not actually parsing
>> > each and every request.
>> >
>> > The proxy only "speaks" Pulsar protocol initial Connect/Connected
>> > handshake, in which the proxy forwards the client credentials and
>> > route it through the appropriate broker.
>> >
>> > After the initial handshake, the proxy is essentially degrading itself
>> > into a 1-1 TCP proxy, patching 2 TCP connections through
>> > without checking the commands anymore. That's the reason we only have
>> > metrics around bytes/s and "operation/s" where
>> > operation maps to a "buffer" we're getting from a socket (with no
>> > direct relation to IP packets).
>> >
>> >
>> > --
>> > Matteo Merli
>> > <ma...@gmail.com>
>> >
>> > On Wed, Jan 30, 2019 at 9:19 AM Sijie Guo <gu...@gmail.com> wrote:
>> > >
>> > > Yao,
>> > >
>> > > Thank you for your proposal! The proposal looks good to me +1.
>> > >
>> > > In general, the ASF is using lazy consensus for a lot of things, like
>> > > adopting PIPs. basically, if there is no objection coming up within a
>> > > period (typically 1~2 days), you are good to pull the trigger and send
>> > PRs
>> > > :-)
>> > >
>> > > - Sijie
>> > >
>> > > On Sun, Jan 27, 2019 at 11:39 AM sun yao <foreversunyao@yahoo.com
>> > .invalid>
>> > > wrote:
>> > >
>> > > >
>> > > >
>> > > > Hi folks,
>> > > > Pulsar Proxy is almost a gateway for all pulsar requests, it
>> > would
>> > > > be helpful if it can record more details for the traffic, like source,
>> > > > target, session id, response time(different stage) for each request,
>> > even
>> > > > for message body. I am proposing an improve for pulsar proxy, for
>> > > > more information : PIP:
>> > > >
>> > https://github.com/apache/pulsar/wiki/PIP-28%3A-Improve-pulsar-proxy-log-messages
>> > > > Feel free to let me know any ideas or suggestions for
>> > this
>> > > > feature. Thanks.
>> > > >
>> >
Re: PIP 28: Improve pulsar proxy log messages
Posted by Sijie Guo <gu...@gmail.com>.
I think if this feature is controlled under some flags/settings. It should
be a good tradeoff between performance concerns and
debuggability/visibility.
As what Yao said, people can choose what level of details that they would
like to see. It is similar to log4j's debug level.
Thanks,
Sijie
On Thu, Jan 31, 2019 at 2:30 PM sun yao <fo...@yahoo.com> wrote:
> Hi Sijie, Matteo and Joe ,
>
> Thanks for your kind and professional opinions .
>
> For performance concern from Joe, what I think is like this, different
> log levels for different detail outputs and let dev decide which is good
> for them, and I also mentioned in PIP about we can disable this feature by
> parameter in some critical scenarios.
>
> For protocol implement from Matteo and Joe, we could try to
> implement it for we need to get from the proxy, like what it did in other
> proxies, proxysql <https://github.com/sysown/proxysql/wiki/Query-Logging>
> and twemproxy
> <https://github.com/twitter/twemproxy/blob/c5c725d2b45715e378909ebc909b0492591c4715/notes/recommendation.md> ,
> or any others.
>
> The point for this feature is we could have more details for our
> traffic, not only for performances, and also resources usage and msg
> details for debug and audit, like client issues, proxy issue or broker
> issues or ...
>
> Thanks.
>
>
>
>
>
> On Wednesday, January 30, 2019, 11:26:07 AM PST, Joe F <
> joefrancisk@gmail.com> wrote:
>
>
> I run Pulsar proxy in production, and the same concern here.
>
> I don't think we can get any of these metrics unless we start parsing
> protocol, and definitely its going to make everything slower, and create
> additional memory and GC pressures.
>
> Joe
>
>
> On Wed, Jan 30, 2019 at 11:08 AM Matteo Merli <ma...@gmail.com>
> wrote:
>
> > Missed to comment on this :)
> >
> > One issue might arise from the fact that proxy is not actually parsing
> > each and every request.
> >
> > The proxy only "speaks" Pulsar protocol initial Connect/Connected
> > handshake, in which the proxy forwards the client credentials and
> > route it through the appropriate broker.
> >
> > After the initial handshake, the proxy is essentially degrading itself
> > into a 1-1 TCP proxy, patching 2 TCP connections through
> > without checking the commands anymore. That's the reason we only have
> > metrics around bytes/s and "operation/s" where
> > operation maps to a "buffer" we're getting from a socket (with no
> > direct relation to IP packets).
> >
> >
> > --
> > Matteo Merli
> > <ma...@gmail.com>
> >
> > On Wed, Jan 30, 2019 at 9:19 AM Sijie Guo <gu...@gmail.com> wrote:
> > >
> > > Yao,
> > >
> > > Thank you for your proposal! The proposal looks good to me +1.
> > >
> > > In general, the ASF is using lazy consensus for a lot of things, like
> > > adopting PIPs. basically, if there is no objection coming up within a
> > > period (typically 1~2 days), you are good to pull the trigger and send
> > PRs
> > > :-)
> > >
> > > - Sijie
> > >
> > > On Sun, Jan 27, 2019 at 11:39 AM sun yao <foreversunyao@yahoo.com
> > .invalid>
> > > wrote:
> > >
> > > >
> > > >
> > > > Hi folks,
> > > > Pulsar Proxy is almost a gateway for all pulsar requests, it
> > would
> > > > be helpful if it can record more details for the traffic, like
> source,
> > > > target, session id, response time(different stage) for each request,
> > even
> > > > for message body. I am proposing an improve for pulsar proxy,
> for
> > > > more information : PIP:
> > > >
> >
> https://github.com/apache/pulsar/wiki/PIP-28%3A-Improve-pulsar-proxy-log-messages
> > > > Feel free to let me know any ideas or suggestions for
> > this
> > > > feature. Thanks.
> > > >
> >
>
Re: PIP 28: Improve pulsar proxy log messages
Posted by sun yao <fo...@yahoo.com.INVALID>.
Hi Sijie, Matteo and Joe ,
Thanks for your kind and professional opinions .
For performance concern from Joe, what I think is like this, different log levels for different detail outputs and let dev decide which is good for them, and I also mentioned in PIP about we can disable this feature by parameter in some critical scenarios.
For protocol implement from Matteo and Joe, we could try to implement it for we need to get from the proxy, like what it did in other proxies, proxysql and twemproxy , or any others.
The point for this feature is we could have more details for our traffic, not only for performances, and also resources usage and msg details for debug and audit, like client issues, proxy issue or broker issues or ...
Thanks.
On Wednesday, January 30, 2019, 11:26:07 AM PST, Joe F <jo...@gmail.com> wrote:
I run Pulsar proxy in production, and the same concern here.
I don't think we can get any of these metrics unless we start parsing
protocol, and definitely its going to make everything slower, and create
additional memory and GC pressures.
Joe
On Wed, Jan 30, 2019 at 11:08 AM Matteo Merli <ma...@gmail.com>
wrote:
> Missed to comment on this :)
>
> One issue might arise from the fact that proxy is not actually parsing
> each and every request.
>
> The proxy only "speaks" Pulsar protocol initial Connect/Connected
> handshake, in which the proxy forwards the client credentials and
> route it through the appropriate broker.
>
> After the initial handshake, the proxy is essentially degrading itself
> into a 1-1 TCP proxy, patching 2 TCP connections through
> without checking the commands anymore. That's the reason we only have
> metrics around bytes/s and "operation/s" where
> operation maps to a "buffer" we're getting from a socket (with no
> direct relation to IP packets).
>
>
> --
> Matteo Merli
> <ma...@gmail.com>
>
> On Wed, Jan 30, 2019 at 9:19 AM Sijie Guo <gu...@gmail.com> wrote:
> >
> > Yao,
> >
> > Thank you for your proposal! The proposal looks good to me +1.
> >
> > In general, the ASF is using lazy consensus for a lot of things, like
> > adopting PIPs. basically, if there is no objection coming up within a
> > period (typically 1~2 days), you are good to pull the trigger and send
> PRs
> > :-)
> >
> > - Sijie
> >
> > On Sun, Jan 27, 2019 at 11:39 AM sun yao <foreversunyao@yahoo.com
> .invalid>
> > wrote:
> >
> > >
> > >
> > > Hi folks,
> > > Pulsar Proxy is almost a gateway for all pulsar requests, it
> would
> > > be helpful if it can record more details for the traffic, like source,
> > > target, session id, response time(different stage) for each request,
> even
> > > for message body. I am proposing an improve for pulsar proxy, for
> > > more information : PIP:
> > >
> https://github.com/apache/pulsar/wiki/PIP-28%3A-Improve-pulsar-proxy-log-messages
> > > Feel free to let me know any ideas or suggestions for
> this
> > > feature. Thanks.
> > >
>
Re: PIP 28: Improve pulsar proxy log messages
Posted by Joe F <jo...@gmail.com>.
I run Pulsar proxy in production, and the same concern here.
I don't think we can get any of these metrics unless we start parsing
protocol, and definitely its going to make everything slower, and create
additional memory and GC pressures.
Joe
On Wed, Jan 30, 2019 at 11:08 AM Matteo Merli <ma...@gmail.com>
wrote:
> Missed to comment on this :)
>
> One issue might arise from the fact that proxy is not actually parsing
> each and every request.
>
> The proxy only "speaks" Pulsar protocol initial Connect/Connected
> handshake, in which the proxy forwards the client credentials and
> route it through the appropriate broker.
>
> After the initial handshake, the proxy is essentially degrading itself
> into a 1-1 TCP proxy, patching 2 TCP connections through
> without checking the commands anymore. That's the reason we only have
> metrics around bytes/s and "operation/s" where
> operation maps to a "buffer" we're getting from a socket (with no
> direct relation to IP packets).
>
>
> --
> Matteo Merli
> <ma...@gmail.com>
>
> On Wed, Jan 30, 2019 at 9:19 AM Sijie Guo <gu...@gmail.com> wrote:
> >
> > Yao,
> >
> > Thank you for your proposal! The proposal looks good to me +1.
> >
> > In general, the ASF is using lazy consensus for a lot of things, like
> > adopting PIPs. basically, if there is no objection coming up within a
> > period (typically 1~2 days), you are good to pull the trigger and send
> PRs
> > :-)
> >
> > - Sijie
> >
> > On Sun, Jan 27, 2019 at 11:39 AM sun yao <foreversunyao@yahoo.com
> .invalid>
> > wrote:
> >
> > >
> > >
> > > Hi folks,
> > > Pulsar Proxy is almost a gateway for all pulsar requests, it
> would
> > > be helpful if it can record more details for the traffic, like source,
> > > target, session id, response time(different stage) for each request,
> even
> > > for message body. I am proposing an improve for pulsar proxy, for
> > > more information : PIP:
> > >
> https://github.com/apache/pulsar/wiki/PIP-28%3A-Improve-pulsar-proxy-log-messages
> > > Feel free to let me know any ideas or suggestions for
> this
> > > feature. Thanks.
> > >
>
Re: PIP 28: Improve pulsar proxy log messages
Posted by Matteo Merli <ma...@gmail.com>.
Missed to comment on this :)
One issue might arise from the fact that proxy is not actually parsing
each and every request.
The proxy only "speaks" Pulsar protocol initial Connect/Connected
handshake, in which the proxy forwards the client credentials and
route it through the appropriate broker.
After the initial handshake, the proxy is essentially degrading itself
into a 1-1 TCP proxy, patching 2 TCP connections through
without checking the commands anymore. That's the reason we only have
metrics around bytes/s and "operation/s" where
operation maps to a "buffer" we're getting from a socket (with no
direct relation to IP packets).
--
Matteo Merli
<ma...@gmail.com>
On Wed, Jan 30, 2019 at 9:19 AM Sijie Guo <gu...@gmail.com> wrote:
>
> Yao,
>
> Thank you for your proposal! The proposal looks good to me +1.
>
> In general, the ASF is using lazy consensus for a lot of things, like
> adopting PIPs. basically, if there is no objection coming up within a
> period (typically 1~2 days), you are good to pull the trigger and send PRs
> :-)
>
> - Sijie
>
> On Sun, Jan 27, 2019 at 11:39 AM sun yao <fo...@yahoo.com.invalid>
> wrote:
>
> >
> >
> > Hi folks,
> > Pulsar Proxy is almost a gateway for all pulsar requests, it would
> > be helpful if it can record more details for the traffic, like source,
> > target, session id, response time(different stage) for each request, even
> > for message body. I am proposing an improve for pulsar proxy, for
> > more information : PIP:
> > https://github.com/apache/pulsar/wiki/PIP-28%3A-Improve-pulsar-proxy-log-messages
> > Feel free to let me know any ideas or suggestions for this
> > feature. Thanks.
> >
Re: PIP 28: Improve pulsar proxy log messages
Posted by Sijie Guo <gu...@gmail.com>.
Yao,
Thank you for your proposal! The proposal looks good to me +1.
In general, the ASF is using lazy consensus for a lot of things, like
adopting PIPs. basically, if there is no objection coming up within a
period (typically 1~2 days), you are good to pull the trigger and send PRs
:-)
- Sijie
On Sun, Jan 27, 2019 at 11:39 AM sun yao <fo...@yahoo.com.invalid>
wrote:
>
>
> Hi folks,
> Pulsar Proxy is almost a gateway for all pulsar requests, it would
> be helpful if it can record more details for the traffic, like source,
> target, session id, response time(different stage) for each request, even
> for message body. I am proposing an improve for pulsar proxy, for
> more information : PIP:
> https://github.com/apache/pulsar/wiki/PIP-28%3A-Improve-pulsar-proxy-log-messages
> Feel free to let me know any ideas or suggestions for this
> feature. Thanks.
>