You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Yong Zhang <zh...@gmail.com> on 2019/05/06 10:03:41 UTC
PIP 36: Max message size
hi all,
Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t be modified
in server configuration. So there is no way when user want to modify the
limit to transfer larger size message.
Hence i propose adding a `MaxMessageSize` config in `broker.conf` to solve
this problem. Because broker server will decide how much message size will
be received so client need know how much message client can be sent.
for details track following pip:
https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a
Re: PIP 36: Max message size
Posted by Matteo Merli <ma...@gmail.com>.
> I think there should be more conditions specified in the PIP and we should have
a clear documentation on what the behavior will be and what kind of errors an
application will see. Especially we should cover how this will
interact with proxy.
Just to clarify my thoughts here, I was thinking of what happens in these cases
* Old client doesn't adapt to smaller limit from broker
* Will it keep reconnecting (in a flood, possibly) when the producer
retries on the same "big" message?
--
Matteo Merli
<ma...@gmail.com>
On Thu, May 9, 2019 at 1:32 PM Matteo Merli <ma...@gmail.com> wrote:
>
> > I think 5MB is too large and it should be reduced. :-) . I am with Kafka
> on this one.. 1MB is good enough.
>
> It would be easy to configure in such way if broker advertises 1MB and
> client complies with it.
>
> --
> Matteo Merli
> <ma...@gmail.com>
>
> On Thu, May 9, 2019 at 12:52 PM Joe F <jo...@apache.org> wrote:
> >
> > I think 5MB is too large and it should be reduced. :-) . I am with Kafka
> > on this one.. 1MB is good enough.
> >
> > Have we completely ruled out a split/join abstraction on the Pulsar
> > produce/consumer APIs? No backward compatibility, no client-server
> > capability mismatches, now that we have dedup, split/join should even be
> > resumable on an n/w interruption ..
> >
> >
> > Joe
> >
> > On Thu, May 9, 2019 at 10:43 AM Matteo Merli <ma...@gmail.com> wrote:
> >
> > > Thanks Yong for starting the work on this.
> > >
> > > I think there should be more conditions specified in the PIP and we should
> > > have
> > > a clear documentation on what the behavior will be and what kind of errors
> > > an
> > > application will see. Especially we should cover how this will
> > > interact with proxy.
> > >
> > > > On client side, client should set max message size in configuration and
> > > it should be smaller than server support.
> > >
> > > If we just let the broker/proxy decide the max size, then we can avoid
> > > the configuration
> > > in the client. That removes a lot of complexity in how to handle
> > > situation when client max-size
> > > is smaller than broker max-size and broker needs to deliver a message
> > > to that consumer.
> > >
> > > Also, it becomes much easier to make the change from ops perspective,
> > > 1 single system instead of having to
> > > configure all the client applications.
> > >
> > > --
> > > Matteo Merli
> > > <ma...@gmail.com>
> > >
> > >
> > > On Wed, May 8, 2019 at 8:34 PM Sijie Guo <gu...@gmail.com> wrote:
> > > >
> > > > Thanks Yong for the PIP.
> > > >
> > > > I moved your gist to Pulsar wiki page:
> > > > https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size
> > > >
> > > > The proposal looks good to me. +1
> > > >
> > > > - Sijie
> > > >
> > > > On Mon, May 6, 2019 at 6:04 PM Yong Zhang <zh...@gmail.com>
> > > > wrote:
> > > >
> > > > > hi all,
> > > > >
> > > > >
> > > > > Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t be
> > > modified
> > > > > in server configuration. So there is no way when user want to modify
> > > the
> > > > > limit to transfer larger size message.
> > > > >
> > > > > Hence i propose adding a `MaxMessageSize` config in `broker.conf` to
> > > solve
> > > > > this problem. Because broker server will decide how much message size
> > > will
> > > > > be received so client need know how much message client can be sent.
> > > > >
> > > > > for details track following pip:
> > > > > https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a
> > > > >
> > >
Re: PIP 36: Max message size
Posted by Matteo Merli <ma...@gmail.com>.
> I think 5MB is too large and it should be reduced. :-) . I am with Kafka
on this one.. 1MB is good enough.
It would be easy to configure in such way if broker advertises 1MB and
client complies with it.
--
Matteo Merli
<ma...@gmail.com>
On Thu, May 9, 2019 at 12:52 PM Joe F <jo...@apache.org> wrote:
>
> I think 5MB is too large and it should be reduced. :-) . I am with Kafka
> on this one.. 1MB is good enough.
>
> Have we completely ruled out a split/join abstraction on the Pulsar
> produce/consumer APIs? No backward compatibility, no client-server
> capability mismatches, now that we have dedup, split/join should even be
> resumable on an n/w interruption ..
>
>
> Joe
>
> On Thu, May 9, 2019 at 10:43 AM Matteo Merli <ma...@gmail.com> wrote:
>
> > Thanks Yong for starting the work on this.
> >
> > I think there should be more conditions specified in the PIP and we should
> > have
> > a clear documentation on what the behavior will be and what kind of errors
> > an
> > application will see. Especially we should cover how this will
> > interact with proxy.
> >
> > > On client side, client should set max message size in configuration and
> > it should be smaller than server support.
> >
> > If we just let the broker/proxy decide the max size, then we can avoid
> > the configuration
> > in the client. That removes a lot of complexity in how to handle
> > situation when client max-size
> > is smaller than broker max-size and broker needs to deliver a message
> > to that consumer.
> >
> > Also, it becomes much easier to make the change from ops perspective,
> > 1 single system instead of having to
> > configure all the client applications.
> >
> > --
> > Matteo Merli
> > <ma...@gmail.com>
> >
> >
> > On Wed, May 8, 2019 at 8:34 PM Sijie Guo <gu...@gmail.com> wrote:
> > >
> > > Thanks Yong for the PIP.
> > >
> > > I moved your gist to Pulsar wiki page:
> > > https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size
> > >
> > > The proposal looks good to me. +1
> > >
> > > - Sijie
> > >
> > > On Mon, May 6, 2019 at 6:04 PM Yong Zhang <zh...@gmail.com>
> > > wrote:
> > >
> > > > hi all,
> > > >
> > > >
> > > > Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t be
> > modified
> > > > in server configuration. So there is no way when user want to modify
> > the
> > > > limit to transfer larger size message.
> > > >
> > > > Hence i propose adding a `MaxMessageSize` config in `broker.conf` to
> > solve
> > > > this problem. Because broker server will decide how much message size
> > will
> > > > be received so client need know how much message client can be sent.
> > > >
> > > > for details track following pip:
> > > > https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a
> > > >
> >
Re: PIP 36: Max message size
Posted by Sijie Guo <gu...@gmail.com>.
On Fri, May 10, 2019 at 3:52 AM Joe F <jo...@apache.org> wrote:
> I think 5MB is too large and it should be reduced. :-) . I am with Kafka
> on this one.. 1MB is good enough.
It seems common to see a message larger than 5MB in MySQL cdc :(
>
> Have we completely ruled out a split/join abstraction on the Pulsar
> produce/consumer APIs? No backward compatibility, no client-server
> capability mismatches, now that we have dedup, split/join should even be
> resumable on an n/w interruption ..
It can be easier to achieve when Transaction is supported. A large message
can be split into a txn of smaller messages.
>
>
> Joe
>
> On Thu, May 9, 2019 at 10:43 AM Matteo Merli <ma...@gmail.com>
> wrote:
>
> > Thanks Yong for starting the work on this.
> >
> > I think there should be more conditions specified in the PIP and we
> should
> > have
> > a clear documentation on what the behavior will be and what kind of
> errors
> > an
> > application will see. Especially we should cover how this will
> > interact with proxy.
> >
> > > On client side, client should set max message size in configuration and
> > it should be smaller than server support.
> >
> > If we just let the broker/proxy decide the max size, then we can avoid
> > the configuration
> > in the client. That removes a lot of complexity in how to handle
> > situation when client max-size
> > is smaller than broker max-size and broker needs to deliver a message
> > to that consumer.
> >
> > Also, it becomes much easier to make the change from ops perspective,
> > 1 single system instead of having to
> > configure all the client applications.
> >
> > --
> > Matteo Merli
> > <ma...@gmail.com>
> >
> >
> > On Wed, May 8, 2019 at 8:34 PM Sijie Guo <gu...@gmail.com> wrote:
> > >
> > > Thanks Yong for the PIP.
> > >
> > > I moved your gist to Pulsar wiki page:
> > > https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size
> > >
> > > The proposal looks good to me. +1
> > >
> > > - Sijie
> > >
> > > On Mon, May 6, 2019 at 6:04 PM Yong Zhang <zh...@gmail.com>
> > > wrote:
> > >
> > > > hi all,
> > > >
> > > >
> > > > Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t be
> > modified
> > > > in server configuration. So there is no way when user want to modify
> > the
> > > > limit to transfer larger size message.
> > > >
> > > > Hence i propose adding a `MaxMessageSize` config in `broker.conf` to
> > solve
> > > > this problem. Because broker server will decide how much message size
> > will
> > > > be received so client need know how much message client can be sent.
> > > >
> > > > for details track following pip:
> > > > https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a
> > > >
> >
>
Re: PIP 36: Max message size
Posted by Joe F <jo...@apache.org>.
I think 5MB is too large and it should be reduced. :-) . I am with Kafka
on this one.. 1MB is good enough.
Have we completely ruled out a split/join abstraction on the Pulsar
produce/consumer APIs? No backward compatibility, no client-server
capability mismatches, now that we have dedup, split/join should even be
resumable on an n/w interruption ..
Joe
On Thu, May 9, 2019 at 10:43 AM Matteo Merli <ma...@gmail.com> wrote:
> Thanks Yong for starting the work on this.
>
> I think there should be more conditions specified in the PIP and we should
> have
> a clear documentation on what the behavior will be and what kind of errors
> an
> application will see. Especially we should cover how this will
> interact with proxy.
>
> > On client side, client should set max message size in configuration and
> it should be smaller than server support.
>
> If we just let the broker/proxy decide the max size, then we can avoid
> the configuration
> in the client. That removes a lot of complexity in how to handle
> situation when client max-size
> is smaller than broker max-size and broker needs to deliver a message
> to that consumer.
>
> Also, it becomes much easier to make the change from ops perspective,
> 1 single system instead of having to
> configure all the client applications.
>
> --
> Matteo Merli
> <ma...@gmail.com>
>
>
> On Wed, May 8, 2019 at 8:34 PM Sijie Guo <gu...@gmail.com> wrote:
> >
> > Thanks Yong for the PIP.
> >
> > I moved your gist to Pulsar wiki page:
> > https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size
> >
> > The proposal looks good to me. +1
> >
> > - Sijie
> >
> > On Mon, May 6, 2019 at 6:04 PM Yong Zhang <zh...@gmail.com>
> > wrote:
> >
> > > hi all,
> > >
> > >
> > > Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t be
> modified
> > > in server configuration. So there is no way when user want to modify
> the
> > > limit to transfer larger size message.
> > >
> > > Hence i propose adding a `MaxMessageSize` config in `broker.conf` to
> solve
> > > this problem. Because broker server will decide how much message size
> will
> > > be received so client need know how much message client can be sent.
> > >
> > > for details track following pip:
> > > https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a
> > >
>
Re: PIP 36: Max message size
Posted by Rajan Dhabalia <rd...@apache.org>.
>> Also I have a few questions regarding chunking. I will leave it in your
doc.
can you please use below link as previous doc-link was not allowing to
comment.
https://docs.google.com/document/d/13F2QLcOBVUJhVKQJVsV7yajYpfdp5pDL0qmMbTb5A3Q/edit?usp=sharing
>> Since this is a different topic than configuring max message size, does
it
make sense to move it to a separated topic?
Sure, will create a separate PIP for it.
Thanks,
Rajan
On Tue, May 14, 2019 at 3:14 PM Sijie Guo <gu...@gmail.com> wrote:
> Rajan,
>
> Since this is a different topic than configuring max message size, does it
> make sense to move it to a separated topic?
>
> Also I have a few questions regarding chunking. I will leave it in your
> doc.
>
> - Sijie
>
> On Fri, May 10, 2019 at 1:08 PM Rajan Dhabalia <rd...@apache.org>
> wrote:
>
> > Hi,
> >
> > >> It can be easier to achieve when Transaction is supported. A large
> > message
> > can be split into a txn of smaller messages.
> >
> > I would like to propose another approach "Chunking and combine" to
> publish
> > large size of messages. Transaction deployment might not be convenient
> all
> > the time specially for the large traffic pulsar system due to multiple
> > reasons which I have shared in the doc.
> >
> >
> https://docs.google.com/document/d/13F2QLcOBVUJhVKQJVsV7yajYpfdp5pDL0qmMbTb5A3Q/edit?usp=sharing
> > So, we would like to propose above approach to address publishing of
> large
> > message size.
> >
> > Thanks,
> > Rajan
> >
> > On Thu, May 9, 2019 at 7:38 PM Yong Zhang <zh...@gmail.com>
> > wrote:
> >
> > > >
> > > > If we just let the broker/proxy decide the max size, then we can
> avoid
> > > > the configuration
> > > > in the client. That removes a lot of complexity in how to handle
> > > > situation when client max-size
> > > > is smaller than broker max-size and broker needs to deliver a message
> > > > to that consumer.
> > > >
> > >
> > > Thank you. Actually implement as this way. I will fix the description
> in
> > > pip.
> > > On Fri, 10 May 2019 at 01:43, Matteo Merli <ma...@gmail.com>
> > wrote:
> > >
> > > > Thanks Yong for starting the work on this.
> > > >
> > > > I think there should be more conditions specified in the PIP and we
> > > should
> > > > have
> > > > a clear documentation on what the behavior will be and what kind of
> > > errors
> > > > an
> > > > application will see. Especially we should cover how this will
> > > > interact with proxy.
> > > >
> > > > > On client side, client should set max message size in configuration
> > and
> > > > it should be smaller than server support.
> > > >
> > > > If we just let the broker/proxy decide the max size, then we can
> avoid
> > > > the configuration
> > > > in the client. That removes a lot of complexity in how to handle
> > > > situation when client max-size
> > > > is smaller than broker max-size and broker needs to deliver a message
> > > > to that consumer.
> > > >
> > > > Also, it becomes much easier to make the change from ops perspective,
> > > > 1 single system instead of having to
> > > > configure all the client applications.
> > > >
> > > > --
> > > > Matteo Merli
> > > > <ma...@gmail.com>
> > > >
> > > >
> > > > On Wed, May 8, 2019 at 8:34 PM Sijie Guo <gu...@gmail.com> wrote:
> > > > >
> > > > > Thanks Yong for the PIP.
> > > > >
> > > > > I moved your gist to Pulsar wiki page:
> > > > > https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size
> > > > >
> > > > > The proposal looks good to me. +1
> > > > >
> > > > > - Sijie
> > > > >
> > > > > On Mon, May 6, 2019 at 6:04 PM Yong Zhang <
> > zhangyong1025.zy@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > hi all,
> > > > > >
> > > > > >
> > > > > > Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t be
> > > > modified
> > > > > > in server configuration. So there is no way when user want to
> > modify
> > > > the
> > > > > > limit to transfer larger size message.
> > > > > >
> > > > > > Hence i propose adding a `MaxMessageSize` config in `broker.conf`
> > to
> > > > solve
> > > > > > this problem. Because broker server will decide how much message
> > size
> > > > will
> > > > > > be received so client need know how much message client can be
> > sent.
> > > > > >
> > > > > > for details track following pip:
> > > > > > https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a
> > > > > >
> > > >
> > >
> >
>
Re: PIP 36: Max message size
Posted by Matteo Merli <ma...@gmail.com>.
Once we have automatic break down and reassembly of bigger messages,
then the client will be able to use the "discovered" max message size
in the same way a TCP client uses the MSS.
--
Matteo Merli
<ma...@gmail.com>
On Tue, May 14, 2019 at 5:08 PM Joe Francis
<jo...@verizonmedia.com.invalid> wrote:
>
> They are not unrelated, because msg size automatically server double duty
> as a resource unit now, for all buffer allocations, and decoupling the two
> is good.
> It's perfectly reasonable to have one resource unit and be able to support
> small and large messages, instead of having to run separate instances tuned
> for each size
>
> On Tue, May 14, 2019 at 3:51 PM Matteo Merli <ma...@gmail.com> wrote:
>
> > On Tue, May 14, 2019 at 3:14 PM Sijie Guo <gu...@gmail.com> wrote:
> >
> > > Rajan,
> > >
> > > Since this is a different topic than configuring max message size, does
> > it
> > > make sense to move it to a separated topic?
> >
> >
> >
> > +1
> >
> > having the max message size configurable is a good thing anyways. Admins
> > can decide what they want to support.
> >
> > How to handle “huge” messages is a different discussion.
> >
> >
> > >
> > > Also I have a few questions regarding chunking. I will leave it in your
> > > doc.
> > >
> > > - Sijie
> > >
> > > On Fri, May 10, 2019 at 1:08 PM Rajan Dhabalia <rd...@apache.org>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > >> It can be easier to achieve when Transaction is supported. A large
> > > > message
> > > > can be split into a txn of smaller messages.
> > > >
> > > > I would like to propose another approach "Chunking and combine" to
> > > publish
> > > > large size of messages. Transaction deployment might not be convenient
> > > all
> > > > the time specially for the large traffic pulsar system due to multiple
> > > > reasons which I have shared in the doc.
> > > >
> > > >
> > >
> > https://docs.google.com/document/d/13F2QLcOBVUJhVKQJVsV7yajYpfdp5pDL0qmMbTb5A3Q/edit?usp=sharing
> > > > So, we would like to propose above approach to address publishing of
> > > large
> > > > message size.
> > > >
> > > > Thanks,
> > > > Rajan
> > > >
> > > > On Thu, May 9, 2019 at 7:38 PM Yong Zhang <zh...@gmail.com>
> > > > wrote:
> > > >
> > > > > >
> > > > > > If we just let the broker/proxy decide the max size, then we can
> > > avoid
> > > > > > the configuration
> > > > > > in the client. That removes a lot of complexity in how to handle
> > > > > > situation when client max-size
> > > > > > is smaller than broker max-size and broker needs to deliver a
> > message
> > > > > > to that consumer.
> > > > > >
> > > > >
> > > > > Thank you. Actually implement as this way. I will fix the description
> > > in
> > > > > pip.
> > > > > On Fri, 10 May 2019 at 01:43, Matteo Merli <ma...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Thanks Yong for starting the work on this.
> > > > > >
> > > > > > I think there should be more conditions specified in the PIP and we
> > > > > should
> > > > > > have
> > > > > > a clear documentation on what the behavior will be and what kind of
> > > > > errors
> > > > > > an
> > > > > > application will see. Especially we should cover how this will
> > > > > > interact with proxy.
> > > > > >
> > > > > > > On client side, client should set max message size in
> > configuration
> > > > and
> > > > > > it should be smaller than server support.
> > > > > >
> > > > > > If we just let the broker/proxy decide the max size, then we can
> > > avoid
> > > > > > the configuration
> > > > > > in the client. That removes a lot of complexity in how to handle
> > > > > > situation when client max-size
> > > > > > is smaller than broker max-size and broker needs to deliver a
> > message
> > > > > > to that consumer.
> > > > > >
> > > > > > Also, it becomes much easier to make the change from ops
> > perspective,
> > > > > > 1 single system instead of having to
> > > > > > configure all the client applications.
> > > > > >
> > > > > > --
> > > > > > Matteo Merli
> > > > > > <ma...@gmail.com>
> > > > > >
> > > > > >
> > > > > > On Wed, May 8, 2019 at 8:34 PM Sijie Guo <gu...@gmail.com>
> > wrote:
> > > > > > >
> > > > > > > Thanks Yong for the PIP.
> > > > > > >
> > > > > > > I moved your gist to Pulsar wiki page:
> > > > > > > https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size
> > > > > > >
> > > > > > > The proposal looks good to me. +1
> > > > > > >
> > > > > > > - Sijie
> > > > > > >
> > > > > > > On Mon, May 6, 2019 at 6:04 PM Yong Zhang <
> > > > zhangyong1025.zy@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > hi all,
> > > > > > > >
> > > > > > > >
> > > > > > > > Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t
> > be
> > > > > > modified
> > > > > > > > in server configuration. So there is no way when user want to
> > > > modify
> > > > > > the
> > > > > > > > limit to transfer larger size message.
> > > > > > > >
> > > > > > > > Hence i propose adding a `MaxMessageSize` config in
> > `broker.conf`
> > > > to
> > > > > > solve
> > > > > > > > this problem. Because broker server will decide how much
> > message
> > > > size
> > > > > > will
> > > > > > > > be received so client need know how much message client can be
> > > > sent.
> > > > > > > >
> > > > > > > > for details track following pip:
> > > > > > > > https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > --
> > --
> > Matteo Merli
> > <ma...@gmail.com>
> >
Re: PIP 36: Max message size
Posted by Joe Francis <jo...@verizonmedia.com.INVALID>.
They are not unrelated, because msg size automatically server double duty
as a resource unit now, for all buffer allocations, and decoupling the two
is good.
It's perfectly reasonable to have one resource unit and be able to support
small and large messages, instead of having to run separate instances tuned
for each size
On Tue, May 14, 2019 at 3:51 PM Matteo Merli <ma...@gmail.com> wrote:
> On Tue, May 14, 2019 at 3:14 PM Sijie Guo <gu...@gmail.com> wrote:
>
> > Rajan,
> >
> > Since this is a different topic than configuring max message size, does
> it
> > make sense to move it to a separated topic?
>
>
>
> +1
>
> having the max message size configurable is a good thing anyways. Admins
> can decide what they want to support.
>
> How to handle “huge” messages is a different discussion.
>
>
> >
> > Also I have a few questions regarding chunking. I will leave it in your
> > doc.
> >
> > - Sijie
> >
> > On Fri, May 10, 2019 at 1:08 PM Rajan Dhabalia <rd...@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > >> It can be easier to achieve when Transaction is supported. A large
> > > message
> > > can be split into a txn of smaller messages.
> > >
> > > I would like to propose another approach "Chunking and combine" to
> > publish
> > > large size of messages. Transaction deployment might not be convenient
> > all
> > > the time specially for the large traffic pulsar system due to multiple
> > > reasons which I have shared in the doc.
> > >
> > >
> >
> https://docs.google.com/document/d/13F2QLcOBVUJhVKQJVsV7yajYpfdp5pDL0qmMbTb5A3Q/edit?usp=sharing
> > > So, we would like to propose above approach to address publishing of
> > large
> > > message size.
> > >
> > > Thanks,
> > > Rajan
> > >
> > > On Thu, May 9, 2019 at 7:38 PM Yong Zhang <zh...@gmail.com>
> > > wrote:
> > >
> > > > >
> > > > > If we just let the broker/proxy decide the max size, then we can
> > avoid
> > > > > the configuration
> > > > > in the client. That removes a lot of complexity in how to handle
> > > > > situation when client max-size
> > > > > is smaller than broker max-size and broker needs to deliver a
> message
> > > > > to that consumer.
> > > > >
> > > >
> > > > Thank you. Actually implement as this way. I will fix the description
> > in
> > > > pip.
> > > > On Fri, 10 May 2019 at 01:43, Matteo Merli <ma...@gmail.com>
> > > wrote:
> > > >
> > > > > Thanks Yong for starting the work on this.
> > > > >
> > > > > I think there should be more conditions specified in the PIP and we
> > > > should
> > > > > have
> > > > > a clear documentation on what the behavior will be and what kind of
> > > > errors
> > > > > an
> > > > > application will see. Especially we should cover how this will
> > > > > interact with proxy.
> > > > >
> > > > > > On client side, client should set max message size in
> configuration
> > > and
> > > > > it should be smaller than server support.
> > > > >
> > > > > If we just let the broker/proxy decide the max size, then we can
> > avoid
> > > > > the configuration
> > > > > in the client. That removes a lot of complexity in how to handle
> > > > > situation when client max-size
> > > > > is smaller than broker max-size and broker needs to deliver a
> message
> > > > > to that consumer.
> > > > >
> > > > > Also, it becomes much easier to make the change from ops
> perspective,
> > > > > 1 single system instead of having to
> > > > > configure all the client applications.
> > > > >
> > > > > --
> > > > > Matteo Merli
> > > > > <ma...@gmail.com>
> > > > >
> > > > >
> > > > > On Wed, May 8, 2019 at 8:34 PM Sijie Guo <gu...@gmail.com>
> wrote:
> > > > > >
> > > > > > Thanks Yong for the PIP.
> > > > > >
> > > > > > I moved your gist to Pulsar wiki page:
> > > > > > https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size
> > > > > >
> > > > > > The proposal looks good to me. +1
> > > > > >
> > > > > > - Sijie
> > > > > >
> > > > > > On Mon, May 6, 2019 at 6:04 PM Yong Zhang <
> > > zhangyong1025.zy@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > hi all,
> > > > > > >
> > > > > > >
> > > > > > > Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t
> be
> > > > > modified
> > > > > > > in server configuration. So there is no way when user want to
> > > modify
> > > > > the
> > > > > > > limit to transfer larger size message.
> > > > > > >
> > > > > > > Hence i propose adding a `MaxMessageSize` config in
> `broker.conf`
> > > to
> > > > > solve
> > > > > > > this problem. Because broker server will decide how much
> message
> > > size
> > > > > will
> > > > > > > be received so client need know how much message client can be
> > > sent.
> > > > > > >
> > > > > > > for details track following pip:
> > > > > > > https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a
> > > > > > >
> > > > >
> > > >
> > >
> >
> --
> --
> Matteo Merli
> <ma...@gmail.com>
>
Re: PIP 36: Max message size
Posted by Matteo Merli <ma...@gmail.com>.
On Tue, May 14, 2019 at 3:14 PM Sijie Guo <gu...@gmail.com> wrote:
> Rajan,
>
> Since this is a different topic than configuring max message size, does it
> make sense to move it to a separated topic?
+1
having the max message size configurable is a good thing anyways. Admins
can decide what they want to support.
How to handle “huge” messages is a different discussion.
>
> Also I have a few questions regarding chunking. I will leave it in your
> doc.
>
> - Sijie
>
> On Fri, May 10, 2019 at 1:08 PM Rajan Dhabalia <rd...@apache.org>
> wrote:
>
> > Hi,
> >
> > >> It can be easier to achieve when Transaction is supported. A large
> > message
> > can be split into a txn of smaller messages.
> >
> > I would like to propose another approach "Chunking and combine" to
> publish
> > large size of messages. Transaction deployment might not be convenient
> all
> > the time specially for the large traffic pulsar system due to multiple
> > reasons which I have shared in the doc.
> >
> >
> https://docs.google.com/document/d/13F2QLcOBVUJhVKQJVsV7yajYpfdp5pDL0qmMbTb5A3Q/edit?usp=sharing
> > So, we would like to propose above approach to address publishing of
> large
> > message size.
> >
> > Thanks,
> > Rajan
> >
> > On Thu, May 9, 2019 at 7:38 PM Yong Zhang <zh...@gmail.com>
> > wrote:
> >
> > > >
> > > > If we just let the broker/proxy decide the max size, then we can
> avoid
> > > > the configuration
> > > > in the client. That removes a lot of complexity in how to handle
> > > > situation when client max-size
> > > > is smaller than broker max-size and broker needs to deliver a message
> > > > to that consumer.
> > > >
> > >
> > > Thank you. Actually implement as this way. I will fix the description
> in
> > > pip.
> > > On Fri, 10 May 2019 at 01:43, Matteo Merli <ma...@gmail.com>
> > wrote:
> > >
> > > > Thanks Yong for starting the work on this.
> > > >
> > > > I think there should be more conditions specified in the PIP and we
> > > should
> > > > have
> > > > a clear documentation on what the behavior will be and what kind of
> > > errors
> > > > an
> > > > application will see. Especially we should cover how this will
> > > > interact with proxy.
> > > >
> > > > > On client side, client should set max message size in configuration
> > and
> > > > it should be smaller than server support.
> > > >
> > > > If we just let the broker/proxy decide the max size, then we can
> avoid
> > > > the configuration
> > > > in the client. That removes a lot of complexity in how to handle
> > > > situation when client max-size
> > > > is smaller than broker max-size and broker needs to deliver a message
> > > > to that consumer.
> > > >
> > > > Also, it becomes much easier to make the change from ops perspective,
> > > > 1 single system instead of having to
> > > > configure all the client applications.
> > > >
> > > > --
> > > > Matteo Merli
> > > > <ma...@gmail.com>
> > > >
> > > >
> > > > On Wed, May 8, 2019 at 8:34 PM Sijie Guo <gu...@gmail.com> wrote:
> > > > >
> > > > > Thanks Yong for the PIP.
> > > > >
> > > > > I moved your gist to Pulsar wiki page:
> > > > > https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size
> > > > >
> > > > > The proposal looks good to me. +1
> > > > >
> > > > > - Sijie
> > > > >
> > > > > On Mon, May 6, 2019 at 6:04 PM Yong Zhang <
> > zhangyong1025.zy@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > hi all,
> > > > > >
> > > > > >
> > > > > > Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t be
> > > > modified
> > > > > > in server configuration. So there is no way when user want to
> > modify
> > > > the
> > > > > > limit to transfer larger size message.
> > > > > >
> > > > > > Hence i propose adding a `MaxMessageSize` config in `broker.conf`
> > to
> > > > solve
> > > > > > this problem. Because broker server will decide how much message
> > size
> > > > will
> > > > > > be received so client need know how much message client can be
> > sent.
> > > > > >
> > > > > > for details track following pip:
> > > > > > https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a
> > > > > >
> > > >
> > >
> >
>
--
--
Matteo Merli
<ma...@gmail.com>
Re: PIP 36: Max message size
Posted by Sijie Guo <gu...@gmail.com>.
Rajan,
Since this is a different topic than configuring max message size, does it
make sense to move it to a separated topic?
Also I have a few questions regarding chunking. I will leave it in your
doc.
- Sijie
On Fri, May 10, 2019 at 1:08 PM Rajan Dhabalia <rd...@apache.org> wrote:
> Hi,
>
> >> It can be easier to achieve when Transaction is supported. A large
> message
> can be split into a txn of smaller messages.
>
> I would like to propose another approach "Chunking and combine" to publish
> large size of messages. Transaction deployment might not be convenient all
> the time specially for the large traffic pulsar system due to multiple
> reasons which I have shared in the doc.
>
> https://docs.google.com/document/d/13F2QLcOBVUJhVKQJVsV7yajYpfdp5pDL0qmMbTb5A3Q/edit?usp=sharing
> So, we would like to propose above approach to address publishing of large
> message size.
>
> Thanks,
> Rajan
>
> On Thu, May 9, 2019 at 7:38 PM Yong Zhang <zh...@gmail.com>
> wrote:
>
> > >
> > > If we just let the broker/proxy decide the max size, then we can avoid
> > > the configuration
> > > in the client. That removes a lot of complexity in how to handle
> > > situation when client max-size
> > > is smaller than broker max-size and broker needs to deliver a message
> > > to that consumer.
> > >
> >
> > Thank you. Actually implement as this way. I will fix the description in
> > pip.
> > On Fri, 10 May 2019 at 01:43, Matteo Merli <ma...@gmail.com>
> wrote:
> >
> > > Thanks Yong for starting the work on this.
> > >
> > > I think there should be more conditions specified in the PIP and we
> > should
> > > have
> > > a clear documentation on what the behavior will be and what kind of
> > errors
> > > an
> > > application will see. Especially we should cover how this will
> > > interact with proxy.
> > >
> > > > On client side, client should set max message size in configuration
> and
> > > it should be smaller than server support.
> > >
> > > If we just let the broker/proxy decide the max size, then we can avoid
> > > the configuration
> > > in the client. That removes a lot of complexity in how to handle
> > > situation when client max-size
> > > is smaller than broker max-size and broker needs to deliver a message
> > > to that consumer.
> > >
> > > Also, it becomes much easier to make the change from ops perspective,
> > > 1 single system instead of having to
> > > configure all the client applications.
> > >
> > > --
> > > Matteo Merli
> > > <ma...@gmail.com>
> > >
> > >
> > > On Wed, May 8, 2019 at 8:34 PM Sijie Guo <gu...@gmail.com> wrote:
> > > >
> > > > Thanks Yong for the PIP.
> > > >
> > > > I moved your gist to Pulsar wiki page:
> > > > https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size
> > > >
> > > > The proposal looks good to me. +1
> > > >
> > > > - Sijie
> > > >
> > > > On Mon, May 6, 2019 at 6:04 PM Yong Zhang <
> zhangyong1025.zy@gmail.com>
> > > > wrote:
> > > >
> > > > > hi all,
> > > > >
> > > > >
> > > > > Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t be
> > > modified
> > > > > in server configuration. So there is no way when user want to
> modify
> > > the
> > > > > limit to transfer larger size message.
> > > > >
> > > > > Hence i propose adding a `MaxMessageSize` config in `broker.conf`
> to
> > > solve
> > > > > this problem. Because broker server will decide how much message
> size
> > > will
> > > > > be received so client need know how much message client can be
> sent.
> > > > >
> > > > > for details track following pip:
> > > > > https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a
> > > > >
> > >
> >
>
Re: PIP 36: Max message size
Posted by Rajan Dhabalia <rd...@apache.org>.
Hi,
>> It can be easier to achieve when Transaction is supported. A large
message
can be split into a txn of smaller messages.
I would like to propose another approach "Chunking and combine" to publish
large size of messages. Transaction deployment might not be convenient all
the time specially for the large traffic pulsar system due to multiple
reasons which I have shared in the doc.
https://docs.google.com/document/d/13F2QLcOBVUJhVKQJVsV7yajYpfdp5pDL0qmMbTb5A3Q/edit?usp=sharing
So, we would like to propose above approach to address publishing of large
message size.
Thanks,
Rajan
On Thu, May 9, 2019 at 7:38 PM Yong Zhang <zh...@gmail.com>
wrote:
> >
> > If we just let the broker/proxy decide the max size, then we can avoid
> > the configuration
> > in the client. That removes a lot of complexity in how to handle
> > situation when client max-size
> > is smaller than broker max-size and broker needs to deliver a message
> > to that consumer.
> >
>
> Thank you. Actually implement as this way. I will fix the description in
> pip.
> On Fri, 10 May 2019 at 01:43, Matteo Merli <ma...@gmail.com> wrote:
>
> > Thanks Yong for starting the work on this.
> >
> > I think there should be more conditions specified in the PIP and we
> should
> > have
> > a clear documentation on what the behavior will be and what kind of
> errors
> > an
> > application will see. Especially we should cover how this will
> > interact with proxy.
> >
> > > On client side, client should set max message size in configuration and
> > it should be smaller than server support.
> >
> > If we just let the broker/proxy decide the max size, then we can avoid
> > the configuration
> > in the client. That removes a lot of complexity in how to handle
> > situation when client max-size
> > is smaller than broker max-size and broker needs to deliver a message
> > to that consumer.
> >
> > Also, it becomes much easier to make the change from ops perspective,
> > 1 single system instead of having to
> > configure all the client applications.
> >
> > --
> > Matteo Merli
> > <ma...@gmail.com>
> >
> >
> > On Wed, May 8, 2019 at 8:34 PM Sijie Guo <gu...@gmail.com> wrote:
> > >
> > > Thanks Yong for the PIP.
> > >
> > > I moved your gist to Pulsar wiki page:
> > > https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size
> > >
> > > The proposal looks good to me. +1
> > >
> > > - Sijie
> > >
> > > On Mon, May 6, 2019 at 6:04 PM Yong Zhang <zh...@gmail.com>
> > > wrote:
> > >
> > > > hi all,
> > > >
> > > >
> > > > Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t be
> > modified
> > > > in server configuration. So there is no way when user want to modify
> > the
> > > > limit to transfer larger size message.
> > > >
> > > > Hence i propose adding a `MaxMessageSize` config in `broker.conf` to
> > solve
> > > > this problem. Because broker server will decide how much message size
> > will
> > > > be received so client need know how much message client can be sent.
> > > >
> > > > for details track following pip:
> > > > https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a
> > > >
> >
>
Re: PIP 36: Max message size
Posted by Yong Zhang <zh...@gmail.com>.
>
> If we just let the broker/proxy decide the max size, then we can avoid
> the configuration
> in the client. That removes a lot of complexity in how to handle
> situation when client max-size
> is smaller than broker max-size and broker needs to deliver a message
> to that consumer.
>
Thank you. Actually implement as this way. I will fix the description in
pip.
On Fri, 10 May 2019 at 01:43, Matteo Merli <ma...@gmail.com> wrote:
> Thanks Yong for starting the work on this.
>
> I think there should be more conditions specified in the PIP and we should
> have
> a clear documentation on what the behavior will be and what kind of errors
> an
> application will see. Especially we should cover how this will
> interact with proxy.
>
> > On client side, client should set max message size in configuration and
> it should be smaller than server support.
>
> If we just let the broker/proxy decide the max size, then we can avoid
> the configuration
> in the client. That removes a lot of complexity in how to handle
> situation when client max-size
> is smaller than broker max-size and broker needs to deliver a message
> to that consumer.
>
> Also, it becomes much easier to make the change from ops perspective,
> 1 single system instead of having to
> configure all the client applications.
>
> --
> Matteo Merli
> <ma...@gmail.com>
>
>
> On Wed, May 8, 2019 at 8:34 PM Sijie Guo <gu...@gmail.com> wrote:
> >
> > Thanks Yong for the PIP.
> >
> > I moved your gist to Pulsar wiki page:
> > https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size
> >
> > The proposal looks good to me. +1
> >
> > - Sijie
> >
> > On Mon, May 6, 2019 at 6:04 PM Yong Zhang <zh...@gmail.com>
> > wrote:
> >
> > > hi all,
> > >
> > >
> > > Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t be
> modified
> > > in server configuration. So there is no way when user want to modify
> the
> > > limit to transfer larger size message.
> > >
> > > Hence i propose adding a `MaxMessageSize` config in `broker.conf` to
> solve
> > > this problem. Because broker server will decide how much message size
> will
> > > be received so client need know how much message client can be sent.
> > >
> > > for details track following pip:
> > > https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a
> > >
>
Re: PIP 36: Max message size
Posted by Matteo Merli <ma...@gmail.com>.
Thanks Yong for starting the work on this.
I think there should be more conditions specified in the PIP and we should have
a clear documentation on what the behavior will be and what kind of errors an
application will see. Especially we should cover how this will
interact with proxy.
> On client side, client should set max message size in configuration and it should be smaller than server support.
If we just let the broker/proxy decide the max size, then we can avoid
the configuration
in the client. That removes a lot of complexity in how to handle
situation when client max-size
is smaller than broker max-size and broker needs to deliver a message
to that consumer.
Also, it becomes much easier to make the change from ops perspective,
1 single system instead of having to
configure all the client applications.
--
Matteo Merli
<ma...@gmail.com>
On Wed, May 8, 2019 at 8:34 PM Sijie Guo <gu...@gmail.com> wrote:
>
> Thanks Yong for the PIP.
>
> I moved your gist to Pulsar wiki page:
> https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size
>
> The proposal looks good to me. +1
>
> - Sijie
>
> On Mon, May 6, 2019 at 6:04 PM Yong Zhang <zh...@gmail.com>
> wrote:
>
> > hi all,
> >
> >
> > Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t be modified
> > in server configuration. So there is no way when user want to modify the
> > limit to transfer larger size message.
> >
> > Hence i propose adding a `MaxMessageSize` config in `broker.conf` to solve
> > this problem. Because broker server will decide how much message size will
> > be received so client need know how much message client can be sent.
> >
> > for details track following pip:
> > https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a
> >
Re: PIP 36: Max message size
Posted by Sijie Guo <gu...@gmail.com>.
Thanks Yong for the PIP.
I moved your gist to Pulsar wiki page:
https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size
The proposal looks good to me. +1
- Sijie
On Mon, May 6, 2019 at 6:04 PM Yong Zhang <zh...@gmail.com>
wrote:
> hi all,
>
>
> Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t be modified
> in server configuration. So there is no way when user want to modify the
> limit to transfer larger size message.
>
> Hence i propose adding a `MaxMessageSize` config in `broker.conf` to solve
> this problem. Because broker server will decide how much message size will
> be received so client need know how much message client can be sent.
>
> for details track following pip:
> https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a
>