You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by xiaolong ran <ra...@gmail.com> on 2020/08/06 01:53:56 UTC
PIP 69: Schema design for pulsar-client-go
Hello all:
Type safety is extremely important in any application built around a message bus like Pulsar.
Currently, Apache Pulsar supports the function of Schema Registry. And Java, CPP and
Python clients already support schema registry related functions, In order to further improve
the function of Go Client, we need to support the function of Schema Registry in Go Client.
The proposal is in:
https://gist.github.com/wolfstudy/e0c9b349cd405715a9e04dd8658510bc <https://gist.github.com/wolfstudy/e0c9b349cd405715a9e04dd8658510bc>
Looking forward to any feedback.
--
Thanks
Xiaolong Ran
Re: PIP 69: Schema design for pulsar-client-go
Posted by Matteo Merli <ma...@gmail.com>.
Thanks Xiaolong for starting this.
I think one other aspect to look into is how to fit in nicely in the
language conventions. For example, in Go, there's already a
pre-defined way to configure JSON serialization for a struct.
>> func NewJsonSchema(jsonAvroSchemaDef string, properties map[string]string) *JsonSchema {
A user, in most cases should not be required to know the AVRO
definition for its own JSON struct, rather we should extract
automatically.
Similarly, Avro library might have a way to generate a schema from a
struct definition.
Also, since the encode/decode are done in library, the way to pass the
message content is different. Right now we're taking `[]byte` only.
It would be good to have the entire example on how it would look like
end to end.
Finally, I'd say that if we can find a good solution, we could also
potentially "break" the API at this point if the need arises (Since
we're at 0.1.1 version).
--
Matteo Merli
<ma...@gmail.com>
On Thu, Aug 6, 2020 at 1:54 PM Sijie Guo <gu...@gmail.com> wrote:
>
> I think we can follow what Java client does for supporting multiple
> schemas. Does it work for you?
>
> - Sijie
>
> On Thu, Aug 6, 2020 at 1:12 PM Yuva raj <uv...@gmail.com> wrote:
>
> > Thanks for bringing this up xiaolong. This is very much needed feature.
> > Overall looks good . Adding schema in producer options will make one schema
> > bound to one producer. This works most of the time. But We produce multiple
> > different types of messages in single topic (Ex avro: userCreated ,
> > accountCreated). It would be great if we can allow providing schema
> > details when sending an message so that we don't need to create one
> > producer per type. Please let me know your thoughts on this.
> >
> > On Thu, Aug 6, 2020, 7:24 AM xiaolong ran <ra...@gmail.com>
> > wrote:
> >
> > > Hello all:
> > >
> > > Type safety is extremely important in any application built around a
> > > message bus like Pulsar.
> > > Currently, Apache Pulsar supports the function of Schema Registry. And
> > > Java, CPP and
> > > Python clients already support schema registry related functions, In
> > order
> > > to further improve
> > > the function of Go Client, we need to support the function of Schema
> > > Registry in Go Client.
> > >
> > >
> > > The proposal is in:
> > >
> > > https://gist.github.com/wolfstudy/e0c9b349cd405715a9e04dd8658510bc <
> > > https://gist.github.com/wolfstudy/e0c9b349cd405715a9e04dd8658510bc>
> > >
> > > Looking forward to any feedback.
> > >
> > >
> > > --
> > > Thanks
> > > Xiaolong Ran
> >
Re: PIP 69: Schema design for pulsar-client-go
Posted by Yuva raj <uv...@gmail.com>.
Hi Xiaolong, is there any progress made on this ? We would like to help on
this . Please let me know how we can proceed further
On Fri, 7 Aug 2020 at 08:34, Yuva raj <uv...@gmail.com> wrote:
>
> https://github.com/apache/pulsar/wiki/PIP-43%3A-producer-send-message-with-different-schema
>
> Yes the above PIP works for us Sijie.
>
> On Fri, Aug 7, 2020, 2:24 AM Sijie Guo <gu...@gmail.com> wrote:
>
>> I think we can follow what Java client does for supporting multiple
>> schemas. Does it work for you?
>>
>> - Sijie
>>
>> On Thu, Aug 6, 2020 at 1:12 PM Yuva raj <uv...@gmail.com> wrote:
>>
>> > Thanks for bringing this up xiaolong. This is very much needed feature.
>> > Overall looks good . Adding schema in producer options will make one
>> schema
>> > bound to one producer. This works most of the time. But We produce
>> multiple
>> > different types of messages in single topic (Ex avro: userCreated ,
>> > accountCreated). It would be great if we can allow providing schema
>> > details when sending an message so that we don't need to create one
>> > producer per type. Please let me know your thoughts on this.
>> >
>> > On Thu, Aug 6, 2020, 7:24 AM xiaolong ran <ra...@gmail.com>
>> > wrote:
>> >
>> > > Hello all:
>> > >
>> > > Type safety is extremely important in any application built around a
>> > > message bus like Pulsar.
>> > > Currently, Apache Pulsar supports the function of Schema Registry. And
>> > > Java, CPP and
>> > > Python clients already support schema registry related functions, In
>> > order
>> > > to further improve
>> > > the function of Go Client, we need to support the function of Schema
>> > > Registry in Go Client.
>> > >
>> > >
>> > > The proposal is in:
>> > >
>> > > https://gist.github.com/wolfstudy/e0c9b349cd405715a9e04dd8658510bc <
>> > > https://gist.github.com/wolfstudy/e0c9b349cd405715a9e04dd8658510bc>
>> > >
>> > > Looking forward to any feedback.
>> > >
>> > >
>> > > --
>> > > Thanks
>> > > Xiaolong Ran
>> >
>>
>
--
*Thanks*
*Yuvaraj L*
Re: PIP 69: Schema design for pulsar-client-go
Posted by Yuva raj <uv...@gmail.com>.
https://github.com/apache/pulsar/wiki/PIP-43%3A-producer-send-message-with-different-schema
Yes the above PIP works for us Sijie.
On Fri, Aug 7, 2020, 2:24 AM Sijie Guo <gu...@gmail.com> wrote:
> I think we can follow what Java client does for supporting multiple
> schemas. Does it work for you?
>
> - Sijie
>
> On Thu, Aug 6, 2020 at 1:12 PM Yuva raj <uv...@gmail.com> wrote:
>
> > Thanks for bringing this up xiaolong. This is very much needed feature.
> > Overall looks good . Adding schema in producer options will make one
> schema
> > bound to one producer. This works most of the time. But We produce
> multiple
> > different types of messages in single topic (Ex avro: userCreated ,
> > accountCreated). It would be great if we can allow providing schema
> > details when sending an message so that we don't need to create one
> > producer per type. Please let me know your thoughts on this.
> >
> > On Thu, Aug 6, 2020, 7:24 AM xiaolong ran <ra...@gmail.com>
> > wrote:
> >
> > > Hello all:
> > >
> > > Type safety is extremely important in any application built around a
> > > message bus like Pulsar.
> > > Currently, Apache Pulsar supports the function of Schema Registry. And
> > > Java, CPP and
> > > Python clients already support schema registry related functions, In
> > order
> > > to further improve
> > > the function of Go Client, we need to support the function of Schema
> > > Registry in Go Client.
> > >
> > >
> > > The proposal is in:
> > >
> > > https://gist.github.com/wolfstudy/e0c9b349cd405715a9e04dd8658510bc <
> > > https://gist.github.com/wolfstudy/e0c9b349cd405715a9e04dd8658510bc>
> > >
> > > Looking forward to any feedback.
> > >
> > >
> > > --
> > > Thanks
> > > Xiaolong Ran
> >
>
Re: PIP 69: Schema design for pulsar-client-go
Posted by Sijie Guo <gu...@gmail.com>.
I think we can follow what Java client does for supporting multiple
schemas. Does it work for you?
- Sijie
On Thu, Aug 6, 2020 at 1:12 PM Yuva raj <uv...@gmail.com> wrote:
> Thanks for bringing this up xiaolong. This is very much needed feature.
> Overall looks good . Adding schema in producer options will make one schema
> bound to one producer. This works most of the time. But We produce multiple
> different types of messages in single topic (Ex avro: userCreated ,
> accountCreated). It would be great if we can allow providing schema
> details when sending an message so that we don't need to create one
> producer per type. Please let me know your thoughts on this.
>
> On Thu, Aug 6, 2020, 7:24 AM xiaolong ran <ra...@gmail.com>
> wrote:
>
> > Hello all:
> >
> > Type safety is extremely important in any application built around a
> > message bus like Pulsar.
> > Currently, Apache Pulsar supports the function of Schema Registry. And
> > Java, CPP and
> > Python clients already support schema registry related functions, In
> order
> > to further improve
> > the function of Go Client, we need to support the function of Schema
> > Registry in Go Client.
> >
> >
> > The proposal is in:
> >
> > https://gist.github.com/wolfstudy/e0c9b349cd405715a9e04dd8658510bc <
> > https://gist.github.com/wolfstudy/e0c9b349cd405715a9e04dd8658510bc>
> >
> > Looking forward to any feedback.
> >
> >
> > --
> > Thanks
> > Xiaolong Ran
>
Re: PIP 69: Schema design for pulsar-client-go
Posted by Yuva raj <uv...@gmail.com>.
Thanks for bringing this up xiaolong. This is very much needed feature.
Overall looks good . Adding schema in producer options will make one schema
bound to one producer. This works most of the time. But We produce multiple
different types of messages in single topic (Ex avro: userCreated ,
accountCreated). It would be great if we can allow providing schema
details when sending an message so that we don't need to create one
producer per type. Please let me know your thoughts on this.
On Thu, Aug 6, 2020, 7:24 AM xiaolong ran <ra...@gmail.com> wrote:
> Hello all:
>
> Type safety is extremely important in any application built around a
> message bus like Pulsar.
> Currently, Apache Pulsar supports the function of Schema Registry. And
> Java, CPP and
> Python clients already support schema registry related functions, In order
> to further improve
> the function of Go Client, we need to support the function of Schema
> Registry in Go Client.
>
>
> The proposal is in:
>
> https://gist.github.com/wolfstudy/e0c9b349cd405715a9e04dd8658510bc <
> https://gist.github.com/wolfstudy/e0c9b349cd405715a9e04dd8658510bc>
>
> Looking forward to any feedback.
>
>
> --
> Thanks
> Xiaolong Ran