You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Anastasia Vela <av...@confluent.io> on 2020/09/30 18:18:51 UTC
[VOTE] KIP-673: Emit JSONs with new auto-generated schema
Hi everyone,
Thanks again for the discussion. I'd like to start the vote for this KIP.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema
Thanks,
Anastasia
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Posted by Lucas Bradstreet <lu...@confluent.io>.
Hi Anastasia,
This looks like a great change that will make debugging cluster behavior
much easier.
+1 non binding.
Lucas
On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela <av...@confluent.io> wrote:
> Hi everyone,
>
> Thanks again for the discussion. I'd like to start the vote for this KIP.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema
>
> Thanks,
> Anastasia
>
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Posted by Lucas Bradstreet <lu...@confluent.io>.
Hi David,
That is a good point that we should clarify. I think we should not commit
to guaranteeing the names or the format of the particular request and
response schemas themselves, though we should guarantee that they are
parseable as JSON. The pre-existing trace logging did not guarantee this
either and it was more difficult to parse as it was not well structured. I
believe for most structured logging use cases where we are analyzing
cluster behavior this should be sufficient.
Thanks,
Lucas
On Thu, Oct 29, 2020 at 3:47 AM David Jacot <dj...@confluent.io> wrote:
> Hi folks,
>
> I have looked at Anastasia's PR to implement this KIP and I was wondering
> how far we want to go with the backward compatibility of this in the
> future.
> Now that we rely on the auto-generated protocol, the outputted requests and
> responses use the name of the fields defined in the schema. Until now, we
> have been renaming fields rather easily in the schemas as they were purely
> internal. With this KIP, renaming a field will break the structured request
> log.
>
> Does this imply that we will now consider the schemas as part of our public
> API? We don't discuss this point in the KIP so it is subject to
> interpretation.
>
> I think that we should be clear on that point and either commit to only
> making
> the request log parsable while not guaranteeing the format of the requests
> and
> the responses; or commit to making the schemas part of our public API.
>
> Personally, I lean towards the former at the moment. That is
> probably sufficient
> for the targeted use cases. Breaking changes are annoying but as this is
> intended to be used for debugging purposes, that may be OK.
>
> What do you guys think?
>
> Best,
> David
>
>
> On Thu, Oct 8, 2020 at 7:19 PM Anastasia Vela <av...@confluent.io> wrote:
>
> > Thanks everyone for the votes. In summary, the vote passed and the KIP
> was
> > accepted with 3 binding and 3 non-binding +1s.
> >
> > Best,
> > Anastasia
> >
> > On Mon, Oct 5, 2020 at 2:50 PM Jason Gustafson <ja...@confluent.io>
> wrote:
> >
> > > +1 Thanks for the KIP!
> > >
> > > On Fri, Oct 2, 2020 at 4:37 PM Colin McCabe <cm...@apache.org>
> wrote:
> > >
> > > > Thanks, Anastasia! This will be a lot easier to maintain.
> > > >
> > > > +1 (binding)
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Wed, Sep 30, 2020, at 23:57, David Jacot wrote:
> > > > > Thanks for the KIP, Anastasia.
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > On Thu, Oct 1, 2020 at 8:06 AM Tom Bentley <tb...@redhat.com>
> > > wrote:
> > > > >
> > > > > > Thanks Anastasia,
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira <gw...@confluent.io>
> > > wrote:
> > > > > >
> > > > > > > Thank you, this will be quite helpful.
> > > > > > >
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela <
> > > avela@confluent.io>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > Thanks again for the discussion. I'd like to start the vote
> for
> > > > this
> > > > > > KIP.
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Anastasia
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Gwen Shapira
> > > > > > > Engineering Manager | Confluent
> > > > > > > 650.450.2760 | @gwenshap
> > > > > > > Follow us: Twitter | blog
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Posted by David Jacot <dj...@confluent.io>.
Hi folks,
I have looked at Anastasia's PR to implement this KIP and I was wondering
how far we want to go with the backward compatibility of this in the future.
Now that we rely on the auto-generated protocol, the outputted requests and
responses use the name of the fields defined in the schema. Until now, we
have been renaming fields rather easily in the schemas as they were purely
internal. With this KIP, renaming a field will break the structured request
log.
Does this imply that we will now consider the schemas as part of our public
API? We don't discuss this point in the KIP so it is subject to
interpretation.
I think that we should be clear on that point and either commit to only
making
the request log parsable while not guaranteeing the format of the requests
and
the responses; or commit to making the schemas part of our public API.
Personally, I lean towards the former at the moment. That is
probably sufficient
for the targeted use cases. Breaking changes are annoying but as this is
intended to be used for debugging purposes, that may be OK.
What do you guys think?
Best,
David
On Thu, Oct 8, 2020 at 7:19 PM Anastasia Vela <av...@confluent.io> wrote:
> Thanks everyone for the votes. In summary, the vote passed and the KIP was
> accepted with 3 binding and 3 non-binding +1s.
>
> Best,
> Anastasia
>
> On Mon, Oct 5, 2020 at 2:50 PM Jason Gustafson <ja...@confluent.io> wrote:
>
> > +1 Thanks for the KIP!
> >
> > On Fri, Oct 2, 2020 at 4:37 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > Thanks, Anastasia! This will be a lot easier to maintain.
> > >
> > > +1 (binding)
> > >
> > > best,
> > > Colin
> > >
> > > On Wed, Sep 30, 2020, at 23:57, David Jacot wrote:
> > > > Thanks for the KIP, Anastasia.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > On Thu, Oct 1, 2020 at 8:06 AM Tom Bentley <tb...@redhat.com>
> > wrote:
> > > >
> > > > > Thanks Anastasia,
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > >
> > > > > On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira <gw...@confluent.io>
> > wrote:
> > > > >
> > > > > > Thank you, this will be quite helpful.
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela <
> > avela@confluent.io>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Thanks again for the discussion. I'd like to start the vote for
> > > this
> > > > > KIP.
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Anastasia
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Gwen Shapira
> > > > > > Engineering Manager | Confluent
> > > > > > 650.450.2760 | @gwenshap
> > > > > > Follow us: Twitter | blog
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Posted by Anastasia Vela <av...@confluent.io>.
Thanks everyone for the votes. In summary, the vote passed and the KIP was
accepted with 3 binding and 3 non-binding +1s.
Best,
Anastasia
On Mon, Oct 5, 2020 at 2:50 PM Jason Gustafson <ja...@confluent.io> wrote:
> +1 Thanks for the KIP!
>
> On Fri, Oct 2, 2020 at 4:37 PM Colin McCabe <cm...@apache.org> wrote:
>
> > Thanks, Anastasia! This will be a lot easier to maintain.
> >
> > +1 (binding)
> >
> > best,
> > Colin
> >
> > On Wed, Sep 30, 2020, at 23:57, David Jacot wrote:
> > > Thanks for the KIP, Anastasia.
> > >
> > > +1 (non-binding)
> > >
> > > On Thu, Oct 1, 2020 at 8:06 AM Tom Bentley <tb...@redhat.com>
> wrote:
> > >
> > > > Thanks Anastasia,
> > > >
> > > > +1 (non-binding)
> > > >
> > > >
> > > > On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira <gw...@confluent.io>
> wrote:
> > > >
> > > > > Thank you, this will be quite helpful.
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela <
> avela@confluent.io>
> > > > > wrote:
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Thanks again for the discussion. I'd like to start the vote for
> > this
> > > > KIP.
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema
> > > > > >
> > > > > > Thanks,
> > > > > > Anastasia
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Gwen Shapira
> > > > > Engineering Manager | Confluent
> > > > > 650.450.2760 | @gwenshap
> > > > > Follow us: Twitter | blog
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Posted by Jason Gustafson <ja...@confluent.io>.
+1 Thanks for the KIP!
On Fri, Oct 2, 2020 at 4:37 PM Colin McCabe <cm...@apache.org> wrote:
> Thanks, Anastasia! This will be a lot easier to maintain.
>
> +1 (binding)
>
> best,
> Colin
>
> On Wed, Sep 30, 2020, at 23:57, David Jacot wrote:
> > Thanks for the KIP, Anastasia.
> >
> > +1 (non-binding)
> >
> > On Thu, Oct 1, 2020 at 8:06 AM Tom Bentley <tb...@redhat.com> wrote:
> >
> > > Thanks Anastasia,
> > >
> > > +1 (non-binding)
> > >
> > >
> > > On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira <gw...@confluent.io> wrote:
> > >
> > > > Thank you, this will be quite helpful.
> > > >
> > > > +1 (binding)
> > > >
> > > > On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela <av...@confluent.io>
> > > > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > Thanks again for the discussion. I'd like to start the vote for
> this
> > > KIP.
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema
> > > > >
> > > > > Thanks,
> > > > > Anastasia
> > > >
> > > >
> > > >
> > > > --
> > > > Gwen Shapira
> > > > Engineering Manager | Confluent
> > > > 650.450.2760 | @gwenshap
> > > > Follow us: Twitter | blog
> > > >
> > > >
> > >
> >
>
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Posted by Colin McCabe <cm...@apache.org>.
Thanks, Anastasia! This will be a lot easier to maintain.
+1 (binding)
best,
Colin
On Wed, Sep 30, 2020, at 23:57, David Jacot wrote:
> Thanks for the KIP, Anastasia.
>
> +1 (non-binding)
>
> On Thu, Oct 1, 2020 at 8:06 AM Tom Bentley <tb...@redhat.com> wrote:
>
> > Thanks Anastasia,
> >
> > +1 (non-binding)
> >
> >
> > On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira <gw...@confluent.io> wrote:
> >
> > > Thank you, this will be quite helpful.
> > >
> > > +1 (binding)
> > >
> > > On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela <av...@confluent.io>
> > > wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > Thanks again for the discussion. I'd like to start the vote for this
> > KIP.
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema
> > > >
> > > > Thanks,
> > > > Anastasia
> > >
> > >
> > >
> > > --
> > > Gwen Shapira
> > > Engineering Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter | blog
> > >
> > >
> >
>
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Posted by David Jacot <dj...@confluent.io>.
Thanks for the KIP, Anastasia.
+1 (non-binding)
On Thu, Oct 1, 2020 at 8:06 AM Tom Bentley <tb...@redhat.com> wrote:
> Thanks Anastasia,
>
> +1 (non-binding)
>
>
> On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira <gw...@confluent.io> wrote:
>
> > Thank you, this will be quite helpful.
> >
> > +1 (binding)
> >
> > On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela <av...@confluent.io>
> > wrote:
> > >
> > > Hi everyone,
> > >
> > > Thanks again for the discussion. I'd like to start the vote for this
> KIP.
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema
> > >
> > > Thanks,
> > > Anastasia
> >
> >
> >
> > --
> > Gwen Shapira
> > Engineering Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
> >
>
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Posted by Tom Bentley <tb...@redhat.com>.
Thanks Anastasia,
+1 (non-binding)
On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira <gw...@confluent.io> wrote:
> Thank you, this will be quite helpful.
>
> +1 (binding)
>
> On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela <av...@confluent.io>
> wrote:
> >
> > Hi everyone,
> >
> > Thanks again for the discussion. I'd like to start the vote for this KIP.
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema
> >
> > Thanks,
> > Anastasia
>
>
>
> --
> Gwen Shapira
> Engineering Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
>
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Posted by Gwen Shapira <gw...@confluent.io>.
Thank you, this will be quite helpful.
+1 (binding)
On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela <av...@confluent.io> wrote:
>
> Hi everyone,
>
> Thanks again for the discussion. I'd like to start the vote for this KIP.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema
>
> Thanks,
> Anastasia
--
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog