You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Yuriy Badalyantc <lm...@gmail.com> on 2020/08/05 06:24:49 UTC
Re: [DISCUSS] KIP-616: Rename implicit Serdes instances in kafka-streams-scala
Hi John!
I understand your point. But, I'm still not sure that having a Unit serde
is a good idea because it could be ambiguous for the user. In a KStream
[Long, Unit] valid value could mean one the following:
1. Empty byte array
2. null
3. Any value (value ignored)
As a user of a library, I think all these options are more or less valid.
That's because the Unit type in my opinion has no clear representation in
terms of values and deserialization.
About Null serde - I'm ok with this. But, it should be a low priority
implicit to not intersect will other implicits. I will add it.
-Yuriy
On Sat, Jul 11, 2020 at 9:18 PM John Roesler <vv...@apache.org> wrote:
> Thanks, Yuriy,
>
> IIRC, I added VoidSerde because there are times when the key or value is
> always null, but since ‘null’ has no type in Java, we had used the
> ByteArraySerde or some other proxy. This is confusing, because then the
> type is ‘byte[]’ although we intended it to always be null. It also delays
> finding bugs because serdes themselves force runtime type checks, but the
> ByteArraySerde accepts all data.
>
> Anyway, Void isn’t the type of null, but it’s close enough. Although
> Scala’s equivalent to “void” method is Unit methods, and like Void, Unit is
> uninstantiable, Unit has a value, also called Unit or “()”, whereas Void
> has no value in Java. But because Void is still a reference type, it can be
> null (I.e., it can only be null), so I’m some sense, it’s closer to the
> Scala type Null.
>
> I guess Scala users would benefit from both a NullSerde and a UnitSerde,
> but not a VoidSerde.
>
> Thanks for bringing it up, Matthias.
>
> -John
>
> On Fri, Jul 10, 2020, at 21:49, Yuriy Badalyantc wrote:
> > Ok, I mentioned adding missing serdes in the Proposed Change paragraph.
> >
> > About VoidSerde. I didn't add it intentionally. The semantic of the Unit
> > (scala's void) type is not clear in terms of the data. If kafka topic
> > contains messages of type Unit, what does it actually means? That there
> is
> > always null? Well, for that we have a Null type. That it's an empty byte
> > array? For that, we have an Array[Byte]. Empty string? No, it's a String.
> > So, I decided to not include Unit serde in the built-in Serdes. And if a
> > user will want to use the Unit type he can implement its own serde.
> >
> > -Yuriy
> >
> > On Fri, Jul 10, 2020 at 11:21 PM Matthias J. Sax <mj...@apache.org>
> wrote:
> >
> > > Thanks Yuriy!
> > >
> > > What about `VoidSerde` ? It's not listed.
> > >
> > > It might also be nice to add a short sentence and state that in
> addition
> > > to fixing the name collisions, the KIP will also close the gap of
> > > out-of-the-box serdes and add missing Serdes that are offered in Java
> to
> > > Scala.
> > >
> > >
> > > -Matthias
> > >
> > > On 7/10/20 7:51 AM, Yuriy Badalyantc wrote:
> > > > Oh, ok. I have done that. Just didn't know that it was necessary.
> > > >
> > > > -Yuriy
> > > >
> > > > On Fri, Jul 10, 2020 at 9:30 PM John Roesler <vv...@apache.org>
> > > wrote:
> > > >
> > > >> Ah, thanks Yuriy,
> > > >>
> > > >> Sorry if this wasn't clear, but _all_ public API changes have to
> > > >> be explicitly included in the KIP. Can you just enumerate all
> > > >> the contents of the new API?
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >> On Fri, Jul 10, 2020, at 04:54, Yuriy Badalyantc wrote:
> > > >>> Hi, Matthias,
> > > >>>
> > > >>> It's not directly mentioned in the KIP, but I added all missing
> Java
> > > >>> serdes. I mentioned it in the pull request description:
> > > >>> https://github.com/apache/kafka/pull/8955
> > > >>>
> > > >>> And also, this KIP originally was based on a pull request where I
> added
> > > >>> missing java serdes :) https://github.com/apache/kafka/pull/8049
> > > >>>
> > > >>> -Yuriy
> > > >>>
> > > >>> On Fri, Jul 10, 2020 at 3:36 AM Matthias J. Sax <mj...@apache.org>
> > > >> wrote:
> > > >>>
> > > >>>> Yuriy,
> > > >>>>
> > > >>>> thanks for the KIP update. I have one follow up thought: I checked
> > > what
> > > >>>> default Serdes we offer in the Java class
> > > >>>>
> > > >>>> `org.apache.kafka.common.serialization.Serdes`
> > > >>>>
> > > >>>> and I think it would be good if we could close the gap between the
> > > Java
> > > >>>> and Scala code and add the missing Java Serdes in Scala, too.
> > > >>>>
> > > >>>> It seems we are missing `Short` (Java and Scala), `Void`, `UUID`,
> and
> > > >>>> `ByterBuffer`.
> > > >>>>
> > > >>>> Can we add those in addition?
> > > >>>>
> > > >>>>
> > > >>>> -Matthias
> > > >>>>
> > > >>>> On 7/8/20 6:45 AM, John Roesler wrote:
> > > >>>>> Hi Yuriy,
> > > >>>>>
> > > >>>>> Once it seems like there’s general agreement in the discussion,
> you
> > > >> can
> > > >>>> start a voting thread. You can find examples on the mailing list
> of
> > > >> what to
> > > >>>> say in the first message. It’s basically just a message with the
> > > >> subject
> > > >>>> line changed from “[DISCUSS]...” to “[VOTE]...”, and then stating
> that
> > > >>>> you’d like to start the vote. It’s nice to link to the kip
> document
> > > >> again.
> > > >>>>>
> > > >>>>> The rules for the vote are at the top of the “Kafka Improvement
> > > >> Process”
> > > >>>> page, but you basically need 3 binding +1 votes and no binding -1
> > > >> votes.
> > > >>>> You also need to wait at least three days from when you start the
> vote
> > > >>>> before you can declare it accepted. There’s no upper time limit.
> > > >>>>>
> > > >>>>> If you’re unsure of who has a binding vote, it’s just the people
> > > >> listed
> > > >>>> on the Apache Kafka Committers page.
> > > >>>>>
> > > >>>>> If people are slow to vote, feel free to keep bumping the thread,
> > > >> just
> > > >>>> like with the discussion.
> > > >>>>>
> > > >>>>> Thanks again for getting involved!
> > > >>>>> -John
> > > >>>>>
> > > >>>>> On Tue, Jul 7, 2020, at 01:51, Yuriy Badalyantc wrote:
> > > >>>>>> So, what's next? It's my first KIP and I'm not familiar with all
> > > >>>> processes.
> > > >>>>>>
> > > >>>>>> -Yuriy
> > > >>>>>>
> > > >>>>>> On Mon, Jul 6, 2020 at 1:32 AM John Roesler <
> vvcephei@apache.org>
> > > >>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hi Yuriy,
> > > >>>>>>>
> > > >>>>>>> Thanks for the update! It looks good to me.
> > > >>>>>>>
> > > >>>>>>> Thanks,
> > > >>>>>>> John
> > > >>>>>>>
> > > >>>>>>> On Sun, Jul 5, 2020, at 03:27, Yuriy Badalyantc wrote:
> > > >>>>>>>> Hi John.
> > > >>>>>>>>
> > > >>>>>>>> I updated the KIP. An old proposed implementation is now in
> the
> > > >>>> rejected
> > > >>>>>>>> alternatives.
> > > >>>>>>>>
> > > >>>>>>>> - Yuriy
> > > >>>>>>>>
> > > >>>>>>>> On Sun, Jul 5, 2020 at 12:03 AM John Roesler <
> vvcephei@apache.org
> > > >>>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi Yuriy,
> > > >>>>>>>>>
> > > >>>>>>>>> I agree, we can keep them separate. I just wanted to make you
> > > >> aware
> > > >>>> of
> > > >>>>>>> it.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks for the PR, it looks the way I expected.
> > > >>>>>>>>>
> > > >>>>>>>>> I just read over the KIP document again. I think it needs to
> be
> > > >>>>>>> updated to
> > > >>>>>>>>> the current proposal, and then we’ll be able to start the
> vote.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks,
> > > >>>>>>>>> John
> > > >>>>>>>>>
> > > >>>>>>>>> On Tue, Jun 30, 2020, at 04:58, Yuriy Badalyantc wrote:
> > > >>>>>>>>>> Hi everybody!
> > > >>>>>>>>>>
> > > >>>>>>>>>> Looks like a discussion about KIP-513 could take a while. I
> > > >> think we
> > > >>>>>>>>> should
> > > >>>>>>>>>> move forward with KIP-616 without waiting for KIP-513.
> > > >>>>>>>>>>
> > > >>>>>>>>>> I created a new pull request for KIP-616:
> > > >>>>>>>>>> https://github.com/apache/kafka/pull/8955. It contains a
> new
> > > >>>>>>>>>> `org.apache.kafka.streams.scala.serialization.Serdes` object
> > > >> without
> > > >>>>>>> name
> > > >>>>>>>>>> clash. An old one was marked as deprecated. This change is
> > > >> backward
> > > >>>>>>>>>> compatible and it could be merged in any further release.
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Wed, Jun 3, 2020 at 12:41 PM Yuriy Badalyantc <
> > > >> lmnet89@gmail.com
> > > >>>>>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hi, John
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks for pointing that out. I expressed my thoughts about
> > > >>>>>>> KIP-513 and
> > > >>>>>>>>>>> its connection to KIP-616 in the KIP-513 mail list.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - Yuriy
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Sun, May 31, 2020 at 1:26 AM John Roesler <
> > > >> vvcephei@apache.org>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi Yuriy,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I was just looking back at KIP-513, and I’m wondering if
> > > >> there’s
> > > >>>>>>> any
> > > >>>>>>>>>>>> overlap we should consider here, or if they are just
> > > >> orthogonal.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>> -John
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Thu, May 28, 2020, at 21:36, Yuriy Badalyantc wrote:
> > > >>>>>>>>>>>>> At the current moment, I think John's plan is better
> than the
> > > >>>>>>>>> original
> > > >>>>>>>>>>>> plan
> > > >>>>>>>>>>>>> described in the KIP. I think we should create a new
> > > >> `Serdes` in
> > > >>>>>>>>> another
> > > >>>>>>>>>>>>> package. The old one will be deprecated.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> - Yuriy
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Fri, May 29, 2020 at 8:58 AM John Roesler <
> > > >>>>>>> vvcephei@apache.org>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks, Matthias,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> If we go with the approach Yuriy and I agreed on, to
> > > >>>>>>> deprecate and
> > > >>>>>>>>>>>> replace
> > > >>>>>>>>>>>>>> the whole class and not just a few of the methods, then
> the
> > > >>>>>>>>> timeline
> > > >>>>>>>>>>>> is
> > > >>>>>>>>>>>>>> less of a concern. Under that plan, Yuriy can just
> write the
> > > >>>>>>> new
> > > >>>>>>>>> class
> > > >>>>>>>>>>>>>> exactly the way he wants and people can cleanly swap
> over to
> > > >>>>>>> the
> > > >>>>>>>>> new
> > > >>>>>>>>>>>>>> pattern when they are ready.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> The timeline was more significant if we were just going
> to
> > > >>>>>>>>> deprecate
> > > >>>>>>>>>>>> some
> > > >>>>>>>>>>>>>> methods and add new methods to the existing class. That
> plan
> > > >>>>>>>>> requires
> > > >>>>>>>>>>>> two
> > > >>>>>>>>>>>>>> implementation phases, where we first deprecate the
> existing
> > > >>>>>>>>> methods
> > > >>>>>>>>>>>> and
> > > >>>>>>>>>>>>>> later swap the implicits at the same time we remove the
> > > >>>>>>> deprecated
> > > >>>>>>>>>>>> members.
> > > >>>>>>>>>>>>>> Aside from the complexity of that approach, it’s not a
> > > >>>>>>> breakage
> > > >>>>>>>>> free
> > > >>>>>>>>>>>> path,
> > > >>>>>>>>>>>>>> as some users would be forced to continue using the
> > > >> deprecated
> > > >>>>>>>>> members
> > > >>>>>>>>>>>>>> until a future release drops them, breaking their source
> > > >>>>>>> code, and
> > > >>>>>>>>>>>> only
> > > >>>>>>>>>>>>>> then can they update their code.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> That wouldn’t be the end of the world, and we’ve had to
> do
> > > >> the
> > > >>>>>>>>> same
> > > >>>>>>>>>>>> thing
> > > >>>>>>>>>>>>>> in the past with the implicit conversations, but this
> is a
> > > >>>>>>> much
> > > >>>>>>>>> wider
> > > >>>>>>>>>>>>>> scope, since it’s all the serdes. I’m happy with the new
> > > >> plan,
> > > >>>>>>>>> since
> > > >>>>>>>>>>>> it’s
> > > >>>>>>>>>>>>>> not only one step, but also it provides everyone a
> > > >>>>>>> breakage-free
> > > >>>>>>>>> path.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> We can still consider dropping the deprecated class in
> 3.0;
> > > >> I
> > > >>>>>>> just
> > > >>>>>>>>>>>> wanted
> > > >>>>>>>>>>>>>> to clarify how the timeline issue has changed.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Thu, May 28, 2020, at 20:34, Matthias J. Sax wrote:
> > > >>>>>>>>>>>>>>> I am not a Scale person, so I cannot really contribute
> > > >> much.
> > > >>>>>>>>>>>> However for
> > > >>>>>>>>>>>>>>> the deprecation period, if we get the change into 2.7,
> it
> > > >>>>>>> might
> > > >>>>>>>>> be
> > > >>>>>>>>>>>> ok to
> > > >>>>>>>>>>>>>>> remove the deprecated classed in 3.0.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> It would only be one minor release in between what is a
> > > >>>>>>> little
> > > >>>>>>>>> bit
> > > >>>>>>>>>>>> short
> > > >>>>>>>>>>>>>>> (we usually prefer at least two minor released, better
> > > >>>>>>> three),
> > > >>>>>>>>> but
> > > >>>>>>>>>>>> if we
> > > >>>>>>>>>>>>>>> have a good reason for it, it might be ok.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> If we cannot remove it in 3.0, it seems there would be
> a
> > > >>>>>>> 4.0 in
> > > >>>>>>>>>>>> about a
> > > >>>>>>>>>>>>>>> year(?) when ZK removal is finished and we can remove
> the
> > > >>>>>>>>> deprecated
> > > >>>>>>>>>>>>>>> code than.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> -Matthias
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On 5/28/20 7:39 AM, John Roesler wrote:
> > > >>>>>>>>>>>>>>>> Hi Yuriy,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Sounds good to me! I had a feeling we were bringing
> > > >>>>>>> different
> > > >>>>>>>>>>>> context
> > > >>>>>>>>>>>>>>>> to the discussion; thanks for sticking with the
> > > >>>>>>> conversation
> > > >>>>>>>>>>>> until we
> > > >>>>>>>>>>>>>> got
> > > >>>>>>>>>>>>>>>> it hashed out.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I'm glad you prefer Serde*s*, since having multiple
> > > >>>>>>> different
> > > >>>>>>>>>>>> classes
> > > >>>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>>>> the same name leads to all kinds of trouble. "Serdes"
> > > >>>>>>> seems
> > > >>>>>>>>>>>> relatively
> > > >>>>>>>>>>>>>>>> safe because people in the Scala lib won't be using
> the
> > > >>>>>>> Java
> > > >>>>>>>>>>>> Serdes
> > > >>>>>>>>>>>>>> class,
> > > >>>>>>>>>>>>>>>> and they won't be using the deprecated and
> non-deprecated
> > > >>>>>>> one
> > > >>>>>>>>> at
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> same time.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thank again,
> > > >>>>>>>>>>>>>>>> -John
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Thu, May 28, 2020, at 02:21, Yuriy Badalyantc
> wrote:
> > > >>>>>>>>>>>>>>>>> Ok, I understood you, John. I wasn't sure about kafka
> > > >>>>>>>>> deprecation
> > > >>>>>>>>>>>>>> policy
> > > >>>>>>>>>>>>>>>>> and thought that the full cycle could be done with
> 2.7
> > > >>>>>>>>> version.
> > > >>>>>>>>>>>>>> Waiting for
> > > >>>>>>>>>>>>>>>>> 3.0 is too much, I agree with it.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> So, I think creating one more `Serdes` in another
> > > >>>>>>> package is
> > > >>>>>>>>> our
> > > >>>>>>>>>>>> way.
> > > >>>>>>>>>>>>>> I
> > > >>>>>>>>>>>>>>>>> suggest one of the following:
> > > >>>>>>>>>>>>>>>>> 1. `org.apache.kafka.streams.scala.serde.Serdes`
> > > >>>>>>>>>>>>>>>>> 2.
> `org.apache.kafka.streams.scala.serialization.Serdes`
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> About `Serde` vs `Serdes`. I'm strongly against
> `Serde`
> > > >>>>>>>>> because
> > > >>>>>>>>>>>> it
> > > >>>>>>>>>>>>>> would
> > > >>>>>>>>>>>>>>>>> lead to a new name clash with the
> > > >>>>>>>>>>>>>>>>> `org.apache.kafka.common.serialization.Serde`.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> - Yuriy
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Thu, May 28, 2020 at 11:12 AM John Roesler <
> > > >>>>>>>>>>>> vvcephei@apache.org>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Hi Yuriy,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Thanks for the clarification.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I guess my concern is twofold:
> > > >>>>>>>>>>>>>>>>>> 1. We typically leave deprecated methods in place
> for at
> > > >>>>>>>>> least a
> > > >>>>>>>>>>>>>> major
> > > >>>>>>>>>>>>>>>>>> release cycle before removing them, so it would seem
> > > >>>>>>> abrupt
> > > >>>>>>>>> to
> > > >>>>>>>>>>>> have a
> > > >>>>>>>>>>>>>>>>>> deprecation period of only one minor release. If we
> > > >>>>>>> follow
> > > >>>>>>>>> the
> > > >>>>>>>>>>>> same
> > > >>>>>>>>>>>>>> pattern
> > > >>>>>>>>>>>>>>>>>> here, it would take over a year to finish this KIP.
> > > >>>>>>>>>>>>>>>>>> 2. It doesn’t seem like there is a nonbreaking
> > > >>>>>>> deprecation
> > > >>>>>>>>> path
> > > >>>>>>>>>>>> at
> > > >>>>>>>>>>>>>> all if
> > > >>>>>>>>>>>>>>>>>> people enumerate their imports (if they don’t use a
> > > >>>>>>>>> wildcard).
> > > >>>>>>>>>>>> In
> > > >>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>> case, they would have no path to implicitly use the
> > > >>>>>>> newly
> > > >>>>>>>>> named
> > > >>>>>>>>>>>>>> serdes, and
> > > >>>>>>>>>>>>>>>>>> therefore they would have no way to avoid
> continuing to
> > > >>>>>>> use
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>>> deprecated
> > > >>>>>>>>>>>>>>>>>> ones.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Since you mentioned that your reason is mainly the
> > > >>>>>>>>> preference
> > > >>>>>>>>>>>> for
> > > >>>>>>>>>>>>>> the name
> > > >>>>>>>>>>>>>>>>>> “Serde” or “Serdes”, can we explore just using one
> of
> > > >>>>>>> those?
> > > >>>>>>>>>>>> Would
> > > >>>>>>>>>>>>>> it cause
> > > >>>>>>>>>>>>>>>>>> some kind of conflict to use
> > > >>>>>>>>>>>> org.apache.kafka.streams.scala.Serde or
> > > >>>>>>>>>>>>>> to use
> > > >>>>>>>>>>>>>>>>>> Serdes in a different package, like
> > > >>>>>>>>>>>>>>>>>> org.apache.kafka.streams.scala.implicit.Serdes?
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I empathize with this desire. I faced the same
> dilemma
> > > >>>>>>> when
> > > >>>>>>>>> I
> > > >>>>>>>>>>>> wanted
> > > >>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>> replace Processor but keep the class name in
> KIP-478. I
> > > >>>>>>>>> wound up
> > > >>>>>>>>>>>>>> creating a
> > > >>>>>>>>>>>>>>>>>> new package for the new Processor.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On Wed, May 27, 2020, at 22:20, Yuriy Badalyantc
> wrote:
> > > >>>>>>>>>>>>>>>>>>> Hi John,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> I'm stick with the
> > > >>>>>>> `org.apache.kafka.streams.scala.Serdes`
> > > >>>>>>>>>>>> because
> > > >>>>>>>>>>>>>> it's
> > > >>>>>>>>>>>>>>>>>>> sort of conventional in the scala community. If you
> > > >>>>>>> have a
> > > >>>>>>>>>>>> typeclass
> > > >>>>>>>>>>>>>>>>>> `Foo`,
> > > >>>>>>>>>>>>>>>>>>> you probably will search `Foo` related stuff in the
> > > >>>>>>> `Foo`
> > > >>>>>>>>> or
> > > >>>>>>>>>>>> maybe
> > > >>>>>>>>>>>>>> `Foos`
> > > >>>>>>>>>>>>>>>>>>> (plural). All other places are far less
> discoverable
> > > >>>>>>> for
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>>> developers.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> I agree that the migration path is a bit complex
> for
> > > >>>>>>> such
> > > >>>>>>>>>>>> change.
> > > >>>>>>>>>>>>>> But I
> > > >>>>>>>>>>>>>>>>>>> think it's more important to provide good developer
> > > >>>>>>>>> experience
> > > >>>>>>>>>>>> than
> > > >>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>> simplify migration. Also, I think it's debatable
> which
> > > >>>>>>>>>>>> migration
> > > >>>>>>>>>>>>>> path is
> > > >>>>>>>>>>>>>>>>>>> better for library users. If we would create, for
> > > >>>>>>> example,
> > > >>>>>>>>>>>>>> `Serdes2`,
> > > >>>>>>>>>>>>>>>>>>> library users will have to modify their code if
> they
> > > >>>>>>> used
> > > >>>>>>>>> any
> > > >>>>>>>>>>>> part
> > > >>>>>>>>>>>>>> of the
> > > >>>>>>>>>>>>>>>>>>> old `Serde`. With my approach, most of the old code
> > > >>>>>>> will
> > > >>>>>>>>> still
> > > >>>>>>>>>>>> work
> > > >>>>>>>>>>>>>>>>>> without
> > > >>>>>>>>>>>>>>>>>>> changes. Only explicit usage of implicits will
> need to
> > > >>>>>>> be
> > > >>>>>>>>> fixed
> > > >>>>>>>>>>>>>> (because
> > > >>>>>>>>>>>>>>>>>>> names will be changed, and old names will be
> > > >>>>>>> deprecated).
> > > >>>>>>>>>>>> Wildcard
> > > >>>>>>>>>>>>>>>>>> imports
> > > >>>>>>>>>>>>>>>>>>> will work without changes and will not lead to a
> name
> > > >>>>>>>>> clash.
> > > >>>>>>>>>>>>>> Moreover,
> > > >>>>>>>>>>>>>>>>>> many
> > > >>>>>>>>>>>>>>>>>>> users may not notice name clash problems. And with
> my
> > > >>>>>>>>> migration
> > > >>>>>>>>>>>>>> path,
> > > >>>>>>>>>>>>>>>>>> they
> > > >>>>>>>>>>>>>>>>>>> will not notice any changes at all.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> - Yuriy
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On Thu, May 28, 2020 at 7:48 AM John Roesler <
> > > >>>>>>>>>>>> vvcephei@apache.org>
> > > >>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Hi Yuriy,
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Thanks for the reply. I guess I've been out of the
> > > >>>>>>> Scala
> > > >>>>>>>>> game
> > > >>>>>>>>>>>> for a
> > > >>>>>>>>>>>>>>>>>>>> while; all this summoner business is totally new
> to
> > > >>>>>>> me.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> I think I followed the rationale you provided,
> but I
> > > >>>>>>> still
> > > >>>>>>>>>>>> don't
> > > >>>>>>>>>>>>>> see
> > > >>>>>>>>>>>>>>>>>>>> why you can't implement your whole plan in a new
> > > >>>>>>> class.
> > > >>>>>>>>> What
> > > >>>>>>>>>>>>>>>>>>>> is special about the existing Serdes class?
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>> -John
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> On Tue, May 19, 2020, at 01:18, Yuriy Badalyantc
> > > >>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>> Hi John,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Your suggestion looks interesting. I think it's
> > > >>>>>>>>> technically
> > > >>>>>>>>>>>>>> doable.
> > > >>>>>>>>>>>>>>>>>> But
> > > >>>>>>>>>>>>>>>>>>>> I'm
> > > >>>>>>>>>>>>>>>>>>>>> not sure that this is the better solution. I will
> > > >>>>>>> try to
> > > >>>>>>>>>>>> explain.
> > > >>>>>>>>>>>>>>>>>> From
> > > >>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>> scala developers' perspective, `Serde` looks
> really
> > > >>>>>>> like
> > > >>>>>>>>> a
> > > >>>>>>>>>>>>>> typeclass.
> > > >>>>>>>>>>>>>>>>>>>>> Typical typeclass in pure scala will look like
> this:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> ```
> > > >>>>>>>>>>>>>>>>>>>>> trait Serde[A] {
> > > >>>>>>>>>>>>>>>>>>>>> def serialize(data: A): Array[Byte]
> > > >>>>>>>>>>>>>>>>>>>>> def deserialize(data: Array[Byte]): A
> > > >>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>> object Serde extends DefaultSerdes {
> > > >>>>>>>>>>>>>>>>>>>>> // "summoner" function. With this I can write
> > > >>>>>>>>> `Serde[A]`
> > > >>>>>>>>>>>> and
> > > >>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>> serde
> > > >>>>>>>>>>>>>>>>>>>>> will be implicitly summonned.
> > > >>>>>>>>>>>>>>>>>>>>> def apply[A](implicit ev: Serde[A]): Serde[A]
> = ev
> > > >>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> trait DefaultSerdes {
> > > >>>>>>>>>>>>>>>>>>>>> // default instances here
> > > >>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>> ```
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Usage example (note, that there are no wildcards
> > > >>>>>>> imports
> > > >>>>>>>>>>>> here):
> > > >>>>>>>>>>>>>>>>>>>>> ```
> > > >>>>>>>>>>>>>>>>>>>>> object Main extends App {
> > > >>>>>>>>>>>>>>>>>>>>> import Serde // not wildcard import
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> // explicit summonning:
> > > >>>>>>>>>>>>>>>>>>>>> val stringSerde = Serde[String] // using
> summoner
> > > >>>>>>>>>>>>>>>>>>>>> stringSerde.serialize(???)
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> // implicit summonning
> > > >>>>>>>>>>>>>>>>>>>>> def serialize[A: Serde](a: A) = {
> > > >>>>>>>>>>>>>>>>>>>>> Serde[A].serialize(a) // summoner again
> > > >>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>> serialize("foo")
> > > >>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>> ```
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Examples are pretty silly, but I just want to
> show
> > > >>>>>>> common
> > > >>>>>>>>>>>>>> patterns of
> > > >>>>>>>>>>>>>>>>>>>>> working with typeclasses in scala. All default
> > > >>>>>>> instances
> > > >>>>>>>>> in
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> usage
> > > >>>>>>>>>>>>>>>>>>>>> examples are found using implicits searching
> > > >>>>>>> mechanism.
> > > >>>>>>>>> Scala
> > > >>>>>>>>>>>>>>>>>> compiler
> > > >>>>>>>>>>>>>>>>>>>>> searches implicits in a lot of places. Including
> > > >>>>>>>>> companion
> > > >>>>>>>>>>>>>> objects.
> > > >>>>>>>>>>>>>>>>>> In my
> > > >>>>>>>>>>>>>>>>>>>>> examples compiler will found `Serde[String]`
> > > >>>>>>> instance in
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> companion
> > > >>>>>>>>>>>>>>>>>>>>> object of `Serde` typeclass.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Also, I want to pay attention to the summoner
> > > >>>>>>> function.
> > > >>>>>>>>> It
> > > >>>>>>>>>>>> makes
> > > >>>>>>>>>>>>>>>>>> usage of
> > > >>>>>>>>>>>>>>>>>>>>> typeclasses very neat and clear.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> The example above was the example of the perfect
> > > >>>>>>> solution
> > > >>>>>>>>>>>> for the
> > > >>>>>>>>>>>>>>>>>> scala
> > > >>>>>>>>>>>>>>>>>>>>> developers. But this solution requires to create
> > > >>>>>>> separate
> > > >>>>>>>>>>>> `Serde`
> > > >>>>>>>>>>>>>>>>>>>>> typeclass, to make all this implicit searching
> stuff
> > > >>>>>>>>> works. I
> > > >>>>>>>>>>>>>> don't
> > > >>>>>>>>>>>>>>>>>> think
> > > >>>>>>>>>>>>>>>>>>>>> that it worth it, because a lot of code should be
> > > >>>>>>>>>>>> reimplemented
> > > >>>>>>>>>>>>>> using
> > > >>>>>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>> new typeclass. But the main point of my example
> is to
> > > >>>>>>>>> show
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> perfect
> > > >>>>>>>>>>>>>>>>>>>>> solution. And I think we should strive to provide
> > > >>>>>>>>> developer
> > > >>>>>>>>>>>>>>>>>> experience
> > > >>>>>>>>>>>>>>>>>>>>> close to this.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> It's a bit out of the scope of my KIP, but I
> have a
> > > >>>>>>> plan
> > > >>>>>>>>> to
> > > >>>>>>>>>>>> make
> > > >>>>>>>>>>>>>>>>>>>>> `org.apache.kafka.streams.scala.Serdes` more
> closer
> > > >>>>>>> to
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>>> solution
> > > >>>>>>>>>>>>>>>>>>>> above.
> > > >>>>>>>>>>>>>>>>>>>>> It could be done in 2 steps:
> > > >>>>>>>>>>>>>>>>>>>>> 1. Fix implicit names.
> > > >>>>>>>>>>>>>>>>>>>>> 2. Add summoner function.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> And with this scala developers will be able to
> write
> > > >>>>>>>>> almost
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> same
> > > >>>>>>>>>>>>>>>>>> code
> > > >>>>>>>>>>>>>>>>>>>>> as in the example above:
> > > >>>>>>>>>>>>>>>>>>>>> ```
> > > >>>>>>>>>>>>>>>>>>>>> object Main extends App {
> > > >>>>>>>>>>>>>>>>>>>>> import org.apache.kafka.streams.scala.Serdes
> // not
> > > >>>>>>>>>>>> wildcard
> > > >>>>>>>>>>>>>> import
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> val stringSerde = Serdes[String]
> > > >>>>>>>>>>>>>>>>>>>>> stringSerde.serialize(???)
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> def serialize[A: Serde](a: A) = {
> > > >>>>>>>>>>>>>>>>>>>>> Serdes[A].serialize(a)
> > > >>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>> serialize("foo")
> > > >>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>> ```
> > > >>>>>>>>>>>>>>>>>>>>> Of course, wildcard import will still work.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Other names will make this new entity (containing
> > > >>>>>>> default
> > > >>>>>>>>>>>>>> implicits)
> > > >>>>>>>>>>>>>>>>>> less
> > > >>>>>>>>>>>>>>>>>>>>> discoverable. And summoner usage, in this case,
> will
> > > >>>>>>> look
> > > >>>>>>>>>>>> weird:
> > > >>>>>>>>>>>>>>>>>>>>> ```
> > > >>>>>>>>>>>>>>>>>>>>> object Main extends App {
> > > >>>>>>>>>>>>>>>>>>>>> import
> > > >>>>>>> org.apache.kafka.streams.scala.DefaultSerdes //
> > > >>>>>>>>> not
> > > >>>>>>>>>>>>>> wildcard
> > > >>>>>>>>>>>>>>>>>>>> import
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> val stringSerde = DefaultSerdes[String]
> > > >>>>>>>>>>>>>>>>>>>>> stringSerde.serialize(???)
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> def serialize[A: Serde](a: A) = {
> > > >>>>>>>>>>>>>>>>>>>>> DefaultSerdes[A].serialize(a)
> > > >>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>> serialize("foo")
> > > >>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>> ```
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> So, I think it's more important to provide a
> solid
> > > >>>>>>> and
> > > >>>>>>>>>>>> familiar
> > > >>>>>>>>>>>>>>>>>> developer
> > > >>>>>>>>>>>>>>>>>>>>> experience for the scala developer. And renaming
> (or
> > > >>>>>>>>>>>> creating a
> > > >>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>>> version) of `Serdes` will not help here.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> -Yuriy
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> On Tue, May 19, 2020 at 11:56 AM John Roesler <
> > > >>>>>>>>>>>>>> vvcephei@apache.org>
> > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Hi Yuriy,
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Thanks so much for the KIP! I didn’t anticipate
> the
> > > >>>>>>>>> problem
> > > >>>>>>>>>>>> you
> > > >>>>>>>>>>>>>>>>>> laid
> > > >>>>>>>>>>>>>>>>>>>> out
> > > >>>>>>>>>>>>>>>>>>>>>> in the KIP, but I find it very plausible.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Thanks for pushing back on the “convention” and
> > > >>>>>>> raising
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>>> issue,
> > > >>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>> also volunteering a solution!
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> I’m wondering if we can “fix” it in one shot by
> just
> > > >>>>>>>>>>>> deprecating
> > > >>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> whole
> > > >>>>>>>>>>>>>>>>>>>>>> Serdes class and replacing it with a new one
> > > >>>>>>> containing
> > > >>>>>>>>> the
> > > >>>>>>>>>>>> defs
> > > >>>>>>>>>>>>>>>>>> you
> > > >>>>>>>>>>>>>>>>>>>>>> proposed. Then, people could just switch their
> > > >>>>>>> import to
> > > >>>>>>>>>>>> the new
> > > >>>>>>>>>>>>>>>>>> one.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Of course the new class needs to have a
> different
> > > >>>>>>> name,
> > > >>>>>>>>>>>> which is
> > > >>>>>>>>>>>>>>>>>>>> always a
> > > >>>>>>>>>>>>>>>>>>>>>> challenge in situations like this, so I might
> just
> > > >>>>>>>>> throw out
> > > >>>>>>>>>>>>>>>>>>>> ImplicitSerdes
> > > >>>>>>>>>>>>>>>>>>>>>> as an option.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Do you think this would work?
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Thanks again,
> > > >>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> On Mon, May 18, 2020, at 23:35, Yuriy Badalyantc
> > > >>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>> Hi,
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> I would like to propose KIP-616 to fix naming
> > > >>>>>>> clash in
> > > >>>>>>>>> the
> > > >>>>>>>>>>>> kafka
> > > >>>>>>>>>>>>>>>>>>>>>>> streams scala API:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-616%3A+Rename+implicit+Serdes+instances+in+kafka-streams-scala
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Looking forward to your feedback.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> -Yuriy
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Attachments:
> > > >>>>>>>>>>>>>>> * signature.asc
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
>
Re: [DISCUSS] KIP-616: Rename implicit Serdes instances in kafka-streams-scala
Posted by John Roesler <vv...@apache.org>.
Thanks, Yuriy,
That sounds good to me.
-John
On Wed, Aug 5, 2020, at 00:24, Yuriy Badalyantc wrote:
> Hi John!
>
> I understand your point. But, I'm still not sure that having a Unit serde
> is a good idea because it could be ambiguous for the user. In a KStream
> [Long, Unit] valid value could mean one the following:
> 1. Empty byte array
> 2. null
> 3. Any value (value ignored)
>
> As a user of a library, I think all these options are more or less valid.
> That's because the Unit type in my opinion has no clear representation in
> terms of values and deserialization.
>
> About Null serde - I'm ok with this. But, it should be a low priority
> implicit to not intersect will other implicits. I will add it.
>
> -Yuriy
>
> On Sat, Jul 11, 2020 at 9:18 PM John Roesler <vv...@apache.org> wrote:
>
> > Thanks, Yuriy,
> >
> > IIRC, I added VoidSerde because there are times when the key or value is
> > always null, but since ‘null’ has no type in Java, we had used the
> > ByteArraySerde or some other proxy. This is confusing, because then the
> > type is ‘byte[]’ although we intended it to always be null. It also delays
> > finding bugs because serdes themselves force runtime type checks, but the
> > ByteArraySerde accepts all data.
> >
> > Anyway, Void isn’t the type of null, but it’s close enough. Although
> > Scala’s equivalent to “void” method is Unit methods, and like Void, Unit is
> > uninstantiable, Unit has a value, also called Unit or “()”, whereas Void
> > has no value in Java. But because Void is still a reference type, it can be
> > null (I.e., it can only be null), so I’m some sense, it’s closer to the
> > Scala type Null.
> >
> > I guess Scala users would benefit from both a NullSerde and a UnitSerde,
> > but not a VoidSerde.
> >
> > Thanks for bringing it up, Matthias.
> >
> > -John
> >
> > On Fri, Jul 10, 2020, at 21:49, Yuriy Badalyantc wrote:
> > > Ok, I mentioned adding missing serdes in the Proposed Change paragraph.
> > >
> > > About VoidSerde. I didn't add it intentionally. The semantic of the Unit
> > > (scala's void) type is not clear in terms of the data. If kafka topic
> > > contains messages of type Unit, what does it actually means? That there
> > is
> > > always null? Well, for that we have a Null type. That it's an empty byte
> > > array? For that, we have an Array[Byte]. Empty string? No, it's a String.
> > > So, I decided to not include Unit serde in the built-in Serdes. And if a
> > > user will want to use the Unit type he can implement its own serde.
> > >
> > > -Yuriy
> > >
> > > On Fri, Jul 10, 2020 at 11:21 PM Matthias J. Sax <mj...@apache.org>
> > wrote:
> > >
> > > > Thanks Yuriy!
> > > >
> > > > What about `VoidSerde` ? It's not listed.
> > > >
> > > > It might also be nice to add a short sentence and state that in
> > addition
> > > > to fixing the name collisions, the KIP will also close the gap of
> > > > out-of-the-box serdes and add missing Serdes that are offered in Java
> > to
> > > > Scala.
> > > >
> > > >
> > > > -Matthias
> > > >
> > > > On 7/10/20 7:51 AM, Yuriy Badalyantc wrote:
> > > > > Oh, ok. I have done that. Just didn't know that it was necessary.
> > > > >
> > > > > -Yuriy
> > > > >
> > > > > On Fri, Jul 10, 2020 at 9:30 PM John Roesler <vv...@apache.org>
> > > > wrote:
> > > > >
> > > > >> Ah, thanks Yuriy,
> > > > >>
> > > > >> Sorry if this wasn't clear, but _all_ public API changes have to
> > > > >> be explicitly included in the KIP. Can you just enumerate all
> > > > >> the contents of the new API?
> > > > >>
> > > > >> Thanks,
> > > > >> John
> > > > >>
> > > > >> On Fri, Jul 10, 2020, at 04:54, Yuriy Badalyantc wrote:
> > > > >>> Hi, Matthias,
> > > > >>>
> > > > >>> It's not directly mentioned in the KIP, but I added all missing
> > Java
> > > > >>> serdes. I mentioned it in the pull request description:
> > > > >>> https://github.com/apache/kafka/pull/8955
> > > > >>>
> > > > >>> And also, this KIP originally was based on a pull request where I
> > added
> > > > >>> missing java serdes :) https://github.com/apache/kafka/pull/8049
> > > > >>>
> > > > >>> -Yuriy
> > > > >>>
> > > > >>> On Fri, Jul 10, 2020 at 3:36 AM Matthias J. Sax <mj...@apache.org>
> > > > >> wrote:
> > > > >>>
> > > > >>>> Yuriy,
> > > > >>>>
> > > > >>>> thanks for the KIP update. I have one follow up thought: I checked
> > > > what
> > > > >>>> default Serdes we offer in the Java class
> > > > >>>>
> > > > >>>> `org.apache.kafka.common.serialization.Serdes`
> > > > >>>>
> > > > >>>> and I think it would be good if we could close the gap between the
> > > > Java
> > > > >>>> and Scala code and add the missing Java Serdes in Scala, too.
> > > > >>>>
> > > > >>>> It seems we are missing `Short` (Java and Scala), `Void`, `UUID`,
> > and
> > > > >>>> `ByterBuffer`.
> > > > >>>>
> > > > >>>> Can we add those in addition?
> > > > >>>>
> > > > >>>>
> > > > >>>> -Matthias
> > > > >>>>
> > > > >>>> On 7/8/20 6:45 AM, John Roesler wrote:
> > > > >>>>> Hi Yuriy,
> > > > >>>>>
> > > > >>>>> Once it seems like there’s general agreement in the discussion,
> > you
> > > > >> can
> > > > >>>> start a voting thread. You can find examples on the mailing list
> > of
> > > > >> what to
> > > > >>>> say in the first message. It’s basically just a message with the
> > > > >> subject
> > > > >>>> line changed from “[DISCUSS]...” to “[VOTE]...”, and then stating
> > that
> > > > >>>> you’d like to start the vote. It’s nice to link to the kip
> > document
> > > > >> again.
> > > > >>>>>
> > > > >>>>> The rules for the vote are at the top of the “Kafka Improvement
> > > > >> Process”
> > > > >>>> page, but you basically need 3 binding +1 votes and no binding -1
> > > > >> votes.
> > > > >>>> You also need to wait at least three days from when you start the
> > vote
> > > > >>>> before you can declare it accepted. There’s no upper time limit.
> > > > >>>>>
> > > > >>>>> If you’re unsure of who has a binding vote, it’s just the people
> > > > >> listed
> > > > >>>> on the Apache Kafka Committers page.
> > > > >>>>>
> > > > >>>>> If people are slow to vote, feel free to keep bumping the thread,
> > > > >> just
> > > > >>>> like with the discussion.
> > > > >>>>>
> > > > >>>>> Thanks again for getting involved!
> > > > >>>>> -John
> > > > >>>>>
> > > > >>>>> On Tue, Jul 7, 2020, at 01:51, Yuriy Badalyantc wrote:
> > > > >>>>>> So, what's next? It's my first KIP and I'm not familiar with all
> > > > >>>> processes.
> > > > >>>>>>
> > > > >>>>>> -Yuriy
> > > > >>>>>>
> > > > >>>>>> On Mon, Jul 6, 2020 at 1:32 AM John Roesler <
> > vvcephei@apache.org>
> > > > >>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hi Yuriy,
> > > > >>>>>>>
> > > > >>>>>>> Thanks for the update! It looks good to me.
> > > > >>>>>>>
> > > > >>>>>>> Thanks,
> > > > >>>>>>> John
> > > > >>>>>>>
> > > > >>>>>>> On Sun, Jul 5, 2020, at 03:27, Yuriy Badalyantc wrote:
> > > > >>>>>>>> Hi John.
> > > > >>>>>>>>
> > > > >>>>>>>> I updated the KIP. An old proposed implementation is now in
> > the
> > > > >>>> rejected
> > > > >>>>>>>> alternatives.
> > > > >>>>>>>>
> > > > >>>>>>>> - Yuriy
> > > > >>>>>>>>
> > > > >>>>>>>> On Sun, Jul 5, 2020 at 12:03 AM John Roesler <
> > vvcephei@apache.org
> > > > >>>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Hi Yuriy,
> > > > >>>>>>>>>
> > > > >>>>>>>>> I agree, we can keep them separate. I just wanted to make you
> > > > >> aware
> > > > >>>> of
> > > > >>>>>>> it.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks for the PR, it looks the way I expected.
> > > > >>>>>>>>>
> > > > >>>>>>>>> I just read over the KIP document again. I think it needs to
> > be
> > > > >>>>>>> updated to
> > > > >>>>>>>>> the current proposal, and then we’ll be able to start the
> > vote.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks,
> > > > >>>>>>>>> John
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Tue, Jun 30, 2020, at 04:58, Yuriy Badalyantc wrote:
> > > > >>>>>>>>>> Hi everybody!
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Looks like a discussion about KIP-513 could take a while. I
> > > > >> think we
> > > > >>>>>>>>> should
> > > > >>>>>>>>>> move forward with KIP-616 without waiting for KIP-513.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I created a new pull request for KIP-616:
> > > > >>>>>>>>>> https://github.com/apache/kafka/pull/8955. It contains a
> > new
> > > > >>>>>>>>>> `org.apache.kafka.streams.scala.serialization.Serdes` object
> > > > >> without
> > > > >>>>>>> name
> > > > >>>>>>>>>> clash. An old one was marked as deprecated. This change is
> > > > >> backward
> > > > >>>>>>>>>> compatible and it could be merged in any further release.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Wed, Jun 3, 2020 at 12:41 PM Yuriy Badalyantc <
> > > > >> lmnet89@gmail.com
> > > > >>>>>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Hi, John
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Thanks for pointing that out. I expressed my thoughts about
> > > > >>>>>>> KIP-513 and
> > > > >>>>>>>>>>> its connection to KIP-616 in the KIP-513 mail list.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> - Yuriy
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Sun, May 31, 2020 at 1:26 AM John Roesler <
> > > > >> vvcephei@apache.org>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Hi Yuriy,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I was just looking back at KIP-513, and I’m wondering if
> > > > >> there’s
> > > > >>>>>>> any
> > > > >>>>>>>>>>>> overlap we should consider here, or if they are just
> > > > >> orthogonal.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>> -John
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Thu, May 28, 2020, at 21:36, Yuriy Badalyantc wrote:
> > > > >>>>>>>>>>>>> At the current moment, I think John's plan is better
> > than the
> > > > >>>>>>>>> original
> > > > >>>>>>>>>>>> plan
> > > > >>>>>>>>>>>>> described in the KIP. I think we should create a new
> > > > >> `Serdes` in
> > > > >>>>>>>>> another
> > > > >>>>>>>>>>>>> package. The old one will be deprecated.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> - Yuriy
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Fri, May 29, 2020 at 8:58 AM John Roesler <
> > > > >>>>>>> vvcephei@apache.org>
> > > > >>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Thanks, Matthias,
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> If we go with the approach Yuriy and I agreed on, to
> > > > >>>>>>> deprecate and
> > > > >>>>>>>>>>>> replace
> > > > >>>>>>>>>>>>>> the whole class and not just a few of the methods, then
> > the
> > > > >>>>>>>>> timeline
> > > > >>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>> less of a concern. Under that plan, Yuriy can just
> > write the
> > > > >>>>>>> new
> > > > >>>>>>>>> class
> > > > >>>>>>>>>>>>>> exactly the way he wants and people can cleanly swap
> > over to
> > > > >>>>>>> the
> > > > >>>>>>>>> new
> > > > >>>>>>>>>>>>>> pattern when they are ready.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> The timeline was more significant if we were just going
> > to
> > > > >>>>>>>>> deprecate
> > > > >>>>>>>>>>>> some
> > > > >>>>>>>>>>>>>> methods and add new methods to the existing class. That
> > plan
> > > > >>>>>>>>> requires
> > > > >>>>>>>>>>>> two
> > > > >>>>>>>>>>>>>> implementation phases, where we first deprecate the
> > existing
> > > > >>>>>>>>> methods
> > > > >>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>> later swap the implicits at the same time we remove the
> > > > >>>>>>> deprecated
> > > > >>>>>>>>>>>> members.
> > > > >>>>>>>>>>>>>> Aside from the complexity of that approach, it’s not a
> > > > >>>>>>> breakage
> > > > >>>>>>>>> free
> > > > >>>>>>>>>>>> path,
> > > > >>>>>>>>>>>>>> as some users would be forced to continue using the
> > > > >> deprecated
> > > > >>>>>>>>> members
> > > > >>>>>>>>>>>>>> until a future release drops them, breaking their source
> > > > >>>>>>> code, and
> > > > >>>>>>>>>>>> only
> > > > >>>>>>>>>>>>>> then can they update their code.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> That wouldn’t be the end of the world, and we’ve had to
> > do
> > > > >> the
> > > > >>>>>>>>> same
> > > > >>>>>>>>>>>> thing
> > > > >>>>>>>>>>>>>> in the past with the implicit conversations, but this
> > is a
> > > > >>>>>>> much
> > > > >>>>>>>>> wider
> > > > >>>>>>>>>>>>>> scope, since it’s all the serdes. I’m happy with the new
> > > > >> plan,
> > > > >>>>>>>>> since
> > > > >>>>>>>>>>>> it’s
> > > > >>>>>>>>>>>>>> not only one step, but also it provides everyone a
> > > > >>>>>>> breakage-free
> > > > >>>>>>>>> path.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> We can still consider dropping the deprecated class in
> > 3.0;
> > > > >> I
> > > > >>>>>>> just
> > > > >>>>>>>>>>>> wanted
> > > > >>>>>>>>>>>>>> to clarify how the timeline issue has changed.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On Thu, May 28, 2020, at 20:34, Matthias J. Sax wrote:
> > > > >>>>>>>>>>>>>>> I am not a Scale person, so I cannot really contribute
> > > > >> much.
> > > > >>>>>>>>>>>> However for
> > > > >>>>>>>>>>>>>>> the deprecation period, if we get the change into 2.7,
> > it
> > > > >>>>>>> might
> > > > >>>>>>>>> be
> > > > >>>>>>>>>>>> ok to
> > > > >>>>>>>>>>>>>>> remove the deprecated classed in 3.0.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> It would only be one minor release in between what is a
> > > > >>>>>>> little
> > > > >>>>>>>>> bit
> > > > >>>>>>>>>>>> short
> > > > >>>>>>>>>>>>>>> (we usually prefer at least two minor released, better
> > > > >>>>>>> three),
> > > > >>>>>>>>> but
> > > > >>>>>>>>>>>> if we
> > > > >>>>>>>>>>>>>>> have a good reason for it, it might be ok.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> If we cannot remove it in 3.0, it seems there would be
> > a
> > > > >>>>>>> 4.0 in
> > > > >>>>>>>>>>>> about a
> > > > >>>>>>>>>>>>>>> year(?) when ZK removal is finished and we can remove
> > the
> > > > >>>>>>>>> deprecated
> > > > >>>>>>>>>>>>>>> code than.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> -Matthias
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On 5/28/20 7:39 AM, John Roesler wrote:
> > > > >>>>>>>>>>>>>>>> Hi Yuriy,
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Sounds good to me! I had a feeling we were bringing
> > > > >>>>>>> different
> > > > >>>>>>>>>>>> context
> > > > >>>>>>>>>>>>>>>> to the discussion; thanks for sticking with the
> > > > >>>>>>> conversation
> > > > >>>>>>>>>>>> until we
> > > > >>>>>>>>>>>>>> got
> > > > >>>>>>>>>>>>>>>> it hashed out.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> I'm glad you prefer Serde*s*, since having multiple
> > > > >>>>>>> different
> > > > >>>>>>>>>>>> classes
> > > > >>>>>>>>>>>>>> with
> > > > >>>>>>>>>>>>>>>> the same name leads to all kinds of trouble. "Serdes"
> > > > >>>>>>> seems
> > > > >>>>>>>>>>>> relatively
> > > > >>>>>>>>>>>>>>>> safe because people in the Scala lib won't be using
> > the
> > > > >>>>>>> Java
> > > > >>>>>>>>>>>> Serdes
> > > > >>>>>>>>>>>>>> class,
> > > > >>>>>>>>>>>>>>>> and they won't be using the deprecated and
> > non-deprecated
> > > > >>>>>>> one
> > > > >>>>>>>>> at
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>> same time.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Thank again,
> > > > >>>>>>>>>>>>>>>> -John
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On Thu, May 28, 2020, at 02:21, Yuriy Badalyantc
> > wrote:
> > > > >>>>>>>>>>>>>>>>> Ok, I understood you, John. I wasn't sure about kafka
> > > > >>>>>>>>> deprecation
> > > > >>>>>>>>>>>>>> policy
> > > > >>>>>>>>>>>>>>>>> and thought that the full cycle could be done with
> > 2.7
> > > > >>>>>>>>> version.
> > > > >>>>>>>>>>>>>> Waiting for
> > > > >>>>>>>>>>>>>>>>> 3.0 is too much, I agree with it.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> So, I think creating one more `Serdes` in another
> > > > >>>>>>> package is
> > > > >>>>>>>>> our
> > > > >>>>>>>>>>>> way.
> > > > >>>>>>>>>>>>>> I
> > > > >>>>>>>>>>>>>>>>> suggest one of the following:
> > > > >>>>>>>>>>>>>>>>> 1. `org.apache.kafka.streams.scala.serde.Serdes`
> > > > >>>>>>>>>>>>>>>>> 2.
> > `org.apache.kafka.streams.scala.serialization.Serdes`
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> About `Serde` vs `Serdes`. I'm strongly against
> > `Serde`
> > > > >>>>>>>>> because
> > > > >>>>>>>>>>>> it
> > > > >>>>>>>>>>>>>> would
> > > > >>>>>>>>>>>>>>>>> lead to a new name clash with the
> > > > >>>>>>>>>>>>>>>>> `org.apache.kafka.common.serialization.Serde`.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> - Yuriy
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> On Thu, May 28, 2020 at 11:12 AM John Roesler <
> > > > >>>>>>>>>>>> vvcephei@apache.org>
> > > > >>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Hi Yuriy,
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Thanks for the clarification.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> I guess my concern is twofold:
> > > > >>>>>>>>>>>>>>>>>> 1. We typically leave deprecated methods in place
> > for at
> > > > >>>>>>>>> least a
> > > > >>>>>>>>>>>>>> major
> > > > >>>>>>>>>>>>>>>>>> release cycle before removing them, so it would seem
> > > > >>>>>>> abrupt
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>> have a
> > > > >>>>>>>>>>>>>>>>>> deprecation period of only one minor release. If we
> > > > >>>>>>> follow
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>> same
> > > > >>>>>>>>>>>>>> pattern
> > > > >>>>>>>>>>>>>>>>>> here, it would take over a year to finish this KIP.
> > > > >>>>>>>>>>>>>>>>>> 2. It doesn’t seem like there is a nonbreaking
> > > > >>>>>>> deprecation
> > > > >>>>>>>>> path
> > > > >>>>>>>>>>>> at
> > > > >>>>>>>>>>>>>> all if
> > > > >>>>>>>>>>>>>>>>>> people enumerate their imports (if they don’t use a
> > > > >>>>>>>>> wildcard).
> > > > >>>>>>>>>>>> In
> > > > >>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>> case, they would have no path to implicitly use the
> > > > >>>>>>> newly
> > > > >>>>>>>>> named
> > > > >>>>>>>>>>>>>> serdes, and
> > > > >>>>>>>>>>>>>>>>>> therefore they would have no way to avoid
> > continuing to
> > > > >>>>>>> use
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>>>> deprecated
> > > > >>>>>>>>>>>>>>>>>> ones.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Since you mentioned that your reason is mainly the
> > > > >>>>>>>>> preference
> > > > >>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>> the name
> > > > >>>>>>>>>>>>>>>>>> “Serde” or “Serdes”, can we explore just using one
> > of
> > > > >>>>>>> those?
> > > > >>>>>>>>>>>> Would
> > > > >>>>>>>>>>>>>> it cause
> > > > >>>>>>>>>>>>>>>>>> some kind of conflict to use
> > > > >>>>>>>>>>>> org.apache.kafka.streams.scala.Serde or
> > > > >>>>>>>>>>>>>> to use
> > > > >>>>>>>>>>>>>>>>>> Serdes in a different package, like
> > > > >>>>>>>>>>>>>>>>>> org.apache.kafka.streams.scala.implicit.Serdes?
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> I empathize with this desire. I faced the same
> > dilemma
> > > > >>>>>>> when
> > > > >>>>>>>>> I
> > > > >>>>>>>>>>>> wanted
> > > > >>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>> replace Processor but keep the class name in
> > KIP-478. I
> > > > >>>>>>>>> wound up
> > > > >>>>>>>>>>>>>> creating a
> > > > >>>>>>>>>>>>>>>>>> new package for the new Processor.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> On Wed, May 27, 2020, at 22:20, Yuriy Badalyantc
> > wrote:
> > > > >>>>>>>>>>>>>>>>>>> Hi John,
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> I'm stick with the
> > > > >>>>>>> `org.apache.kafka.streams.scala.Serdes`
> > > > >>>>>>>>>>>> because
> > > > >>>>>>>>>>>>>> it's
> > > > >>>>>>>>>>>>>>>>>>> sort of conventional in the scala community. If you
> > > > >>>>>>> have a
> > > > >>>>>>>>>>>> typeclass
> > > > >>>>>>>>>>>>>>>>>> `Foo`,
> > > > >>>>>>>>>>>>>>>>>>> you probably will search `Foo` related stuff in the
> > > > >>>>>>> `Foo`
> > > > >>>>>>>>> or
> > > > >>>>>>>>>>>> maybe
> > > > >>>>>>>>>>>>>> `Foos`
> > > > >>>>>>>>>>>>>>>>>>> (plural). All other places are far less
> > discoverable
> > > > >>>>>>> for
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>>>> developers.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> I agree that the migration path is a bit complex
> > for
> > > > >>>>>>> such
> > > > >>>>>>>>>>>> change.
> > > > >>>>>>>>>>>>>> But I
> > > > >>>>>>>>>>>>>>>>>>> think it's more important to provide good developer
> > > > >>>>>>>>> experience
> > > > >>>>>>>>>>>> than
> > > > >>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>> simplify migration. Also, I think it's debatable
> > which
> > > > >>>>>>>>>>>> migration
> > > > >>>>>>>>>>>>>> path is
> > > > >>>>>>>>>>>>>>>>>>> better for library users. If we would create, for
> > > > >>>>>>> example,
> > > > >>>>>>>>>>>>>> `Serdes2`,
> > > > >>>>>>>>>>>>>>>>>>> library users will have to modify their code if
> > they
> > > > >>>>>>> used
> > > > >>>>>>>>> any
> > > > >>>>>>>>>>>> part
> > > > >>>>>>>>>>>>>> of the
> > > > >>>>>>>>>>>>>>>>>>> old `Serde`. With my approach, most of the old code
> > > > >>>>>>> will
> > > > >>>>>>>>> still
> > > > >>>>>>>>>>>> work
> > > > >>>>>>>>>>>>>>>>>> without
> > > > >>>>>>>>>>>>>>>>>>> changes. Only explicit usage of implicits will
> > need to
> > > > >>>>>>> be
> > > > >>>>>>>>> fixed
> > > > >>>>>>>>>>>>>> (because
> > > > >>>>>>>>>>>>>>>>>>> names will be changed, and old names will be
> > > > >>>>>>> deprecated).
> > > > >>>>>>>>>>>> Wildcard
> > > > >>>>>>>>>>>>>>>>>> imports
> > > > >>>>>>>>>>>>>>>>>>> will work without changes and will not lead to a
> > name
> > > > >>>>>>>>> clash.
> > > > >>>>>>>>>>>>>> Moreover,
> > > > >>>>>>>>>>>>>>>>>> many
> > > > >>>>>>>>>>>>>>>>>>> users may not notice name clash problems. And with
> > my
> > > > >>>>>>>>> migration
> > > > >>>>>>>>>>>>>> path,
> > > > >>>>>>>>>>>>>>>>>> they
> > > > >>>>>>>>>>>>>>>>>>> will not notice any changes at all.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> - Yuriy
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> On Thu, May 28, 2020 at 7:48 AM John Roesler <
> > > > >>>>>>>>>>>> vvcephei@apache.org>
> > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Hi Yuriy,
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Thanks for the reply. I guess I've been out of the
> > > > >>>>>>> Scala
> > > > >>>>>>>>> game
> > > > >>>>>>>>>>>> for a
> > > > >>>>>>>>>>>>>>>>>>>> while; all this summoner business is totally new
> > to
> > > > >>>>>>> me.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> I think I followed the rationale you provided,
> > but I
> > > > >>>>>>> still
> > > > >>>>>>>>>>>> don't
> > > > >>>>>>>>>>>>>> see
> > > > >>>>>>>>>>>>>>>>>>>> why you can't implement your whole plan in a new
> > > > >>>>>>> class.
> > > > >>>>>>>>> What
> > > > >>>>>>>>>>>>>>>>>>>> is special about the existing Serdes class?
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>>>> -John
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> On Tue, May 19, 2020, at 01:18, Yuriy Badalyantc
> > > > >>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>> Hi John,
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Your suggestion looks interesting. I think it's
> > > > >>>>>>>>> technically
> > > > >>>>>>>>>>>>>> doable.
> > > > >>>>>>>>>>>>>>>>>> But
> > > > >>>>>>>>>>>>>>>>>>>> I'm
> > > > >>>>>>>>>>>>>>>>>>>>> not sure that this is the better solution. I will
> > > > >>>>>>> try to
> > > > >>>>>>>>>>>> explain.
> > > > >>>>>>>>>>>>>>>>>> From
> > > > >>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>> scala developers' perspective, `Serde` looks
> > really
> > > > >>>>>>> like
> > > > >>>>>>>>> a
> > > > >>>>>>>>>>>>>> typeclass.
> > > > >>>>>>>>>>>>>>>>>>>>> Typical typeclass in pure scala will look like
> > this:
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> ```
> > > > >>>>>>>>>>>>>>>>>>>>> trait Serde[A] {
> > > > >>>>>>>>>>>>>>>>>>>>> def serialize(data: A): Array[Byte]
> > > > >>>>>>>>>>>>>>>>>>>>> def deserialize(data: Array[Byte]): A
> > > > >>>>>>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>>>>> object Serde extends DefaultSerdes {
> > > > >>>>>>>>>>>>>>>>>>>>> // "summoner" function. With this I can write
> > > > >>>>>>>>> `Serde[A]`
> > > > >>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>>>> serde
> > > > >>>>>>>>>>>>>>>>>>>>> will be implicitly summonned.
> > > > >>>>>>>>>>>>>>>>>>>>> def apply[A](implicit ev: Serde[A]): Serde[A]
> > = ev
> > > > >>>>>>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> trait DefaultSerdes {
> > > > >>>>>>>>>>>>>>>>>>>>> // default instances here
> > > > >>>>>>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>>>>> ```
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Usage example (note, that there are no wildcards
> > > > >>>>>>> imports
> > > > >>>>>>>>>>>> here):
> > > > >>>>>>>>>>>>>>>>>>>>> ```
> > > > >>>>>>>>>>>>>>>>>>>>> object Main extends App {
> > > > >>>>>>>>>>>>>>>>>>>>> import Serde // not wildcard import
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> // explicit summonning:
> > > > >>>>>>>>>>>>>>>>>>>>> val stringSerde = Serde[String] // using
> > summoner
> > > > >>>>>>>>>>>>>>>>>>>>> stringSerde.serialize(???)
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> // implicit summonning
> > > > >>>>>>>>>>>>>>>>>>>>> def serialize[A: Serde](a: A) = {
> > > > >>>>>>>>>>>>>>>>>>>>> Serde[A].serialize(a) // summoner again
> > > > >>>>>>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>>>>> serialize("foo")
> > > > >>>>>>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>>>>> ```
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Examples are pretty silly, but I just want to
> > show
> > > > >>>>>>> common
> > > > >>>>>>>>>>>>>> patterns of
> > > > >>>>>>>>>>>>>>>>>>>>> working with typeclasses in scala. All default
> > > > >>>>>>> instances
> > > > >>>>>>>>> in
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>> usage
> > > > >>>>>>>>>>>>>>>>>>>>> examples are found using implicits searching
> > > > >>>>>>> mechanism.
> > > > >>>>>>>>> Scala
> > > > >>>>>>>>>>>>>>>>>> compiler
> > > > >>>>>>>>>>>>>>>>>>>>> searches implicits in a lot of places. Including
> > > > >>>>>>>>> companion
> > > > >>>>>>>>>>>>>> objects.
> > > > >>>>>>>>>>>>>>>>>> In my
> > > > >>>>>>>>>>>>>>>>>>>>> examples compiler will found `Serde[String]`
> > > > >>>>>>> instance in
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>> companion
> > > > >>>>>>>>>>>>>>>>>>>>> object of `Serde` typeclass.
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Also, I want to pay attention to the summoner
> > > > >>>>>>> function.
> > > > >>>>>>>>> It
> > > > >>>>>>>>>>>> makes
> > > > >>>>>>>>>>>>>>>>>> usage of
> > > > >>>>>>>>>>>>>>>>>>>>> typeclasses very neat and clear.
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> The example above was the example of the perfect
> > > > >>>>>>> solution
> > > > >>>>>>>>>>>> for the
> > > > >>>>>>>>>>>>>>>>>> scala
> > > > >>>>>>>>>>>>>>>>>>>>> developers. But this solution requires to create
> > > > >>>>>>> separate
> > > > >>>>>>>>>>>> `Serde`
> > > > >>>>>>>>>>>>>>>>>>>>> typeclass, to make all this implicit searching
> > stuff
> > > > >>>>>>>>> works. I
> > > > >>>>>>>>>>>>>> don't
> > > > >>>>>>>>>>>>>>>>>> think
> > > > >>>>>>>>>>>>>>>>>>>>> that it worth it, because a lot of code should be
> > > > >>>>>>>>>>>> reimplemented
> > > > >>>>>>>>>>>>>> using
> > > > >>>>>>>>>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>>>>>>> new typeclass. But the main point of my example
> > is to
> > > > >>>>>>>>> show
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>> perfect
> > > > >>>>>>>>>>>>>>>>>>>>> solution. And I think we should strive to provide
> > > > >>>>>>>>> developer
> > > > >>>>>>>>>>>>>>>>>> experience
> > > > >>>>>>>>>>>>>>>>>>>>> close to this.
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> It's a bit out of the scope of my KIP, but I
> > have a
> > > > >>>>>>> plan
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>> make
> > > > >>>>>>>>>>>>>>>>>>>>> `org.apache.kafka.streams.scala.Serdes` more
> > closer
> > > > >>>>>>> to
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>>>> solution
> > > > >>>>>>>>>>>>>>>>>>>> above.
> > > > >>>>>>>>>>>>>>>>>>>>> It could be done in 2 steps:
> > > > >>>>>>>>>>>>>>>>>>>>> 1. Fix implicit names.
> > > > >>>>>>>>>>>>>>>>>>>>> 2. Add summoner function.
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> And with this scala developers will be able to
> > write
> > > > >>>>>>>>> almost
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>> same
> > > > >>>>>>>>>>>>>>>>>> code
> > > > >>>>>>>>>>>>>>>>>>>>> as in the example above:
> > > > >>>>>>>>>>>>>>>>>>>>> ```
> > > > >>>>>>>>>>>>>>>>>>>>> object Main extends App {
> > > > >>>>>>>>>>>>>>>>>>>>> import org.apache.kafka.streams.scala.Serdes
> > // not
> > > > >>>>>>>>>>>> wildcard
> > > > >>>>>>>>>>>>>> import
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> val stringSerde = Serdes[String]
> > > > >>>>>>>>>>>>>>>>>>>>> stringSerde.serialize(???)
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> def serialize[A: Serde](a: A) = {
> > > > >>>>>>>>>>>>>>>>>>>>> Serdes[A].serialize(a)
> > > > >>>>>>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>>>>> serialize("foo")
> > > > >>>>>>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>>>>> ```
> > > > >>>>>>>>>>>>>>>>>>>>> Of course, wildcard import will still work.
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Other names will make this new entity (containing
> > > > >>>>>>> default
> > > > >>>>>>>>>>>>>> implicits)
> > > > >>>>>>>>>>>>>>>>>> less
> > > > >>>>>>>>>>>>>>>>>>>>> discoverable. And summoner usage, in this case,
> > will
> > > > >>>>>>> look
> > > > >>>>>>>>>>>> weird:
> > > > >>>>>>>>>>>>>>>>>>>>> ```
> > > > >>>>>>>>>>>>>>>>>>>>> object Main extends App {
> > > > >>>>>>>>>>>>>>>>>>>>> import
> > > > >>>>>>> org.apache.kafka.streams.scala.DefaultSerdes //
> > > > >>>>>>>>> not
> > > > >>>>>>>>>>>>>> wildcard
> > > > >>>>>>>>>>>>>>>>>>>> import
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> val stringSerde = DefaultSerdes[String]
> > > > >>>>>>>>>>>>>>>>>>>>> stringSerde.serialize(???)
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> def serialize[A: Serde](a: A) = {
> > > > >>>>>>>>>>>>>>>>>>>>> DefaultSerdes[A].serialize(a)
> > > > >>>>>>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>>>>> serialize("foo")
> > > > >>>>>>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>>>>> ```
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> So, I think it's more important to provide a
> > solid
> > > > >>>>>>> and
> > > > >>>>>>>>>>>> familiar
> > > > >>>>>>>>>>>>>>>>>> developer
> > > > >>>>>>>>>>>>>>>>>>>>> experience for the scala developer. And renaming
> > (or
> > > > >>>>>>>>>>>> creating a
> > > > >>>>>>>>>>>>>> new
> > > > >>>>>>>>>>>>>>>>>>>>> version) of `Serdes` will not help here.
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> -Yuriy
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> On Tue, May 19, 2020 at 11:56 AM John Roesler <
> > > > >>>>>>>>>>>>>> vvcephei@apache.org>
> > > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Hi Yuriy,
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Thanks so much for the KIP! I didn’t anticipate
> > the
> > > > >>>>>>>>> problem
> > > > >>>>>>>>>>>> you
> > > > >>>>>>>>>>>>>>>>>> laid
> > > > >>>>>>>>>>>>>>>>>>>> out
> > > > >>>>>>>>>>>>>>>>>>>>>> in the KIP, but I find it very plausible.
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Thanks for pushing back on the “convention” and
> > > > >>>>>>> raising
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>>>> issue,
> > > > >>>>>>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>>>>>>> also volunteering a solution!
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> I’m wondering if we can “fix” it in one shot by
> > just
> > > > >>>>>>>>>>>> deprecating
> > > > >>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>> whole
> > > > >>>>>>>>>>>>>>>>>>>>>> Serdes class and replacing it with a new one
> > > > >>>>>>> containing
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>> defs
> > > > >>>>>>>>>>>>>>>>>> you
> > > > >>>>>>>>>>>>>>>>>>>>>> proposed. Then, people could just switch their
> > > > >>>>>>> import to
> > > > >>>>>>>>>>>> the new
> > > > >>>>>>>>>>>>>>>>>> one.
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Of course the new class needs to have a
> > different
> > > > >>>>>>> name,
> > > > >>>>>>>>>>>> which is
> > > > >>>>>>>>>>>>>>>>>>>> always a
> > > > >>>>>>>>>>>>>>>>>>>>>> challenge in situations like this, so I might
> > just
> > > > >>>>>>>>> throw out
> > > > >>>>>>>>>>>>>>>>>>>> ImplicitSerdes
> > > > >>>>>>>>>>>>>>>>>>>>>> as an option.
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Do you think this would work?
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Thanks again,
> > > > >>>>>>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> On Mon, May 18, 2020, at 23:35, Yuriy Badalyantc
> > > > >>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>> Hi,
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> I would like to propose KIP-616 to fix naming
> > > > >>>>>>> clash in
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>> kafka
> > > > >>>>>>>>>>>>>>>>>>>>>>> streams scala API:
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>
> > > > >>
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-616%3A+Rename+implicit+Serdes+instances+in+kafka-streams-scala
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Looking forward to your feedback.
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> -Yuriy
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Attachments:
> > > > >>>>>>>>>>>>>>> * signature.asc
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > > >
> > > >
> > > >
> > >
> >
>