You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Debasish Ghosh <de...@lightbend.com> on 2018/03/16 15:25:44 UTC
[DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Hi -
A new KIP, KIP-270 is up for discussion:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
The relevant JIRA issue: https://issues.apache.org/jira/browse/KAFKA-6670
The library as proposed in the KIP has been implemented at
https://github.com/lightbend/kafka-streams-scala and the current release is
0.2.0 (https://github.com/lightbend/kafka-streams-scala/releases/tag/v0.2.0).
We at Lightbend has been using it since quite some time now.
regards.
--
Debasish Ghosh
Principal Engineer
Twitter: @debasishg
Blog: http://debasishg.blogspot.com
Code: https://github.com/debasishg
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Debasish Ghosh <de...@lightbend.com>.
Submitted the PR .. https://github.com/apache/kafka/pull/4756 and updated
the KIP
https://cwiki.apache.org/confluence/display/KAFKA/KIP-270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
regards.
On Tue, Mar 20, 2018 at 4:06 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Matthias,
>
> You have to choose between the package and class names when it comes to
> adding a suffix (or prefix). I think the package name is the lesser of two
> evils, but it would be interesting to know what others think.
>
> I agree that we should include the information in the KIP (whatever we
> decide).
>
> Ismael
>
> On Mon, Mar 19, 2018 at 3:30 PM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
> > About the package name.
> >
> > Would it be better/cleaner to omit the `scala` sub-package? Or is this
> > required to avoid naming conflicts with the Java classes? If yes, please
> > point it out in the KIP.
> >
> >
> > -Matthias
> >
> > On 3/19/18 11:24 AM, Debasish Ghosh wrote:
> > > Cool .. so the changes that u suggested e.g the WordCount example,
> > package
> > > name changes etc. will be in the PR only and not in the KIP document ..
> > >
> > > Or we keep the KIP document updated as well ?
> > >
> > > regards.
> > >
> > > On Mon, 19 Mar 2018 at 11:46 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > >> Hi Debasish,
> > >>
> > >> You can work on the PR in parallel to the KIP discussion so that
> people
> > can
> > >> start reviewing it. The only restriction is that the PR cannot be
> merged
> > >> until the KIP is accepted.
> > >>
> > >>
> > >> Guozhang
> > >>
> > >>
> > >> On Mon, Mar 19, 2018 at 11:09 AM, Debasish Ghosh <
> > >> debasish.ghosh@lightbend.com> wrote:
> > >>
> > >>> Based on this discussion I have a question ..
> > >>>
> > >>> For the actual implementation we need to have a PR on
> > >>> https://github.com/apache/kafka and the new library will be in the
> > >> package
> > >>> structure org.apache.kafka.streams.scala. My question is, is this PR
> > part
> > >>> of the KIP ? Or we work on the PR once the KIP is accepted ..
> > >>>
> > >>> regards.
> > >>>
> > >>> On Mon, Mar 19, 2018 at 11:28 PM, Debasish Ghosh <
> > >>> debasish.ghosh@lightbend.com> wrote:
> > >>>
> > >>>> Thanks Guozhang for the comments .. we will start working on them ..
> > >>>>
> > >>>> regards.
> > >>>>
> > >>>> On Mon, Mar 19, 2018 at 11:02 PM, Guozhang Wang <wangguoz@gmail.com
> >
> > >>>> wrote:
> > >>>>
> > >>>>> And one more comment about the type safety:
> > >>>>>
> > >>>>> 7. I'm in favor of the approach of enforcing type safety at compile
> > >>> time,
> > >>>>> since with Scala's strong typing system I think it makes more sense
> > to
> > >>> get
> > >>>>> rid of config-based serde specifications that can cause runtime
> error
> > >> in
> > >>>>> the Scala wrapper.
> > >>>>>
> > >>>>>
> > >>>>> Guozhang
> > >>>>>
> > >>>>>
> > >>>>> On Mon, Mar 19, 2018 at 10:28 AM, Guozhang Wang <
> wangguoz@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hello Debasish,
> > >>>>>>
> > >>>>>> Thanks for the KIP. Here are some comments:
> > >>>>>>
> > >>>>>> 1. About the naming: I'd also vote for option 2 and enforce users
> to
> > >>>>>> rename when import since this scenario seems rare to me.
> > >>>>>>
> > >>>>>> 2. About the dependency: since this KIP can only be merged as
> early
> > >> as
> > >>>>>> 1.2, it means that for users who wants to use this artifact, say
> for
> > >>>>>> version 1.2, she would need to bring in "kafka-streams" version
> 1.2.
> > >>> In
> > >>>>>> addition, if we change the Java API in the future versions we'd
> also
> > >>>>> need
> > >>>>>> to update the Scala wrapper at the same version as well, so in
> other
> > >>>>> words
> > >>>>>> "kafka-streams-scala" version X have to be with "kafka-streams"
> > >>> version
> > >>>>> X
> > >>>>>> anyways. So I'd suggest we remove the version number in
> > >>>>> "kafka-streams-scala
> > >>>>>> only depends on the Scala standard library and Kafka Streams" as
> it
> > >>> may
> > >>>>>> be confusing to readers.
> > >>>>>>
> > >>>>>> 3. For the KIP discussion, it need be based on the proposed state
> of
> > >>>>> this
> > >>>>>> project in AK. More specifically, for the groupId in maven /
> gradle,
> > >>> it
> > >>>>>> need to be "org.apache.kafka"; for the version, it need to be
> Kafka
> > >>>>> release
> > >>>>>> versions, e.g. 1.2.0.
> > >>>>>>
> > >>>>>> 4. In "New or Changed Public Interfaces" section, it does not
> > >> mention
> > >>>>> the
> > >>>>>> serde classes like "DefaultSerdes", but I think they should also
> be
> > >>>>>> considered public APIs as users would most likely import these in
> > >> her
> > >>>>>> implementation.
> > >>>>>>
> > >>>>>> 5. Could you also list the changes you'd propose to made to
> > >>> build.gradle
> > >>>>>> in kafka for adding this artifact? More details will help readers
> to
> > >>>>> better
> > >>>>>> understand your proposal.
> > >>>>>>
> > >>>>>> 6. I think it'd will be good to have a WordCount example code as
> > >> part
> > >>> of
> > >>>>>> this KIP, to illustrate how to code in this Scala wrapper, e.g. in
> > >>>>>> o.a.k.scala.examples. But for this class we probably do not need
> to
> > >>>>> have a
> > >>>>>> separate artifact for it as we did in kafka-streams-examples.
> > >>>>>>
> > >>>>>>
> > >>>>>> Guozhang
> > >>>>>>
> > >>>>>>
> > >>>>>> On Mon, Mar 19, 2018 at 9:16 AM, Ted Yu <yu...@gmail.com>
> > >> wrote:
> > >>>>>>
> > >>>>>>> I agree with Sean on name unification.
> > >>>>>>>
> > >>>>>>> +1 to option 2.
> > >>>>>>>
> > >>>>>>> On Mon, Mar 19, 2018 at 7:23 AM, Sean Glover <
> > >>>>> sean.glover@lightbend.com>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Type names: I vote for option 2. The user must explicitly add a
> > >>>>>>> dependency
> > >>>>>>>> to this library and the wrapper types are in a different
> package.
> > >>> It
> > >>>>>>> seems
> > >>>>>>>> reasonable to expect them to do an import rename if there's a
> > >> need
> > >>> to
> > >>>>>>> drop
> > >>>>>>>> down to the Java API.
> > >>>>>>>>
> > >>>>>>>> Test Utils: The test utils in kafka-streams-scala are nice and
> > >>> lean,
> > >>>>> but
> > >>>>>>>> I'm not sure if it provides much more value than other options
> > >> that
> > >>>>>>> exist
> > >>>>>>>> in the community. There's an embedded Kafka/ZK project
> > >>>>> implementation
> > >>>>>>> for
> > >>>>>>>> ScalaTest that's popular and active: manub/scalatest-embedded-
> > >>> kakfa.
> > >>>>>>> It
> > >>>>>>>> implies you must also use ScalaTest, which I acknowledge isn't
> > >>>>>>> everyone's
> > >>>>>>>> first choice for Scala test framework, but it probably is one
> of,
> > >>> if
> > >>>>> not
> > >>>>>>>> the most, popular library. It includes a DSL for Kafka Streams
> > >>>>> too. If
> > >>>>>>>> this KIP is accepted then perhaps a PR to that project could be
> > >>> made
> > >>>>> to
> > >>>>>>>> support the new wrapper implementations.
> > >>>>>>>>
> > >>>>>>>> https://github.com/manub/scalatest-embedded-kafka#
> > >>>>>>>> scalatest-embedded-kafka-streams
> > >>>>>>>>
> > >>>>>>>> Sean
> > >>>>>>>>
> > >>>>>>>> On Sun, Mar 18, 2018 at 4:05 AM, Debasish Ghosh <
> > >>>>>>>> debasish.ghosh@lightbend.com> wrote:
> > >>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> Should this be 1.2 (maybe it's even better to not put any
> > >>>>>>> version at
> > >>>>>>>>>> all)
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Actually wanted to emphasize that the support is from 1.0.0
> > >>>>> onwards ..
> > >>>>>>>>> Should we make that explicit ? Like ..
> > >>>>>>>>>
> > >>>>>>>>> kafka-streams-scala only depends on the Scala standard library
> > >>> and
> > >>>>>>> Kafka
> > >>>>>>>>>> Streams 1.0.0+.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> In 1.1 release, we add a new module `kafka-streams-test-utils`
> > >>> to
> > >>>>>>>> simplify
> > >>>>>>>>>> testing for Kafka Streams applications. Are those test utils
> > >>>>>>> suitable
> > >>>>>>>> for
> > >>>>>>>>>> Scala users or should we add Scala wrappers for those, too?
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> I will check up and let you know ..
> > >>>>>>>>>
> > >>>>>>>>> Also I am not clear about the decision on renaming of Scala
> > >>>>>>> abstractions.
> > >>>>>>>>> Can we have a consensus on this ? Here's the summary ..
> > >>>>>>>>>
> > >>>>>>>>> *Option 1:* Keep names separate (KStream for Java class,
> > >> KStreamS
> > >>>>> for
> > >>>>>>>>> Scala). No renaming of imports required.
> > >>>>>>>>> *Option 2:* Unify names (KStream for Java and Scala class
> > >> names).
> > >>>>> No
> > >>>>>>>>> conflict since they will reside in different packages. But if
> > >> we
> > >>>>> need
> > >>>>>>> to
> > >>>>>>>>> use both abstractions, renaming of imports are required. But
> > >>> again,
> > >>>>>>> this
> > >>>>>>>>> may not be a too frequent use case.
> > >>>>>>>>>
> > >>>>>>>>> Suggestions ?
> > >>>>>>>>>
> > >>>>>>>>> regards.
> > >>>>>>>>>
> > >>>>>>>>> On Sat, Mar 17, 2018 at 3:07 AM, Matthias J. Sax <
> > >>>>>>> matthias@confluent.io>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Thanks a lot for the KIP! Two questions:
> > >>>>>>>>>>
> > >>>>>>>>>> 1) the KIP states:
> > >>>>>>>>>>
> > >>>>>>>>>>> kafka-streams-scala only depends on the Scala standard
> > >>> library
> > >>>>> and
> > >>>>>>>>> Kafka
> > >>>>>>>>>> Streams 1.0.0.
> > >>>>>>>>>>
> > >>>>>>>>>> Should this be 1.2 (maybe it's even better to not put any
> > >>>>> version
> > >>>>>>> at
> > >>>>>>>>> all)
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> 2) In 1.1 release, we add a new module
> > >>>>> `kafka-streams-test-utils` to
> > >>>>>>>>>> simplify testing for Kafka Streams applications. Are those
> > >> test
> > >>>>>>> utils
> > >>>>>>>>>> suitable for Scala users or should we add Scala wrappers for
> > >>>>> those,
> > >>>>>>>> too?
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> -Matthias
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On 3/16/18 11:10 AM, Ted Yu wrote:
> > >>>>>>>>>>> Import renames seem to be fine.
> > >>>>>>>>>>>
> > >>>>>>>>>>> The class names with trailing 'S' look clean.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Cheers
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Fri, Mar 16, 2018 at 11:04 AM, Ismael Juma <
> > >>>>> ismael@juma.me.uk>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> If this is rare (as it sounds), relying on import renames
> > >>>>> seems
> > >>>>>>> fine
> > >>>>>>>>> to
> > >>>>>>>>>> me.
> > >>>>>>>>>>>> Let's see what others think.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Ismael
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
> > >>>>>>>>>>>> debasish.ghosh@lightbend.com> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> I am not sure if this is practical or not. But
> > >>> theoretically
> > >>>>> a
> > >>>>>>> user
> > >>>>>>>>> may
> > >>>>>>>>>>>>> want to extract the unsafe Java abstraction from the
> > >> Scala
> > >>>>> ones
> > >>>>>>> and
> > >>>>>>>>> use
> > >>>>>>>>>>>>> Java APIs on them .. e.g.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> val userClicksStream: KStreamS[String, Long] =
> > >>>>>>>>>>>>> builder.stream(userClicksTopic) // Scala abstraction
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> val jStream: KStream[String, Long] =
> > >> userClicksStream.inner
> > >>>>> //
> > >>>>>>>>>> publishes
> > >>>>>>>>>>>>> the underlying Java abstraction
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> //.. work with Java, may be pass to some function written
> > >>> in
> > >>>>>>> Java
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I do realize this is somewhat of a convoluted use case
> > >> and
> > >>>>> may
> > >>>>>>> not
> > >>>>>>>> be
> > >>>>>>>>>>>>> practically useful ..
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Otherwise we can very well work on the suggested approach
> > >>> of
> > >>>>>>>> unifying
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> names ..
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> regards.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <
> > >>>>>>> ismael@juma.me.uk>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> What does "mixed mode application" mean? What are the
> > >>> cases
> > >>>>>>> where
> > >>>>>>>> a
> > >>>>>>>>>>>> user
> > >>>>>>>>>>>>>> would want to use both APIs? I think that would help
> > >>>>> understand
> > >>>>>>>> the
> > >>>>>>>>>>>>>> reasoning.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>> Ismael
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
> > >>>>>>>>>>>>>> debasish.ghosh@lightbend.com> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi Damian -
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> We could. But in case the user wants to use both Scala
> > >>> and
> > >>>>>>> Java
> > >>>>>>>>> APIs
> > >>>>>>>>>>>>> (may
> > >>>>>>>>>>>>>>> be for some mixed mode application), won't that be
> > >>>>> confusing ?
> > >>>>>>>> She
> > >>>>>>>>>>>> will
> > >>>>>>>>>>>>>>> have to do something like ..
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> import o.a.k.s.scala.{KStream => KStreamS}
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> to rename Scala imports or the other way round for
> > >>> imported
> > >>>>>>> Java
> > >>>>>>>>>>>>> classes.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> regards.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <
> > >>>>>>>> damian.guy@gmail.com>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi Debasish,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks for the KIP - will be a great addition to
> > >>> streams.
> > >>>>>>> I've
> > >>>>>>>>> only
> > >>>>>>>>>>>>>> had a
> > >>>>>>>>>>>>>>>> quick scan, but seeing as the Scala classes are going
> > >> to
> > >>>>> be
> > >>>>>>> in
> > >>>>>>>>>>>> their
> > >>>>>>>>>>>>>> own
> > >>>>>>>>>>>>>>>> package could we drop the S at the end of the class
> > >>> names?
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>> Damian
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> > >>>>>>>>>>>>>>> debasish.ghosh@lightbend.com>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi -
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> A new KIP, KIP-270 is up for discussion:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >>>>>>>>>>>>>>>> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> The relevant JIRA issue: https://issues.apache.org/
> > >>>>>>>>>>>>>>>> jira/browse/KAFKA-6670
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> The library as proposed in the KIP has been
> > >> implemented
> > >>>>> at
> > >>>>>>>>>>>>>>>>> https://github.com/lightbend/kafka-streams-scala and
> > >>> the
> > >>>>>>>> current
> > >>>>>>>>>>>>>>> release
> > >>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>> 0.2.0 (
> > >>>>>>>>>>>>>>>>> https://github.com/lightbend/k
> > >>>>> afka-streams-scala/releases/
> > >>>>>>>>>>>>> tag/v0.2.0
> > >>>>>>>>>>>>>> ).
> > >>>>>>>>>>>>>>>>> We at Lightbend has been using it since quite some
> > >> time
> > >>>>> now.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> regards.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>>> Debasish Ghosh
> > >>>>>>>>>>>>>>>>> Principal Engineer
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Twitter: @debasishg
> > >>>>>>>>>>>>>>>>> Blog: http://debasishg.blogspot.com
> > >>>>>>>>>>>>>>>>> Code: https://github.com/debasishg
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>> Debasish Ghosh
> > >>>>>>>>>>>>>>> Principal Engineer
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Twitter: @debasishg
> > >>>>>>>>>>>>>>> Blog: http://debasishg.blogspot.com
> > >>>>>>>>>>>>>>> Code: https://github.com/debasishg
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> --
> > >>>>>>>>>>>>> Debasish Ghosh
> > >>>>>>>>>>>>> Principal Engineer
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Twitter: @debasishg
> > >>>>>>>>>>>>> Blog: http://debasishg.blogspot.com
> > >>>>>>>>>>>>> Code: https://github.com/debasishg
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>> Debasish Ghosh
> > >>>>>>>>> Principal Engineer
> > >>>>>>>>>
> > >>>>>>>>> Twitter: @debasishg
> > >>>>>>>>> Blog: http://debasishg.blogspot.com
> > >>>>>>>>> Code: https://github.com/debasishg
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> --
> > >>>>>>>> Senior Software Engineer, Lightbend, Inc.
> > >>>>>>>>
> > >>>>>>>> <http://lightbend.com>
> > >>>>>>>>
> > >>>>>>>> @seg1o <https://twitter.com/seg1o>, in/seanaglover
> > >>>>>>>> <https://www.linkedin.com/in/seanaglover/>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> -- Guozhang
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> -- Guozhang
> > >>>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Debasish Ghosh
> > >>>> Principal Engineer
> > >>>>
> > >>>> Twitter: @debasishg
> > >>>> Blog: http://debasishg.blogspot.com
> > >>>> Code: https://github.com/debasishg
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>> Debasish Ghosh
> > >>> Principal Engineer
> > >>>
> > >>> Twitter: @debasishg
> > >>> Blog: http://debasishg.blogspot.com
> > >>> Code: https://github.com/debasishg
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> >
> >
>
--
Debasish Ghosh
Principal Engineer
Twitter: @debasishg
Blog: http://debasishg.blogspot.com
Code: https://github.com/debasishg
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Ismael Juma <is...@juma.me.uk>.
Matthias,
You have to choose between the package and class names when it comes to
adding a suffix (or prefix). I think the package name is the lesser of two
evils, but it would be interesting to know what others think.
I agree that we should include the information in the KIP (whatever we
decide).
Ismael
On Mon, Mar 19, 2018 at 3:30 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> About the package name.
>
> Would it be better/cleaner to omit the `scala` sub-package? Or is this
> required to avoid naming conflicts with the Java classes? If yes, please
> point it out in the KIP.
>
>
> -Matthias
>
> On 3/19/18 11:24 AM, Debasish Ghosh wrote:
> > Cool .. so the changes that u suggested e.g the WordCount example,
> package
> > name changes etc. will be in the PR only and not in the KIP document ..
> >
> > Or we keep the KIP document updated as well ?
> >
> > regards.
> >
> > On Mon, 19 Mar 2018 at 11:46 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> >> Hi Debasish,
> >>
> >> You can work on the PR in parallel to the KIP discussion so that people
> can
> >> start reviewing it. The only restriction is that the PR cannot be merged
> >> until the KIP is accepted.
> >>
> >>
> >> Guozhang
> >>
> >>
> >> On Mon, Mar 19, 2018 at 11:09 AM, Debasish Ghosh <
> >> debasish.ghosh@lightbend.com> wrote:
> >>
> >>> Based on this discussion I have a question ..
> >>>
> >>> For the actual implementation we need to have a PR on
> >>> https://github.com/apache/kafka and the new library will be in the
> >> package
> >>> structure org.apache.kafka.streams.scala. My question is, is this PR
> part
> >>> of the KIP ? Or we work on the PR once the KIP is accepted ..
> >>>
> >>> regards.
> >>>
> >>> On Mon, Mar 19, 2018 at 11:28 PM, Debasish Ghosh <
> >>> debasish.ghosh@lightbend.com> wrote:
> >>>
> >>>> Thanks Guozhang for the comments .. we will start working on them ..
> >>>>
> >>>> regards.
> >>>>
> >>>> On Mon, Mar 19, 2018 at 11:02 PM, Guozhang Wang <wa...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> And one more comment about the type safety:
> >>>>>
> >>>>> 7. I'm in favor of the approach of enforcing type safety at compile
> >>> time,
> >>>>> since with Scala's strong typing system I think it makes more sense
> to
> >>> get
> >>>>> rid of config-based serde specifications that can cause runtime error
> >> in
> >>>>> the Scala wrapper.
> >>>>>
> >>>>>
> >>>>> Guozhang
> >>>>>
> >>>>>
> >>>>> On Mon, Mar 19, 2018 at 10:28 AM, Guozhang Wang <wa...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hello Debasish,
> >>>>>>
> >>>>>> Thanks for the KIP. Here are some comments:
> >>>>>>
> >>>>>> 1. About the naming: I'd also vote for option 2 and enforce users to
> >>>>>> rename when import since this scenario seems rare to me.
> >>>>>>
> >>>>>> 2. About the dependency: since this KIP can only be merged as early
> >> as
> >>>>>> 1.2, it means that for users who wants to use this artifact, say for
> >>>>>> version 1.2, she would need to bring in "kafka-streams" version 1.2.
> >>> In
> >>>>>> addition, if we change the Java API in the future versions we'd also
> >>>>> need
> >>>>>> to update the Scala wrapper at the same version as well, so in other
> >>>>> words
> >>>>>> "kafka-streams-scala" version X have to be with "kafka-streams"
> >>> version
> >>>>> X
> >>>>>> anyways. So I'd suggest we remove the version number in
> >>>>> "kafka-streams-scala
> >>>>>> only depends on the Scala standard library and Kafka Streams" as it
> >>> may
> >>>>>> be confusing to readers.
> >>>>>>
> >>>>>> 3. For the KIP discussion, it need be based on the proposed state of
> >>>>> this
> >>>>>> project in AK. More specifically, for the groupId in maven / gradle,
> >>> it
> >>>>>> need to be "org.apache.kafka"; for the version, it need to be Kafka
> >>>>> release
> >>>>>> versions, e.g. 1.2.0.
> >>>>>>
> >>>>>> 4. In "New or Changed Public Interfaces" section, it does not
> >> mention
> >>>>> the
> >>>>>> serde classes like "DefaultSerdes", but I think they should also be
> >>>>>> considered public APIs as users would most likely import these in
> >> her
> >>>>>> implementation.
> >>>>>>
> >>>>>> 5. Could you also list the changes you'd propose to made to
> >>> build.gradle
> >>>>>> in kafka for adding this artifact? More details will help readers to
> >>>>> better
> >>>>>> understand your proposal.
> >>>>>>
> >>>>>> 6. I think it'd will be good to have a WordCount example code as
> >> part
> >>> of
> >>>>>> this KIP, to illustrate how to code in this Scala wrapper, e.g. in
> >>>>>> o.a.k.scala.examples. But for this class we probably do not need to
> >>>>> have a
> >>>>>> separate artifact for it as we did in kafka-streams-examples.
> >>>>>>
> >>>>>>
> >>>>>> Guozhang
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Mar 19, 2018 at 9:16 AM, Ted Yu <yu...@gmail.com>
> >> wrote:
> >>>>>>
> >>>>>>> I agree with Sean on name unification.
> >>>>>>>
> >>>>>>> +1 to option 2.
> >>>>>>>
> >>>>>>> On Mon, Mar 19, 2018 at 7:23 AM, Sean Glover <
> >>>>> sean.glover@lightbend.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Type names: I vote for option 2. The user must explicitly add a
> >>>>>>> dependency
> >>>>>>>> to this library and the wrapper types are in a different package.
> >>> It
> >>>>>>> seems
> >>>>>>>> reasonable to expect them to do an import rename if there's a
> >> need
> >>> to
> >>>>>>> drop
> >>>>>>>> down to the Java API.
> >>>>>>>>
> >>>>>>>> Test Utils: The test utils in kafka-streams-scala are nice and
> >>> lean,
> >>>>> but
> >>>>>>>> I'm not sure if it provides much more value than other options
> >> that
> >>>>>>> exist
> >>>>>>>> in the community. There's an embedded Kafka/ZK project
> >>>>> implementation
> >>>>>>> for
> >>>>>>>> ScalaTest that's popular and active: manub/scalatest-embedded-
> >>> kakfa.
> >>>>>>> It
> >>>>>>>> implies you must also use ScalaTest, which I acknowledge isn't
> >>>>>>> everyone's
> >>>>>>>> first choice for Scala test framework, but it probably is one of,
> >>> if
> >>>>> not
> >>>>>>>> the most, popular library. It includes a DSL for Kafka Streams
> >>>>> too. If
> >>>>>>>> this KIP is accepted then perhaps a PR to that project could be
> >>> made
> >>>>> to
> >>>>>>>> support the new wrapper implementations.
> >>>>>>>>
> >>>>>>>> https://github.com/manub/scalatest-embedded-kafka#
> >>>>>>>> scalatest-embedded-kafka-streams
> >>>>>>>>
> >>>>>>>> Sean
> >>>>>>>>
> >>>>>>>> On Sun, Mar 18, 2018 at 4:05 AM, Debasish Ghosh <
> >>>>>>>> debasish.ghosh@lightbend.com> wrote:
> >>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Should this be 1.2 (maybe it's even better to not put any
> >>>>>>> version at
> >>>>>>>>>> all)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Actually wanted to emphasize that the support is from 1.0.0
> >>>>> onwards ..
> >>>>>>>>> Should we make that explicit ? Like ..
> >>>>>>>>>
> >>>>>>>>> kafka-streams-scala only depends on the Scala standard library
> >>> and
> >>>>>>> Kafka
> >>>>>>>>>> Streams 1.0.0+.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> In 1.1 release, we add a new module `kafka-streams-test-utils`
> >>> to
> >>>>>>>> simplify
> >>>>>>>>>> testing for Kafka Streams applications. Are those test utils
> >>>>>>> suitable
> >>>>>>>> for
> >>>>>>>>>> Scala users or should we add Scala wrappers for those, too?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I will check up and let you know ..
> >>>>>>>>>
> >>>>>>>>> Also I am not clear about the decision on renaming of Scala
> >>>>>>> abstractions.
> >>>>>>>>> Can we have a consensus on this ? Here's the summary ..
> >>>>>>>>>
> >>>>>>>>> *Option 1:* Keep names separate (KStream for Java class,
> >> KStreamS
> >>>>> for
> >>>>>>>>> Scala). No renaming of imports required.
> >>>>>>>>> *Option 2:* Unify names (KStream for Java and Scala class
> >> names).
> >>>>> No
> >>>>>>>>> conflict since they will reside in different packages. But if
> >> we
> >>>>> need
> >>>>>>> to
> >>>>>>>>> use both abstractions, renaming of imports are required. But
> >>> again,
> >>>>>>> this
> >>>>>>>>> may not be a too frequent use case.
> >>>>>>>>>
> >>>>>>>>> Suggestions ?
> >>>>>>>>>
> >>>>>>>>> regards.
> >>>>>>>>>
> >>>>>>>>> On Sat, Mar 17, 2018 at 3:07 AM, Matthias J. Sax <
> >>>>>>> matthias@confluent.io>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Thanks a lot for the KIP! Two questions:
> >>>>>>>>>>
> >>>>>>>>>> 1) the KIP states:
> >>>>>>>>>>
> >>>>>>>>>>> kafka-streams-scala only depends on the Scala standard
> >>> library
> >>>>> and
> >>>>>>>>> Kafka
> >>>>>>>>>> Streams 1.0.0.
> >>>>>>>>>>
> >>>>>>>>>> Should this be 1.2 (maybe it's even better to not put any
> >>>>> version
> >>>>>>> at
> >>>>>>>>> all)
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> 2) In 1.1 release, we add a new module
> >>>>> `kafka-streams-test-utils` to
> >>>>>>>>>> simplify testing for Kafka Streams applications. Are those
> >> test
> >>>>>>> utils
> >>>>>>>>>> suitable for Scala users or should we add Scala wrappers for
> >>>>> those,
> >>>>>>>> too?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> -Matthias
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 3/16/18 11:10 AM, Ted Yu wrote:
> >>>>>>>>>>> Import renames seem to be fine.
> >>>>>>>>>>>
> >>>>>>>>>>> The class names with trailing 'S' look clean.
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Mar 16, 2018 at 11:04 AM, Ismael Juma <
> >>>>> ismael@juma.me.uk>
> >>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> If this is rare (as it sounds), relying on import renames
> >>>>> seems
> >>>>>>> fine
> >>>>>>>>> to
> >>>>>>>>>> me.
> >>>>>>>>>>>> Let's see what others think.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Ismael
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
> >>>>>>>>>>>> debasish.ghosh@lightbend.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I am not sure if this is practical or not. But
> >>> theoretically
> >>>>> a
> >>>>>>> user
> >>>>>>>>> may
> >>>>>>>>>>>>> want to extract the unsafe Java abstraction from the
> >> Scala
> >>>>> ones
> >>>>>>> and
> >>>>>>>>> use
> >>>>>>>>>>>>> Java APIs on them .. e.g.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> val userClicksStream: KStreamS[String, Long] =
> >>>>>>>>>>>>> builder.stream(userClicksTopic) // Scala abstraction
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> val jStream: KStream[String, Long] =
> >> userClicksStream.inner
> >>>>> //
> >>>>>>>>>> publishes
> >>>>>>>>>>>>> the underlying Java abstraction
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> //.. work with Java, may be pass to some function written
> >>> in
> >>>>>>> Java
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I do realize this is somewhat of a convoluted use case
> >> and
> >>>>> may
> >>>>>>> not
> >>>>>>>> be
> >>>>>>>>>>>>> practically useful ..
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Otherwise we can very well work on the suggested approach
> >>> of
> >>>>>>>> unifying
> >>>>>>>>>> the
> >>>>>>>>>>>>> names ..
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> regards.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <
> >>>>>>> ismael@juma.me.uk>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> What does "mixed mode application" mean? What are the
> >>> cases
> >>>>>>> where
> >>>>>>>> a
> >>>>>>>>>>>> user
> >>>>>>>>>>>>>> would want to use both APIs? I think that would help
> >>>>> understand
> >>>>>>>> the
> >>>>>>>>>>>>>> reasoning.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> Ismael
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
> >>>>>>>>>>>>>> debasish.ghosh@lightbend.com> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Damian -
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> We could. But in case the user wants to use both Scala
> >>> and
> >>>>>>> Java
> >>>>>>>>> APIs
> >>>>>>>>>>>>> (may
> >>>>>>>>>>>>>>> be for some mixed mode application), won't that be
> >>>>> confusing ?
> >>>>>>>> She
> >>>>>>>>>>>> will
> >>>>>>>>>>>>>>> have to do something like ..
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> import o.a.k.s.scala.{KStream => KStreamS}
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> to rename Scala imports or the other way round for
> >>> imported
> >>>>>>> Java
> >>>>>>>>>>>>> classes.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> regards.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <
> >>>>>>>> damian.guy@gmail.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Debasish,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the KIP - will be a great addition to
> >>> streams.
> >>>>>>> I've
> >>>>>>>>> only
> >>>>>>>>>>>>>> had a
> >>>>>>>>>>>>>>>> quick scan, but seeing as the Scala classes are going
> >> to
> >>>>> be
> >>>>>>> in
> >>>>>>>>>>>> their
> >>>>>>>>>>>>>> own
> >>>>>>>>>>>>>>>> package could we drop the S at the end of the class
> >>> names?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>> Damian
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> >>>>>>>>>>>>>>> debasish.ghosh@lightbend.com>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi -
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> A new KIP, KIP-270 is up for discussion:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>>>>>>>>>>>> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The relevant JIRA issue: https://issues.apache.org/
> >>>>>>>>>>>>>>>> jira/browse/KAFKA-6670
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The library as proposed in the KIP has been
> >> implemented
> >>>>> at
> >>>>>>>>>>>>>>>>> https://github.com/lightbend/kafka-streams-scala and
> >>> the
> >>>>>>>> current
> >>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> 0.2.0 (
> >>>>>>>>>>>>>>>>> https://github.com/lightbend/k
> >>>>> afka-streams-scala/releases/
> >>>>>>>>>>>>> tag/v0.2.0
> >>>>>>>>>>>>>> ).
> >>>>>>>>>>>>>>>>> We at Lightbend has been using it since quite some
> >> time
> >>>>> now.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> regards.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>> Debasish Ghosh
> >>>>>>>>>>>>>>>>> Principal Engineer
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Twitter: @debasishg
> >>>>>>>>>>>>>>>>> Blog: http://debasishg.blogspot.com
> >>>>>>>>>>>>>>>>> Code: https://github.com/debasishg
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> Debasish Ghosh
> >>>>>>>>>>>>>>> Principal Engineer
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Twitter: @debasishg
> >>>>>>>>>>>>>>> Blog: http://debasishg.blogspot.com
> >>>>>>>>>>>>>>> Code: https://github.com/debasishg
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> Debasish Ghosh
> >>>>>>>>>>>>> Principal Engineer
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Twitter: @debasishg
> >>>>>>>>>>>>> Blog: http://debasishg.blogspot.com
> >>>>>>>>>>>>> Code: https://github.com/debasishg
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Debasish Ghosh
> >>>>>>>>> Principal Engineer
> >>>>>>>>>
> >>>>>>>>> Twitter: @debasishg
> >>>>>>>>> Blog: http://debasishg.blogspot.com
> >>>>>>>>> Code: https://github.com/debasishg
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Senior Software Engineer, Lightbend, Inc.
> >>>>>>>>
> >>>>>>>> <http://lightbend.com>
> >>>>>>>>
> >>>>>>>> @seg1o <https://twitter.com/seg1o>, in/seanaglover
> >>>>>>>> <https://www.linkedin.com/in/seanaglover/>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> -- Guozhang
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> -- Guozhang
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Debasish Ghosh
> >>>> Principal Engineer
> >>>>
> >>>> Twitter: @debasishg
> >>>> Blog: http://debasishg.blogspot.com
> >>>> Code: https://github.com/debasishg
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Debasish Ghosh
> >>> Principal Engineer
> >>>
> >>> Twitter: @debasishg
> >>> Blog: http://debasishg.blogspot.com
> >>> Code: https://github.com/debasishg
> >>>
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
>
>
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by "Matthias J. Sax" <ma...@confluent.io>.
About the package name.
Would it be better/cleaner to omit the `scala` sub-package? Or is this
required to avoid naming conflicts with the Java classes? If yes, please
point it out in the KIP.
-Matthias
On 3/19/18 11:24 AM, Debasish Ghosh wrote:
> Cool .. so the changes that u suggested e.g the WordCount example, package
> name changes etc. will be in the PR only and not in the KIP document ..
>
> Or we keep the KIP document updated as well ?
>
> regards.
>
> On Mon, 19 Mar 2018 at 11:46 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
>> Hi Debasish,
>>
>> You can work on the PR in parallel to the KIP discussion so that people can
>> start reviewing it. The only restriction is that the PR cannot be merged
>> until the KIP is accepted.
>>
>>
>> Guozhang
>>
>>
>> On Mon, Mar 19, 2018 at 11:09 AM, Debasish Ghosh <
>> debasish.ghosh@lightbend.com> wrote:
>>
>>> Based on this discussion I have a question ..
>>>
>>> For the actual implementation we need to have a PR on
>>> https://github.com/apache/kafka and the new library will be in the
>> package
>>> structure org.apache.kafka.streams.scala. My question is, is this PR part
>>> of the KIP ? Or we work on the PR once the KIP is accepted ..
>>>
>>> regards.
>>>
>>> On Mon, Mar 19, 2018 at 11:28 PM, Debasish Ghosh <
>>> debasish.ghosh@lightbend.com> wrote:
>>>
>>>> Thanks Guozhang for the comments .. we will start working on them ..
>>>>
>>>> regards.
>>>>
>>>> On Mon, Mar 19, 2018 at 11:02 PM, Guozhang Wang <wa...@gmail.com>
>>>> wrote:
>>>>
>>>>> And one more comment about the type safety:
>>>>>
>>>>> 7. I'm in favor of the approach of enforcing type safety at compile
>>> time,
>>>>> since with Scala's strong typing system I think it makes more sense to
>>> get
>>>>> rid of config-based serde specifications that can cause runtime error
>> in
>>>>> the Scala wrapper.
>>>>>
>>>>>
>>>>> Guozhang
>>>>>
>>>>>
>>>>> On Mon, Mar 19, 2018 at 10:28 AM, Guozhang Wang <wa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello Debasish,
>>>>>>
>>>>>> Thanks for the KIP. Here are some comments:
>>>>>>
>>>>>> 1. About the naming: I'd also vote for option 2 and enforce users to
>>>>>> rename when import since this scenario seems rare to me.
>>>>>>
>>>>>> 2. About the dependency: since this KIP can only be merged as early
>> as
>>>>>> 1.2, it means that for users who wants to use this artifact, say for
>>>>>> version 1.2, she would need to bring in "kafka-streams" version 1.2.
>>> In
>>>>>> addition, if we change the Java API in the future versions we'd also
>>>>> need
>>>>>> to update the Scala wrapper at the same version as well, so in other
>>>>> words
>>>>>> "kafka-streams-scala" version X have to be with "kafka-streams"
>>> version
>>>>> X
>>>>>> anyways. So I'd suggest we remove the version number in
>>>>> "kafka-streams-scala
>>>>>> only depends on the Scala standard library and Kafka Streams" as it
>>> may
>>>>>> be confusing to readers.
>>>>>>
>>>>>> 3. For the KIP discussion, it need be based on the proposed state of
>>>>> this
>>>>>> project in AK. More specifically, for the groupId in maven / gradle,
>>> it
>>>>>> need to be "org.apache.kafka"; for the version, it need to be Kafka
>>>>> release
>>>>>> versions, e.g. 1.2.0.
>>>>>>
>>>>>> 4. In "New or Changed Public Interfaces" section, it does not
>> mention
>>>>> the
>>>>>> serde classes like "DefaultSerdes", but I think they should also be
>>>>>> considered public APIs as users would most likely import these in
>> her
>>>>>> implementation.
>>>>>>
>>>>>> 5. Could you also list the changes you'd propose to made to
>>> build.gradle
>>>>>> in kafka for adding this artifact? More details will help readers to
>>>>> better
>>>>>> understand your proposal.
>>>>>>
>>>>>> 6. I think it'd will be good to have a WordCount example code as
>> part
>>> of
>>>>>> this KIP, to illustrate how to code in this Scala wrapper, e.g. in
>>>>>> o.a.k.scala.examples. But for this class we probably do not need to
>>>>> have a
>>>>>> separate artifact for it as we did in kafka-streams-examples.
>>>>>>
>>>>>>
>>>>>> Guozhang
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 19, 2018 at 9:16 AM, Ted Yu <yu...@gmail.com>
>> wrote:
>>>>>>
>>>>>>> I agree with Sean on name unification.
>>>>>>>
>>>>>>> +1 to option 2.
>>>>>>>
>>>>>>> On Mon, Mar 19, 2018 at 7:23 AM, Sean Glover <
>>>>> sean.glover@lightbend.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Type names: I vote for option 2. The user must explicitly add a
>>>>>>> dependency
>>>>>>>> to this library and the wrapper types are in a different package.
>>> It
>>>>>>> seems
>>>>>>>> reasonable to expect them to do an import rename if there's a
>> need
>>> to
>>>>>>> drop
>>>>>>>> down to the Java API.
>>>>>>>>
>>>>>>>> Test Utils: The test utils in kafka-streams-scala are nice and
>>> lean,
>>>>> but
>>>>>>>> I'm not sure if it provides much more value than other options
>> that
>>>>>>> exist
>>>>>>>> in the community. There's an embedded Kafka/ZK project
>>>>> implementation
>>>>>>> for
>>>>>>>> ScalaTest that's popular and active: manub/scalatest-embedded-
>>> kakfa.
>>>>>>> It
>>>>>>>> implies you must also use ScalaTest, which I acknowledge isn't
>>>>>>> everyone's
>>>>>>>> first choice for Scala test framework, but it probably is one of,
>>> if
>>>>> not
>>>>>>>> the most, popular library. It includes a DSL for Kafka Streams
>>>>> too. If
>>>>>>>> this KIP is accepted then perhaps a PR to that project could be
>>> made
>>>>> to
>>>>>>>> support the new wrapper implementations.
>>>>>>>>
>>>>>>>> https://github.com/manub/scalatest-embedded-kafka#
>>>>>>>> scalatest-embedded-kafka-streams
>>>>>>>>
>>>>>>>> Sean
>>>>>>>>
>>>>>>>> On Sun, Mar 18, 2018 at 4:05 AM, Debasish Ghosh <
>>>>>>>> debasish.ghosh@lightbend.com> wrote:
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Should this be 1.2 (maybe it's even better to not put any
>>>>>>> version at
>>>>>>>>>> all)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Actually wanted to emphasize that the support is from 1.0.0
>>>>> onwards ..
>>>>>>>>> Should we make that explicit ? Like ..
>>>>>>>>>
>>>>>>>>> kafka-streams-scala only depends on the Scala standard library
>>> and
>>>>>>> Kafka
>>>>>>>>>> Streams 1.0.0+.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In 1.1 release, we add a new module `kafka-streams-test-utils`
>>> to
>>>>>>>> simplify
>>>>>>>>>> testing for Kafka Streams applications. Are those test utils
>>>>>>> suitable
>>>>>>>> for
>>>>>>>>>> Scala users or should we add Scala wrappers for those, too?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I will check up and let you know ..
>>>>>>>>>
>>>>>>>>> Also I am not clear about the decision on renaming of Scala
>>>>>>> abstractions.
>>>>>>>>> Can we have a consensus on this ? Here's the summary ..
>>>>>>>>>
>>>>>>>>> *Option 1:* Keep names separate (KStream for Java class,
>> KStreamS
>>>>> for
>>>>>>>>> Scala). No renaming of imports required.
>>>>>>>>> *Option 2:* Unify names (KStream for Java and Scala class
>> names).
>>>>> No
>>>>>>>>> conflict since they will reside in different packages. But if
>> we
>>>>> need
>>>>>>> to
>>>>>>>>> use both abstractions, renaming of imports are required. But
>>> again,
>>>>>>> this
>>>>>>>>> may not be a too frequent use case.
>>>>>>>>>
>>>>>>>>> Suggestions ?
>>>>>>>>>
>>>>>>>>> regards.
>>>>>>>>>
>>>>>>>>> On Sat, Mar 17, 2018 at 3:07 AM, Matthias J. Sax <
>>>>>>> matthias@confluent.io>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks a lot for the KIP! Two questions:
>>>>>>>>>>
>>>>>>>>>> 1) the KIP states:
>>>>>>>>>>
>>>>>>>>>>> kafka-streams-scala only depends on the Scala standard
>>> library
>>>>> and
>>>>>>>>> Kafka
>>>>>>>>>> Streams 1.0.0.
>>>>>>>>>>
>>>>>>>>>> Should this be 1.2 (maybe it's even better to not put any
>>>>> version
>>>>>>> at
>>>>>>>>> all)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2) In 1.1 release, we add a new module
>>>>> `kafka-streams-test-utils` to
>>>>>>>>>> simplify testing for Kafka Streams applications. Are those
>> test
>>>>>>> utils
>>>>>>>>>> suitable for Scala users or should we add Scala wrappers for
>>>>> those,
>>>>>>>> too?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -Matthias
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 3/16/18 11:10 AM, Ted Yu wrote:
>>>>>>>>>>> Import renames seem to be fine.
>>>>>>>>>>>
>>>>>>>>>>> The class names with trailing 'S' look clean.
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Mar 16, 2018 at 11:04 AM, Ismael Juma <
>>>>> ismael@juma.me.uk>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> If this is rare (as it sounds), relying on import renames
>>>>> seems
>>>>>>> fine
>>>>>>>>> to
>>>>>>>>>> me.
>>>>>>>>>>>> Let's see what others think.
>>>>>>>>>>>>
>>>>>>>>>>>> Ismael
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
>>>>>>>>>>>> debasish.ghosh@lightbend.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I am not sure if this is practical or not. But
>>> theoretically
>>>>> a
>>>>>>> user
>>>>>>>>> may
>>>>>>>>>>>>> want to extract the unsafe Java abstraction from the
>> Scala
>>>>> ones
>>>>>>> and
>>>>>>>>> use
>>>>>>>>>>>>> Java APIs on them .. e.g.
>>>>>>>>>>>>>
>>>>>>>>>>>>> val userClicksStream: KStreamS[String, Long] =
>>>>>>>>>>>>> builder.stream(userClicksTopic) // Scala abstraction
>>>>>>>>>>>>>
>>>>>>>>>>>>> val jStream: KStream[String, Long] =
>> userClicksStream.inner
>>>>> //
>>>>>>>>>> publishes
>>>>>>>>>>>>> the underlying Java abstraction
>>>>>>>>>>>>>
>>>>>>>>>>>>> //.. work with Java, may be pass to some function written
>>> in
>>>>>>> Java
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do realize this is somewhat of a convoluted use case
>> and
>>>>> may
>>>>>>> not
>>>>>>>> be
>>>>>>>>>>>>> practically useful ..
>>>>>>>>>>>>>
>>>>>>>>>>>>> Otherwise we can very well work on the suggested approach
>>> of
>>>>>>>> unifying
>>>>>>>>>> the
>>>>>>>>>>>>> names ..
>>>>>>>>>>>>>
>>>>>>>>>>>>> regards.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <
>>>>>>> ismael@juma.me.uk>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> What does "mixed mode application" mean? What are the
>>> cases
>>>>>>> where
>>>>>>>> a
>>>>>>>>>>>> user
>>>>>>>>>>>>>> would want to use both APIs? I think that would help
>>>>> understand
>>>>>>>> the
>>>>>>>>>>>>>> reasoning.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Ismael
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
>>>>>>>>>>>>>> debasish.ghosh@lightbend.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Damian -
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We could. But in case the user wants to use both Scala
>>> and
>>>>>>> Java
>>>>>>>>> APIs
>>>>>>>>>>>>> (may
>>>>>>>>>>>>>>> be for some mixed mode application), won't that be
>>>>> confusing ?
>>>>>>>> She
>>>>>>>>>>>> will
>>>>>>>>>>>>>>> have to do something like ..
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> import o.a.k.s.scala.{KStream => KStreamS}
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> to rename Scala imports or the other way round for
>>> imported
>>>>>>> Java
>>>>>>>>>>>>> classes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> regards.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <
>>>>>>>> damian.guy@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Debasish,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the KIP - will be a great addition to
>>> streams.
>>>>>>> I've
>>>>>>>>> only
>>>>>>>>>>>>>> had a
>>>>>>>>>>>>>>>> quick scan, but seeing as the Scala classes are going
>> to
>>>>> be
>>>>>>> in
>>>>>>>>>>>> their
>>>>>>>>>>>>>> own
>>>>>>>>>>>>>>>> package could we drop the S at the end of the class
>>> names?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Damian
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
>>>>>>>>>>>>>>> debasish.ghosh@lightbend.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi -
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> A new KIP, KIP-270 is up for discussion:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>>>>>>>>> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The relevant JIRA issue: https://issues.apache.org/
>>>>>>>>>>>>>>>> jira/browse/KAFKA-6670
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The library as proposed in the KIP has been
>> implemented
>>>>> at
>>>>>>>>>>>>>>>>> https://github.com/lightbend/kafka-streams-scala and
>>> the
>>>>>>>> current
>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> 0.2.0 (
>>>>>>>>>>>>>>>>> https://github.com/lightbend/k
>>>>> afka-streams-scala/releases/
>>>>>>>>>>>>> tag/v0.2.0
>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>> We at Lightbend has been using it since quite some
>> time
>>>>> now.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> regards.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Debasish Ghosh
>>>>>>>>>>>>>>>>> Principal Engineer
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Twitter: @debasishg
>>>>>>>>>>>>>>>>> Blog: http://debasishg.blogspot.com
>>>>>>>>>>>>>>>>> Code: https://github.com/debasishg
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Debasish Ghosh
>>>>>>>>>>>>>>> Principal Engineer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Twitter: @debasishg
>>>>>>>>>>>>>>> Blog: http://debasishg.blogspot.com
>>>>>>>>>>>>>>> Code: https://github.com/debasishg
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Debasish Ghosh
>>>>>>>>>>>>> Principal Engineer
>>>>>>>>>>>>>
>>>>>>>>>>>>> Twitter: @debasishg
>>>>>>>>>>>>> Blog: http://debasishg.blogspot.com
>>>>>>>>>>>>> Code: https://github.com/debasishg
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Debasish Ghosh
>>>>>>>>> Principal Engineer
>>>>>>>>>
>>>>>>>>> Twitter: @debasishg
>>>>>>>>> Blog: http://debasishg.blogspot.com
>>>>>>>>> Code: https://github.com/debasishg
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Senior Software Engineer, Lightbend, Inc.
>>>>>>>>
>>>>>>>> <http://lightbend.com>
>>>>>>>>
>>>>>>>> @seg1o <https://twitter.com/seg1o>, in/seanaglover
>>>>>>>> <https://www.linkedin.com/in/seanaglover/>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -- Guozhang
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -- Guozhang
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Debasish Ghosh
>>>> Principal Engineer
>>>>
>>>> Twitter: @debasishg
>>>> Blog: http://debasishg.blogspot.com
>>>> Code: https://github.com/debasishg
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Debasish Ghosh
>>> Principal Engineer
>>>
>>> Twitter: @debasishg
>>> Blog: http://debasishg.blogspot.com
>>> Code: https://github.com/debasishg
>>>
>>
>>
>>
>> --
>> -- Guozhang
>>
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Debasish Ghosh <de...@lightbend.com>.
Cool .. so the changes that u suggested e.g the WordCount example, package
name changes etc. will be in the PR only and not in the KIP document ..
Or we keep the KIP document updated as well ?
regards.
On Mon, 19 Mar 2018 at 11:46 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Debasish,
>
> You can work on the PR in parallel to the KIP discussion so that people can
> start reviewing it. The only restriction is that the PR cannot be merged
> until the KIP is accepted.
>
>
> Guozhang
>
>
> On Mon, Mar 19, 2018 at 11:09 AM, Debasish Ghosh <
> debasish.ghosh@lightbend.com> wrote:
>
> > Based on this discussion I have a question ..
> >
> > For the actual implementation we need to have a PR on
> > https://github.com/apache/kafka and the new library will be in the
> package
> > structure org.apache.kafka.streams.scala. My question is, is this PR part
> > of the KIP ? Or we work on the PR once the KIP is accepted ..
> >
> > regards.
> >
> > On Mon, Mar 19, 2018 at 11:28 PM, Debasish Ghosh <
> > debasish.ghosh@lightbend.com> wrote:
> >
> > > Thanks Guozhang for the comments .. we will start working on them ..
> > >
> > > regards.
> > >
> > > On Mon, Mar 19, 2018 at 11:02 PM, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > >
> > >> And one more comment about the type safety:
> > >>
> > >> 7. I'm in favor of the approach of enforcing type safety at compile
> > time,
> > >> since with Scala's strong typing system I think it makes more sense to
> > get
> > >> rid of config-based serde specifications that can cause runtime error
> in
> > >> the Scala wrapper.
> > >>
> > >>
> > >> Guozhang
> > >>
> > >>
> > >> On Mon, Mar 19, 2018 at 10:28 AM, Guozhang Wang <wa...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hello Debasish,
> > >> >
> > >> > Thanks for the KIP. Here are some comments:
> > >> >
> > >> > 1. About the naming: I'd also vote for option 2 and enforce users to
> > >> > rename when import since this scenario seems rare to me.
> > >> >
> > >> > 2. About the dependency: since this KIP can only be merged as early
> as
> > >> > 1.2, it means that for users who wants to use this artifact, say for
> > >> > version 1.2, she would need to bring in "kafka-streams" version 1.2.
> > In
> > >> > addition, if we change the Java API in the future versions we'd also
> > >> need
> > >> > to update the Scala wrapper at the same version as well, so in other
> > >> words
> > >> > "kafka-streams-scala" version X have to be with "kafka-streams"
> > version
> > >> X
> > >> > anyways. So I'd suggest we remove the version number in
> > >> "kafka-streams-scala
> > >> > only depends on the Scala standard library and Kafka Streams" as it
> > may
> > >> > be confusing to readers.
> > >> >
> > >> > 3. For the KIP discussion, it need be based on the proposed state of
> > >> this
> > >> > project in AK. More specifically, for the groupId in maven / gradle,
> > it
> > >> > need to be "org.apache.kafka"; for the version, it need to be Kafka
> > >> release
> > >> > versions, e.g. 1.2.0.
> > >> >
> > >> > 4. In "New or Changed Public Interfaces" section, it does not
> mention
> > >> the
> > >> > serde classes like "DefaultSerdes", but I think they should also be
> > >> > considered public APIs as users would most likely import these in
> her
> > >> > implementation.
> > >> >
> > >> > 5. Could you also list the changes you'd propose to made to
> > build.gradle
> > >> > in kafka for adding this artifact? More details will help readers to
> > >> better
> > >> > understand your proposal.
> > >> >
> > >> > 6. I think it'd will be good to have a WordCount example code as
> part
> > of
> > >> > this KIP, to illustrate how to code in this Scala wrapper, e.g. in
> > >> > o.a.k.scala.examples. But for this class we probably do not need to
> > >> have a
> > >> > separate artifact for it as we did in kafka-streams-examples.
> > >> >
> > >> >
> > >> > Guozhang
> > >> >
> > >> >
> > >> > On Mon, Mar 19, 2018 at 9:16 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > >> >
> > >> >> I agree with Sean on name unification.
> > >> >>
> > >> >> +1 to option 2.
> > >> >>
> > >> >> On Mon, Mar 19, 2018 at 7:23 AM, Sean Glover <
> > >> sean.glover@lightbend.com>
> > >> >> wrote:
> > >> >>
> > >> >> > Type names: I vote for option 2. The user must explicitly add a
> > >> >> dependency
> > >> >> > to this library and the wrapper types are in a different package.
> > It
> > >> >> seems
> > >> >> > reasonable to expect them to do an import rename if there's a
> need
> > to
> > >> >> drop
> > >> >> > down to the Java API.
> > >> >> >
> > >> >> > Test Utils: The test utils in kafka-streams-scala are nice and
> > lean,
> > >> but
> > >> >> > I'm not sure if it provides much more value than other options
> that
> > >> >> exist
> > >> >> > in the community. There's an embedded Kafka/ZK project
> > >> implementation
> > >> >> for
> > >> >> > ScalaTest that's popular and active: manub/scalatest-embedded-
> > kakfa.
> > >> >> It
> > >> >> > implies you must also use ScalaTest, which I acknowledge isn't
> > >> >> everyone's
> > >> >> > first choice for Scala test framework, but it probably is one of,
> > if
> > >> not
> > >> >> > the most, popular library. It includes a DSL for Kafka Streams
> > >> too. If
> > >> >> > this KIP is accepted then perhaps a PR to that project could be
> > made
> > >> to
> > >> >> > support the new wrapper implementations.
> > >> >> >
> > >> >> > https://github.com/manub/scalatest-embedded-kafka#
> > >> >> > scalatest-embedded-kafka-streams
> > >> >> >
> > >> >> > Sean
> > >> >> >
> > >> >> > On Sun, Mar 18, 2018 at 4:05 AM, Debasish Ghosh <
> > >> >> > debasish.ghosh@lightbend.com> wrote:
> > >> >> >
> > >> >> > > >
> > >> >> > > > > Should this be 1.2 (maybe it's even better to not put any
> > >> >> version at
> > >> >> > > > all)
> > >> >> > >
> > >> >> > >
> > >> >> > > Actually wanted to emphasize that the support is from 1.0.0
> > >> onwards ..
> > >> >> > > Should we make that explicit ? Like ..
> > >> >> > >
> > >> >> > > kafka-streams-scala only depends on the Scala standard library
> > and
> > >> >> Kafka
> > >> >> > > > Streams 1.0.0+.
> > >> >> > >
> > >> >> > >
> > >> >> > > In 1.1 release, we add a new module `kafka-streams-test-utils`
> > to
> > >> >> > simplify
> > >> >> > > > testing for Kafka Streams applications. Are those test utils
> > >> >> suitable
> > >> >> > for
> > >> >> > > > Scala users or should we add Scala wrappers for those, too?
> > >> >> > >
> > >> >> > >
> > >> >> > > I will check up and let you know ..
> > >> >> > >
> > >> >> > > Also I am not clear about the decision on renaming of Scala
> > >> >> abstractions.
> > >> >> > > Can we have a consensus on this ? Here's the summary ..
> > >> >> > >
> > >> >> > > *Option 1:* Keep names separate (KStream for Java class,
> KStreamS
> > >> for
> > >> >> > > Scala). No renaming of imports required.
> > >> >> > > *Option 2:* Unify names (KStream for Java and Scala class
> names).
> > >> No
> > >> >> > > conflict since they will reside in different packages. But if
> we
> > >> need
> > >> >> to
> > >> >> > > use both abstractions, renaming of imports are required. But
> > again,
> > >> >> this
> > >> >> > > may not be a too frequent use case.
> > >> >> > >
> > >> >> > > Suggestions ?
> > >> >> > >
> > >> >> > > regards.
> > >> >> > >
> > >> >> > > On Sat, Mar 17, 2018 at 3:07 AM, Matthias J. Sax <
> > >> >> matthias@confluent.io>
> > >> >> > > wrote:
> > >> >> > >
> > >> >> > > > Thanks a lot for the KIP! Two questions:
> > >> >> > > >
> > >> >> > > > 1) the KIP states:
> > >> >> > > >
> > >> >> > > > > kafka-streams-scala only depends on the Scala standard
> > library
> > >> and
> > >> >> > > Kafka
> > >> >> > > > Streams 1.0.0.
> > >> >> > > >
> > >> >> > > > Should this be 1.2 (maybe it's even better to not put any
> > >> version
> > >> >> at
> > >> >> > > all)
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > 2) In 1.1 release, we add a new module
> > >> `kafka-streams-test-utils` to
> > >> >> > > > simplify testing for Kafka Streams applications. Are those
> test
> > >> >> utils
> > >> >> > > > suitable for Scala users or should we add Scala wrappers for
> > >> those,
> > >> >> > too?
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > -Matthias
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > On 3/16/18 11:10 AM, Ted Yu wrote:
> > >> >> > > > > Import renames seem to be fine.
> > >> >> > > > >
> > >> >> > > > > The class names with trailing 'S' look clean.
> > >> >> > > > >
> > >> >> > > > > Cheers
> > >> >> > > > >
> > >> >> > > > > On Fri, Mar 16, 2018 at 11:04 AM, Ismael Juma <
> > >> ismael@juma.me.uk>
> > >> >> > > wrote:
> > >> >> > > > >
> > >> >> > > > >> If this is rare (as it sounds), relying on import renames
> > >> seems
> > >> >> fine
> > >> >> > > to
> > >> >> > > > me.
> > >> >> > > > >> Let's see what others think.
> > >> >> > > > >>
> > >> >> > > > >> Ismael
> > >> >> > > > >>
> > >> >> > > > >> On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
> > >> >> > > > >> debasish.ghosh@lightbend.com> wrote:
> > >> >> > > > >>
> > >> >> > > > >>> I am not sure if this is practical or not. But
> > theoretically
> > >> a
> > >> >> user
> > >> >> > > may
> > >> >> > > > >>> want to extract the unsafe Java abstraction from the
> Scala
> > >> ones
> > >> >> and
> > >> >> > > use
> > >> >> > > > >>> Java APIs on them .. e.g.
> > >> >> > > > >>>
> > >> >> > > > >>> val userClicksStream: KStreamS[String, Long] =
> > >> >> > > > >>> builder.stream(userClicksTopic) // Scala abstraction
> > >> >> > > > >>>
> > >> >> > > > >>> val jStream: KStream[String, Long] =
> userClicksStream.inner
> > >> //
> > >> >> > > > publishes
> > >> >> > > > >>> the underlying Java abstraction
> > >> >> > > > >>>
> > >> >> > > > >>> //.. work with Java, may be pass to some function written
> > in
> > >> >> Java
> > >> >> > > > >>>
> > >> >> > > > >>> I do realize this is somewhat of a convoluted use case
> and
> > >> may
> > >> >> not
> > >> >> > be
> > >> >> > > > >>> practically useful ..
> > >> >> > > > >>>
> > >> >> > > > >>> Otherwise we can very well work on the suggested approach
> > of
> > >> >> > unifying
> > >> >> > > > the
> > >> >> > > > >>> names ..
> > >> >> > > > >>>
> > >> >> > > > >>> regards.
> > >> >> > > > >>>
> > >> >> > > > >>>
> > >> >> > > > >>>
> > >> >> > > > >>> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <
> > >> >> ismael@juma.me.uk>
> > >> >> > > > wrote:
> > >> >> > > > >>>
> > >> >> > > > >>>> What does "mixed mode application" mean? What are the
> > cases
> > >> >> where
> > >> >> > a
> > >> >> > > > >> user
> > >> >> > > > >>>> would want to use both APIs? I think that would help
> > >> understand
> > >> >> > the
> > >> >> > > > >>>> reasoning.
> > >> >> > > > >>>>
> > >> >> > > > >>>> Thanks,
> > >> >> > > > >>>> Ismael
> > >> >> > > > >>>>
> > >> >> > > > >>>> On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
> > >> >> > > > >>>> debasish.ghosh@lightbend.com> wrote:
> > >> >> > > > >>>>
> > >> >> > > > >>>>> Hi Damian -
> > >> >> > > > >>>>>
> > >> >> > > > >>>>> We could. But in case the user wants to use both Scala
> > and
> > >> >> Java
> > >> >> > > APIs
> > >> >> > > > >>> (may
> > >> >> > > > >>>>> be for some mixed mode application), won't that be
> > >> confusing ?
> > >> >> > She
> > >> >> > > > >> will
> > >> >> > > > >>>>> have to do something like ..
> > >> >> > > > >>>>>
> > >> >> > > > >>>>> import o.a.k.s.scala.{KStream => KStreamS}
> > >> >> > > > >>>>>
> > >> >> > > > >>>>> to rename Scala imports or the other way round for
> > imported
> > >> >> Java
> > >> >> > > > >>> classes.
> > >> >> > > > >>>>>
> > >> >> > > > >>>>> regards.
> > >> >> > > > >>>>>
> > >> >> > > > >>>>>
> > >> >> > > > >>>>>
> > >> >> > > > >>>>> On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <
> > >> >> > damian.guy@gmail.com>
> > >> >> > > > >>>> wrote:
> > >> >> > > > >>>>>
> > >> >> > > > >>>>>> Hi Debasish,
> > >> >> > > > >>>>>>
> > >> >> > > > >>>>>> Thanks for the KIP - will be a great addition to
> > streams.
> > >> >> I've
> > >> >> > > only
> > >> >> > > > >>>> had a
> > >> >> > > > >>>>>> quick scan, but seeing as the Scala classes are going
> to
> > >> be
> > >> >> in
> > >> >> > > > >> their
> > >> >> > > > >>>> own
> > >> >> > > > >>>>>> package could we drop the S at the end of the class
> > names?
> > >> >> > > > >>>>>>
> > >> >> > > > >>>>>> Thanks,
> > >> >> > > > >>>>>> Damian
> > >> >> > > > >>>>>>
> > >> >> > > > >>>>>>
> > >> >> > > > >>>>>> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> > >> >> > > > >>>>> debasish.ghosh@lightbend.com>
> > >> >> > > > >>>>>> wrote:
> > >> >> > > > >>>>>>
> > >> >> > > > >>>>>>> Hi -
> > >> >> > > > >>>>>>>
> > >> >> > > > >>>>>>> A new KIP, KIP-270 is up for discussion:
> > >> >> > > > >>>>>>>
> > >> >> > > > >>>>>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> >> > > > >>>>>> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> > >> >> > > > >>>>>>>
> > >> >> > > > >>>>>>> The relevant JIRA issue: https://issues.apache.org/
> > >> >> > > > >>>>>> jira/browse/KAFKA-6670
> > >> >> > > > >>>>>>>
> > >> >> > > > >>>>>>> The library as proposed in the KIP has been
> implemented
> > >> at
> > >> >> > > > >>>>>>> https://github.com/lightbend/kafka-streams-scala and
> > the
> > >> >> > current
> > >> >> > > > >>>>> release
> > >> >> > > > >>>>>>> is
> > >> >> > > > >>>>>>> 0.2.0 (
> > >> >> > > > >>>>>>> https://github.com/lightbend/k
> > >> afka-streams-scala/releases/
> > >> >> > > > >>> tag/v0.2.0
> > >> >> > > > >>>> ).
> > >> >> > > > >>>>>>> We at Lightbend has been using it since quite some
> time
> > >> now.
> > >> >> > > > >>>>>>>
> > >> >> > > > >>>>>>> regards.
> > >> >> > > > >>>>>>>
> > >> >> > > > >>>>>>> --
> > >> >> > > > >>>>>>> Debasish Ghosh
> > >> >> > > > >>>>>>> Principal Engineer
> > >> >> > > > >>>>>>>
> > >> >> > > > >>>>>>> Twitter: @debasishg
> > >> >> > > > >>>>>>> Blog: http://debasishg.blogspot.com
> > >> >> > > > >>>>>>> Code: https://github.com/debasishg
> > >> >> > > > >>>>>>>
> > >> >> > > > >>>>>>
> > >> >> > > > >>>>>
> > >> >> > > > >>>>>
> > >> >> > > > >>>>>
> > >> >> > > > >>>>> --
> > >> >> > > > >>>>> Debasish Ghosh
> > >> >> > > > >>>>> Principal Engineer
> > >> >> > > > >>>>>
> > >> >> > > > >>>>> Twitter: @debasishg
> > >> >> > > > >>>>> Blog: http://debasishg.blogspot.com
> > >> >> > > > >>>>> Code: https://github.com/debasishg
> > >> >> > > > >>>>>
> > >> >> > > > >>>>
> > >> >> > > > >>>
> > >> >> > > > >>>
> > >> >> > > > >>>
> > >> >> > > > >>> --
> > >> >> > > > >>> Debasish Ghosh
> > >> >> > > > >>> Principal Engineer
> > >> >> > > > >>>
> > >> >> > > > >>> Twitter: @debasishg
> > >> >> > > > >>> Blog: http://debasishg.blogspot.com
> > >> >> > > > >>> Code: https://github.com/debasishg
> > >> >> > > > >>>
> > >> >> > > > >>
> > >> >> > > > >
> > >> >> > > >
> > >> >> > > >
> > >> >> > >
> > >> >> > >
> > >> >> > > --
> > >> >> > > Debasish Ghosh
> > >> >> > > Principal Engineer
> > >> >> > >
> > >> >> > > Twitter: @debasishg
> > >> >> > > Blog: http://debasishg.blogspot.com
> > >> >> > > Code: https://github.com/debasishg
> > >> >> > >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > --
> > >> >> > Senior Software Engineer, Lightbend, Inc.
> > >> >> >
> > >> >> > <http://lightbend.com>
> > >> >> >
> > >> >> > @seg1o <https://twitter.com/seg1o>, in/seanaglover
> > >> >> > <https://www.linkedin.com/in/seanaglover/>
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > -- Guozhang
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> > >
> > >
> > > --
> > > Debasish Ghosh
> > > Principal Engineer
> > >
> > > Twitter: @debasishg
> > > Blog: http://debasishg.blogspot.com
> > > Code: https://github.com/debasishg
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Debasish Ghosh
> > Principal Engineer
> >
> > Twitter: @debasishg
> > Blog: http://debasishg.blogspot.com
> > Code: https://github.com/debasishg
> >
>
>
>
> --
> -- Guozhang
>
--
Debasish Ghosh
Principal Engineer
Twitter: @debasishg
Blog: http://debasishg.blogspot.com
Code: https://github.com/debasishg
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Debasish,
You can work on the PR in parallel to the KIP discussion so that people can
start reviewing it. The only restriction is that the PR cannot be merged
until the KIP is accepted.
Guozhang
On Mon, Mar 19, 2018 at 11:09 AM, Debasish Ghosh <
debasish.ghosh@lightbend.com> wrote:
> Based on this discussion I have a question ..
>
> For the actual implementation we need to have a PR on
> https://github.com/apache/kafka and the new library will be in the package
> structure org.apache.kafka.streams.scala. My question is, is this PR part
> of the KIP ? Or we work on the PR once the KIP is accepted ..
>
> regards.
>
> On Mon, Mar 19, 2018 at 11:28 PM, Debasish Ghosh <
> debasish.ghosh@lightbend.com> wrote:
>
> > Thanks Guozhang for the comments .. we will start working on them ..
> >
> > regards.
> >
> > On Mon, Mar 19, 2018 at 11:02 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> >
> >> And one more comment about the type safety:
> >>
> >> 7. I'm in favor of the approach of enforcing type safety at compile
> time,
> >> since with Scala's strong typing system I think it makes more sense to
> get
> >> rid of config-based serde specifications that can cause runtime error in
> >> the Scala wrapper.
> >>
> >>
> >> Guozhang
> >>
> >>
> >> On Mon, Mar 19, 2018 at 10:28 AM, Guozhang Wang <wa...@gmail.com>
> >> wrote:
> >>
> >> > Hello Debasish,
> >> >
> >> > Thanks for the KIP. Here are some comments:
> >> >
> >> > 1. About the naming: I'd also vote for option 2 and enforce users to
> >> > rename when import since this scenario seems rare to me.
> >> >
> >> > 2. About the dependency: since this KIP can only be merged as early as
> >> > 1.2, it means that for users who wants to use this artifact, say for
> >> > version 1.2, she would need to bring in "kafka-streams" version 1.2.
> In
> >> > addition, if we change the Java API in the future versions we'd also
> >> need
> >> > to update the Scala wrapper at the same version as well, so in other
> >> words
> >> > "kafka-streams-scala" version X have to be with "kafka-streams"
> version
> >> X
> >> > anyways. So I'd suggest we remove the version number in
> >> "kafka-streams-scala
> >> > only depends on the Scala standard library and Kafka Streams" as it
> may
> >> > be confusing to readers.
> >> >
> >> > 3. For the KIP discussion, it need be based on the proposed state of
> >> this
> >> > project in AK. More specifically, for the groupId in maven / gradle,
> it
> >> > need to be "org.apache.kafka"; for the version, it need to be Kafka
> >> release
> >> > versions, e.g. 1.2.0.
> >> >
> >> > 4. In "New or Changed Public Interfaces" section, it does not mention
> >> the
> >> > serde classes like "DefaultSerdes", but I think they should also be
> >> > considered public APIs as users would most likely import these in her
> >> > implementation.
> >> >
> >> > 5. Could you also list the changes you'd propose to made to
> build.gradle
> >> > in kafka for adding this artifact? More details will help readers to
> >> better
> >> > understand your proposal.
> >> >
> >> > 6. I think it'd will be good to have a WordCount example code as part
> of
> >> > this KIP, to illustrate how to code in this Scala wrapper, e.g. in
> >> > o.a.k.scala.examples. But for this class we probably do not need to
> >> have a
> >> > separate artifact for it as we did in kafka-streams-examples.
> >> >
> >> >
> >> > Guozhang
> >> >
> >> >
> >> > On Mon, Mar 19, 2018 at 9:16 AM, Ted Yu <yu...@gmail.com> wrote:
> >> >
> >> >> I agree with Sean on name unification.
> >> >>
> >> >> +1 to option 2.
> >> >>
> >> >> On Mon, Mar 19, 2018 at 7:23 AM, Sean Glover <
> >> sean.glover@lightbend.com>
> >> >> wrote:
> >> >>
> >> >> > Type names: I vote for option 2. The user must explicitly add a
> >> >> dependency
> >> >> > to this library and the wrapper types are in a different package.
> It
> >> >> seems
> >> >> > reasonable to expect them to do an import rename if there's a need
> to
> >> >> drop
> >> >> > down to the Java API.
> >> >> >
> >> >> > Test Utils: The test utils in kafka-streams-scala are nice and
> lean,
> >> but
> >> >> > I'm not sure if it provides much more value than other options that
> >> >> exist
> >> >> > in the community. There's an embedded Kafka/ZK project
> >> implementation
> >> >> for
> >> >> > ScalaTest that's popular and active: manub/scalatest-embedded-
> kakfa.
> >> >> It
> >> >> > implies you must also use ScalaTest, which I acknowledge isn't
> >> >> everyone's
> >> >> > first choice for Scala test framework, but it probably is one of,
> if
> >> not
> >> >> > the most, popular library. It includes a DSL for Kafka Streams
> >> too. If
> >> >> > this KIP is accepted then perhaps a PR to that project could be
> made
> >> to
> >> >> > support the new wrapper implementations.
> >> >> >
> >> >> > https://github.com/manub/scalatest-embedded-kafka#
> >> >> > scalatest-embedded-kafka-streams
> >> >> >
> >> >> > Sean
> >> >> >
> >> >> > On Sun, Mar 18, 2018 at 4:05 AM, Debasish Ghosh <
> >> >> > debasish.ghosh@lightbend.com> wrote:
> >> >> >
> >> >> > > >
> >> >> > > > > Should this be 1.2 (maybe it's even better to not put any
> >> >> version at
> >> >> > > > all)
> >> >> > >
> >> >> > >
> >> >> > > Actually wanted to emphasize that the support is from 1.0.0
> >> onwards ..
> >> >> > > Should we make that explicit ? Like ..
> >> >> > >
> >> >> > > kafka-streams-scala only depends on the Scala standard library
> and
> >> >> Kafka
> >> >> > > > Streams 1.0.0+.
> >> >> > >
> >> >> > >
> >> >> > > In 1.1 release, we add a new module `kafka-streams-test-utils`
> to
> >> >> > simplify
> >> >> > > > testing for Kafka Streams applications. Are those test utils
> >> >> suitable
> >> >> > for
> >> >> > > > Scala users or should we add Scala wrappers for those, too?
> >> >> > >
> >> >> > >
> >> >> > > I will check up and let you know ..
> >> >> > >
> >> >> > > Also I am not clear about the decision on renaming of Scala
> >> >> abstractions.
> >> >> > > Can we have a consensus on this ? Here's the summary ..
> >> >> > >
> >> >> > > *Option 1:* Keep names separate (KStream for Java class, KStreamS
> >> for
> >> >> > > Scala). No renaming of imports required.
> >> >> > > *Option 2:* Unify names (KStream for Java and Scala class names).
> >> No
> >> >> > > conflict since they will reside in different packages. But if we
> >> need
> >> >> to
> >> >> > > use both abstractions, renaming of imports are required. But
> again,
> >> >> this
> >> >> > > may not be a too frequent use case.
> >> >> > >
> >> >> > > Suggestions ?
> >> >> > >
> >> >> > > regards.
> >> >> > >
> >> >> > > On Sat, Mar 17, 2018 at 3:07 AM, Matthias J. Sax <
> >> >> matthias@confluent.io>
> >> >> > > wrote:
> >> >> > >
> >> >> > > > Thanks a lot for the KIP! Two questions:
> >> >> > > >
> >> >> > > > 1) the KIP states:
> >> >> > > >
> >> >> > > > > kafka-streams-scala only depends on the Scala standard
> library
> >> and
> >> >> > > Kafka
> >> >> > > > Streams 1.0.0.
> >> >> > > >
> >> >> > > > Should this be 1.2 (maybe it's even better to not put any
> >> version
> >> >> at
> >> >> > > all)
> >> >> > > >
> >> >> > > >
> >> >> > > > 2) In 1.1 release, we add a new module
> >> `kafka-streams-test-utils` to
> >> >> > > > simplify testing for Kafka Streams applications. Are those test
> >> >> utils
> >> >> > > > suitable for Scala users or should we add Scala wrappers for
> >> those,
> >> >> > too?
> >> >> > > >
> >> >> > > >
> >> >> > > > -Matthias
> >> >> > > >
> >> >> > > >
> >> >> > > > On 3/16/18 11:10 AM, Ted Yu wrote:
> >> >> > > > > Import renames seem to be fine.
> >> >> > > > >
> >> >> > > > > The class names with trailing 'S' look clean.
> >> >> > > > >
> >> >> > > > > Cheers
> >> >> > > > >
> >> >> > > > > On Fri, Mar 16, 2018 at 11:04 AM, Ismael Juma <
> >> ismael@juma.me.uk>
> >> >> > > wrote:
> >> >> > > > >
> >> >> > > > >> If this is rare (as it sounds), relying on import renames
> >> seems
> >> >> fine
> >> >> > > to
> >> >> > > > me.
> >> >> > > > >> Let's see what others think.
> >> >> > > > >>
> >> >> > > > >> Ismael
> >> >> > > > >>
> >> >> > > > >> On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
> >> >> > > > >> debasish.ghosh@lightbend.com> wrote:
> >> >> > > > >>
> >> >> > > > >>> I am not sure if this is practical or not. But
> theoretically
> >> a
> >> >> user
> >> >> > > may
> >> >> > > > >>> want to extract the unsafe Java abstraction from the Scala
> >> ones
> >> >> and
> >> >> > > use
> >> >> > > > >>> Java APIs on them .. e.g.
> >> >> > > > >>>
> >> >> > > > >>> val userClicksStream: KStreamS[String, Long] =
> >> >> > > > >>> builder.stream(userClicksTopic) // Scala abstraction
> >> >> > > > >>>
> >> >> > > > >>> val jStream: KStream[String, Long] = userClicksStream.inner
> >> //
> >> >> > > > publishes
> >> >> > > > >>> the underlying Java abstraction
> >> >> > > > >>>
> >> >> > > > >>> //.. work with Java, may be pass to some function written
> in
> >> >> Java
> >> >> > > > >>>
> >> >> > > > >>> I do realize this is somewhat of a convoluted use case and
> >> may
> >> >> not
> >> >> > be
> >> >> > > > >>> practically useful ..
> >> >> > > > >>>
> >> >> > > > >>> Otherwise we can very well work on the suggested approach
> of
> >> >> > unifying
> >> >> > > > the
> >> >> > > > >>> names ..
> >> >> > > > >>>
> >> >> > > > >>> regards.
> >> >> > > > >>>
> >> >> > > > >>>
> >> >> > > > >>>
> >> >> > > > >>> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <
> >> >> ismael@juma.me.uk>
> >> >> > > > wrote:
> >> >> > > > >>>
> >> >> > > > >>>> What does "mixed mode application" mean? What are the
> cases
> >> >> where
> >> >> > a
> >> >> > > > >> user
> >> >> > > > >>>> would want to use both APIs? I think that would help
> >> understand
> >> >> > the
> >> >> > > > >>>> reasoning.
> >> >> > > > >>>>
> >> >> > > > >>>> Thanks,
> >> >> > > > >>>> Ismael
> >> >> > > > >>>>
> >> >> > > > >>>> On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
> >> >> > > > >>>> debasish.ghosh@lightbend.com> wrote:
> >> >> > > > >>>>
> >> >> > > > >>>>> Hi Damian -
> >> >> > > > >>>>>
> >> >> > > > >>>>> We could. But in case the user wants to use both Scala
> and
> >> >> Java
> >> >> > > APIs
> >> >> > > > >>> (may
> >> >> > > > >>>>> be for some mixed mode application), won't that be
> >> confusing ?
> >> >> > She
> >> >> > > > >> will
> >> >> > > > >>>>> have to do something like ..
> >> >> > > > >>>>>
> >> >> > > > >>>>> import o.a.k.s.scala.{KStream => KStreamS}
> >> >> > > > >>>>>
> >> >> > > > >>>>> to rename Scala imports or the other way round for
> imported
> >> >> Java
> >> >> > > > >>> classes.
> >> >> > > > >>>>>
> >> >> > > > >>>>> regards.
> >> >> > > > >>>>>
> >> >> > > > >>>>>
> >> >> > > > >>>>>
> >> >> > > > >>>>> On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <
> >> >> > damian.guy@gmail.com>
> >> >> > > > >>>> wrote:
> >> >> > > > >>>>>
> >> >> > > > >>>>>> Hi Debasish,
> >> >> > > > >>>>>>
> >> >> > > > >>>>>> Thanks for the KIP - will be a great addition to
> streams.
> >> >> I've
> >> >> > > only
> >> >> > > > >>>> had a
> >> >> > > > >>>>>> quick scan, but seeing as the Scala classes are going to
> >> be
> >> >> in
> >> >> > > > >> their
> >> >> > > > >>>> own
> >> >> > > > >>>>>> package could we drop the S at the end of the class
> names?
> >> >> > > > >>>>>>
> >> >> > > > >>>>>> Thanks,
> >> >> > > > >>>>>> Damian
> >> >> > > > >>>>>>
> >> >> > > > >>>>>>
> >> >> > > > >>>>>> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> >> >> > > > >>>>> debasish.ghosh@lightbend.com>
> >> >> > > > >>>>>> wrote:
> >> >> > > > >>>>>>
> >> >> > > > >>>>>>> Hi -
> >> >> > > > >>>>>>>
> >> >> > > > >>>>>>> A new KIP, KIP-270 is up for discussion:
> >> >> > > > >>>>>>>
> >> >> > > > >>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> >> > > > >>>>>> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> >> >> > > > >>>>>>>
> >> >> > > > >>>>>>> The relevant JIRA issue: https://issues.apache.org/
> >> >> > > > >>>>>> jira/browse/KAFKA-6670
> >> >> > > > >>>>>>>
> >> >> > > > >>>>>>> The library as proposed in the KIP has been implemented
> >> at
> >> >> > > > >>>>>>> https://github.com/lightbend/kafka-streams-scala and
> the
> >> >> > current
> >> >> > > > >>>>> release
> >> >> > > > >>>>>>> is
> >> >> > > > >>>>>>> 0.2.0 (
> >> >> > > > >>>>>>> https://github.com/lightbend/k
> >> afka-streams-scala/releases/
> >> >> > > > >>> tag/v0.2.0
> >> >> > > > >>>> ).
> >> >> > > > >>>>>>> We at Lightbend has been using it since quite some time
> >> now.
> >> >> > > > >>>>>>>
> >> >> > > > >>>>>>> regards.
> >> >> > > > >>>>>>>
> >> >> > > > >>>>>>> --
> >> >> > > > >>>>>>> Debasish Ghosh
> >> >> > > > >>>>>>> Principal Engineer
> >> >> > > > >>>>>>>
> >> >> > > > >>>>>>> Twitter: @debasishg
> >> >> > > > >>>>>>> Blog: http://debasishg.blogspot.com
> >> >> > > > >>>>>>> Code: https://github.com/debasishg
> >> >> > > > >>>>>>>
> >> >> > > > >>>>>>
> >> >> > > > >>>>>
> >> >> > > > >>>>>
> >> >> > > > >>>>>
> >> >> > > > >>>>> --
> >> >> > > > >>>>> Debasish Ghosh
> >> >> > > > >>>>> Principal Engineer
> >> >> > > > >>>>>
> >> >> > > > >>>>> Twitter: @debasishg
> >> >> > > > >>>>> Blog: http://debasishg.blogspot.com
> >> >> > > > >>>>> Code: https://github.com/debasishg
> >> >> > > > >>>>>
> >> >> > > > >>>>
> >> >> > > > >>>
> >> >> > > > >>>
> >> >> > > > >>>
> >> >> > > > >>> --
> >> >> > > > >>> Debasish Ghosh
> >> >> > > > >>> Principal Engineer
> >> >> > > > >>>
> >> >> > > > >>> Twitter: @debasishg
> >> >> > > > >>> Blog: http://debasishg.blogspot.com
> >> >> > > > >>> Code: https://github.com/debasishg
> >> >> > > > >>>
> >> >> > > > >>
> >> >> > > > >
> >> >> > > >
> >> >> > > >
> >> >> > >
> >> >> > >
> >> >> > > --
> >> >> > > Debasish Ghosh
> >> >> > > Principal Engineer
> >> >> > >
> >> >> > > Twitter: @debasishg
> >> >> > > Blog: http://debasishg.blogspot.com
> >> >> > > Code: https://github.com/debasishg
> >> >> > >
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Senior Software Engineer, Lightbend, Inc.
> >> >> >
> >> >> > <http://lightbend.com>
> >> >> >
> >> >> > @seg1o <https://twitter.com/seg1o>, in/seanaglover
> >> >> > <https://www.linkedin.com/in/seanaglover/>
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > -- Guozhang
> >> >
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
> >
> >
> > --
> > Debasish Ghosh
> > Principal Engineer
> >
> > Twitter: @debasishg
> > Blog: http://debasishg.blogspot.com
> > Code: https://github.com/debasishg
> >
> >
> >
> >
>
>
> --
> Debasish Ghosh
> Principal Engineer
>
> Twitter: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: https://github.com/debasishg
>
--
-- Guozhang
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Debasish Ghosh <de...@lightbend.com>.
Based on this discussion I have a question ..
For the actual implementation we need to have a PR on
https://github.com/apache/kafka and the new library will be in the package
structure org.apache.kafka.streams.scala. My question is, is this PR part
of the KIP ? Or we work on the PR once the KIP is accepted ..
regards.
On Mon, Mar 19, 2018 at 11:28 PM, Debasish Ghosh <
debasish.ghosh@lightbend.com> wrote:
> Thanks Guozhang for the comments .. we will start working on them ..
>
> regards.
>
> On Mon, Mar 19, 2018 at 11:02 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
>
>> And one more comment about the type safety:
>>
>> 7. I'm in favor of the approach of enforcing type safety at compile time,
>> since with Scala's strong typing system I think it makes more sense to get
>> rid of config-based serde specifications that can cause runtime error in
>> the Scala wrapper.
>>
>>
>> Guozhang
>>
>>
>> On Mon, Mar 19, 2018 at 10:28 AM, Guozhang Wang <wa...@gmail.com>
>> wrote:
>>
>> > Hello Debasish,
>> >
>> > Thanks for the KIP. Here are some comments:
>> >
>> > 1. About the naming: I'd also vote for option 2 and enforce users to
>> > rename when import since this scenario seems rare to me.
>> >
>> > 2. About the dependency: since this KIP can only be merged as early as
>> > 1.2, it means that for users who wants to use this artifact, say for
>> > version 1.2, she would need to bring in "kafka-streams" version 1.2. In
>> > addition, if we change the Java API in the future versions we'd also
>> need
>> > to update the Scala wrapper at the same version as well, so in other
>> words
>> > "kafka-streams-scala" version X have to be with "kafka-streams" version
>> X
>> > anyways. So I'd suggest we remove the version number in
>> "kafka-streams-scala
>> > only depends on the Scala standard library and Kafka Streams" as it may
>> > be confusing to readers.
>> >
>> > 3. For the KIP discussion, it need be based on the proposed state of
>> this
>> > project in AK. More specifically, for the groupId in maven / gradle, it
>> > need to be "org.apache.kafka"; for the version, it need to be Kafka
>> release
>> > versions, e.g. 1.2.0.
>> >
>> > 4. In "New or Changed Public Interfaces" section, it does not mention
>> the
>> > serde classes like "DefaultSerdes", but I think they should also be
>> > considered public APIs as users would most likely import these in her
>> > implementation.
>> >
>> > 5. Could you also list the changes you'd propose to made to build.gradle
>> > in kafka for adding this artifact? More details will help readers to
>> better
>> > understand your proposal.
>> >
>> > 6. I think it'd will be good to have a WordCount example code as part of
>> > this KIP, to illustrate how to code in this Scala wrapper, e.g. in
>> > o.a.k.scala.examples. But for this class we probably do not need to
>> have a
>> > separate artifact for it as we did in kafka-streams-examples.
>> >
>> >
>> > Guozhang
>> >
>> >
>> > On Mon, Mar 19, 2018 at 9:16 AM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> >> I agree with Sean on name unification.
>> >>
>> >> +1 to option 2.
>> >>
>> >> On Mon, Mar 19, 2018 at 7:23 AM, Sean Glover <
>> sean.glover@lightbend.com>
>> >> wrote:
>> >>
>> >> > Type names: I vote for option 2. The user must explicitly add a
>> >> dependency
>> >> > to this library and the wrapper types are in a different package. It
>> >> seems
>> >> > reasonable to expect them to do an import rename if there's a need to
>> >> drop
>> >> > down to the Java API.
>> >> >
>> >> > Test Utils: The test utils in kafka-streams-scala are nice and lean,
>> but
>> >> > I'm not sure if it provides much more value than other options that
>> >> exist
>> >> > in the community. There's an embedded Kafka/ZK project
>> implementation
>> >> for
>> >> > ScalaTest that's popular and active: manub/scalatest-embedded-kakfa.
>> >> It
>> >> > implies you must also use ScalaTest, which I acknowledge isn't
>> >> everyone's
>> >> > first choice for Scala test framework, but it probably is one of, if
>> not
>> >> > the most, popular library. It includes a DSL for Kafka Streams
>> too. If
>> >> > this KIP is accepted then perhaps a PR to that project could be made
>> to
>> >> > support the new wrapper implementations.
>> >> >
>> >> > https://github.com/manub/scalatest-embedded-kafka#
>> >> > scalatest-embedded-kafka-streams
>> >> >
>> >> > Sean
>> >> >
>> >> > On Sun, Mar 18, 2018 at 4:05 AM, Debasish Ghosh <
>> >> > debasish.ghosh@lightbend.com> wrote:
>> >> >
>> >> > > >
>> >> > > > > Should this be 1.2 (maybe it's even better to not put any
>> >> version at
>> >> > > > all)
>> >> > >
>> >> > >
>> >> > > Actually wanted to emphasize that the support is from 1.0.0
>> onwards ..
>> >> > > Should we make that explicit ? Like ..
>> >> > >
>> >> > > kafka-streams-scala only depends on the Scala standard library and
>> >> Kafka
>> >> > > > Streams 1.0.0+.
>> >> > >
>> >> > >
>> >> > > In 1.1 release, we add a new module `kafka-streams-test-utils` to
>> >> > simplify
>> >> > > > testing for Kafka Streams applications. Are those test utils
>> >> suitable
>> >> > for
>> >> > > > Scala users or should we add Scala wrappers for those, too?
>> >> > >
>> >> > >
>> >> > > I will check up and let you know ..
>> >> > >
>> >> > > Also I am not clear about the decision on renaming of Scala
>> >> abstractions.
>> >> > > Can we have a consensus on this ? Here's the summary ..
>> >> > >
>> >> > > *Option 1:* Keep names separate (KStream for Java class, KStreamS
>> for
>> >> > > Scala). No renaming of imports required.
>> >> > > *Option 2:* Unify names (KStream for Java and Scala class names).
>> No
>> >> > > conflict since they will reside in different packages. But if we
>> need
>> >> to
>> >> > > use both abstractions, renaming of imports are required. But again,
>> >> this
>> >> > > may not be a too frequent use case.
>> >> > >
>> >> > > Suggestions ?
>> >> > >
>> >> > > regards.
>> >> > >
>> >> > > On Sat, Mar 17, 2018 at 3:07 AM, Matthias J. Sax <
>> >> matthias@confluent.io>
>> >> > > wrote:
>> >> > >
>> >> > > > Thanks a lot for the KIP! Two questions:
>> >> > > >
>> >> > > > 1) the KIP states:
>> >> > > >
>> >> > > > > kafka-streams-scala only depends on the Scala standard library
>> and
>> >> > > Kafka
>> >> > > > Streams 1.0.0.
>> >> > > >
>> >> > > > Should this be 1.2 (maybe it's even better to not put any
>> version
>> >> at
>> >> > > all)
>> >> > > >
>> >> > > >
>> >> > > > 2) In 1.1 release, we add a new module
>> `kafka-streams-test-utils` to
>> >> > > > simplify testing for Kafka Streams applications. Are those test
>> >> utils
>> >> > > > suitable for Scala users or should we add Scala wrappers for
>> those,
>> >> > too?
>> >> > > >
>> >> > > >
>> >> > > > -Matthias
>> >> > > >
>> >> > > >
>> >> > > > On 3/16/18 11:10 AM, Ted Yu wrote:
>> >> > > > > Import renames seem to be fine.
>> >> > > > >
>> >> > > > > The class names with trailing 'S' look clean.
>> >> > > > >
>> >> > > > > Cheers
>> >> > > > >
>> >> > > > > On Fri, Mar 16, 2018 at 11:04 AM, Ismael Juma <
>> ismael@juma.me.uk>
>> >> > > wrote:
>> >> > > > >
>> >> > > > >> If this is rare (as it sounds), relying on import renames
>> seems
>> >> fine
>> >> > > to
>> >> > > > me.
>> >> > > > >> Let's see what others think.
>> >> > > > >>
>> >> > > > >> Ismael
>> >> > > > >>
>> >> > > > >> On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
>> >> > > > >> debasish.ghosh@lightbend.com> wrote:
>> >> > > > >>
>> >> > > > >>> I am not sure if this is practical or not. But theoretically
>> a
>> >> user
>> >> > > may
>> >> > > > >>> want to extract the unsafe Java abstraction from the Scala
>> ones
>> >> and
>> >> > > use
>> >> > > > >>> Java APIs on them .. e.g.
>> >> > > > >>>
>> >> > > > >>> val userClicksStream: KStreamS[String, Long] =
>> >> > > > >>> builder.stream(userClicksTopic) // Scala abstraction
>> >> > > > >>>
>> >> > > > >>> val jStream: KStream[String, Long] = userClicksStream.inner
>> //
>> >> > > > publishes
>> >> > > > >>> the underlying Java abstraction
>> >> > > > >>>
>> >> > > > >>> //.. work with Java, may be pass to some function written in
>> >> Java
>> >> > > > >>>
>> >> > > > >>> I do realize this is somewhat of a convoluted use case and
>> may
>> >> not
>> >> > be
>> >> > > > >>> practically useful ..
>> >> > > > >>>
>> >> > > > >>> Otherwise we can very well work on the suggested approach of
>> >> > unifying
>> >> > > > the
>> >> > > > >>> names ..
>> >> > > > >>>
>> >> > > > >>> regards.
>> >> > > > >>>
>> >> > > > >>>
>> >> > > > >>>
>> >> > > > >>> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <
>> >> ismael@juma.me.uk>
>> >> > > > wrote:
>> >> > > > >>>
>> >> > > > >>>> What does "mixed mode application" mean? What are the cases
>> >> where
>> >> > a
>> >> > > > >> user
>> >> > > > >>>> would want to use both APIs? I think that would help
>> understand
>> >> > the
>> >> > > > >>>> reasoning.
>> >> > > > >>>>
>> >> > > > >>>> Thanks,
>> >> > > > >>>> Ismael
>> >> > > > >>>>
>> >> > > > >>>> On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
>> >> > > > >>>> debasish.ghosh@lightbend.com> wrote:
>> >> > > > >>>>
>> >> > > > >>>>> Hi Damian -
>> >> > > > >>>>>
>> >> > > > >>>>> We could. But in case the user wants to use both Scala and
>> >> Java
>> >> > > APIs
>> >> > > > >>> (may
>> >> > > > >>>>> be for some mixed mode application), won't that be
>> confusing ?
>> >> > She
>> >> > > > >> will
>> >> > > > >>>>> have to do something like ..
>> >> > > > >>>>>
>> >> > > > >>>>> import o.a.k.s.scala.{KStream => KStreamS}
>> >> > > > >>>>>
>> >> > > > >>>>> to rename Scala imports or the other way round for imported
>> >> Java
>> >> > > > >>> classes.
>> >> > > > >>>>>
>> >> > > > >>>>> regards.
>> >> > > > >>>>>
>> >> > > > >>>>>
>> >> > > > >>>>>
>> >> > > > >>>>> On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <
>> >> > damian.guy@gmail.com>
>> >> > > > >>>> wrote:
>> >> > > > >>>>>
>> >> > > > >>>>>> Hi Debasish,
>> >> > > > >>>>>>
>> >> > > > >>>>>> Thanks for the KIP - will be a great addition to streams.
>> >> I've
>> >> > > only
>> >> > > > >>>> had a
>> >> > > > >>>>>> quick scan, but seeing as the Scala classes are going to
>> be
>> >> in
>> >> > > > >> their
>> >> > > > >>>> own
>> >> > > > >>>>>> package could we drop the S at the end of the class names?
>> >> > > > >>>>>>
>> >> > > > >>>>>> Thanks,
>> >> > > > >>>>>> Damian
>> >> > > > >>>>>>
>> >> > > > >>>>>>
>> >> > > > >>>>>> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
>> >> > > > >>>>> debasish.ghosh@lightbend.com>
>> >> > > > >>>>>> wrote:
>> >> > > > >>>>>>
>> >> > > > >>>>>>> Hi -
>> >> > > > >>>>>>>
>> >> > > > >>>>>>> A new KIP, KIP-270 is up for discussion:
>> >> > > > >>>>>>>
>> >> > > > >>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> >> > > > >>>>>> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
>> >> > > > >>>>>>>
>> >> > > > >>>>>>> The relevant JIRA issue: https://issues.apache.org/
>> >> > > > >>>>>> jira/browse/KAFKA-6670
>> >> > > > >>>>>>>
>> >> > > > >>>>>>> The library as proposed in the KIP has been implemented
>> at
>> >> > > > >>>>>>> https://github.com/lightbend/kafka-streams-scala and the
>> >> > current
>> >> > > > >>>>> release
>> >> > > > >>>>>>> is
>> >> > > > >>>>>>> 0.2.0 (
>> >> > > > >>>>>>> https://github.com/lightbend/k
>> afka-streams-scala/releases/
>> >> > > > >>> tag/v0.2.0
>> >> > > > >>>> ).
>> >> > > > >>>>>>> We at Lightbend has been using it since quite some time
>> now.
>> >> > > > >>>>>>>
>> >> > > > >>>>>>> regards.
>> >> > > > >>>>>>>
>> >> > > > >>>>>>> --
>> >> > > > >>>>>>> Debasish Ghosh
>> >> > > > >>>>>>> Principal Engineer
>> >> > > > >>>>>>>
>> >> > > > >>>>>>> Twitter: @debasishg
>> >> > > > >>>>>>> Blog: http://debasishg.blogspot.com
>> >> > > > >>>>>>> Code: https://github.com/debasishg
>> >> > > > >>>>>>>
>> >> > > > >>>>>>
>> >> > > > >>>>>
>> >> > > > >>>>>
>> >> > > > >>>>>
>> >> > > > >>>>> --
>> >> > > > >>>>> Debasish Ghosh
>> >> > > > >>>>> Principal Engineer
>> >> > > > >>>>>
>> >> > > > >>>>> Twitter: @debasishg
>> >> > > > >>>>> Blog: http://debasishg.blogspot.com
>> >> > > > >>>>> Code: https://github.com/debasishg
>> >> > > > >>>>>
>> >> > > > >>>>
>> >> > > > >>>
>> >> > > > >>>
>> >> > > > >>>
>> >> > > > >>> --
>> >> > > > >>> Debasish Ghosh
>> >> > > > >>> Principal Engineer
>> >> > > > >>>
>> >> > > > >>> Twitter: @debasishg
>> >> > > > >>> Blog: http://debasishg.blogspot.com
>> >> > > > >>> Code: https://github.com/debasishg
>> >> > > > >>>
>> >> > > > >>
>> >> > > > >
>> >> > > >
>> >> > > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Debasish Ghosh
>> >> > > Principal Engineer
>> >> > >
>> >> > > Twitter: @debasishg
>> >> > > Blog: http://debasishg.blogspot.com
>> >> > > Code: https://github.com/debasishg
>> >> > >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Senior Software Engineer, Lightbend, Inc.
>> >> >
>> >> > <http://lightbend.com>
>> >> >
>> >> > @seg1o <https://twitter.com/seg1o>, in/seanaglover
>> >> > <https://www.linkedin.com/in/seanaglover/>
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > -- Guozhang
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>
>
> --
> Debasish Ghosh
> Principal Engineer
>
> Twitter: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: https://github.com/debasishg
>
>
>
>
--
Debasish Ghosh
Principal Engineer
Twitter: @debasishg
Blog: http://debasishg.blogspot.com
Code: https://github.com/debasishg
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Debasish Ghosh <de...@lightbend.com>.
Thanks Guozhang for the comments .. we will start working on them ..
regards.
On Mon, Mar 19, 2018 at 11:02 PM, Guozhang Wang <wa...@gmail.com> wrote:
> And one more comment about the type safety:
>
> 7. I'm in favor of the approach of enforcing type safety at compile time,
> since with Scala's strong typing system I think it makes more sense to get
> rid of config-based serde specifications that can cause runtime error in
> the Scala wrapper.
>
>
> Guozhang
>
>
> On Mon, Mar 19, 2018 at 10:28 AM, Guozhang Wang <wa...@gmail.com>
> wrote:
>
> > Hello Debasish,
> >
> > Thanks for the KIP. Here are some comments:
> >
> > 1. About the naming: I'd also vote for option 2 and enforce users to
> > rename when import since this scenario seems rare to me.
> >
> > 2. About the dependency: since this KIP can only be merged as early as
> > 1.2, it means that for users who wants to use this artifact, say for
> > version 1.2, she would need to bring in "kafka-streams" version 1.2. In
> > addition, if we change the Java API in the future versions we'd also need
> > to update the Scala wrapper at the same version as well, so in other
> words
> > "kafka-streams-scala" version X have to be with "kafka-streams" version X
> > anyways. So I'd suggest we remove the version number in
> "kafka-streams-scala
> > only depends on the Scala standard library and Kafka Streams" as it may
> > be confusing to readers.
> >
> > 3. For the KIP discussion, it need be based on the proposed state of this
> > project in AK. More specifically, for the groupId in maven / gradle, it
> > need to be "org.apache.kafka"; for the version, it need to be Kafka
> release
> > versions, e.g. 1.2.0.
> >
> > 4. In "New or Changed Public Interfaces" section, it does not mention the
> > serde classes like "DefaultSerdes", but I think they should also be
> > considered public APIs as users would most likely import these in her
> > implementation.
> >
> > 5. Could you also list the changes you'd propose to made to build.gradle
> > in kafka for adding this artifact? More details will help readers to
> better
> > understand your proposal.
> >
> > 6. I think it'd will be good to have a WordCount example code as part of
> > this KIP, to illustrate how to code in this Scala wrapper, e.g. in
> > o.a.k.scala.examples. But for this class we probably do not need to have
> a
> > separate artifact for it as we did in kafka-streams-examples.
> >
> >
> > Guozhang
> >
> >
> > On Mon, Mar 19, 2018 at 9:16 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> I agree with Sean on name unification.
> >>
> >> +1 to option 2.
> >>
> >> On Mon, Mar 19, 2018 at 7:23 AM, Sean Glover <sean.glover@lightbend.com
> >
> >> wrote:
> >>
> >> > Type names: I vote for option 2. The user must explicitly add a
> >> dependency
> >> > to this library and the wrapper types are in a different package. It
> >> seems
> >> > reasonable to expect them to do an import rename if there's a need to
> >> drop
> >> > down to the Java API.
> >> >
> >> > Test Utils: The test utils in kafka-streams-scala are nice and lean,
> but
> >> > I'm not sure if it provides much more value than other options that
> >> exist
> >> > in the community. There's an embedded Kafka/ZK project implementation
> >> for
> >> > ScalaTest that's popular and active: manub/scalatest-embedded-kakfa.
> >> It
> >> > implies you must also use ScalaTest, which I acknowledge isn't
> >> everyone's
> >> > first choice for Scala test framework, but it probably is one of, if
> not
> >> > the most, popular library. It includes a DSL for Kafka Streams too.
> If
> >> > this KIP is accepted then perhaps a PR to that project could be made
> to
> >> > support the new wrapper implementations.
> >> >
> >> > https://github.com/manub/scalatest-embedded-kafka#
> >> > scalatest-embedded-kafka-streams
> >> >
> >> > Sean
> >> >
> >> > On Sun, Mar 18, 2018 at 4:05 AM, Debasish Ghosh <
> >> > debasish.ghosh@lightbend.com> wrote:
> >> >
> >> > > >
> >> > > > > Should this be 1.2 (maybe it's even better to not put any
> >> version at
> >> > > > all)
> >> > >
> >> > >
> >> > > Actually wanted to emphasize that the support is from 1.0.0 onwards
> ..
> >> > > Should we make that explicit ? Like ..
> >> > >
> >> > > kafka-streams-scala only depends on the Scala standard library and
> >> Kafka
> >> > > > Streams 1.0.0+.
> >> > >
> >> > >
> >> > > In 1.1 release, we add a new module `kafka-streams-test-utils` to
> >> > simplify
> >> > > > testing for Kafka Streams applications. Are those test utils
> >> suitable
> >> > for
> >> > > > Scala users or should we add Scala wrappers for those, too?
> >> > >
> >> > >
> >> > > I will check up and let you know ..
> >> > >
> >> > > Also I am not clear about the decision on renaming of Scala
> >> abstractions.
> >> > > Can we have a consensus on this ? Here's the summary ..
> >> > >
> >> > > *Option 1:* Keep names separate (KStream for Java class, KStreamS
> for
> >> > > Scala). No renaming of imports required.
> >> > > *Option 2:* Unify names (KStream for Java and Scala class names). No
> >> > > conflict since they will reside in different packages. But if we
> need
> >> to
> >> > > use both abstractions, renaming of imports are required. But again,
> >> this
> >> > > may not be a too frequent use case.
> >> > >
> >> > > Suggestions ?
> >> > >
> >> > > regards.
> >> > >
> >> > > On Sat, Mar 17, 2018 at 3:07 AM, Matthias J. Sax <
> >> matthias@confluent.io>
> >> > > wrote:
> >> > >
> >> > > > Thanks a lot for the KIP! Two questions:
> >> > > >
> >> > > > 1) the KIP states:
> >> > > >
> >> > > > > kafka-streams-scala only depends on the Scala standard library
> and
> >> > > Kafka
> >> > > > Streams 1.0.0.
> >> > > >
> >> > > > Should this be 1.2 (maybe it's even better to not put any version
> >> at
> >> > > all)
> >> > > >
> >> > > >
> >> > > > 2) In 1.1 release, we add a new module `kafka-streams-test-utils`
> to
> >> > > > simplify testing for Kafka Streams applications. Are those test
> >> utils
> >> > > > suitable for Scala users or should we add Scala wrappers for
> those,
> >> > too?
> >> > > >
> >> > > >
> >> > > > -Matthias
> >> > > >
> >> > > >
> >> > > > On 3/16/18 11:10 AM, Ted Yu wrote:
> >> > > > > Import renames seem to be fine.
> >> > > > >
> >> > > > > The class names with trailing 'S' look clean.
> >> > > > >
> >> > > > > Cheers
> >> > > > >
> >> > > > > On Fri, Mar 16, 2018 at 11:04 AM, Ismael Juma <
> ismael@juma.me.uk>
> >> > > wrote:
> >> > > > >
> >> > > > >> If this is rare (as it sounds), relying on import renames seems
> >> fine
> >> > > to
> >> > > > me.
> >> > > > >> Let's see what others think.
> >> > > > >>
> >> > > > >> Ismael
> >> > > > >>
> >> > > > >> On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
> >> > > > >> debasish.ghosh@lightbend.com> wrote:
> >> > > > >>
> >> > > > >>> I am not sure if this is practical or not. But theoretically a
> >> user
> >> > > may
> >> > > > >>> want to extract the unsafe Java abstraction from the Scala
> ones
> >> and
> >> > > use
> >> > > > >>> Java APIs on them .. e.g.
> >> > > > >>>
> >> > > > >>> val userClicksStream: KStreamS[String, Long] =
> >> > > > >>> builder.stream(userClicksTopic) // Scala abstraction
> >> > > > >>>
> >> > > > >>> val jStream: KStream[String, Long] = userClicksStream.inner //
> >> > > > publishes
> >> > > > >>> the underlying Java abstraction
> >> > > > >>>
> >> > > > >>> //.. work with Java, may be pass to some function written in
> >> Java
> >> > > > >>>
> >> > > > >>> I do realize this is somewhat of a convoluted use case and may
> >> not
> >> > be
> >> > > > >>> practically useful ..
> >> > > > >>>
> >> > > > >>> Otherwise we can very well work on the suggested approach of
> >> > unifying
> >> > > > the
> >> > > > >>> names ..
> >> > > > >>>
> >> > > > >>> regards.
> >> > > > >>>
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <
> >> ismael@juma.me.uk>
> >> > > > wrote:
> >> > > > >>>
> >> > > > >>>> What does "mixed mode application" mean? What are the cases
> >> where
> >> > a
> >> > > > >> user
> >> > > > >>>> would want to use both APIs? I think that would help
> understand
> >> > the
> >> > > > >>>> reasoning.
> >> > > > >>>>
> >> > > > >>>> Thanks,
> >> > > > >>>> Ismael
> >> > > > >>>>
> >> > > > >>>> On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
> >> > > > >>>> debasish.ghosh@lightbend.com> wrote:
> >> > > > >>>>
> >> > > > >>>>> Hi Damian -
> >> > > > >>>>>
> >> > > > >>>>> We could. But in case the user wants to use both Scala and
> >> Java
> >> > > APIs
> >> > > > >>> (may
> >> > > > >>>>> be for some mixed mode application), won't that be
> confusing ?
> >> > She
> >> > > > >> will
> >> > > > >>>>> have to do something like ..
> >> > > > >>>>>
> >> > > > >>>>> import o.a.k.s.scala.{KStream => KStreamS}
> >> > > > >>>>>
> >> > > > >>>>> to rename Scala imports or the other way round for imported
> >> Java
> >> > > > >>> classes.
> >> > > > >>>>>
> >> > > > >>>>> regards.
> >> > > > >>>>>
> >> > > > >>>>>
> >> > > > >>>>>
> >> > > > >>>>> On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <
> >> > damian.guy@gmail.com>
> >> > > > >>>> wrote:
> >> > > > >>>>>
> >> > > > >>>>>> Hi Debasish,
> >> > > > >>>>>>
> >> > > > >>>>>> Thanks for the KIP - will be a great addition to streams.
> >> I've
> >> > > only
> >> > > > >>>> had a
> >> > > > >>>>>> quick scan, but seeing as the Scala classes are going to be
> >> in
> >> > > > >> their
> >> > > > >>>> own
> >> > > > >>>>>> package could we drop the S at the end of the class names?
> >> > > > >>>>>>
> >> > > > >>>>>> Thanks,
> >> > > > >>>>>> Damian
> >> > > > >>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> >> > > > >>>>> debasish.ghosh@lightbend.com>
> >> > > > >>>>>> wrote:
> >> > > > >>>>>>
> >> > > > >>>>>>> Hi -
> >> > > > >>>>>>>
> >> > > > >>>>>>> A new KIP, KIP-270 is up for discussion:
> >> > > > >>>>>>>
> >> > > > >>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> > > > >>>>>> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> >> > > > >>>>>>>
> >> > > > >>>>>>> The relevant JIRA issue: https://issues.apache.org/
> >> > > > >>>>>> jira/browse/KAFKA-6670
> >> > > > >>>>>>>
> >> > > > >>>>>>> The library as proposed in the KIP has been implemented at
> >> > > > >>>>>>> https://github.com/lightbend/kafka-streams-scala and the
> >> > current
> >> > > > >>>>> release
> >> > > > >>>>>>> is
> >> > > > >>>>>>> 0.2.0 (
> >> > > > >>>>>>> https://github.com/lightbend/
> kafka-streams-scala/releases/
> >> > > > >>> tag/v0.2.0
> >> > > > >>>> ).
> >> > > > >>>>>>> We at Lightbend has been using it since quite some time
> now.
> >> > > > >>>>>>>
> >> > > > >>>>>>> regards.
> >> > > > >>>>>>>
> >> > > > >>>>>>> --
> >> > > > >>>>>>> Debasish Ghosh
> >> > > > >>>>>>> Principal Engineer
> >> > > > >>>>>>>
> >> > > > >>>>>>> Twitter: @debasishg
> >> > > > >>>>>>> Blog: http://debasishg.blogspot.com
> >> > > > >>>>>>> Code: https://github.com/debasishg
> >> > > > >>>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>
> >> > > > >>>>>
> >> > > > >>>>>
> >> > > > >>>>> --
> >> > > > >>>>> Debasish Ghosh
> >> > > > >>>>> Principal Engineer
> >> > > > >>>>>
> >> > > > >>>>> Twitter: @debasishg
> >> > > > >>>>> Blog: http://debasishg.blogspot.com
> >> > > > >>>>> Code: https://github.com/debasishg
> >> > > > >>>>>
> >> > > > >>>>
> >> > > > >>>
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> --
> >> > > > >>> Debasish Ghosh
> >> > > > >>> Principal Engineer
> >> > > > >>>
> >> > > > >>> Twitter: @debasishg
> >> > > > >>> Blog: http://debasishg.blogspot.com
> >> > > > >>> Code: https://github.com/debasishg
> >> > > > >>>
> >> > > > >>
> >> > > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > Debasish Ghosh
> >> > > Principal Engineer
> >> > >
> >> > > Twitter: @debasishg
> >> > > Blog: http://debasishg.blogspot.com
> >> > > Code: https://github.com/debasishg
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Senior Software Engineer, Lightbend, Inc.
> >> >
> >> > <http://lightbend.com>
> >> >
> >> > @seg1o <https://twitter.com/seg1o>, in/seanaglover
> >> > <https://www.linkedin.com/in/seanaglover/>
> >> >
> >>
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> -- Guozhang
>
--
Debasish Ghosh
Principal Engineer
Twitter: @debasishg
Blog: http://debasishg.blogspot.com
Code: https://github.com/debasishg
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Guozhang Wang <wa...@gmail.com>.
And one more comment about the type safety:
7. I'm in favor of the approach of enforcing type safety at compile time,
since with Scala's strong typing system I think it makes more sense to get
rid of config-based serde specifications that can cause runtime error in
the Scala wrapper.
Guozhang
On Mon, Mar 19, 2018 at 10:28 AM, Guozhang Wang <wa...@gmail.com> wrote:
> Hello Debasish,
>
> Thanks for the KIP. Here are some comments:
>
> 1. About the naming: I'd also vote for option 2 and enforce users to
> rename when import since this scenario seems rare to me.
>
> 2. About the dependency: since this KIP can only be merged as early as
> 1.2, it means that for users who wants to use this artifact, say for
> version 1.2, she would need to bring in "kafka-streams" version 1.2. In
> addition, if we change the Java API in the future versions we'd also need
> to update the Scala wrapper at the same version as well, so in other words
> "kafka-streams-scala" version X have to be with "kafka-streams" version X
> anyways. So I'd suggest we remove the version number in "kafka-streams-scala
> only depends on the Scala standard library and Kafka Streams" as it may
> be confusing to readers.
>
> 3. For the KIP discussion, it need be based on the proposed state of this
> project in AK. More specifically, for the groupId in maven / gradle, it
> need to be "org.apache.kafka"; for the version, it need to be Kafka release
> versions, e.g. 1.2.0.
>
> 4. In "New or Changed Public Interfaces" section, it does not mention the
> serde classes like "DefaultSerdes", but I think they should also be
> considered public APIs as users would most likely import these in her
> implementation.
>
> 5. Could you also list the changes you'd propose to made to build.gradle
> in kafka for adding this artifact? More details will help readers to better
> understand your proposal.
>
> 6. I think it'd will be good to have a WordCount example code as part of
> this KIP, to illustrate how to code in this Scala wrapper, e.g. in
> o.a.k.scala.examples. But for this class we probably do not need to have a
> separate artifact for it as we did in kafka-streams-examples.
>
>
> Guozhang
>
>
> On Mon, Mar 19, 2018 at 9:16 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> I agree with Sean on name unification.
>>
>> +1 to option 2.
>>
>> On Mon, Mar 19, 2018 at 7:23 AM, Sean Glover <se...@lightbend.com>
>> wrote:
>>
>> > Type names: I vote for option 2. The user must explicitly add a
>> dependency
>> > to this library and the wrapper types are in a different package. It
>> seems
>> > reasonable to expect them to do an import rename if there's a need to
>> drop
>> > down to the Java API.
>> >
>> > Test Utils: The test utils in kafka-streams-scala are nice and lean, but
>> > I'm not sure if it provides much more value than other options that
>> exist
>> > in the community. There's an embedded Kafka/ZK project implementation
>> for
>> > ScalaTest that's popular and active: manub/scalatest-embedded-kakfa.
>> It
>> > implies you must also use ScalaTest, which I acknowledge isn't
>> everyone's
>> > first choice for Scala test framework, but it probably is one of, if not
>> > the most, popular library. It includes a DSL for Kafka Streams too. If
>> > this KIP is accepted then perhaps a PR to that project could be made to
>> > support the new wrapper implementations.
>> >
>> > https://github.com/manub/scalatest-embedded-kafka#
>> > scalatest-embedded-kafka-streams
>> >
>> > Sean
>> >
>> > On Sun, Mar 18, 2018 at 4:05 AM, Debasish Ghosh <
>> > debasish.ghosh@lightbend.com> wrote:
>> >
>> > > >
>> > > > > Should this be 1.2 (maybe it's even better to not put any
>> version at
>> > > > all)
>> > >
>> > >
>> > > Actually wanted to emphasize that the support is from 1.0.0 onwards ..
>> > > Should we make that explicit ? Like ..
>> > >
>> > > kafka-streams-scala only depends on the Scala standard library and
>> Kafka
>> > > > Streams 1.0.0+.
>> > >
>> > >
>> > > In 1.1 release, we add a new module `kafka-streams-test-utils` to
>> > simplify
>> > > > testing for Kafka Streams applications. Are those test utils
>> suitable
>> > for
>> > > > Scala users or should we add Scala wrappers for those, too?
>> > >
>> > >
>> > > I will check up and let you know ..
>> > >
>> > > Also I am not clear about the decision on renaming of Scala
>> abstractions.
>> > > Can we have a consensus on this ? Here's the summary ..
>> > >
>> > > *Option 1:* Keep names separate (KStream for Java class, KStreamS for
>> > > Scala). No renaming of imports required.
>> > > *Option 2:* Unify names (KStream for Java and Scala class names). No
>> > > conflict since they will reside in different packages. But if we need
>> to
>> > > use both abstractions, renaming of imports are required. But again,
>> this
>> > > may not be a too frequent use case.
>> > >
>> > > Suggestions ?
>> > >
>> > > regards.
>> > >
>> > > On Sat, Mar 17, 2018 at 3:07 AM, Matthias J. Sax <
>> matthias@confluent.io>
>> > > wrote:
>> > >
>> > > > Thanks a lot for the KIP! Two questions:
>> > > >
>> > > > 1) the KIP states:
>> > > >
>> > > > > kafka-streams-scala only depends on the Scala standard library and
>> > > Kafka
>> > > > Streams 1.0.0.
>> > > >
>> > > > Should this be 1.2 (maybe it's even better to not put any version
>> at
>> > > all)
>> > > >
>> > > >
>> > > > 2) In 1.1 release, we add a new module `kafka-streams-test-utils` to
>> > > > simplify testing for Kafka Streams applications. Are those test
>> utils
>> > > > suitable for Scala users or should we add Scala wrappers for those,
>> > too?
>> > > >
>> > > >
>> > > > -Matthias
>> > > >
>> > > >
>> > > > On 3/16/18 11:10 AM, Ted Yu wrote:
>> > > > > Import renames seem to be fine.
>> > > > >
>> > > > > The class names with trailing 'S' look clean.
>> > > > >
>> > > > > Cheers
>> > > > >
>> > > > > On Fri, Mar 16, 2018 at 11:04 AM, Ismael Juma <is...@juma.me.uk>
>> > > wrote:
>> > > > >
>> > > > >> If this is rare (as it sounds), relying on import renames seems
>> fine
>> > > to
>> > > > me.
>> > > > >> Let's see what others think.
>> > > > >>
>> > > > >> Ismael
>> > > > >>
>> > > > >> On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
>> > > > >> debasish.ghosh@lightbend.com> wrote:
>> > > > >>
>> > > > >>> I am not sure if this is practical or not. But theoretically a
>> user
>> > > may
>> > > > >>> want to extract the unsafe Java abstraction from the Scala ones
>> and
>> > > use
>> > > > >>> Java APIs on them .. e.g.
>> > > > >>>
>> > > > >>> val userClicksStream: KStreamS[String, Long] =
>> > > > >>> builder.stream(userClicksTopic) // Scala abstraction
>> > > > >>>
>> > > > >>> val jStream: KStream[String, Long] = userClicksStream.inner //
>> > > > publishes
>> > > > >>> the underlying Java abstraction
>> > > > >>>
>> > > > >>> //.. work with Java, may be pass to some function written in
>> Java
>> > > > >>>
>> > > > >>> I do realize this is somewhat of a convoluted use case and may
>> not
>> > be
>> > > > >>> practically useful ..
>> > > > >>>
>> > > > >>> Otherwise we can very well work on the suggested approach of
>> > unifying
>> > > > the
>> > > > >>> names ..
>> > > > >>>
>> > > > >>> regards.
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <
>> ismael@juma.me.uk>
>> > > > wrote:
>> > > > >>>
>> > > > >>>> What does "mixed mode application" mean? What are the cases
>> where
>> > a
>> > > > >> user
>> > > > >>>> would want to use both APIs? I think that would help understand
>> > the
>> > > > >>>> reasoning.
>> > > > >>>>
>> > > > >>>> Thanks,
>> > > > >>>> Ismael
>> > > > >>>>
>> > > > >>>> On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
>> > > > >>>> debasish.ghosh@lightbend.com> wrote:
>> > > > >>>>
>> > > > >>>>> Hi Damian -
>> > > > >>>>>
>> > > > >>>>> We could. But in case the user wants to use both Scala and
>> Java
>> > > APIs
>> > > > >>> (may
>> > > > >>>>> be for some mixed mode application), won't that be confusing ?
>> > She
>> > > > >> will
>> > > > >>>>> have to do something like ..
>> > > > >>>>>
>> > > > >>>>> import o.a.k.s.scala.{KStream => KStreamS}
>> > > > >>>>>
>> > > > >>>>> to rename Scala imports or the other way round for imported
>> Java
>> > > > >>> classes.
>> > > > >>>>>
>> > > > >>>>> regards.
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>> On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <
>> > damian.guy@gmail.com>
>> > > > >>>> wrote:
>> > > > >>>>>
>> > > > >>>>>> Hi Debasish,
>> > > > >>>>>>
>> > > > >>>>>> Thanks for the KIP - will be a great addition to streams.
>> I've
>> > > only
>> > > > >>>> had a
>> > > > >>>>>> quick scan, but seeing as the Scala classes are going to be
>> in
>> > > > >> their
>> > > > >>>> own
>> > > > >>>>>> package could we drop the S at the end of the class names?
>> > > > >>>>>>
>> > > > >>>>>> Thanks,
>> > > > >>>>>> Damian
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
>> > > > >>>>> debasish.ghosh@lightbend.com>
>> > > > >>>>>> wrote:
>> > > > >>>>>>
>> > > > >>>>>>> Hi -
>> > > > >>>>>>>
>> > > > >>>>>>> A new KIP, KIP-270 is up for discussion:
>> > > > >>>>>>>
>> > > > >>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > >>>>>> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
>> > > > >>>>>>>
>> > > > >>>>>>> The relevant JIRA issue: https://issues.apache.org/
>> > > > >>>>>> jira/browse/KAFKA-6670
>> > > > >>>>>>>
>> > > > >>>>>>> The library as proposed in the KIP has been implemented at
>> > > > >>>>>>> https://github.com/lightbend/kafka-streams-scala and the
>> > current
>> > > > >>>>> release
>> > > > >>>>>>> is
>> > > > >>>>>>> 0.2.0 (
>> > > > >>>>>>> https://github.com/lightbend/kafka-streams-scala/releases/
>> > > > >>> tag/v0.2.0
>> > > > >>>> ).
>> > > > >>>>>>> We at Lightbend has been using it since quite some time now.
>> > > > >>>>>>>
>> > > > >>>>>>> regards.
>> > > > >>>>>>>
>> > > > >>>>>>> --
>> > > > >>>>>>> Debasish Ghosh
>> > > > >>>>>>> Principal Engineer
>> > > > >>>>>>>
>> > > > >>>>>>> Twitter: @debasishg
>> > > > >>>>>>> Blog: http://debasishg.blogspot.com
>> > > > >>>>>>> Code: https://github.com/debasishg
>> > > > >>>>>>>
>> > > > >>>>>>
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>> --
>> > > > >>>>> Debasish Ghosh
>> > > > >>>>> Principal Engineer
>> > > > >>>>>
>> > > > >>>>> Twitter: @debasishg
>> > > > >>>>> Blog: http://debasishg.blogspot.com
>> > > > >>>>> Code: https://github.com/debasishg
>> > > > >>>>>
>> > > > >>>>
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> --
>> > > > >>> Debasish Ghosh
>> > > > >>> Principal Engineer
>> > > > >>>
>> > > > >>> Twitter: @debasishg
>> > > > >>> Blog: http://debasishg.blogspot.com
>> > > > >>> Code: https://github.com/debasishg
>> > > > >>>
>> > > > >>
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Debasish Ghosh
>> > > Principal Engineer
>> > >
>> > > Twitter: @debasishg
>> > > Blog: http://debasishg.blogspot.com
>> > > Code: https://github.com/debasishg
>> > >
>> >
>> >
>> >
>> > --
>> > Senior Software Engineer, Lightbend, Inc.
>> >
>> > <http://lightbend.com>
>> >
>> > @seg1o <https://twitter.com/seg1o>, in/seanaglover
>> > <https://www.linkedin.com/in/seanaglover/>
>> >
>>
>
>
>
> --
> -- Guozhang
>
--
-- Guozhang
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Debasish,
Thanks for the KIP. Here are some comments:
1. About the naming: I'd also vote for option 2 and enforce users to rename
when import since this scenario seems rare to me.
2. About the dependency: since this KIP can only be merged as early as 1.2,
it means that for users who wants to use this artifact, say for version
1.2, she would need to bring in "kafka-streams" version 1.2. In addition,
if we change the Java API in the future versions we'd also need to update
the Scala wrapper at the same version as well, so in other words
"kafka-streams-scala" version X have to be with "kafka-streams" version X
anyways. So I'd suggest we remove the version number in "kafka-streams-scala
only depends on the Scala standard library and Kafka Streams" as it may be
confusing to readers.
3. For the KIP discussion, it need be based on the proposed state of this
project in AK. More specifically, for the groupId in maven / gradle, it
need to be "org.apache.kafka"; for the version, it need to be Kafka release
versions, e.g. 1.2.0.
4. In "New or Changed Public Interfaces" section, it does not mention the
serde classes like "DefaultSerdes", but I think they should also be
considered public APIs as users would most likely import these in her
implementation.
5. Could you also list the changes you'd propose to made to build.gradle in
kafka for adding this artifact? More details will help readers to better
understand your proposal.
6. I think it'd will be good to have a WordCount example code as part of
this KIP, to illustrate how to code in this Scala wrapper, e.g. in
o.a.k.scala.examples. But for this class we probably do not need to have a
separate artifact for it as we did in kafka-streams-examples.
Guozhang
On Mon, Mar 19, 2018 at 9:16 AM, Ted Yu <yu...@gmail.com> wrote:
> I agree with Sean on name unification.
>
> +1 to option 2.
>
> On Mon, Mar 19, 2018 at 7:23 AM, Sean Glover <se...@lightbend.com>
> wrote:
>
> > Type names: I vote for option 2. The user must explicitly add a
> dependency
> > to this library and the wrapper types are in a different package. It
> seems
> > reasonable to expect them to do an import rename if there's a need to
> drop
> > down to the Java API.
> >
> > Test Utils: The test utils in kafka-streams-scala are nice and lean, but
> > I'm not sure if it provides much more value than other options that exist
> > in the community. There's an embedded Kafka/ZK project implementation
> for
> > ScalaTest that's popular and active: manub/scalatest-embedded-kakfa. It
> > implies you must also use ScalaTest, which I acknowledge isn't everyone's
> > first choice for Scala test framework, but it probably is one of, if not
> > the most, popular library. It includes a DSL for Kafka Streams too. If
> > this KIP is accepted then perhaps a PR to that project could be made to
> > support the new wrapper implementations.
> >
> > https://github.com/manub/scalatest-embedded-kafka#
> > scalatest-embedded-kafka-streams
> >
> > Sean
> >
> > On Sun, Mar 18, 2018 at 4:05 AM, Debasish Ghosh <
> > debasish.ghosh@lightbend.com> wrote:
> >
> > > >
> > > > > Should this be 1.2 (maybe it's even better to not put any version
> at
> > > > all)
> > >
> > >
> > > Actually wanted to emphasize that the support is from 1.0.0 onwards ..
> > > Should we make that explicit ? Like ..
> > >
> > > kafka-streams-scala only depends on the Scala standard library and
> Kafka
> > > > Streams 1.0.0+.
> > >
> > >
> > > In 1.1 release, we add a new module `kafka-streams-test-utils` to
> > simplify
> > > > testing for Kafka Streams applications. Are those test utils suitable
> > for
> > > > Scala users or should we add Scala wrappers for those, too?
> > >
> > >
> > > I will check up and let you know ..
> > >
> > > Also I am not clear about the decision on renaming of Scala
> abstractions.
> > > Can we have a consensus on this ? Here's the summary ..
> > >
> > > *Option 1:* Keep names separate (KStream for Java class, KStreamS for
> > > Scala). No renaming of imports required.
> > > *Option 2:* Unify names (KStream for Java and Scala class names). No
> > > conflict since they will reside in different packages. But if we need
> to
> > > use both abstractions, renaming of imports are required. But again,
> this
> > > may not be a too frequent use case.
> > >
> > > Suggestions ?
> > >
> > > regards.
> > >
> > > On Sat, Mar 17, 2018 at 3:07 AM, Matthias J. Sax <
> matthias@confluent.io>
> > > wrote:
> > >
> > > > Thanks a lot for the KIP! Two questions:
> > > >
> > > > 1) the KIP states:
> > > >
> > > > > kafka-streams-scala only depends on the Scala standard library and
> > > Kafka
> > > > Streams 1.0.0.
> > > >
> > > > Should this be 1.2 (maybe it's even better to not put any version at
> > > all)
> > > >
> > > >
> > > > 2) In 1.1 release, we add a new module `kafka-streams-test-utils` to
> > > > simplify testing for Kafka Streams applications. Are those test utils
> > > > suitable for Scala users or should we add Scala wrappers for those,
> > too?
> > > >
> > > >
> > > > -Matthias
> > > >
> > > >
> > > > On 3/16/18 11:10 AM, Ted Yu wrote:
> > > > > Import renames seem to be fine.
> > > > >
> > > > > The class names with trailing 'S' look clean.
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Fri, Mar 16, 2018 at 11:04 AM, Ismael Juma <is...@juma.me.uk>
> > > wrote:
> > > > >
> > > > >> If this is rare (as it sounds), relying on import renames seems
> fine
> > > to
> > > > me.
> > > > >> Let's see what others think.
> > > > >>
> > > > >> Ismael
> > > > >>
> > > > >> On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
> > > > >> debasish.ghosh@lightbend.com> wrote:
> > > > >>
> > > > >>> I am not sure if this is practical or not. But theoretically a
> user
> > > may
> > > > >>> want to extract the unsafe Java abstraction from the Scala ones
> and
> > > use
> > > > >>> Java APIs on them .. e.g.
> > > > >>>
> > > > >>> val userClicksStream: KStreamS[String, Long] =
> > > > >>> builder.stream(userClicksTopic) // Scala abstraction
> > > > >>>
> > > > >>> val jStream: KStream[String, Long] = userClicksStream.inner //
> > > > publishes
> > > > >>> the underlying Java abstraction
> > > > >>>
> > > > >>> //.. work with Java, may be pass to some function written in Java
> > > > >>>
> > > > >>> I do realize this is somewhat of a convoluted use case and may
> not
> > be
> > > > >>> practically useful ..
> > > > >>>
> > > > >>> Otherwise we can very well work on the suggested approach of
> > unifying
> > > > the
> > > > >>> names ..
> > > > >>>
> > > > >>> regards.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <ismael@juma.me.uk
> >
> > > > wrote:
> > > > >>>
> > > > >>>> What does "mixed mode application" mean? What are the cases
> where
> > a
> > > > >> user
> > > > >>>> would want to use both APIs? I think that would help understand
> > the
> > > > >>>> reasoning.
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>> Ismael
> > > > >>>>
> > > > >>>> On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
> > > > >>>> debasish.ghosh@lightbend.com> wrote:
> > > > >>>>
> > > > >>>>> Hi Damian -
> > > > >>>>>
> > > > >>>>> We could. But in case the user wants to use both Scala and Java
> > > APIs
> > > > >>> (may
> > > > >>>>> be for some mixed mode application), won't that be confusing ?
> > She
> > > > >> will
> > > > >>>>> have to do something like ..
> > > > >>>>>
> > > > >>>>> import o.a.k.s.scala.{KStream => KStreamS}
> > > > >>>>>
> > > > >>>>> to rename Scala imports or the other way round for imported
> Java
> > > > >>> classes.
> > > > >>>>>
> > > > >>>>> regards.
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <
> > damian.guy@gmail.com>
> > > > >>>> wrote:
> > > > >>>>>
> > > > >>>>>> Hi Debasish,
> > > > >>>>>>
> > > > >>>>>> Thanks for the KIP - will be a great addition to streams. I've
> > > only
> > > > >>>> had a
> > > > >>>>>> quick scan, but seeing as the Scala classes are going to be in
> > > > >> their
> > > > >>>> own
> > > > >>>>>> package could we drop the S at the end of the class names?
> > > > >>>>>>
> > > > >>>>>> Thanks,
> > > > >>>>>> Damian
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> > > > >>>>> debasish.ghosh@lightbend.com>
> > > > >>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hi -
> > > > >>>>>>>
> > > > >>>>>>> A new KIP, KIP-270 is up for discussion:
> > > > >>>>>>>
> > > > >>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >>>>>> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> > > > >>>>>>>
> > > > >>>>>>> The relevant JIRA issue: https://issues.apache.org/
> > > > >>>>>> jira/browse/KAFKA-6670
> > > > >>>>>>>
> > > > >>>>>>> The library as proposed in the KIP has been implemented at
> > > > >>>>>>> https://github.com/lightbend/kafka-streams-scala and the
> > current
> > > > >>>>> release
> > > > >>>>>>> is
> > > > >>>>>>> 0.2.0 (
> > > > >>>>>>> https://github.com/lightbend/kafka-streams-scala/releases/
> > > > >>> tag/v0.2.0
> > > > >>>> ).
> > > > >>>>>>> We at Lightbend has been using it since quite some time now.
> > > > >>>>>>>
> > > > >>>>>>> regards.
> > > > >>>>>>>
> > > > >>>>>>> --
> > > > >>>>>>> Debasish Ghosh
> > > > >>>>>>> Principal Engineer
> > > > >>>>>>>
> > > > >>>>>>> Twitter: @debasishg
> > > > >>>>>>> Blog: http://debasishg.blogspot.com
> > > > >>>>>>> Code: https://github.com/debasishg
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> --
> > > > >>>>> Debasish Ghosh
> > > > >>>>> Principal Engineer
> > > > >>>>>
> > > > >>>>> Twitter: @debasishg
> > > > >>>>> Blog: http://debasishg.blogspot.com
> > > > >>>>> Code: https://github.com/debasishg
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Debasish Ghosh
> > > > >>> Principal Engineer
> > > > >>>
> > > > >>> Twitter: @debasishg
> > > > >>> Blog: http://debasishg.blogspot.com
> > > > >>> Code: https://github.com/debasishg
> > > > >>>
> > > > >>
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Debasish Ghosh
> > > Principal Engineer
> > >
> > > Twitter: @debasishg
> > > Blog: http://debasishg.blogspot.com
> > > Code: https://github.com/debasishg
> > >
> >
> >
> >
> > --
> > Senior Software Engineer, Lightbend, Inc.
> >
> > <http://lightbend.com>
> >
> > @seg1o <https://twitter.com/seg1o>, in/seanaglover
> > <https://www.linkedin.com/in/seanaglover/>
> >
>
--
-- Guozhang
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Ted Yu <yu...@gmail.com>.
I agree with Sean on name unification.
+1 to option 2.
On Mon, Mar 19, 2018 at 7:23 AM, Sean Glover <se...@lightbend.com>
wrote:
> Type names: I vote for option 2. The user must explicitly add a dependency
> to this library and the wrapper types are in a different package. It seems
> reasonable to expect them to do an import rename if there's a need to drop
> down to the Java API.
>
> Test Utils: The test utils in kafka-streams-scala are nice and lean, but
> I'm not sure if it provides much more value than other options that exist
> in the community. There's an embedded Kafka/ZK project implementation for
> ScalaTest that's popular and active: manub/scalatest-embedded-kakfa. It
> implies you must also use ScalaTest, which I acknowledge isn't everyone's
> first choice for Scala test framework, but it probably is one of, if not
> the most, popular library. It includes a DSL for Kafka Streams too. If
> this KIP is accepted then perhaps a PR to that project could be made to
> support the new wrapper implementations.
>
> https://github.com/manub/scalatest-embedded-kafka#
> scalatest-embedded-kafka-streams
>
> Sean
>
> On Sun, Mar 18, 2018 at 4:05 AM, Debasish Ghosh <
> debasish.ghosh@lightbend.com> wrote:
>
> > >
> > > > Should this be 1.2 (maybe it's even better to not put any version at
> > > all)
> >
> >
> > Actually wanted to emphasize that the support is from 1.0.0 onwards ..
> > Should we make that explicit ? Like ..
> >
> > kafka-streams-scala only depends on the Scala standard library and Kafka
> > > Streams 1.0.0+.
> >
> >
> > In 1.1 release, we add a new module `kafka-streams-test-utils` to
> simplify
> > > testing for Kafka Streams applications. Are those test utils suitable
> for
> > > Scala users or should we add Scala wrappers for those, too?
> >
> >
> > I will check up and let you know ..
> >
> > Also I am not clear about the decision on renaming of Scala abstractions.
> > Can we have a consensus on this ? Here's the summary ..
> >
> > *Option 1:* Keep names separate (KStream for Java class, KStreamS for
> > Scala). No renaming of imports required.
> > *Option 2:* Unify names (KStream for Java and Scala class names). No
> > conflict since they will reside in different packages. But if we need to
> > use both abstractions, renaming of imports are required. But again, this
> > may not be a too frequent use case.
> >
> > Suggestions ?
> >
> > regards.
> >
> > On Sat, Mar 17, 2018 at 3:07 AM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> > > Thanks a lot for the KIP! Two questions:
> > >
> > > 1) the KIP states:
> > >
> > > > kafka-streams-scala only depends on the Scala standard library and
> > Kafka
> > > Streams 1.0.0.
> > >
> > > Should this be 1.2 (maybe it's even better to not put any version at
> > all)
> > >
> > >
> > > 2) In 1.1 release, we add a new module `kafka-streams-test-utils` to
> > > simplify testing for Kafka Streams applications. Are those test utils
> > > suitable for Scala users or should we add Scala wrappers for those,
> too?
> > >
> > >
> > > -Matthias
> > >
> > >
> > > On 3/16/18 11:10 AM, Ted Yu wrote:
> > > > Import renames seem to be fine.
> > > >
> > > > The class names with trailing 'S' look clean.
> > > >
> > > > Cheers
> > > >
> > > > On Fri, Mar 16, 2018 at 11:04 AM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > >
> > > >> If this is rare (as it sounds), relying on import renames seems fine
> > to
> > > me.
> > > >> Let's see what others think.
> > > >>
> > > >> Ismael
> > > >>
> > > >> On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
> > > >> debasish.ghosh@lightbend.com> wrote:
> > > >>
> > > >>> I am not sure if this is practical or not. But theoretically a user
> > may
> > > >>> want to extract the unsafe Java abstraction from the Scala ones and
> > use
> > > >>> Java APIs on them .. e.g.
> > > >>>
> > > >>> val userClicksStream: KStreamS[String, Long] =
> > > >>> builder.stream(userClicksTopic) // Scala abstraction
> > > >>>
> > > >>> val jStream: KStream[String, Long] = userClicksStream.inner //
> > > publishes
> > > >>> the underlying Java abstraction
> > > >>>
> > > >>> //.. work with Java, may be pass to some function written in Java
> > > >>>
> > > >>> I do realize this is somewhat of a convoluted use case and may not
> be
> > > >>> practically useful ..
> > > >>>
> > > >>> Otherwise we can very well work on the suggested approach of
> unifying
> > > the
> > > >>> names ..
> > > >>>
> > > >>> regards.
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <is...@juma.me.uk>
> > > wrote:
> > > >>>
> > > >>>> What does "mixed mode application" mean? What are the cases where
> a
> > > >> user
> > > >>>> would want to use both APIs? I think that would help understand
> the
> > > >>>> reasoning.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Ismael
> > > >>>>
> > > >>>> On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
> > > >>>> debasish.ghosh@lightbend.com> wrote:
> > > >>>>
> > > >>>>> Hi Damian -
> > > >>>>>
> > > >>>>> We could. But in case the user wants to use both Scala and Java
> > APIs
> > > >>> (may
> > > >>>>> be for some mixed mode application), won't that be confusing ?
> She
> > > >> will
> > > >>>>> have to do something like ..
> > > >>>>>
> > > >>>>> import o.a.k.s.scala.{KStream => KStreamS}
> > > >>>>>
> > > >>>>> to rename Scala imports or the other way round for imported Java
> > > >>> classes.
> > > >>>>>
> > > >>>>> regards.
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <
> damian.guy@gmail.com>
> > > >>>> wrote:
> > > >>>>>
> > > >>>>>> Hi Debasish,
> > > >>>>>>
> > > >>>>>> Thanks for the KIP - will be a great addition to streams. I've
> > only
> > > >>>> had a
> > > >>>>>> quick scan, but seeing as the Scala classes are going to be in
> > > >> their
> > > >>>> own
> > > >>>>>> package could we drop the S at the end of the class names?
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>> Damian
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> > > >>>>> debasish.ghosh@lightbend.com>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hi -
> > > >>>>>>>
> > > >>>>>>> A new KIP, KIP-270 is up for discussion:
> > > >>>>>>>
> > > >>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >>>>>> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> > > >>>>>>>
> > > >>>>>>> The relevant JIRA issue: https://issues.apache.org/
> > > >>>>>> jira/browse/KAFKA-6670
> > > >>>>>>>
> > > >>>>>>> The library as proposed in the KIP has been implemented at
> > > >>>>>>> https://github.com/lightbend/kafka-streams-scala and the
> current
> > > >>>>> release
> > > >>>>>>> is
> > > >>>>>>> 0.2.0 (
> > > >>>>>>> https://github.com/lightbend/kafka-streams-scala/releases/
> > > >>> tag/v0.2.0
> > > >>>> ).
> > > >>>>>>> We at Lightbend has been using it since quite some time now.
> > > >>>>>>>
> > > >>>>>>> regards.
> > > >>>>>>>
> > > >>>>>>> --
> > > >>>>>>> Debasish Ghosh
> > > >>>>>>> Principal Engineer
> > > >>>>>>>
> > > >>>>>>> Twitter: @debasishg
> > > >>>>>>> Blog: http://debasishg.blogspot.com
> > > >>>>>>> Code: https://github.com/debasishg
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> --
> > > >>>>> Debasish Ghosh
> > > >>>>> Principal Engineer
> > > >>>>>
> > > >>>>> Twitter: @debasishg
> > > >>>>> Blog: http://debasishg.blogspot.com
> > > >>>>> Code: https://github.com/debasishg
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Debasish Ghosh
> > > >>> Principal Engineer
> > > >>>
> > > >>> Twitter: @debasishg
> > > >>> Blog: http://debasishg.blogspot.com
> > > >>> Code: https://github.com/debasishg
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
> >
> > --
> > Debasish Ghosh
> > Principal Engineer
> >
> > Twitter: @debasishg
> > Blog: http://debasishg.blogspot.com
> > Code: https://github.com/debasishg
> >
>
>
>
> --
> Senior Software Engineer, Lightbend, Inc.
>
> <http://lightbend.com>
>
> @seg1o <https://twitter.com/seg1o>, in/seanaglover
> <https://www.linkedin.com/in/seanaglover/>
>
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Sean Glover <se...@lightbend.com>.
Type names: I vote for option 2. The user must explicitly add a dependency
to this library and the wrapper types are in a different package. It seems
reasonable to expect them to do an import rename if there's a need to drop
down to the Java API.
Test Utils: The test utils in kafka-streams-scala are nice and lean, but
I'm not sure if it provides much more value than other options that exist
in the community. There's an embedded Kafka/ZK project implementation for
ScalaTest that's popular and active: manub/scalatest-embedded-kakfa. It
implies you must also use ScalaTest, which I acknowledge isn't everyone's
first choice for Scala test framework, but it probably is one of, if not
the most, popular library. It includes a DSL for Kafka Streams too. If
this KIP is accepted then perhaps a PR to that project could be made to
support the new wrapper implementations.
https://github.com/manub/scalatest-embedded-kafka#scalatest-embedded-kafka-streams
Sean
On Sun, Mar 18, 2018 at 4:05 AM, Debasish Ghosh <
debasish.ghosh@lightbend.com> wrote:
> >
> > > Should this be 1.2 (maybe it's even better to not put any version at
> > all)
>
>
> Actually wanted to emphasize that the support is from 1.0.0 onwards ..
> Should we make that explicit ? Like ..
>
> kafka-streams-scala only depends on the Scala standard library and Kafka
> > Streams 1.0.0+.
>
>
> In 1.1 release, we add a new module `kafka-streams-test-utils` to simplify
> > testing for Kafka Streams applications. Are those test utils suitable for
> > Scala users or should we add Scala wrappers for those, too?
>
>
> I will check up and let you know ..
>
> Also I am not clear about the decision on renaming of Scala abstractions.
> Can we have a consensus on this ? Here's the summary ..
>
> *Option 1:* Keep names separate (KStream for Java class, KStreamS for
> Scala). No renaming of imports required.
> *Option 2:* Unify names (KStream for Java and Scala class names). No
> conflict since they will reside in different packages. But if we need to
> use both abstractions, renaming of imports are required. But again, this
> may not be a too frequent use case.
>
> Suggestions ?
>
> regards.
>
> On Sat, Mar 17, 2018 at 3:07 AM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
> > Thanks a lot for the KIP! Two questions:
> >
> > 1) the KIP states:
> >
> > > kafka-streams-scala only depends on the Scala standard library and
> Kafka
> > Streams 1.0.0.
> >
> > Should this be 1.2 (maybe it's even better to not put any version at
> all)
> >
> >
> > 2) In 1.1 release, we add a new module `kafka-streams-test-utils` to
> > simplify testing for Kafka Streams applications. Are those test utils
> > suitable for Scala users or should we add Scala wrappers for those, too?
> >
> >
> > -Matthias
> >
> >
> > On 3/16/18 11:10 AM, Ted Yu wrote:
> > > Import renames seem to be fine.
> > >
> > > The class names with trailing 'S' look clean.
> > >
> > > Cheers
> > >
> > > On Fri, Mar 16, 2018 at 11:04 AM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > >
> > >> If this is rare (as it sounds), relying on import renames seems fine
> to
> > me.
> > >> Let's see what others think.
> > >>
> > >> Ismael
> > >>
> > >> On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
> > >> debasish.ghosh@lightbend.com> wrote:
> > >>
> > >>> I am not sure if this is practical or not. But theoretically a user
> may
> > >>> want to extract the unsafe Java abstraction from the Scala ones and
> use
> > >>> Java APIs on them .. e.g.
> > >>>
> > >>> val userClicksStream: KStreamS[String, Long] =
> > >>> builder.stream(userClicksTopic) // Scala abstraction
> > >>>
> > >>> val jStream: KStream[String, Long] = userClicksStream.inner //
> > publishes
> > >>> the underlying Java abstraction
> > >>>
> > >>> //.. work with Java, may be pass to some function written in Java
> > >>>
> > >>> I do realize this is somewhat of a convoluted use case and may not be
> > >>> practically useful ..
> > >>>
> > >>> Otherwise we can very well work on the suggested approach of unifying
> > the
> > >>> names ..
> > >>>
> > >>> regards.
> > >>>
> > >>>
> > >>>
> > >>> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > >>>
> > >>>> What does "mixed mode application" mean? What are the cases where a
> > >> user
> > >>>> would want to use both APIs? I think that would help understand the
> > >>>> reasoning.
> > >>>>
> > >>>> Thanks,
> > >>>> Ismael
> > >>>>
> > >>>> On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
> > >>>> debasish.ghosh@lightbend.com> wrote:
> > >>>>
> > >>>>> Hi Damian -
> > >>>>>
> > >>>>> We could. But in case the user wants to use both Scala and Java
> APIs
> > >>> (may
> > >>>>> be for some mixed mode application), won't that be confusing ? She
> > >> will
> > >>>>> have to do something like ..
> > >>>>>
> > >>>>> import o.a.k.s.scala.{KStream => KStreamS}
> > >>>>>
> > >>>>> to rename Scala imports or the other way round for imported Java
> > >>> classes.
> > >>>>>
> > >>>>> regards.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <da...@gmail.com>
> > >>>> wrote:
> > >>>>>
> > >>>>>> Hi Debasish,
> > >>>>>>
> > >>>>>> Thanks for the KIP - will be a great addition to streams. I've
> only
> > >>>> had a
> > >>>>>> quick scan, but seeing as the Scala classes are going to be in
> > >> their
> > >>>> own
> > >>>>>> package could we drop the S at the end of the class names?
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Damian
> > >>>>>>
> > >>>>>>
> > >>>>>> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> > >>>>> debasish.ghosh@lightbend.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hi -
> > >>>>>>>
> > >>>>>>> A new KIP, KIP-270 is up for discussion:
> > >>>>>>>
> > >>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >>>>>> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> > >>>>>>>
> > >>>>>>> The relevant JIRA issue: https://issues.apache.org/
> > >>>>>> jira/browse/KAFKA-6670
> > >>>>>>>
> > >>>>>>> The library as proposed in the KIP has been implemented at
> > >>>>>>> https://github.com/lightbend/kafka-streams-scala and the current
> > >>>>> release
> > >>>>>>> is
> > >>>>>>> 0.2.0 (
> > >>>>>>> https://github.com/lightbend/kafka-streams-scala/releases/
> > >>> tag/v0.2.0
> > >>>> ).
> > >>>>>>> We at Lightbend has been using it since quite some time now.
> > >>>>>>>
> > >>>>>>> regards.
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> Debasish Ghosh
> > >>>>>>> Principal Engineer
> > >>>>>>>
> > >>>>>>> Twitter: @debasishg
> > >>>>>>> Blog: http://debasishg.blogspot.com
> > >>>>>>> Code: https://github.com/debasishg
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> Debasish Ghosh
> > >>>>> Principal Engineer
> > >>>>>
> > >>>>> Twitter: @debasishg
> > >>>>> Blog: http://debasishg.blogspot.com
> > >>>>> Code: https://github.com/debasishg
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Debasish Ghosh
> > >>> Principal Engineer
> > >>>
> > >>> Twitter: @debasishg
> > >>> Blog: http://debasishg.blogspot.com
> > >>> Code: https://github.com/debasishg
> > >>>
> > >>
> > >
> >
> >
>
>
> --
> Debasish Ghosh
> Principal Engineer
>
> Twitter: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: https://github.com/debasishg
>
--
Senior Software Engineer, Lightbend, Inc.
<http://lightbend.com>
@seg1o <https://twitter.com/seg1o>, in/seanaglover
<https://www.linkedin.com/in/seanaglover/>
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Debasish Ghosh <de...@lightbend.com>.
>
> > Should this be 1.2 (maybe it's even better to not put any version at
> all)
Actually wanted to emphasize that the support is from 1.0.0 onwards ..
Should we make that explicit ? Like ..
kafka-streams-scala only depends on the Scala standard library and Kafka
> Streams 1.0.0+.
In 1.1 release, we add a new module `kafka-streams-test-utils` to simplify
> testing for Kafka Streams applications. Are those test utils suitable for
> Scala users or should we add Scala wrappers for those, too?
I will check up and let you know ..
Also I am not clear about the decision on renaming of Scala abstractions.
Can we have a consensus on this ? Here's the summary ..
*Option 1:* Keep names separate (KStream for Java class, KStreamS for
Scala). No renaming of imports required.
*Option 2:* Unify names (KStream for Java and Scala class names). No
conflict since they will reside in different packages. But if we need to
use both abstractions, renaming of imports are required. But again, this
may not be a too frequent use case.
Suggestions ?
regards.
On Sat, Mar 17, 2018 at 3:07 AM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Thanks a lot for the KIP! Two questions:
>
> 1) the KIP states:
>
> > kafka-streams-scala only depends on the Scala standard library and Kafka
> Streams 1.0.0.
>
> Should this be 1.2 (maybe it's even better to not put any version at all)
>
>
> 2) In 1.1 release, we add a new module `kafka-streams-test-utils` to
> simplify testing for Kafka Streams applications. Are those test utils
> suitable for Scala users or should we add Scala wrappers for those, too?
>
>
> -Matthias
>
>
> On 3/16/18 11:10 AM, Ted Yu wrote:
> > Import renames seem to be fine.
> >
> > The class names with trailing 'S' look clean.
> >
> > Cheers
> >
> > On Fri, Mar 16, 2018 at 11:04 AM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> >> If this is rare (as it sounds), relying on import renames seems fine to
> me.
> >> Let's see what others think.
> >>
> >> Ismael
> >>
> >> On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
> >> debasish.ghosh@lightbend.com> wrote:
> >>
> >>> I am not sure if this is practical or not. But theoretically a user may
> >>> want to extract the unsafe Java abstraction from the Scala ones and use
> >>> Java APIs on them .. e.g.
> >>>
> >>> val userClicksStream: KStreamS[String, Long] =
> >>> builder.stream(userClicksTopic) // Scala abstraction
> >>>
> >>> val jStream: KStream[String, Long] = userClicksStream.inner //
> publishes
> >>> the underlying Java abstraction
> >>>
> >>> //.. work with Java, may be pass to some function written in Java
> >>>
> >>> I do realize this is somewhat of a convoluted use case and may not be
> >>> practically useful ..
> >>>
> >>> Otherwise we can very well work on the suggested approach of unifying
> the
> >>> names ..
> >>>
> >>> regards.
> >>>
> >>>
> >>>
> >>> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <is...@juma.me.uk>
> wrote:
> >>>
> >>>> What does "mixed mode application" mean? What are the cases where a
> >> user
> >>>> would want to use both APIs? I think that would help understand the
> >>>> reasoning.
> >>>>
> >>>> Thanks,
> >>>> Ismael
> >>>>
> >>>> On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
> >>>> debasish.ghosh@lightbend.com> wrote:
> >>>>
> >>>>> Hi Damian -
> >>>>>
> >>>>> We could. But in case the user wants to use both Scala and Java APIs
> >>> (may
> >>>>> be for some mixed mode application), won't that be confusing ? She
> >> will
> >>>>> have to do something like ..
> >>>>>
> >>>>> import o.a.k.s.scala.{KStream => KStreamS}
> >>>>>
> >>>>> to rename Scala imports or the other way round for imported Java
> >>> classes.
> >>>>>
> >>>>> regards.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <da...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>>> Hi Debasish,
> >>>>>>
> >>>>>> Thanks for the KIP - will be a great addition to streams. I've only
> >>>> had a
> >>>>>> quick scan, but seeing as the Scala classes are going to be in
> >> their
> >>>> own
> >>>>>> package could we drop the S at the end of the class names?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Damian
> >>>>>>
> >>>>>>
> >>>>>> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> >>>>> debasish.ghosh@lightbend.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi -
> >>>>>>>
> >>>>>>> A new KIP, KIP-270 is up for discussion:
> >>>>>>>
> >>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> >>>>>>>
> >>>>>>> The relevant JIRA issue: https://issues.apache.org/
> >>>>>> jira/browse/KAFKA-6670
> >>>>>>>
> >>>>>>> The library as proposed in the KIP has been implemented at
> >>>>>>> https://github.com/lightbend/kafka-streams-scala and the current
> >>>>> release
> >>>>>>> is
> >>>>>>> 0.2.0 (
> >>>>>>> https://github.com/lightbend/kafka-streams-scala/releases/
> >>> tag/v0.2.0
> >>>> ).
> >>>>>>> We at Lightbend has been using it since quite some time now.
> >>>>>>>
> >>>>>>> regards.
> >>>>>>>
> >>>>>>> --
> >>>>>>> Debasish Ghosh
> >>>>>>> Principal Engineer
> >>>>>>>
> >>>>>>> Twitter: @debasishg
> >>>>>>> Blog: http://debasishg.blogspot.com
> >>>>>>> Code: https://github.com/debasishg
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Debasish Ghosh
> >>>>> Principal Engineer
> >>>>>
> >>>>> Twitter: @debasishg
> >>>>> Blog: http://debasishg.blogspot.com
> >>>>> Code: https://github.com/debasishg
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Debasish Ghosh
> >>> Principal Engineer
> >>>
> >>> Twitter: @debasishg
> >>> Blog: http://debasishg.blogspot.com
> >>> Code: https://github.com/debasishg
> >>>
> >>
> >
>
>
--
Debasish Ghosh
Principal Engineer
Twitter: @debasishg
Blog: http://debasishg.blogspot.com
Code: https://github.com/debasishg
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Thanks a lot for the KIP! Two questions:
1) the KIP states:
> kafka-streams-scala only depends on the Scala standard library and Kafka Streams 1.0.0.
Should this be 1.2 (maybe it's even better to not put any version at all)
2) In 1.1 release, we add a new module `kafka-streams-test-utils` to
simplify testing for Kafka Streams applications. Are those test utils
suitable for Scala users or should we add Scala wrappers for those, too?
-Matthias
On 3/16/18 11:10 AM, Ted Yu wrote:
> Import renames seem to be fine.
>
> The class names with trailing 'S' look clean.
>
> Cheers
>
> On Fri, Mar 16, 2018 at 11:04 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
>> If this is rare (as it sounds), relying on import renames seems fine to me.
>> Let's see what others think.
>>
>> Ismael
>>
>> On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
>> debasish.ghosh@lightbend.com> wrote:
>>
>>> I am not sure if this is practical or not. But theoretically a user may
>>> want to extract the unsafe Java abstraction from the Scala ones and use
>>> Java APIs on them .. e.g.
>>>
>>> val userClicksStream: KStreamS[String, Long] =
>>> builder.stream(userClicksTopic) // Scala abstraction
>>>
>>> val jStream: KStream[String, Long] = userClicksStream.inner // publishes
>>> the underlying Java abstraction
>>>
>>> //.. work with Java, may be pass to some function written in Java
>>>
>>> I do realize this is somewhat of a convoluted use case and may not be
>>> practically useful ..
>>>
>>> Otherwise we can very well work on the suggested approach of unifying the
>>> names ..
>>>
>>> regards.
>>>
>>>
>>>
>>> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <is...@juma.me.uk> wrote:
>>>
>>>> What does "mixed mode application" mean? What are the cases where a
>> user
>>>> would want to use both APIs? I think that would help understand the
>>>> reasoning.
>>>>
>>>> Thanks,
>>>> Ismael
>>>>
>>>> On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
>>>> debasish.ghosh@lightbend.com> wrote:
>>>>
>>>>> Hi Damian -
>>>>>
>>>>> We could. But in case the user wants to use both Scala and Java APIs
>>> (may
>>>>> be for some mixed mode application), won't that be confusing ? She
>> will
>>>>> have to do something like ..
>>>>>
>>>>> import o.a.k.s.scala.{KStream => KStreamS}
>>>>>
>>>>> to rename Scala imports or the other way round for imported Java
>>> classes.
>>>>>
>>>>> regards.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <da...@gmail.com>
>>>> wrote:
>>>>>
>>>>>> Hi Debasish,
>>>>>>
>>>>>> Thanks for the KIP - will be a great addition to streams. I've only
>>>> had a
>>>>>> quick scan, but seeing as the Scala classes are going to be in
>> their
>>>> own
>>>>>> package could we drop the S at the end of the class names?
>>>>>>
>>>>>> Thanks,
>>>>>> Damian
>>>>>>
>>>>>>
>>>>>> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
>>>>> debasish.ghosh@lightbend.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi -
>>>>>>>
>>>>>>> A new KIP, KIP-270 is up for discussion:
>>>>>>>
>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
>>>>>>>
>>>>>>> The relevant JIRA issue: https://issues.apache.org/
>>>>>> jira/browse/KAFKA-6670
>>>>>>>
>>>>>>> The library as proposed in the KIP has been implemented at
>>>>>>> https://github.com/lightbend/kafka-streams-scala and the current
>>>>> release
>>>>>>> is
>>>>>>> 0.2.0 (
>>>>>>> https://github.com/lightbend/kafka-streams-scala/releases/
>>> tag/v0.2.0
>>>> ).
>>>>>>> We at Lightbend has been using it since quite some time now.
>>>>>>>
>>>>>>> regards.
>>>>>>>
>>>>>>> --
>>>>>>> Debasish Ghosh
>>>>>>> Principal Engineer
>>>>>>>
>>>>>>> Twitter: @debasishg
>>>>>>> Blog: http://debasishg.blogspot.com
>>>>>>> Code: https://github.com/debasishg
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Debasish Ghosh
>>>>> Principal Engineer
>>>>>
>>>>> Twitter: @debasishg
>>>>> Blog: http://debasishg.blogspot.com
>>>>> Code: https://github.com/debasishg
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Debasish Ghosh
>>> Principal Engineer
>>>
>>> Twitter: @debasishg
>>> Blog: http://debasishg.blogspot.com
>>> Code: https://github.com/debasishg
>>>
>>
>
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Ted Yu <yu...@gmail.com>.
Import renames seem to be fine.
The class names with trailing 'S' look clean.
Cheers
On Fri, Mar 16, 2018 at 11:04 AM, Ismael Juma <is...@juma.me.uk> wrote:
> If this is rare (as it sounds), relying on import renames seems fine to me.
> Let's see what others think.
>
> Ismael
>
> On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
> debasish.ghosh@lightbend.com> wrote:
>
> > I am not sure if this is practical or not. But theoretically a user may
> > want to extract the unsafe Java abstraction from the Scala ones and use
> > Java APIs on them .. e.g.
> >
> > val userClicksStream: KStreamS[String, Long] =
> > builder.stream(userClicksTopic) // Scala abstraction
> >
> > val jStream: KStream[String, Long] = userClicksStream.inner // publishes
> > the underlying Java abstraction
> >
> > //.. work with Java, may be pass to some function written in Java
> >
> > I do realize this is somewhat of a convoluted use case and may not be
> > practically useful ..
> >
> > Otherwise we can very well work on the suggested approach of unifying the
> > names ..
> >
> > regards.
> >
> >
> >
> > On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > What does "mixed mode application" mean? What are the cases where a
> user
> > > would want to use both APIs? I think that would help understand the
> > > reasoning.
> > >
> > > Thanks,
> > > Ismael
> > >
> > > On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
> > > debasish.ghosh@lightbend.com> wrote:
> > >
> > > > Hi Damian -
> > > >
> > > > We could. But in case the user wants to use both Scala and Java APIs
> > (may
> > > > be for some mixed mode application), won't that be confusing ? She
> will
> > > > have to do something like ..
> > > >
> > > > import o.a.k.s.scala.{KStream => KStreamS}
> > > >
> > > > to rename Scala imports or the other way round for imported Java
> > classes.
> > > >
> > > > regards.
> > > >
> > > >
> > > >
> > > > On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <da...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Debasish,
> > > > >
> > > > > Thanks for the KIP - will be a great addition to streams. I've only
> > > had a
> > > > > quick scan, but seeing as the Scala classes are going to be in
> their
> > > own
> > > > > package could we drop the S at the end of the class names?
> > > > >
> > > > > Thanks,
> > > > > Damian
> > > > >
> > > > >
> > > > > On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> > > > debasish.ghosh@lightbend.com>
> > > > > wrote:
> > > > >
> > > > > > Hi -
> > > > > >
> > > > > > A new KIP, KIP-270 is up for discussion:
> > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> > > > > >
> > > > > > The relevant JIRA issue: https://issues.apache.org/
> > > > > jira/browse/KAFKA-6670
> > > > > >
> > > > > > The library as proposed in the KIP has been implemented at
> > > > > > https://github.com/lightbend/kafka-streams-scala and the current
> > > > release
> > > > > > is
> > > > > > 0.2.0 (
> > > > > > https://github.com/lightbend/kafka-streams-scala/releases/
> > tag/v0.2.0
> > > ).
> > > > > > We at Lightbend has been using it since quite some time now.
> > > > > >
> > > > > > regards.
> > > > > >
> > > > > > --
> > > > > > Debasish Ghosh
> > > > > > Principal Engineer
> > > > > >
> > > > > > Twitter: @debasishg
> > > > > > Blog: http://debasishg.blogspot.com
> > > > > > Code: https://github.com/debasishg
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Debasish Ghosh
> > > > Principal Engineer
> > > >
> > > > Twitter: @debasishg
> > > > Blog: http://debasishg.blogspot.com
> > > > Code: https://github.com/debasishg
> > > >
> > >
> >
> >
> >
> > --
> > Debasish Ghosh
> > Principal Engineer
> >
> > Twitter: @debasishg
> > Blog: http://debasishg.blogspot.com
> > Code: https://github.com/debasishg
> >
>
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Ismael Juma <is...@juma.me.uk>.
If this is rare (as it sounds), relying on import renames seems fine to me.
Let's see what others think.
Ismael
On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
debasish.ghosh@lightbend.com> wrote:
> I am not sure if this is practical or not. But theoretically a user may
> want to extract the unsafe Java abstraction from the Scala ones and use
> Java APIs on them .. e.g.
>
> val userClicksStream: KStreamS[String, Long] =
> builder.stream(userClicksTopic) // Scala abstraction
>
> val jStream: KStream[String, Long] = userClicksStream.inner // publishes
> the underlying Java abstraction
>
> //.. work with Java, may be pass to some function written in Java
>
> I do realize this is somewhat of a convoluted use case and may not be
> practically useful ..
>
> Otherwise we can very well work on the suggested approach of unifying the
> names ..
>
> regards.
>
>
>
> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > What does "mixed mode application" mean? What are the cases where a user
> > would want to use both APIs? I think that would help understand the
> > reasoning.
> >
> > Thanks,
> > Ismael
> >
> > On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
> > debasish.ghosh@lightbend.com> wrote:
> >
> > > Hi Damian -
> > >
> > > We could. But in case the user wants to use both Scala and Java APIs
> (may
> > > be for some mixed mode application), won't that be confusing ? She will
> > > have to do something like ..
> > >
> > > import o.a.k.s.scala.{KStream => KStreamS}
> > >
> > > to rename Scala imports or the other way round for imported Java
> classes.
> > >
> > > regards.
> > >
> > >
> > >
> > > On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <da...@gmail.com>
> > wrote:
> > >
> > > > Hi Debasish,
> > > >
> > > > Thanks for the KIP - will be a great addition to streams. I've only
> > had a
> > > > quick scan, but seeing as the Scala classes are going to be in their
> > own
> > > > package could we drop the S at the end of the class names?
> > > >
> > > > Thanks,
> > > > Damian
> > > >
> > > >
> > > > On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> > > debasish.ghosh@lightbend.com>
> > > > wrote:
> > > >
> > > > > Hi -
> > > > >
> > > > > A new KIP, KIP-270 is up for discussion:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> > > > >
> > > > > The relevant JIRA issue: https://issues.apache.org/
> > > > jira/browse/KAFKA-6670
> > > > >
> > > > > The library as proposed in the KIP has been implemented at
> > > > > https://github.com/lightbend/kafka-streams-scala and the current
> > > release
> > > > > is
> > > > > 0.2.0 (
> > > > > https://github.com/lightbend/kafka-streams-scala/releases/
> tag/v0.2.0
> > ).
> > > > > We at Lightbend has been using it since quite some time now.
> > > > >
> > > > > regards.
> > > > >
> > > > > --
> > > > > Debasish Ghosh
> > > > > Principal Engineer
> > > > >
> > > > > Twitter: @debasishg
> > > > > Blog: http://debasishg.blogspot.com
> > > > > Code: https://github.com/debasishg
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Debasish Ghosh
> > > Principal Engineer
> > >
> > > Twitter: @debasishg
> > > Blog: http://debasishg.blogspot.com
> > > Code: https://github.com/debasishg
> > >
> >
>
>
>
> --
> Debasish Ghosh
> Principal Engineer
>
> Twitter: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: https://github.com/debasishg
>
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Debasish Ghosh <de...@lightbend.com>.
I am not sure if this is practical or not. But theoretically a user may
want to extract the unsafe Java abstraction from the Scala ones and use
Java APIs on them .. e.g.
val userClicksStream: KStreamS[String, Long] =
builder.stream(userClicksTopic) // Scala abstraction
val jStream: KStream[String, Long] = userClicksStream.inner // publishes
the underlying Java abstraction
//.. work with Java, may be pass to some function written in Java
I do realize this is somewhat of a convoluted use case and may not be
practically useful ..
Otherwise we can very well work on the suggested approach of unifying the
names ..
regards.
On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma <is...@juma.me.uk> wrote:
> What does "mixed mode application" mean? What are the cases where a user
> would want to use both APIs? I think that would help understand the
> reasoning.
>
> Thanks,
> Ismael
>
> On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
> debasish.ghosh@lightbend.com> wrote:
>
> > Hi Damian -
> >
> > We could. But in case the user wants to use both Scala and Java APIs (may
> > be for some mixed mode application), won't that be confusing ? She will
> > have to do something like ..
> >
> > import o.a.k.s.scala.{KStream => KStreamS}
> >
> > to rename Scala imports or the other way round for imported Java classes.
> >
> > regards.
> >
> >
> >
> > On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <da...@gmail.com>
> wrote:
> >
> > > Hi Debasish,
> > >
> > > Thanks for the KIP - will be a great addition to streams. I've only
> had a
> > > quick scan, but seeing as the Scala classes are going to be in their
> own
> > > package could we drop the S at the end of the class names?
> > >
> > > Thanks,
> > > Damian
> > >
> > >
> > > On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> > debasish.ghosh@lightbend.com>
> > > wrote:
> > >
> > > > Hi -
> > > >
> > > > A new KIP, KIP-270 is up for discussion:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> > > >
> > > > The relevant JIRA issue: https://issues.apache.org/
> > > jira/browse/KAFKA-6670
> > > >
> > > > The library as proposed in the KIP has been implemented at
> > > > https://github.com/lightbend/kafka-streams-scala and the current
> > release
> > > > is
> > > > 0.2.0 (
> > > > https://github.com/lightbend/kafka-streams-scala/releases/tag/v0.2.0
> ).
> > > > We at Lightbend has been using it since quite some time now.
> > > >
> > > > regards.
> > > >
> > > > --
> > > > Debasish Ghosh
> > > > Principal Engineer
> > > >
> > > > Twitter: @debasishg
> > > > Blog: http://debasishg.blogspot.com
> > > > Code: https://github.com/debasishg
> > > >
> > >
> >
> >
> >
> > --
> > Debasish Ghosh
> > Principal Engineer
> >
> > Twitter: @debasishg
> > Blog: http://debasishg.blogspot.com
> > Code: https://github.com/debasishg
> >
>
--
Debasish Ghosh
Principal Engineer
Twitter: @debasishg
Blog: http://debasishg.blogspot.com
Code: https://github.com/debasishg
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Ismael Juma <is...@juma.me.uk>.
What does "mixed mode application" mean? What are the cases where a user
would want to use both APIs? I think that would help understand the
reasoning.
Thanks,
Ismael
On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
debasish.ghosh@lightbend.com> wrote:
> Hi Damian -
>
> We could. But in case the user wants to use both Scala and Java APIs (may
> be for some mixed mode application), won't that be confusing ? She will
> have to do something like ..
>
> import o.a.k.s.scala.{KStream => KStreamS}
>
> to rename Scala imports or the other way round for imported Java classes.
>
> regards.
>
>
>
> On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <da...@gmail.com> wrote:
>
> > Hi Debasish,
> >
> > Thanks for the KIP - will be a great addition to streams. I've only had a
> > quick scan, but seeing as the Scala classes are going to be in their own
> > package could we drop the S at the end of the class names?
> >
> > Thanks,
> > Damian
> >
> >
> > On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> debasish.ghosh@lightbend.com>
> > wrote:
> >
> > > Hi -
> > >
> > > A new KIP, KIP-270 is up for discussion:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> > >
> > > The relevant JIRA issue: https://issues.apache.org/
> > jira/browse/KAFKA-6670
> > >
> > > The library as proposed in the KIP has been implemented at
> > > https://github.com/lightbend/kafka-streams-scala and the current
> release
> > > is
> > > 0.2.0 (
> > > https://github.com/lightbend/kafka-streams-scala/releases/tag/v0.2.0).
> > > We at Lightbend has been using it since quite some time now.
> > >
> > > regards.
> > >
> > > --
> > > Debasish Ghosh
> > > Principal Engineer
> > >
> > > Twitter: @debasishg
> > > Blog: http://debasishg.blogspot.com
> > > Code: https://github.com/debasishg
> > >
> >
>
>
>
> --
> Debasish Ghosh
> Principal Engineer
>
> Twitter: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: https://github.com/debasishg
>
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Debasish Ghosh <de...@lightbend.com>.
Hi Damian -
We could. But in case the user wants to use both Scala and Java APIs (may
be for some mixed mode application), won't that be confusing ? She will
have to do something like ..
import o.a.k.s.scala.{KStream => KStreamS}
to rename Scala imports or the other way round for imported Java classes.
regards.
On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy <da...@gmail.com> wrote:
> Hi Debasish,
>
> Thanks for the KIP - will be a great addition to streams. I've only had a
> quick scan, but seeing as the Scala classes are going to be in their own
> package could we drop the S at the end of the class names?
>
> Thanks,
> Damian
>
>
> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <de...@lightbend.com>
> wrote:
>
> > Hi -
> >
> > A new KIP, KIP-270 is up for discussion:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> >
> > The relevant JIRA issue: https://issues.apache.org/
> jira/browse/KAFKA-6670
> >
> > The library as proposed in the KIP has been implemented at
> > https://github.com/lightbend/kafka-streams-scala and the current release
> > is
> > 0.2.0 (
> > https://github.com/lightbend/kafka-streams-scala/releases/tag/v0.2.0).
> > We at Lightbend has been using it since quite some time now.
> >
> > regards.
> >
> > --
> > Debasish Ghosh
> > Principal Engineer
> >
> > Twitter: @debasishg
> > Blog: http://debasishg.blogspot.com
> > Code: https://github.com/debasishg
> >
>
--
Debasish Ghosh
Principal Engineer
Twitter: @debasishg
Blog: http://debasishg.blogspot.com
Code: https://github.com/debasishg
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Ismael Juma <is...@juma.me.uk>.
Thanks for the KIP! It will be nice to have an API for Kafka Streams that
is more concise _and_ safer.
Like Damian, I would prefer if we didn't have a suffix and relied on
package names. However, if there's a good reason why this won't work well,
we should explain it in the "Rejected Alternatives" section.
Ismael
On Fri, Mar 16, 2018 at 8:37 AM, Damian Guy <da...@gmail.com> wrote:
> Hi Debasish,
>
> Thanks for the KIP - will be a great addition to streams. I've only had a
> quick scan, but seeing as the Scala classes are going to be in their own
> package could we drop the S at the end of the class names?
>
> Thanks,
> Damian
>
>
> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <de...@lightbend.com>
> wrote:
>
> > Hi -
> >
> > A new KIP, KIP-270 is up for discussion:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> >
> > The relevant JIRA issue: https://issues.apache.org/
> jira/browse/KAFKA-6670
> >
> > The library as proposed in the KIP has been implemented at
> > https://github.com/lightbend/kafka-streams-scala and the current release
> > is
> > 0.2.0 (
> > https://github.com/lightbend/kafka-streams-scala/releases/tag/v0.2.0).
> > We at Lightbend has been using it since quite some time now.
> >
> > regards.
> >
> > --
> > Debasish Ghosh
> > Principal Engineer
> >
> > Twitter: @debasishg
> > Blog: http://debasishg.blogspot.com
> > Code: https://github.com/debasishg
> >
>
Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams
Posted by Damian Guy <da...@gmail.com>.
Hi Debasish,
Thanks for the KIP - will be a great addition to streams. I've only had a
quick scan, but seeing as the Scala classes are going to be in their own
package could we drop the S at the end of the class names?
Thanks,
Damian
On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <de...@lightbend.com>
wrote:
> Hi -
>
> A new KIP, KIP-270 is up for discussion:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
>
> The relevant JIRA issue: https://issues.apache.org/jira/browse/KAFKA-6670
>
> The library as proposed in the KIP has been implemented at
> https://github.com/lightbend/kafka-streams-scala and the current release
> is
> 0.2.0 (
> https://github.com/lightbend/kafka-streams-scala/releases/tag/v0.2.0).
> We at Lightbend has been using it since quite some time now.
>
> regards.
>
> --
> Debasish Ghosh
> Principal Engineer
>
> Twitter: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: https://github.com/debasishg
>