code layout

[Jay Kreps <ja...@gmail.com>]

For the new producer code we currently added the new module
   clients
This builds the jar kafka-clients.jar. The core module should be renamed to
kafka-server and producer kafka-server.jar.

It is the intention that the server will end up depending on the clients
but not vice versa (or we could make a separate module for that if we like).

Integration code that tests clients against the server will live with the
server.

There is some shared code between the clients and server. This is the
kafka.common package. Currently this is in the clients module, which is a
little odd. We could alternatively break it into its own module, which
might be nice. However I'm not sure it really warrants its own jar since
there isn't much point to that code on its own and having the clients need
two jars is kind of annoying. But maybe this doesn't matter because
everyone just uses Maven?

So the options are:

A) Leave it the way it is: kafka-clients.jar and kafka-server.jar.
B) Separate out the common code and have kafka-common.jar,
kafka-clients.jar and kafka-server.jar (clients depends on common, and
server depends on clients and common).
C) Make a jar for each client. Currently this would be just producer and
consumer, but in the future we could add a more well-defined an Admin
client for some of the administrative functions. If there ended up being
code that is client-specific but shared by multiple clients this could be
problematic.

Alternately I don't know if the mapping from sub-modules to jars needs to
be one-to-one so we could seperate the clients and common code to enforce
compile dependencies and then glump it all into one client jar. That might
anger the build system, though.

I think I vote for (A) but don't really care much.

-Jay

Re: code layout

[Guozhang Wang <wa...@gmail.com>]

I would vote for A just for simplicity of dependencies.


On Sat, Feb 1, 2014 at 3:06 PM, Neha Narkhede <ne...@gmail.com>wrote:

> I like A. It will be simpler to maintain and evolve when we add the admin
> APIs.
>
>
>
> On Fri, Jan 31, 2014 at 7:32 PM, Steve Morin <st...@stevemorin.com> wrote:
>
> > I like A or C
> >
> > I think it will be clearer for people to separate the two.
> >
> >
> > On Fri, Jan 31, 2014 at 3:29 PM, Jay Kreps <ja...@gmail.com> wrote:
> >
> > > For the new producer code we currently added the new module
> > >    clients
> > > This builds the jar kafka-clients.jar. The core module should be
> renamed
> > to
> > > kafka-server and producer kafka-server.jar.
> > >
> > > It is the intention that the server will end up depending on the
> clients
> > > but not vice versa (or we could make a separate module for that if we
> > > like).
> > >
> > > Integration code that tests clients against the server will live with
> the
> > > server.
> > >
> > > There is some shared code between the clients and server. This is the
> > > kafka.common package. Currently this is in the clients module, which
> is a
> > > little odd. We could alternatively break it into its own module, which
> > > might be nice. However I'm not sure it really warrants its own jar
> since
> > > there isn't much point to that code on its own and having the clients
> > need
> > > two jars is kind of annoying. But maybe this doesn't matter because
> > > everyone just uses Maven?
> > >
> > > So the options are:
> > >
> > > A) Leave it the way it is: kafka-clients.jar and kafka-server.jar.
> > > B) Separate out the common code and have kafka-common.jar,
> > > kafka-clients.jar and kafka-server.jar (clients depends on common, and
> > > server depends on clients and common).
> > > C) Make a jar for each client. Currently this would be just producer
> and
> > > consumer, but in the future we could add a more well-defined an Admin
> > > client for some of the administrative functions. If there ended up
> being
> > > code that is client-specific but shared by multiple clients this could
> be
> > > problematic.
> > >
> > > Alternately I don't know if the mapping from sub-modules to jars needs
> to
> > > be one-to-one so we could seperate the clients and common code to
> enforce
> > > compile dependencies and then glump it all into one client jar. That
> > might
> > > anger the build system, though.
> > >
> > > I think I vote for (A) but don't really care much.
> > >
> > > -Jay
> > >
> >
>



-- 
-- Guozhang

Re: code layout

[Steve Morin <st...@gmail.com>]

That's why I was thinking c) makes sense from a design perspective.

> On Feb 2, 2014, at 21:59, Jay Kreps <ja...@gmail.com> wrote:
> 
> Yes, this is a good point, the admin api could depend on zookeeper. It does
> today. That would not work in the existing clients package.
> 
> -Jay
> 
> 
>> On Sun, Feb 2, 2014 at 4:20 PM, Steve Morin <st...@gmail.com> wrote:
>> 
>> Would the admin api's potentially have different dependencies?
>> 
>>> On Feb 1, 2014, at 15:06, Neha Narkhede <ne...@gmail.com> wrote:
>>> 
>>> I like A. It will be simpler to maintain and evolve when we add the admin
>>> APIs.
>>> 
>>> 
>>> 
>>>> On Fri, Jan 31, 2014 at 7:32 PM, Steve Morin <st...@stevemorin.com>
>> wrote:
>>>> 
>>>> I like A or C
>>>> 
>>>> I think it will be clearer for people to separate the two.
>>>> 
>>>> 
>>>>> On Fri, Jan 31, 2014 at 3:29 PM, Jay Kreps <ja...@gmail.com>
>> wrote:
>>>>> 
>>>>> For the new producer code we currently added the new module
>>>>>  clients
>>>>> This builds the jar kafka-clients.jar. The core module should be
>> renamed
>>>> to
>>>>> kafka-server and producer kafka-server.jar.
>>>>> 
>>>>> It is the intention that the server will end up depending on the
>> clients
>>>>> but not vice versa (or we could make a separate module for that if we
>>>>> like).
>>>>> 
>>>>> Integration code that tests clients against the server will live with
>> the
>>>>> server.
>>>>> 
>>>>> There is some shared code between the clients and server. This is the
>>>>> kafka.common package. Currently this is in the clients module, which
>> is a
>>>>> little odd. We could alternatively break it into its own module, which
>>>>> might be nice. However I'm not sure it really warrants its own jar
>> since
>>>>> there isn't much point to that code on its own and having the clients
>>>> need
>>>>> two jars is kind of annoying. But maybe this doesn't matter because
>>>>> everyone just uses Maven?
>>>>> 
>>>>> So the options are:
>>>>> 
>>>>> A) Leave it the way it is: kafka-clients.jar and kafka-server.jar.
>>>>> B) Separate out the common code and have kafka-common.jar,
>>>>> kafka-clients.jar and kafka-server.jar (clients depends on common, and
>>>>> server depends on clients and common).
>>>>> C) Make a jar for each client. Currently this would be just producer
>> and
>>>>> consumer, but in the future we could add a more well-defined an Admin
>>>>> client for some of the administrative functions. If there ended up
>> being
>>>>> code that is client-specific but shared by multiple clients this could
>> be
>>>>> problematic.
>>>>> 
>>>>> Alternately I don't know if the mapping from sub-modules to jars needs
>> to
>>>>> be one-to-one so we could seperate the clients and common code to
>> enforce
>>>>> compile dependencies and then glump it all into one client jar. That
>>>> might
>>>>> anger the build system, though.
>>>>> 
>>>>> I think I vote for (A) but don't really care much.
>>>>> 
>>>>> -Jay
>> 

Re: code layout

[Jun Rao <ju...@gmail.com>]

Since we don't really have an admin api (just standalone tools), perhaps we
can start with option A. We can potentially add an admin (or tools) jar in
the future when we expose programming apis for admin.

Thanks,

Jun


On Sun, Feb 2, 2014 at 9:59 PM, Jay Kreps <ja...@gmail.com> wrote:

> Yes, this is a good point, the admin api could depend on zookeeper. It does
> today. That would not work in the existing clients package.
>
> -Jay
>
>
> On Sun, Feb 2, 2014 at 4:20 PM, Steve Morin <st...@gmail.com> wrote:
>
> > Would the admin api's potentially have different dependencies?
> >
> > > On Feb 1, 2014, at 15:06, Neha Narkhede <ne...@gmail.com>
> wrote:
> > >
> > > I like A. It will be simpler to maintain and evolve when we add the
> admin
> > > APIs.
> > >
> > >
> > >
> > >> On Fri, Jan 31, 2014 at 7:32 PM, Steve Morin <st...@stevemorin.com>
> > wrote:
> > >>
> > >> I like A or C
> > >>
> > >> I think it will be clearer for people to separate the two.
> > >>
> > >>
> > >>> On Fri, Jan 31, 2014 at 3:29 PM, Jay Kreps <ja...@gmail.com>
> > wrote:
> > >>>
> > >>> For the new producer code we currently added the new module
> > >>>   clients
> > >>> This builds the jar kafka-clients.jar. The core module should be
> > renamed
> > >> to
> > >>> kafka-server and producer kafka-server.jar.
> > >>>
> > >>> It is the intention that the server will end up depending on the
> > clients
> > >>> but not vice versa (or we could make a separate module for that if we
> > >>> like).
> > >>>
> > >>> Integration code that tests clients against the server will live with
> > the
> > >>> server.
> > >>>
> > >>> There is some shared code between the clients and server. This is the
> > >>> kafka.common package. Currently this is in the clients module, which
> > is a
> > >>> little odd. We could alternatively break it into its own module,
> which
> > >>> might be nice. However I'm not sure it really warrants its own jar
> > since
> > >>> there isn't much point to that code on its own and having the clients
> > >> need
> > >>> two jars is kind of annoying. But maybe this doesn't matter because
> > >>> everyone just uses Maven?
> > >>>
> > >>> So the options are:
> > >>>
> > >>> A) Leave it the way it is: kafka-clients.jar and kafka-server.jar.
> > >>> B) Separate out the common code and have kafka-common.jar,
> > >>> kafka-clients.jar and kafka-server.jar (clients depends on common,
> and
> > >>> server depends on clients and common).
> > >>> C) Make a jar for each client. Currently this would be just producer
> > and
> > >>> consumer, but in the future we could add a more well-defined an Admin
> > >>> client for some of the administrative functions. If there ended up
> > being
> > >>> code that is client-specific but shared by multiple clients this
> could
> > be
> > >>> problematic.
> > >>>
> > >>> Alternately I don't know if the mapping from sub-modules to jars
> needs
> > to
> > >>> be one-to-one so we could seperate the clients and common code to
> > enforce
> > >>> compile dependencies and then glump it all into one client jar. That
> > >> might
> > >>> anger the build system, though.
> > >>>
> > >>> I think I vote for (A) but don't really care much.
> > >>>
> > >>> -Jay
> > >>
> >
>

Re: code layout

[Jay Kreps <ja...@gmail.com>]

Yes, this is a good point, the admin api could depend on zookeeper. It does
today. That would not work in the existing clients package.

-Jay


On Sun, Feb 2, 2014 at 4:20 PM, Steve Morin <st...@gmail.com> wrote:

> Would the admin api's potentially have different dependencies?
>
> > On Feb 1, 2014, at 15:06, Neha Narkhede <ne...@gmail.com> wrote:
> >
> > I like A. It will be simpler to maintain and evolve when we add the admin
> > APIs.
> >
> >
> >
> >> On Fri, Jan 31, 2014 at 7:32 PM, Steve Morin <st...@stevemorin.com>
> wrote:
> >>
> >> I like A or C
> >>
> >> I think it will be clearer for people to separate the two.
> >>
> >>
> >>> On Fri, Jan 31, 2014 at 3:29 PM, Jay Kreps <ja...@gmail.com>
> wrote:
> >>>
> >>> For the new producer code we currently added the new module
> >>>   clients
> >>> This builds the jar kafka-clients.jar. The core module should be
> renamed
> >> to
> >>> kafka-server and producer kafka-server.jar.
> >>>
> >>> It is the intention that the server will end up depending on the
> clients
> >>> but not vice versa (or we could make a separate module for that if we
> >>> like).
> >>>
> >>> Integration code that tests clients against the server will live with
> the
> >>> server.
> >>>
> >>> There is some shared code between the clients and server. This is the
> >>> kafka.common package. Currently this is in the clients module, which
> is a
> >>> little odd. We could alternatively break it into its own module, which
> >>> might be nice. However I'm not sure it really warrants its own jar
> since
> >>> there isn't much point to that code on its own and having the clients
> >> need
> >>> two jars is kind of annoying. But maybe this doesn't matter because
> >>> everyone just uses Maven?
> >>>
> >>> So the options are:
> >>>
> >>> A) Leave it the way it is: kafka-clients.jar and kafka-server.jar.
> >>> B) Separate out the common code and have kafka-common.jar,
> >>> kafka-clients.jar and kafka-server.jar (clients depends on common, and
> >>> server depends on clients and common).
> >>> C) Make a jar for each client. Currently this would be just producer
> and
> >>> consumer, but in the future we could add a more well-defined an Admin
> >>> client for some of the administrative functions. If there ended up
> being
> >>> code that is client-specific but shared by multiple clients this could
> be
> >>> problematic.
> >>>
> >>> Alternately I don't know if the mapping from sub-modules to jars needs
> to
> >>> be one-to-one so we could seperate the clients and common code to
> enforce
> >>> compile dependencies and then glump it all into one client jar. That
> >> might
> >>> anger the build system, though.
> >>>
> >>> I think I vote for (A) but don't really care much.
> >>>
> >>> -Jay
> >>
>

Re: code layout

[Steve Morin <st...@gmail.com>]

Would the admin api's potentially have different dependencies?

> On Feb 1, 2014, at 15:06, Neha Narkhede <ne...@gmail.com> wrote:
> 
> I like A. It will be simpler to maintain and evolve when we add the admin
> APIs.
> 
> 
> 
>> On Fri, Jan 31, 2014 at 7:32 PM, Steve Morin <st...@stevemorin.com> wrote:
>> 
>> I like A or C
>> 
>> I think it will be clearer for people to separate the two.
>> 
>> 
>>> On Fri, Jan 31, 2014 at 3:29 PM, Jay Kreps <ja...@gmail.com> wrote:
>>> 
>>> For the new producer code we currently added the new module
>>>   clients
>>> This builds the jar kafka-clients.jar. The core module should be renamed
>> to
>>> kafka-server and producer kafka-server.jar.
>>> 
>>> It is the intention that the server will end up depending on the clients
>>> but not vice versa (or we could make a separate module for that if we
>>> like).
>>> 
>>> Integration code that tests clients against the server will live with the
>>> server.
>>> 
>>> There is some shared code between the clients and server. This is the
>>> kafka.common package. Currently this is in the clients module, which is a
>>> little odd. We could alternatively break it into its own module, which
>>> might be nice. However I'm not sure it really warrants its own jar since
>>> there isn't much point to that code on its own and having the clients
>> need
>>> two jars is kind of annoying. But maybe this doesn't matter because
>>> everyone just uses Maven?
>>> 
>>> So the options are:
>>> 
>>> A) Leave it the way it is: kafka-clients.jar and kafka-server.jar.
>>> B) Separate out the common code and have kafka-common.jar,
>>> kafka-clients.jar and kafka-server.jar (clients depends on common, and
>>> server depends on clients and common).
>>> C) Make a jar for each client. Currently this would be just producer and
>>> consumer, but in the future we could add a more well-defined an Admin
>>> client for some of the administrative functions. If there ended up being
>>> code that is client-specific but shared by multiple clients this could be
>>> problematic.
>>> 
>>> Alternately I don't know if the mapping from sub-modules to jars needs to
>>> be one-to-one so we could seperate the clients and common code to enforce
>>> compile dependencies and then glump it all into one client jar. That
>> might
>>> anger the build system, though.
>>> 
>>> I think I vote for (A) but don't really care much.
>>> 
>>> -Jay
>> 

Re: code layout

[Neha Narkhede <ne...@gmail.com>]

I like A. It will be simpler to maintain and evolve when we add the admin
APIs.



On Fri, Jan 31, 2014 at 7:32 PM, Steve Morin <st...@stevemorin.com> wrote:

> I like A or C
>
> I think it will be clearer for people to separate the two.
>
>
> On Fri, Jan 31, 2014 at 3:29 PM, Jay Kreps <ja...@gmail.com> wrote:
>
> > For the new producer code we currently added the new module
> >    clients
> > This builds the jar kafka-clients.jar. The core module should be renamed
> to
> > kafka-server and producer kafka-server.jar.
> >
> > It is the intention that the server will end up depending on the clients
> > but not vice versa (or we could make a separate module for that if we
> > like).
> >
> > Integration code that tests clients against the server will live with the
> > server.
> >
> > There is some shared code between the clients and server. This is the
> > kafka.common package. Currently this is in the clients module, which is a
> > little odd. We could alternatively break it into its own module, which
> > might be nice. However I'm not sure it really warrants its own jar since
> > there isn't much point to that code on its own and having the clients
> need
> > two jars is kind of annoying. But maybe this doesn't matter because
> > everyone just uses Maven?
> >
> > So the options are:
> >
> > A) Leave it the way it is: kafka-clients.jar and kafka-server.jar.
> > B) Separate out the common code and have kafka-common.jar,
> > kafka-clients.jar and kafka-server.jar (clients depends on common, and
> > server depends on clients and common).
> > C) Make a jar for each client. Currently this would be just producer and
> > consumer, but in the future we could add a more well-defined an Admin
> > client for some of the administrative functions. If there ended up being
> > code that is client-specific but shared by multiple clients this could be
> > problematic.
> >
> > Alternately I don't know if the mapping from sub-modules to jars needs to
> > be one-to-one so we could seperate the clients and common code to enforce
> > compile dependencies and then glump it all into one client jar. That
> might
> > anger the build system, though.
> >
> > I think I vote for (A) but don't really care much.
> >
> > -Jay
> >
>

Re: code layout

[Steve Morin <st...@stevemorin.com>]

I like A or C

I think it will be clearer for people to separate the two.


On Fri, Jan 31, 2014 at 3:29 PM, Jay Kreps <ja...@gmail.com> wrote:

> For the new producer code we currently added the new module
>    clients
> This builds the jar kafka-clients.jar. The core module should be renamed to
> kafka-server and producer kafka-server.jar.
>
> It is the intention that the server will end up depending on the clients
> but not vice versa (or we could make a separate module for that if we
> like).
>
> Integration code that tests clients against the server will live with the
> server.
>
> There is some shared code between the clients and server. This is the
> kafka.common package. Currently this is in the clients module, which is a
> little odd. We could alternatively break it into its own module, which
> might be nice. However I'm not sure it really warrants its own jar since
> there isn't much point to that code on its own and having the clients need
> two jars is kind of annoying. But maybe this doesn't matter because
> everyone just uses Maven?
>
> So the options are:
>
> A) Leave it the way it is: kafka-clients.jar and kafka-server.jar.
> B) Separate out the common code and have kafka-common.jar,
> kafka-clients.jar and kafka-server.jar (clients depends on common, and
> server depends on clients and common).
> C) Make a jar for each client. Currently this would be just producer and
> consumer, but in the future we could add a more well-defined an Admin
> client for some of the administrative functions. If there ended up being
> code that is client-specific but shared by multiple clients this could be
> problematic.
>
> Alternately I don't know if the mapping from sub-modules to jars needs to
> be one-to-one so we could seperate the clients and common code to enforce
> compile dependencies and then glump it all into one client jar. That might
> anger the build system, though.
>
> I think I vote for (A) but don't really care much.
>
> -Jay
>