[DISCUSSION] Boot dependency in the new producer

[Guozhang Wang <wa...@gmail.com>]

Hi all,

I am not sure if we have discussed about this before, but recently I
realized that we have introduced boot dependency of the kafka-server
specified by the "bootstrap.servers" config in the new producer. More
specifically, although in the old producer we also have a similar config
for specifying the broker list, the producer will not try to connect to
those brokers until the first message send call is triggered; whereas in
the new producer, it will try to talk to them in construction time via:

update(Cluster.bootstrap(addresses), time.milliseconds());


I personally am neutral to this change, as in most cases the corresponding
kafka server should be up and running before the producer clients are
deployed, but there are still some corner cases when it is not true, for
example some standalone deployment tests of the app embedded with some
clients, etc. So I would like to bring this up to people's attention if we
have not discussed about it before: do we think this is OK to introduce
this boot dependency in the new producer?

-- Guozhang

Re: [DISCUSSION] Boot dependency in the new producer

[Guozhang Wang <wa...@gmail.com>]

Steven,

You are right, I was wrong about the previous email: it will not set the
flag, the Sender thread will trigger Client.poll() but with Int.Max select
time, hence this should not be an issue. I am closing this discussion now.

Guozhang

On Mon, Jan 26, 2015 at 4:42 PM, Steven Wu <st...@gmail.com> wrote:

> Jay, I don't think this line will bootstrap full metadata (for all topics).
> it will just construct the cluster object with bootstrap host. you need to
> do "metadata.add(topic)"  to set interest of a topic's partition metadata.
>
> Guozhang, I personally think this is ok. it just do a few DNS lookup or TCP
> connection before first send.
>
> On Mon, Jan 26, 2015 at 2:07 PM, Jay Kreps <ja...@gmail.com> wrote:
>
> > Oh, yes, I guess I thought you meant that construction of the client
> would
> > block on the metadata request.
> >
> > I don't personally think that is a problem because if it fails it will
> > retry in the background, right?
> >
> > But actually I think this is probably violating another desirable
> criteria
> > we had talked about which was keeping the producer from bootstrapping the
> > full metadata for all partitions. If it is doing that during construction
> > time presumably the resulting metadata request is for all partitions, no?
> > That isn't a huge problem, but I think isn't what was intended.
> >
> > -Jay
> >
> > On Mon, Jan 26, 2015 at 1:34 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > It will set the needUpdate flag to true and hence the background Sender
> > > will try to talk to the bootstrap servers.
> > >
> > > Guozhang
> > >
> > > On Mon, Jan 26, 2015 at 1:12 PM, Jay Kreps <ja...@gmail.com>
> wrote:
> > >
> > > > Hey Guozhang,
> > > >
> > > > That line shouldn't cause any connections to Kafka to be established,
> > > does
> > > > it? All that is doing is creating the Cluster pojo using the supplied
> > > > addresses. The use of InetSocketAddress may cause some dns stuff to
> > > happen,
> > > > though...
> > > >
> > > > -Jay
> > > >
> > > > On Mon, Jan 26, 2015 at 10:50 AM, Guozhang Wang <wa...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I am not sure if we have discussed about this before, but recently
> I
> > > > > realized that we have introduced boot dependency of the
> kafka-server
> > > > > specified by the "bootstrap.servers" config in the new producer.
> More
> > > > > specifically, although in the old producer we also have a similar
> > > config
> > > > > for specifying the broker list, the producer will not try to
> connect
> > to
> > > > > those brokers until the first message send call is triggered;
> whereas
> > > in
> > > > > the new producer, it will try to talk to them in construction time
> > via:
> > > > >
> > > > > update(Cluster.bootstrap(addresses), time.milliseconds());
> > > > >
> > > > >
> > > > > I personally am neutral to this change, as in most cases the
> > > > corresponding
> > > > > kafka server should be up and running before the producer clients
> are
> > > > > deployed, but there are still some corner cases when it is not
> true,
> > > for
> > > > > example some standalone deployment tests of the app embedded with
> > some
> > > > > clients, etc. So I would like to bring this up to people's
> attention
> > if
> > > > we
> > > > > have not discussed about it before: do we think this is OK to
> > introduce
> > > > > this boot dependency in the new producer?
> > > > >
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>



-- 
-- Guozhang

Re: [DISCUSSION] Boot dependency in the new producer

[Steven Wu <st...@gmail.com>]

Jay, I don't think this line will bootstrap full metadata (for all topics).
it will just construct the cluster object with bootstrap host. you need to
do "metadata.add(topic)"  to set interest of a topic's partition metadata.

Guozhang, I personally think this is ok. it just do a few DNS lookup or TCP
connection before first send.

On Mon, Jan 26, 2015 at 2:07 PM, Jay Kreps <ja...@gmail.com> wrote:

> Oh, yes, I guess I thought you meant that construction of the client would
> block on the metadata request.
>
> I don't personally think that is a problem because if it fails it will
> retry in the background, right?
>
> But actually I think this is probably violating another desirable criteria
> we had talked about which was keeping the producer from bootstrapping the
> full metadata for all partitions. If it is doing that during construction
> time presumably the resulting metadata request is for all partitions, no?
> That isn't a huge problem, but I think isn't what was intended.
>
> -Jay
>
> On Mon, Jan 26, 2015 at 1:34 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > It will set the needUpdate flag to true and hence the background Sender
> > will try to talk to the bootstrap servers.
> >
> > Guozhang
> >
> > On Mon, Jan 26, 2015 at 1:12 PM, Jay Kreps <ja...@gmail.com> wrote:
> >
> > > Hey Guozhang,
> > >
> > > That line shouldn't cause any connections to Kafka to be established,
> > does
> > > it? All that is doing is creating the Cluster pojo using the supplied
> > > addresses. The use of InetSocketAddress may cause some dns stuff to
> > happen,
> > > though...
> > >
> > > -Jay
> > >
> > > On Mon, Jan 26, 2015 at 10:50 AM, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I am not sure if we have discussed about this before, but recently I
> > > > realized that we have introduced boot dependency of the kafka-server
> > > > specified by the "bootstrap.servers" config in the new producer. More
> > > > specifically, although in the old producer we also have a similar
> > config
> > > > for specifying the broker list, the producer will not try to connect
> to
> > > > those brokers until the first message send call is triggered; whereas
> > in
> > > > the new producer, it will try to talk to them in construction time
> via:
> > > >
> > > > update(Cluster.bootstrap(addresses), time.milliseconds());
> > > >
> > > >
> > > > I personally am neutral to this change, as in most cases the
> > > corresponding
> > > > kafka server should be up and running before the producer clients are
> > > > deployed, but there are still some corner cases when it is not true,
> > for
> > > > example some standalone deployment tests of the app embedded with
> some
> > > > clients, etc. So I would like to bring this up to people's attention
> if
> > > we
> > > > have not discussed about it before: do we think this is OK to
> introduce
> > > > this boot dependency in the new producer?
> > > >
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [DISCUSSION] Boot dependency in the new producer

[Jay Kreps <ja...@gmail.com>]

Oh, yes, I guess I thought you meant that construction of the client would
block on the metadata request.

I don't personally think that is a problem because if it fails it will
retry in the background, right?

But actually I think this is probably violating another desirable criteria
we had talked about which was keeping the producer from bootstrapping the
full metadata for all partitions. If it is doing that during construction
time presumably the resulting metadata request is for all partitions, no?
That isn't a huge problem, but I think isn't what was intended.

-Jay

On Mon, Jan 26, 2015 at 1:34 PM, Guozhang Wang <wa...@gmail.com> wrote:

> It will set the needUpdate flag to true and hence the background Sender
> will try to talk to the bootstrap servers.
>
> Guozhang
>
> On Mon, Jan 26, 2015 at 1:12 PM, Jay Kreps <ja...@gmail.com> wrote:
>
> > Hey Guozhang,
> >
> > That line shouldn't cause any connections to Kafka to be established,
> does
> > it? All that is doing is creating the Cluster pojo using the supplied
> > addresses. The use of InetSocketAddress may cause some dns stuff to
> happen,
> > though...
> >
> > -Jay
> >
> > On Mon, Jan 26, 2015 at 10:50 AM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I am not sure if we have discussed about this before, but recently I
> > > realized that we have introduced boot dependency of the kafka-server
> > > specified by the "bootstrap.servers" config in the new producer. More
> > > specifically, although in the old producer we also have a similar
> config
> > > for specifying the broker list, the producer will not try to connect to
> > > those brokers until the first message send call is triggered; whereas
> in
> > > the new producer, it will try to talk to them in construction time via:
> > >
> > > update(Cluster.bootstrap(addresses), time.milliseconds());
> > >
> > >
> > > I personally am neutral to this change, as in most cases the
> > corresponding
> > > kafka server should be up and running before the producer clients are
> > > deployed, but there are still some corner cases when it is not true,
> for
> > > example some standalone deployment tests of the app embedded with some
> > > clients, etc. So I would like to bring this up to people's attention if
> > we
> > > have not discussed about it before: do we think this is OK to introduce
> > > this boot dependency in the new producer?
> > >
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [DISCUSSION] Boot dependency in the new producer

[Guozhang Wang <wa...@gmail.com>]

It will set the needUpdate flag to true and hence the background Sender
will try to talk to the bootstrap servers.

Guozhang

On Mon, Jan 26, 2015 at 1:12 PM, Jay Kreps <ja...@gmail.com> wrote:

> Hey Guozhang,
>
> That line shouldn't cause any connections to Kafka to be established, does
> it? All that is doing is creating the Cluster pojo using the supplied
> addresses. The use of InetSocketAddress may cause some dns stuff to happen,
> though...
>
> -Jay
>
> On Mon, Jan 26, 2015 at 10:50 AM, Guozhang Wang <wa...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I am not sure if we have discussed about this before, but recently I
> > realized that we have introduced boot dependency of the kafka-server
> > specified by the "bootstrap.servers" config in the new producer. More
> > specifically, although in the old producer we also have a similar config
> > for specifying the broker list, the producer will not try to connect to
> > those brokers until the first message send call is triggered; whereas in
> > the new producer, it will try to talk to them in construction time via:
> >
> > update(Cluster.bootstrap(addresses), time.milliseconds());
> >
> >
> > I personally am neutral to this change, as in most cases the
> corresponding
> > kafka server should be up and running before the producer clients are
> > deployed, but there are still some corner cases when it is not true, for
> > example some standalone deployment tests of the app embedded with some
> > clients, etc. So I would like to bring this up to people's attention if
> we
> > have not discussed about it before: do we think this is OK to introduce
> > this boot dependency in the new producer?
> >
> > -- Guozhang
> >
>



-- 
-- Guozhang

Re: [DISCUSSION] Boot dependency in the new producer

[Jay Kreps <ja...@gmail.com>]

Hey Guozhang,

That line shouldn't cause any connections to Kafka to be established, does
it? All that is doing is creating the Cluster pojo using the supplied
addresses. The use of InetSocketAddress may cause some dns stuff to happen,
though...

-Jay

On Mon, Jan 26, 2015 at 10:50 AM, Guozhang Wang <wa...@gmail.com> wrote:

> Hi all,
>
> I am not sure if we have discussed about this before, but recently I
> realized that we have introduced boot dependency of the kafka-server
> specified by the "bootstrap.servers" config in the new producer. More
> specifically, although in the old producer we also have a similar config
> for specifying the broker list, the producer will not try to connect to
> those brokers until the first message send call is triggered; whereas in
> the new producer, it will try to talk to them in construction time via:
>
> update(Cluster.bootstrap(addresses), time.milliseconds());
>
>
> I personally am neutral to this change, as in most cases the corresponding
> kafka server should be up and running before the producer clients are
> deployed, but there are still some corner cases when it is not true, for
> example some standalone deployment tests of the app embedded with some
> clients, etc. So I would like to bring this up to people's attention if we
> have not discussed about it before: do we think this is OK to introduce
> this boot dependency in the new producer?
>
> -- Guozhang
>

Re: [DISCUSSION] Boot dependency in the new producer

[Jay Kreps <ja...@gmail.com>]

Hey Guozhang,

That line shouldn't cause any connections to Kafka to be established, does
it? All that is doing is creating the Cluster pojo using the supplied
addresses. The use of InetSocketAddress may cause some dns stuff to happen,
though...

-Jay

On Mon, Jan 26, 2015 at 10:50 AM, Guozhang Wang <wa...@gmail.com> wrote:

> Hi all,
>
> I am not sure if we have discussed about this before, but recently I
> realized that we have introduced boot dependency of the kafka-server
> specified by the "bootstrap.servers" config in the new producer. More
> specifically, although in the old producer we also have a similar config
> for specifying the broker list, the producer will not try to connect to
> those brokers until the first message send call is triggered; whereas in
> the new producer, it will try to talk to them in construction time via:
>
> update(Cluster.bootstrap(addresses), time.milliseconds());
>
>
> I personally am neutral to this change, as in most cases the corresponding
> kafka server should be up and running before the producer clients are
> deployed, but there are still some corner cases when it is not true, for
> example some standalone deployment tests of the app embedded with some
> clients, etc. So I would like to bring this up to people's attention if we
> have not discussed about it before: do we think this is OK to introduce
> this boot dependency in the new producer?
>
> -- Guozhang
>