You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Bae, Jae Hyeon" <me...@gmail.com> on 2012/11/19 00:37:22 UTC
Why do we need reconnect with an interval?
Hi
I am wondering that we do really need the following producer configuration
val reconnectInterval = Utils.getInt(props, "reconnect.interval", 30000)
/** negative reconnect time interval means disabling this time-based
reconnect feature */
var reconnectTimeInterval = Utils.getInt(props,
"reconnect.time.interval.ms", 1000*1000*10)
If these configurations have the history, please let me know.
Otherwise, I want to disable these default configurations.
Thank you
Best, Jae
Re: Why do we need reconnect with an interval?
Posted by Jun Rao <ju...@gmail.com>.
If use you zk.connect in the producer, you can leave reconnect.interval as
maxInt since we do load balancing using ZK. If you use a single VIP in
broker.list in the producer, reconnect.interval will help load balancing.
Thanks,
Jun
On Sun, Nov 18, 2012 at 10:15 PM, Bae, Jae Hyeon <me...@gmail.com> wrote:
> Thank you for your answer.
>
> SyncProducer doesn't change its host and port when reconnecting. So, I
> don't understand why reconnection can be helpful for load balancing.
> When the new broker is registered in the zookeeper, it will be added
> in the ProducerPool by producerCbk, which does not seem related with
> balancing.
>
> So, is it totally safe to change reconnect.interval as
> Integer.MAX_VALUE and reconnect.time.interval.ms as -1 in the
> environment where the broker can be newly created?
>
> On Sun, Nov 18, 2012 at 9:57 PM, Jun Rao <ju...@gmail.com> wrote:
> > In addition to that, reconnect interval is useful for load balancing. If
> a
> > broker is restarted, initially, no existing producers are connected to
> it.
> > Reconnect interval gives the producer a chance to connect to the new
> broker
> > periodically.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Nov 18, 2012 at 3:47 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
> >
> >> Hi,
> >>
> >> This config is really relevant to 0.7 Kafka users using a VIP to talk
> >> to the Kafka cluster. VIP has a timeout for every TCP connection,
> >> where it cuts the connection that has been idle for a certain time.
> >> Most VIPs are configured to not send TCP RST back to the client, so
> >> the client does not know that the tcp connection is no more. The next
> >> time the client tries to write to the socket, it gets into a really
> >> long TCP RST retry loop. To avoid this, we added a producer config
> >> that would reconnect to the VIP after n requests or m ms, whatever
> >> comes earlier.
> >>
> >> This config is no longer required in 0.8 and has been removed.
> >>
> >> Thanks,
> >> Neha
> >>
> >> On Sun, Nov 18, 2012 at 3:37 PM, Bae, Jae Hyeon <me...@gmail.com>
> >> wrote:
> >> > Hi
> >> >
> >> > I am wondering that we do really need the following producer
> >> configuration
> >> >
> >> > val reconnectInterval = Utils.getInt(props, "reconnect.interval",
> >> 30000)
> >> >
> >> > /** negative reconnect time interval means disabling this time-based
> >> > reconnect feature */
> >> > var reconnectTimeInterval = Utils.getInt(props,
> >> > "reconnect.time.interval.ms", 1000*1000*10)
> >> >
> >> > If these configurations have the history, please let me know.
> >> > Otherwise, I want to disable these default configurations.
> >> >
> >> > Thank you
> >> > Best, Jae
> >>
>
Re: Why do we need reconnect with an interval?
Posted by "Bae, Jae Hyeon" <me...@gmail.com>.
Thank you for your answer.
SyncProducer doesn't change its host and port when reconnecting. So, I
don't understand why reconnection can be helpful for load balancing.
When the new broker is registered in the zookeeper, it will be added
in the ProducerPool by producerCbk, which does not seem related with
balancing.
So, is it totally safe to change reconnect.interval as
Integer.MAX_VALUE and reconnect.time.interval.ms as -1 in the
environment where the broker can be newly created?
On Sun, Nov 18, 2012 at 9:57 PM, Jun Rao <ju...@gmail.com> wrote:
> In addition to that, reconnect interval is useful for load balancing. If a
> broker is restarted, initially, no existing producers are connected to it.
> Reconnect interval gives the producer a chance to connect to the new broker
> periodically.
>
> Thanks,
>
> Jun
>
> On Sun, Nov 18, 2012 at 3:47 PM, Neha Narkhede <ne...@gmail.com>wrote:
>
>> Hi,
>>
>> This config is really relevant to 0.7 Kafka users using a VIP to talk
>> to the Kafka cluster. VIP has a timeout for every TCP connection,
>> where it cuts the connection that has been idle for a certain time.
>> Most VIPs are configured to not send TCP RST back to the client, so
>> the client does not know that the tcp connection is no more. The next
>> time the client tries to write to the socket, it gets into a really
>> long TCP RST retry loop. To avoid this, we added a producer config
>> that would reconnect to the VIP after n requests or m ms, whatever
>> comes earlier.
>>
>> This config is no longer required in 0.8 and has been removed.
>>
>> Thanks,
>> Neha
>>
>> On Sun, Nov 18, 2012 at 3:37 PM, Bae, Jae Hyeon <me...@gmail.com>
>> wrote:
>> > Hi
>> >
>> > I am wondering that we do really need the following producer
>> configuration
>> >
>> > val reconnectInterval = Utils.getInt(props, "reconnect.interval",
>> 30000)
>> >
>> > /** negative reconnect time interval means disabling this time-based
>> > reconnect feature */
>> > var reconnectTimeInterval = Utils.getInt(props,
>> > "reconnect.time.interval.ms", 1000*1000*10)
>> >
>> > If these configurations have the history, please let me know.
>> > Otherwise, I want to disable these default configurations.
>> >
>> > Thank you
>> > Best, Jae
>>
Re: Why do we need reconnect with an interval?
Posted by Jun Rao <ju...@gmail.com>.
In addition to that, reconnect interval is useful for load balancing. If a
broker is restarted, initially, no existing producers are connected to it.
Reconnect interval gives the producer a chance to connect to the new broker
periodically.
Thanks,
Jun
On Sun, Nov 18, 2012 at 3:47 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Hi,
>
> This config is really relevant to 0.7 Kafka users using a VIP to talk
> to the Kafka cluster. VIP has a timeout for every TCP connection,
> where it cuts the connection that has been idle for a certain time.
> Most VIPs are configured to not send TCP RST back to the client, so
> the client does not know that the tcp connection is no more. The next
> time the client tries to write to the socket, it gets into a really
> long TCP RST retry loop. To avoid this, we added a producer config
> that would reconnect to the VIP after n requests or m ms, whatever
> comes earlier.
>
> This config is no longer required in 0.8 and has been removed.
>
> Thanks,
> Neha
>
> On Sun, Nov 18, 2012 at 3:37 PM, Bae, Jae Hyeon <me...@gmail.com>
> wrote:
> > Hi
> >
> > I am wondering that we do really need the following producer
> configuration
> >
> > val reconnectInterval = Utils.getInt(props, "reconnect.interval",
> 30000)
> >
> > /** negative reconnect time interval means disabling this time-based
> > reconnect feature */
> > var reconnectTimeInterval = Utils.getInt(props,
> > "reconnect.time.interval.ms", 1000*1000*10)
> >
> > If these configurations have the history, please let me know.
> > Otherwise, I want to disable these default configurations.
> >
> > Thank you
> > Best, Jae
>
Re: Why do we need reconnect with an interval?
Posted by Neha Narkhede <ne...@gmail.com>.
Hi,
This config is really relevant to 0.7 Kafka users using a VIP to talk
to the Kafka cluster. VIP has a timeout for every TCP connection,
where it cuts the connection that has been idle for a certain time.
Most VIPs are configured to not send TCP RST back to the client, so
the client does not know that the tcp connection is no more. The next
time the client tries to write to the socket, it gets into a really
long TCP RST retry loop. To avoid this, we added a producer config
that would reconnect to the VIP after n requests or m ms, whatever
comes earlier.
This config is no longer required in 0.8 and has been removed.
Thanks,
Neha
On Sun, Nov 18, 2012 at 3:37 PM, Bae, Jae Hyeon <me...@gmail.com> wrote:
> Hi
>
> I am wondering that we do really need the following producer configuration
>
> val reconnectInterval = Utils.getInt(props, "reconnect.interval", 30000)
>
> /** negative reconnect time interval means disabling this time-based
> reconnect feature */
> var reconnectTimeInterval = Utils.getInt(props,
> "reconnect.time.interval.ms", 1000*1000*10)
>
> If these configurations have the history, please let me know.
> Otherwise, I want to disable these default configurations.
>
> Thank you
> Best, Jae