You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by kant kodali <ka...@gmail.com> on 2016/09/17 06:45:17 UTC
any update on this?
https://issues.apache.org/jira/browse/KAFKA-1793
It would be great to use Consul instead of Zookeeper for Kafka and I think it
would benefit Kafka a lot from the exponentially growing consul community.
Re: any update on this?
Posted by F21 <f2...@gmail.com>.
One possible solution might be to use parkeeper, which uses consul as
the backend and exposes a facade that looks like zookeeper:
https://github.com/glerchundi/parkeeper
The project doesn't seem very active though and it is unclear whether it
supports all the features that are used by kafka.
It might be worth a try though.
Francis
On 20/09/2016 6:29 PM, Jens Rantil wrote:
>> not aware of any shortfall with zookeeper so perhaps you can suggest
> advantages for Consul vs Zookeeper?
>
> Maybe it's somewhat off-topic here, but Consul has several advantages over
> Zookeeper:
> * It's IMHO easier to maintain, add leader nodes, remove leader nodes etc.
> * Has high level service discovery mechanisms supporting both healthchecks
> and and a flexible DNS hook to load balance to services.
> * Talks HTTP, so it's much easier to integrate with any programming
> language/platform.
> * Support for multiple datacenters.
> * Zookeeper supports keep-alive as healthcheck. Consul supports more
> high-level healthchecks.
>
> While Consul is more opinionated, it also has more features and highlevel.
> IMHO, that makes it easier to work with not have to write thick clients.
> You can read more about Consul vs. Zookeeper here:
> https://www.consul.io/intro/vs/zookeeper.html
>
> Above said, just like Kant wrote, I don't think moving from Zookeeper to
> Consul should be a goal. The above Consul pros are reasons for systems to
> use Consul instead of Zookeeper. We would be happy if we could ditch
> Zookeeper and use Consul for all coordination to avoid the overhead of
> having to maintain two highly consistent datastores.
>
> Cheers,
> Jens
>
> On Mon, Sep 19, 2016 at 3:31 PM Martin Gainty <mg...@hotmail.com> wrote:
>
>> Jens/Kant
>> not aware of any shortfall with zookeeper so perhaps you can suggest
>> advantages for Consul vs Zookeeper?
>> Maven (I am building, testing and running kafka internally with maven)
>> implements wagon-providers for URLConnection vs HttpURLConnection
>> wagonshttps://maven.apache.org/guides/mini/guide-wagon-providers.html
>> Thinking a network_provider should work for integrating external network
>> provider. how would you architect this integration?
>>
>> would a configurable network-provider such as maven-wagon-provider work
>> for kafka?Martin
>>
>>> From: kanth909@gmail.com
>>> To: users@kafka.apache.org
>>> Subject: Re: any update on this?
>>> Date: Mon, 19 Sep 2016 09:41:10 +0000
>>>
>>> Yes ofcourse the goal shouldn't be moving towards consul. It should just
>> be
>>> flexible enough for users to pick any distributed coordinated system.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Sep 19, 2016 2:23 AM, Jens Rantil jens.rantil@tink.se
>>> wrote:
>>> I think I read somewhere that the long-term goal is to make Kafka
>>>
>>> independent of Zookeeper alltogether. Maybe not worth spending time on
>>>
>>> migrating to Consul in that case.
>>>
>>>
>>>
>>>
>>> Cheers,
>>>
>>> Jens
>>>
>>>
>>>
>>>
>>> On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain <jfountain@meetme.com
>>>
>>>
>>> wrote:
>>>
>>>
>>>
>>>
>>>> +2 watching.
>>>> On Sat, Sep 17, 2016 at 2:45 AM, kant kodali <ka...@gmail.com>
>> wrote:
>>>>> https://issues.apache.org/jira/browse/KAFKA-1793
>>>>> It would be great to use Consul instead of Zookeeper for Kafka and I
>>>> think
>>>>> it
>>>>> would benefit Kafka a lot from the exponentially growing consul
>>>> community.
>>>> --
>>>> Jennifer Fountain
>>>> DevOPS
>>> --
>>>
>>>
>>>
>>>
>>> Jens Rantil
>>>
>>> Backend Developer @ Tink
>>>
>>>
>>>
>>>
>>> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
>>>
>>> For urgent matters you can reach me at +46-708-84 18 32.
Re: any update on this?
Posted by Jens Rantil <je...@tink.se>.
> not aware of any shortfall with zookeeper so perhaps you can suggest
advantages for Consul vs Zookeeper?
Maybe it's somewhat off-topic here, but Consul has several advantages over
Zookeeper:
* It's IMHO easier to maintain, add leader nodes, remove leader nodes etc.
* Has high level service discovery mechanisms supporting both healthchecks
and and a flexible DNS hook to load balance to services.
* Talks HTTP, so it's much easier to integrate with any programming
language/platform.
* Support for multiple datacenters.
* Zookeeper supports keep-alive as healthcheck. Consul supports more
high-level healthchecks.
While Consul is more opinionated, it also has more features and highlevel.
IMHO, that makes it easier to work with not have to write thick clients.
You can read more about Consul vs. Zookeeper here:
https://www.consul.io/intro/vs/zookeeper.html
Above said, just like Kant wrote, I don't think moving from Zookeeper to
Consul should be a goal. The above Consul pros are reasons for systems to
use Consul instead of Zookeeper. We would be happy if we could ditch
Zookeeper and use Consul for all coordination to avoid the overhead of
having to maintain two highly consistent datastores.
Cheers,
Jens
On Mon, Sep 19, 2016 at 3:31 PM Martin Gainty <mg...@hotmail.com> wrote:
> Jens/Kant
> not aware of any shortfall with zookeeper so perhaps you can suggest
> advantages for Consul vs Zookeeper?
> Maven (I am building, testing and running kafka internally with maven)
> implements wagon-providers for URLConnection vs HttpURLConnection
> wagonshttps://maven.apache.org/guides/mini/guide-wagon-providers.html
> Thinking a network_provider should work for integrating external network
> provider. how would you architect this integration?
>
> would a configurable network-provider such as maven-wagon-provider work
> for kafka?Martin
>
> > From: kanth909@gmail.com
> > To: users@kafka.apache.org
> > Subject: Re: any update on this?
> > Date: Mon, 19 Sep 2016 09:41:10 +0000
> >
> > Yes ofcourse the goal shouldn't be moving towards consul. It should just
> be
> > flexible enough for users to pick any distributed coordinated system.
> >
> >
> >
> >
> >
> >
> > On Mon, Sep 19, 2016 2:23 AM, Jens Rantil jens.rantil@tink.se
> > wrote:
> > I think I read somewhere that the long-term goal is to make Kafka
> >
> > independent of Zookeeper alltogether. Maybe not worth spending time on
> >
> > migrating to Consul in that case.
> >
> >
> >
> >
> > Cheers,
> >
> > Jens
> >
> >
> >
> >
> > On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain <jfountain@meetme.com
> >
> >
> > wrote:
> >
> >
> >
> >
> > > +2 watching.
> >
> > >
> >
> > > On Sat, Sep 17, 2016 at 2:45 AM, kant kodali <ka...@gmail.com>
> wrote:
> >
> > >
> >
> > > > https://issues.apache.org/jira/browse/KAFKA-1793
> >
> > > > It would be great to use Consul instead of Zookeeper for Kafka and I
> >
> > > think
> >
> > > > it
> >
> > > > would benefit Kafka a lot from the exponentially growing consul
> >
> > > community.
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > --
> >
> > >
> >
> > >
> >
> > > Jennifer Fountain
> >
> > > DevOPS
> >
> > >
> >
> > --
> >
> >
> >
> >
> > Jens Rantil
> >
> > Backend Developer @ Tink
> >
> >
> >
> >
> > Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
> >
> > For urgent matters you can reach me at +46-708-84 18 32.
>
--
Jens Rantil
Backend Developer @ Tink
Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.
RE: any update on this?
Posted by Martin Gainty <mg...@hotmail.com>.
Jens/Kant
not aware of any shortfall with zookeeper so perhaps you can suggest advantages for Consul vs Zookeeper?
Maven (I am building, testing and running kafka internally with maven) implements wagon-providers for URLConnection vs HttpURLConnection wagonshttps://maven.apache.org/guides/mini/guide-wagon-providers.html
Thinking a network_provider should work for integrating external network provider. how would you architect this integration?
would a configurable network-provider such as maven-wagon-provider work for kafka?Martin
> From: kanth909@gmail.com
> To: users@kafka.apache.org
> Subject: Re: any update on this?
> Date: Mon, 19 Sep 2016 09:41:10 +0000
>
> Yes ofcourse the goal shouldn't be moving towards consul. It should just be
> flexible enough for users to pick any distributed coordinated system.
>
>
>
>
>
>
> On Mon, Sep 19, 2016 2:23 AM, Jens Rantil jens.rantil@tink.se
> wrote:
> I think I read somewhere that the long-term goal is to make Kafka
>
> independent of Zookeeper alltogether. Maybe not worth spending time on
>
> migrating to Consul in that case.
>
>
>
>
> Cheers,
>
> Jens
>
>
>
>
> On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain <jf...@meetme.com>
>
> wrote:
>
>
>
>
> > +2 watching.
>
> >
>
> > On Sat, Sep 17, 2016 at 2:45 AM, kant kodali <ka...@gmail.com> wrote:
>
> >
>
> > > https://issues.apache.org/jira/browse/KAFKA-1793
>
> > > It would be great to use Consul instead of Zookeeper for Kafka and I
>
> > think
>
> > > it
>
> > > would benefit Kafka a lot from the exponentially growing consul
>
> > community.
>
> >
>
> >
>
> >
>
> >
>
> > --
>
> >
>
> >
>
> > Jennifer Fountain
>
> > DevOPS
>
> >
>
> --
>
>
>
>
> Jens Rantil
>
> Backend Developer @ Tink
>
>
>
>
> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
>
> For urgent matters you can reach me at +46-708-84 18 32.
Re: any update on this?
Posted by kant kodali <ka...@gmail.com>.
Yes ofcourse the goal shouldn't be moving towards consul. It should just be
flexible enough for users to pick any distributed coordinated system.
On Mon, Sep 19, 2016 2:23 AM, Jens Rantil jens.rantil@tink.se
wrote:
I think I read somewhere that the long-term goal is to make Kafka
independent of Zookeeper alltogether. Maybe not worth spending time on
migrating to Consul in that case.
Cheers,
Jens
On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain <jf...@meetme.com>
wrote:
> +2 watching.
>
> On Sat, Sep 17, 2016 at 2:45 AM, kant kodali <ka...@gmail.com> wrote:
>
> > https://issues.apache.org/jira/browse/KAFKA-1793
> > It would be great to use Consul instead of Zookeeper for Kafka and I
> think
> > it
> > would benefit Kafka a lot from the exponentially growing consul
> community.
>
>
>
>
> --
>
>
> Jennifer Fountain
> DevOPS
>
--
Jens Rantil
Backend Developer @ Tink
Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.
Re: any update on this?
Posted by Jens Rantil <je...@tink.se>.
I think I read somewhere that the long-term goal is to make Kafka
independent of Zookeeper alltogether. Maybe not worth spending time on
migrating to Consul in that case.
Cheers,
Jens
On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain <jf...@meetme.com>
wrote:
> +2 watching.
>
> On Sat, Sep 17, 2016 at 2:45 AM, kant kodali <ka...@gmail.com> wrote:
>
> > https://issues.apache.org/jira/browse/KAFKA-1793
> > It would be great to use Consul instead of Zookeeper for Kafka and I
> think
> > it
> > would benefit Kafka a lot from the exponentially growing consul
> community.
>
>
>
>
> --
>
>
> Jennifer Fountain
> DevOPS
>
--
Jens Rantil
Backend Developer @ Tink
Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.
Re: any update on this?
Posted by Jennifer Fountain <jf...@meetme.com>.
+2 watching.
On Sat, Sep 17, 2016 at 2:45 AM, kant kodali <ka...@gmail.com> wrote:
> https://issues.apache.org/jira/browse/KAFKA-1793
> It would be great to use Consul instead of Zookeeper for Kafka and I think
> it
> would benefit Kafka a lot from the exponentially growing consul community.
--
Jennifer Fountain
DevOPS