You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by lb...@gmail.com,
lb...@gmail.com on 2018/09/14 07:46:51 UTC
Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper
On 2018/05/09 14:29:17, Molnár Bálint <mo...@gmail.com> wrote:
> Hi,
> I just rebased the Etcd implementation proposal on trunk. Pinging to see if
> anyone has feedback on my questions from my previous email.
>
> Molnár Bálint <mo...@gmail.com> ezt írta (időpont: 2018. ápr. 4.,
> Sze, 10:08):
>
> > Hi,
> > Thanks again for the feedback.
> >
> > Is there already ongoing work for having an own consensus implementation
> > within Kafka?
> > If that work haven't started yet, we think there is value in having an
> > interim solution, that allows the use of another consensus system besides
> > Zookeeper.
> >
> > We ask the community to take a look at the Etcd implementation proposal
> > we created and provide feedback on that.
> > This helps to asses rather this approach is viable at all.
> >
> > We are open to collaborate on integrating our proposed Etcd implementation
> > into any integration test system, to certify that all use cases works as
> > expected.
> >
> > Balint
> >
> > 2018-03-30 22:21 GMT+02:00 Gwen Shapira <gw...@confluent.io>:
> >
> >> Hi,
> >>
> >> I had an offline discussion with Ismael and wanted to summarize the
> >> comments and questions he raised so we are all on the same page.
> >>
> >> The core issue is that this change adds a new public API. Since we already
> >> know that the goal for the next 1-2 years is to get rid of ZK completely.
> >> Do we want to go to the effort of adding (and discussing and reviewing) a
> >> new public API, knowing that it will be completely removed in a year? And
> >> since building and testing a plugin also involves effort, will anyone do
> >> it
> >> for something that is going to be temporary by design?
> >>
> >> Ismael, correct me if this isn't a fair representation of your concerns.
> >>
> >> Gwen
> >>
> >>
> >>
> >> On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira <gw...@confluent.io> wrote:
> >>
> >> > Few other concerns that were raised in the previous discussion were
> >> around
> >> > the challenges both to maintainers and users in making this API
> >> pluggable
> >> > and how does making the interface pluggable aligns with future goals for
> >> > the project. At the time this was difficult to discuss because there
> >> wasn't
> >> > a concrete proposal. I want to discuss these points in the context of
> >> this
> >> > specific proposal:
> >> >
> >> > 1. Problem: Pluggable APIs mean larger surface testing area and multiple
> >> > implementations to cover.
> >> > In this case: At the time, the Kafka project didn't have much
> >> > experience with pluggable APIs and components, so the concerns were very
> >> > valid. Right now Kafka has multiple pluggable components - Connectors,
> >> > converters, transformations, authentication protocols, authorization
> >> > database, coordination protocol, serializers, etc. I think that as a
> >> > community we gotten better at testing the interface, testing the very
> >> few
> >> > implementations that are included in Apache Kafka itself and allowing
> >> the
> >> > community to innovate and validate outside of the Kafka project. I don't
> >> > recall major issues either from lack of testing or from usability
> >> > perspective.
> >> >
> >> > 2. Problem: Users don't want to choose a consensus implementation, they
> >> > just don't want ZK.
> >> > In this case: I agree that users don't actually want to spend time
> >> > choosing consensus implementation and a simpler deployment model would
> >> > serve them better. IMO, if Apache Kafka ships with our well-tested ZK
> >> > implementation, 99% of the users will choose to use that (a vast
> >> majority
> >> > uses our less-than-amazing authorization plugin), and the few that
> >> really
> >> > need something else for whatever reason, will be able to get what they
> >> > need. As Jake said, we need to face the fact that development
> >> trajectory of
> >> > ZK isn't amazing at the moment, that it is lacking features our users
> >> need
> >> > (SSL) and it will be good to allow the user community to explore
> >> > alternatives.
> >> >
> >> > 3. Problem: Why got to the effort of refactoring if we know we want to
> >> get
> >> > rid of ZK.
> >> > In this case: This change isn't huge, it doesn't rewrite large
> >> > portions of Kafka and it does not make the future direction any more
> >> > difficult. If in 2 weeks or 2 month or 2 years we'll have a ZK-less
> >> > solution, applying it on Kafka with this KIP isn't any more challenging
> >> > than applying it to Kafka without this KIP. It is a step in an
> >> orthogonal
> >> > direction, but not opposite direction. I think that letting the perfect
> >> > become the enemy of the good is a repeated failure mode in this
> >> community.
> >> > Can we discuss whether this proposal is good even if there is a more
> >> > complex proposal that may be better? As long as they don't conflict?
> >> >
> >> > Gwen
> >> >
> >> > On Thu, Mar 29, 2018 at 8:31 AM, Molnár Bálint <mo...@gmail.com>
> >> > wrote:
> >> >
> >> >> Thanks, for the feedback.
> >> >>
> >> >> Developing an internal consensus service inside Kafka would require a
> >> team
> >> >> dedicated to this task.
> >> >> We second what Flavio said in
> >> >> https://lists.apache.org/thread.html/24ae56e073104c4531cf64f
> >> >> 7a1f1c0a84f895d139d334c88e9fe6028@1449008733@%3Cdev.kafka.apache.org
> >> %3E
> >> >> that
> >> >> getting an implementation which really works and is maintainable is
> >> >> difficult.
> >> >> We think that Kafka being able to use another widely used consensus
> >> system
> >> >> beside Zookeeper its a safer and workable solution.
> >> >> It will be easier for users to use a consensus system with Kafka that
> >> they
> >> >> are already familiar with.
> >> >>
> >> >>
> >> >> The implementation found here:
> >> >> https://github.com/banzaicloud/apache-kafka-on-k8s/tree/
> >> >> kafka-on-etcd/core/src/main/scala/kafka/etcd
> >> >> is a first version of enabling Etcd in Kafka.
> >> >> This implementation hooked in Etcd with a slight change in the existing
> >> >> interfaces. While this implementation works its far from being
> >> complete.
> >> >> Ideally existing interfaces should be reworked to abstract out the used
> >> >> consensus system.
> >> >> We opened this KIP to start a discussion and the community to have a
> >> look
> >> >> at the initial implementation and receive feedback if this initiative
> >> is
> >> >> viable.
> >> >>
> >> >> Balint
> >> >>
> >> >> 2018-03-29 11:23 GMT+02:00 Jakub Scholz <ja...@scholz.cz>:
> >> >>
> >> >> > I can understand the concerns about the plugability of
> >> Zookeeper/Etcd.
> >> >> It
> >> >> > would not be good for Kafka community if it splits into several
> >> groups
> >> >> > using different implementations.
> >> >> >
> >> >> > On the other hand, Zookeeper development seems to be a bit stalled.
> >> So
> >> >> > maybe there should be at least a discussion whether it makes sense to
> >> >> > replace Zookeeper with something like Etcd or not.
> >> >> >
> >> >> > JAkub
> >> >> >
> >> >> > On Wed, Mar 28, 2018 at 6:18 PM, Molnár Bálint <
> >> molnarcsirke@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> > > Hi all,
> >> >> > >
> >> >> > > I have created KIP-273: Kafka to support using ETCD beside
> >> Zookeeper
> >> >> > >
> >> >> > > Here is the link to the KIP:
> >> >> > >
> >> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> >> > > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
> >> >> > >
> >> >> > > Looking forward to the discussion.
> >> >> > >
> >> >> > > Thanks,
> >> >> > > Balint
> >> >> > >
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > *Gwen Shapira*
> >> > Product Manager | Confluent
> >> > 650.450.2760 <(650)%20450-2760> | @gwenshap
> >> > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> >> > <http://www.confluent.io/blog>
> >> >
> >> >
> >>
> >>
> >> --
> >> *Gwen Shapira*
> >> Product Manager | Confluent
> >> 650.450.2760 | @gwenshap
> >> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> >> <http://www.confluent.io/blog>
> >>
> >
> >
>
Hello, Colin!
I have read the discussion on KIP-273 and want to raise this question. Deploying Kafka to Kubernetes is a crucial problem in my current work. And usage of ZooKeeper makes troubles, because it works unstable in container.
In this discussion you have mentioned that the goal is to get rid of it in future releases. But since then there were no posts about it. Also, i can not see such plan in a roadmap. Cuold you please tell us more about work made in this stream.
Thanks,
Leonid
Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper
Posted by lb...@gmail.com,
lb...@gmail.com.
On 2018/09/14 21:00:10, Colin McCabe <cm...@apache.org> wrote:
> On Fri, Sep 14, 2018, at 00:46, lbarikyn@gmail.com wrote:
> > Hello, Colin!
> >
> > I have read the discussion on KIP-273 and want to raise this question.
> > Deploying Kafka to Kubernetes is a crucial problem in my current work.
> > And usage of ZooKeeper makes troubles, because it works unstable in
> > container.
>
> Hi Ibarikyn,
>
> I haven't heard any reports of ZooKeeper being unstable in containers. There are a lot of companies already deploying Kafka and ZooKeeper this way, so I am surprised to hear that you had trouble. Can you be a little more clear about what issues you are seeing with ZooKeeper in containers?
>
> > In this discussion you have mentioned that the goal is to get rid of it
> > in future releases. But since then there were no posts about it. Also, i
> > can not see such plan in a roadmap. Cuold you please tell us more about
> > work made in this stream.
>
> ZooKeeper (and etcd, and consul, and the others) were originally designed as configuration management systems. They're really not set up to do what Kafka does with them, which is use them as one half of a data management system. We need things like uniform access control, the ability to cache metadata locally, the ability to propagate only what has changed to brokers, etc. That requires us to manage our own metadata log. Kafka is a system for managing logs, so this is a natural design.
>
> There have been a lot of discussions about pluggable consensus before on the mailing list. Even if we wanted to keep using a configuration management system for metadata, pluggable consensus is a really bad idea. It means we would have to support multiple systems, getting correspondingly less testing in each. We also would have to use least common denominator APIs, since each system is somewhat different. And configuration would get harder for new users.
>
> best,
> Colin
>
Hello,
Thank you for the answer!
I can see your point and agree with some ideas (eg complexity of configuration custom systems for users).
Some troubles using Zk that I mentioned:
- it doesnt suuport ssl (at least the version that is used in Kafka)
- cluster being unstable
- dns names confusing
But I talking about another issue. As far as I am concerned, getting rid of dependency to Zk(and other configuration management) is the right direction. But you did not evaluate the time when we will see it in Kafka. I would like to know, is it WIP or smth like that. Do u have any evaluations when this feature will be released?
BTW, how Zk-less version is supposed to work? What technology will be used as a metadata container?
As far as I am very interested in this feature, I d like to try to contribute my work. Could u show me issues/tickets so i can get acquainted with the situation and maybe start work.
Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper
Posted by Colin McCabe <cm...@apache.org>.
On Fri, Sep 14, 2018, at 00:46, lbarikyn@gmail.com wrote:
> Hello, Colin!
>
> I have read the discussion on KIP-273 and want to raise this question.
> Deploying Kafka to Kubernetes is a crucial problem in my current work.
> And usage of ZooKeeper makes troubles, because it works unstable in
> container.
Hi Ibarikyn,
I haven't heard any reports of ZooKeeper being unstable in containers. There are a lot of companies already deploying Kafka and ZooKeeper this way, so I am surprised to hear that you had trouble. Can you be a little more clear about what issues you are seeing with ZooKeeper in containers?
> In this discussion you have mentioned that the goal is to get rid of it
> in future releases. But since then there were no posts about it. Also, i
> can not see such plan in a roadmap. Cuold you please tell us more about
> work made in this stream.
ZooKeeper (and etcd, and consul, and the others) were originally designed as configuration management systems. They're really not set up to do what Kafka does with them, which is use them as one half of a data management system. We need things like uniform access control, the ability to cache metadata locally, the ability to propagate only what has changed to brokers, etc. That requires us to manage our own metadata log. Kafka is a system for managing logs, so this is a natural design.
There have been a lot of discussions about pluggable consensus before on the mailing list. Even if we wanted to keep using a configuration management system for metadata, pluggable consensus is a really bad idea. It means we would have to support multiple systems, getting correspondingly less testing in each. We also would have to use least common denominator APIs, since each system is somewhat different. And configuration would get harder for new users.
best,
Colin