You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openwhisk.apache.org by Michele Sciabarra <mi...@sciabarra.com> on 2022/12/18 13:35:56 UTC
Updating Kafka getting rid of Zookeeper
Hello,
I was trying to run OpenWhisk with a newer version of Kafka and also with RedPanda. And I failed.
I found there is a dependency on version 4 of curator that in turns blocks zookeper to version 3.4 and newer version of Kafka uses 3.5 with a different protocol.
In the code, the curator is only used here:
core/invoker/src/main/scala/org/apache/openwhisk/core/invoker/InstanceIdAssigner.scala
what is exactly the purpose of this class?
From what I can undestand, it is storing in zooker the id of the invoker for the purpose of guaranteeing
Can we replace with something else that does not use zookeeper? We have now etcd for the scheduler if I am not wrong. Can we use etcd and get rid of the dependency on zookeepr 3.4 that holds back also the version of kafka, so we can experiment with Kafka Kraft and RedPanda (that does not have zookeeper at all)?
--
Michele Sciabarra
michele@sciabarra.com
Re: Updating Kafka getting rid of Zookeeper
Posted by Matt Sicker <bo...@gmail.com>.
There’s also Apache Ratis if you want a RAFT library. Looks promising.
—
Matt Sicker
> On Dec 20, 2022, at 20:00, Brendan Doyle <bd...@gmail.com> wrote:
>
> Actually to Dom’s point, rather than UUID you can probably use the ip or
> display name value to store as the key to guarantee uniqueness, but will be
> reused on restart. And still no longer have to worry about tracking the
> counter.
>
> Brendan
>
>> On Tue, Dec 20, 2022 at 8:54 PM Brendan Doyle <bd...@gmail.com> wrote:
>>
>> I think it would be nice to make the invoker ids “memoryless” in that they
>> just get a new id on restart. And this value can just be stored in etcd or
>> any key value data store. The two major things that need to be dealt with
>> on the new architecture to deal with that are 1. The healthcheck function
>> is created using the invoker id as the name of the function so we need to
>> make sure the function is deleted otherwise you’ll endlessly create the
>> healthcheck functions and 2. the container creation kafka topics since
>> they’re somewhat tied to the invoker and it’s assigned id. A new container
>> creation topic would get created every time the invoker is restarted if it
>> gets a new uuid every time (which maybe that’s okay as long as we can clean
>> up old ones). Basically this all just boils down to we just need to make
>> sure we’re accounting for cleaning up all side effect data that gets
>> created by a new invoker uuid when it gets shut down.
>>
>> The ack / results kafka topics wouldn’t have to be adjusted to achieve
>> this because they’re coordinated by the controllers.
>>
>> On the topic of removing kafka from the critical path, I’m in total
>> agreement with Dom I think that would be a great architectural improvement
>> goal for 2023. Since openwhisk is already at most once at this point, I
>> don’t think kafka is adding any value from a persistence perspective and we
>> now have zero down time graceful restarts to drain any in progress work
>> before stopping. I presume most of the original value was from an ordering
>> perspective giving some level of fairness of processing between components.
>>
>>
>>
>>> On Tue, Dec 20, 2022 at 7:48 PM Dominic Kim <st...@gmail.com> wrote:
>>>
>>> We can also consider ETCD.
>>> ETCD supports a transaction API so we can make sure only one invoker can
>>> be
>>> assigned an IP at a time.
>>> (Though ETCD is currently only enabled when a scheduler is enabled.)
>>>
>>> Aside from this, I also wanted to get rid of Kafka from the critical path
>>> too as we can now use the scheduler(the new messaging queue).
>>> Since Kafka is being used to deliver ContainerCreation messages and Ack +
>>> Result messages, it would not be a simple task but we are already using
>>> Akka-grpc and Akka-remote(cluster).
>>> We can migrate to them.
>>>
>>> -dom
>>>
>>>
>>>
>>>
>>>
>>>
>>> [image: Mailtrack]
>>> <
>>> https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&
>>>>
>>> Sender
>>> notified by
>>> Mailtrack
>>> <
>>> https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&
>>>>
>>> 22.
>>> 12. 21. 오전 09:45:27
>>>
>>> 2022년 12월 21일 (수) 오전 6:25, Michele Sciabarra <mi...@sciabarra.com>님이
>>> 작성:
>>>
>>>> I have a question: what if we just use UUID?
>>>> That would be easy to implement.
>>>> Unless there are other reasons because the id should be incremental
>>>> numbers.
>>>> I would like to try to get rid of this dependency because I really want
>>> to
>>>> update Kafka maybe using RedPanda or Kafka Kraft...
>>>>
>>>> --
>>>> Michele Sciabarra
>>>> michele@sciabarra.com
>>>>
>>>> ----- Original message -----
>>>> From: Brendan Doyle <bd...@gmail.com>
>>>> To: dev@openwhisk.apache.org
>>>> Subject: Re: Updating Kafka getting rid of Zookeeper
>>>> Date: Sunday, December 18, 2022 4:36 PM
>>>>
>>>> Yes this is one of the things I had in my list on slack of what I would
>>>> like to see accomplished in 2023 since the only zookeeper use is storing
>>>> the invoker id mappings. I think it should be doable, but the one tricky
>>>> part is the synchronization that the zookeeper counter provides.
>>> Whatever
>>>> new mechanism that is used, it must be guaranteed that two new invokers
>>>> starting at the same time wouldn’t have a race condition where they read
>>>> the current counter value and assign themselves the same id. Or just the
>>>> way invoker ids are represented could be changed entirely as well to be
>>>> more accommodating to auto scaling up and down too; since part of the
>>>> reason for the original incrementing integer invoker ids was the hashing
>>>> algorithm to schedule activations to invokers which now no longer
>>> exists. I
>>>> think an invoker node also no longer has to have a persisted id that is
>>>> reused on restart because there’s no longer an invoker activation kafka
>>>> topic with the invoker id where there may be activations left to still
>>> be
>>>> processed after restart.
>>>>
>>>> But of all the things work wise I have on my list, this is probably the
>>>> easiest value proposition to remove an entire dependency from they
>>> system.
>>>>
>>>> Brendan
>>>>
>>>> On Sun, Dec 18, 2022 at 8:36 AM Michele Sciabarra <
>>> michele@sciabarra.com>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I was trying to run OpenWhisk with a newer version of Kafka and also
>>> with
>>>>> RedPanda. And I failed.
>>>>> I found there is a dependency on version 4 of curator that in turns
>>>> blocks
>>>>> zookeper to version 3.4 and newer version of Kafka uses 3.5 with a
>>>>> different protocol.
>>>>>
>>>>> In the code, the curator is only used here:
>>>>>
>>>>>
>>>>>
>>>>
>>> core/invoker/src/main/scala/org/apache/openwhisk/core/invoker/InstanceIdAssigner.scala
>>>>>
>>>>> what is exactly the purpose of this class?
>>>>>
>>>>> From what I can undestand, it is storing in zooker the id of the
>>> invoker
>>>>> for the purpose of guaranteeing
>>>>>
>>>>> Can we replace with something else that does not use zookeeper? We
>>> have
>>>>> now etcd for the scheduler if I am not wrong. Can we use etcd and get
>>> rid
>>>>> of the dependency on zookeepr 3.4 that holds back also the version of
>>>>> kafka, so we can experiment with Kafka Kraft and RedPanda (that does
>>> not
>>>>> have zookeeper at all)?
>>>>>
>>>>> --
>>>>> Michele Sciabarra
>>>>> michele@sciabarra.com
>>>>>
>>>>
>>>
>>
Re: Updating Kafka getting rid of Zookeeper
Posted by Brendan Doyle <bd...@gmail.com>.
Actually to Dom’s point, rather than UUID you can probably use the ip or
display name value to store as the key to guarantee uniqueness, but will be
reused on restart. And still no longer have to worry about tracking the
counter.
Brendan
On Tue, Dec 20, 2022 at 8:54 PM Brendan Doyle <bd...@gmail.com> wrote:
> I think it would be nice to make the invoker ids “memoryless” in that they
> just get a new id on restart. And this value can just be stored in etcd or
> any key value data store. The two major things that need to be dealt with
> on the new architecture to deal with that are 1. The healthcheck function
> is created using the invoker id as the name of the function so we need to
> make sure the function is deleted otherwise you’ll endlessly create the
> healthcheck functions and 2. the container creation kafka topics since
> they’re somewhat tied to the invoker and it’s assigned id. A new container
> creation topic would get created every time the invoker is restarted if it
> gets a new uuid every time (which maybe that’s okay as long as we can clean
> up old ones). Basically this all just boils down to we just need to make
> sure we’re accounting for cleaning up all side effect data that gets
> created by a new invoker uuid when it gets shut down.
>
> The ack / results kafka topics wouldn’t have to be adjusted to achieve
> this because they’re coordinated by the controllers.
>
> On the topic of removing kafka from the critical path, I’m in total
> agreement with Dom I think that would be a great architectural improvement
> goal for 2023. Since openwhisk is already at most once at this point, I
> don’t think kafka is adding any value from a persistence perspective and we
> now have zero down time graceful restarts to drain any in progress work
> before stopping. I presume most of the original value was from an ordering
> perspective giving some level of fairness of processing between components.
>
>
>
> On Tue, Dec 20, 2022 at 7:48 PM Dominic Kim <st...@gmail.com> wrote:
>
>> We can also consider ETCD.
>> ETCD supports a transaction API so we can make sure only one invoker can
>> be
>> assigned an IP at a time.
>> (Though ETCD is currently only enabled when a scheduler is enabled.)
>>
>> Aside from this, I also wanted to get rid of Kafka from the critical path
>> too as we can now use the scheduler(the new messaging queue).
>> Since Kafka is being used to deliver ContainerCreation messages and Ack +
>> Result messages, it would not be a simple task but we are already using
>> Akka-grpc and Akka-remote(cluster).
>> We can migrate to them.
>>
>> -dom
>>
>>
>>
>>
>>
>>
>> [image: Mailtrack]
>> <
>> https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&
>> >
>> Sender
>> notified by
>> Mailtrack
>> <
>> https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&
>> >
>> 22.
>> 12. 21. 오전 09:45:27
>>
>> 2022년 12월 21일 (수) 오전 6:25, Michele Sciabarra <mi...@sciabarra.com>님이
>> 작성:
>>
>> > I have a question: what if we just use UUID?
>> > That would be easy to implement.
>> > Unless there are other reasons because the id should be incremental
>> > numbers.
>> > I would like to try to get rid of this dependency because I really want
>> to
>> > update Kafka maybe using RedPanda or Kafka Kraft...
>> >
>> > --
>> > Michele Sciabarra
>> > michele@sciabarra.com
>> >
>> > ----- Original message -----
>> > From: Brendan Doyle <bd...@gmail.com>
>> > To: dev@openwhisk.apache.org
>> > Subject: Re: Updating Kafka getting rid of Zookeeper
>> > Date: Sunday, December 18, 2022 4:36 PM
>> >
>> > Yes this is one of the things I had in my list on slack of what I would
>> > like to see accomplished in 2023 since the only zookeeper use is storing
>> > the invoker id mappings. I think it should be doable, but the one tricky
>> > part is the synchronization that the zookeeper counter provides.
>> Whatever
>> > new mechanism that is used, it must be guaranteed that two new invokers
>> > starting at the same time wouldn’t have a race condition where they read
>> > the current counter value and assign themselves the same id. Or just the
>> > way invoker ids are represented could be changed entirely as well to be
>> > more accommodating to auto scaling up and down too; since part of the
>> > reason for the original incrementing integer invoker ids was the hashing
>> > algorithm to schedule activations to invokers which now no longer
>> exists. I
>> > think an invoker node also no longer has to have a persisted id that is
>> > reused on restart because there’s no longer an invoker activation kafka
>> > topic with the invoker id where there may be activations left to still
>> be
>> > processed after restart.
>> >
>> > But of all the things work wise I have on my list, this is probably the
>> > easiest value proposition to remove an entire dependency from they
>> system.
>> >
>> > Brendan
>> >
>> > On Sun, Dec 18, 2022 at 8:36 AM Michele Sciabarra <
>> michele@sciabarra.com>
>> > wrote:
>> >
>> > > Hello,
>> > >
>> > > I was trying to run OpenWhisk with a newer version of Kafka and also
>> with
>> > > RedPanda. And I failed.
>> > > I found there is a dependency on version 4 of curator that in turns
>> > blocks
>> > > zookeper to version 3.4 and newer version of Kafka uses 3.5 with a
>> > > different protocol.
>> > >
>> > > In the code, the curator is only used here:
>> > >
>> > >
>> > >
>> >
>> core/invoker/src/main/scala/org/apache/openwhisk/core/invoker/InstanceIdAssigner.scala
>> > >
>> > > what is exactly the purpose of this class?
>> > >
>> > > From what I can undestand, it is storing in zooker the id of the
>> invoker
>> > > for the purpose of guaranteeing
>> > >
>> > > Can we replace with something else that does not use zookeeper? We
>> have
>> > > now etcd for the scheduler if I am not wrong. Can we use etcd and get
>> rid
>> > > of the dependency on zookeepr 3.4 that holds back also the version of
>> > > kafka, so we can experiment with Kafka Kraft and RedPanda (that does
>> not
>> > > have zookeeper at all)?
>> > >
>> > > --
>> > > Michele Sciabarra
>> > > michele@sciabarra.com
>> > >
>> >
>>
>
Re: Updating Kafka getting rid of Zookeeper
Posted by Brendan Doyle <bd...@gmail.com>.
I think it would be nice to make the invoker ids “memoryless” in that they
just get a new id on restart. And this value can just be stored in etcd or
any key value data store. The two major things that need to be dealt with
on the new architecture to deal with that are 1. The healthcheck function
is created using the invoker id as the name of the function so we need to
make sure the function is deleted otherwise you’ll endlessly create the
healthcheck functions and 2. the container creation kafka topics since
they’re somewhat tied to the invoker and it’s assigned id. A new container
creation topic would get created every time the invoker is restarted if it
gets a new uuid every time (which maybe that’s okay as long as we can clean
up old ones). Basically this all just boils down to we just need to make
sure we’re accounting for cleaning up all side effect data that gets
created by a new invoker uuid when it gets shut down.
The ack / results kafka topics wouldn’t have to be adjusted to achieve this
because they’re coordinated by the controllers.
On the topic of removing kafka from the critical path, I’m in total
agreement with Dom I think that would be a great architectural improvement
goal for 2023. Since openwhisk is already at most once at this point, I
don’t think kafka is adding any value from a persistence perspective and we
now have zero down time graceful restarts to drain any in progress work
before stopping. I presume most of the original value was from an ordering
perspective giving some level of fairness of processing between components.
On Tue, Dec 20, 2022 at 7:48 PM Dominic Kim <st...@gmail.com> wrote:
> We can also consider ETCD.
> ETCD supports a transaction API so we can make sure only one invoker can be
> assigned an IP at a time.
> (Though ETCD is currently only enabled when a scheduler is enabled.)
>
> Aside from this, I also wanted to get rid of Kafka from the critical path
> too as we can now use the scheduler(the new messaging queue).
> Since Kafka is being used to deliver ContainerCreation messages and Ack +
> Result messages, it would not be a simple task but we are already using
> Akka-grpc and Akka-remote(cluster).
> We can migrate to them.
>
> -dom
>
>
>
>
>
>
> [image: Mailtrack]
> <
> https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&
> >
> Sender
> notified by
> Mailtrack
> <
> https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&
> >
> 22.
> 12. 21. 오전 09:45:27
>
> 2022년 12월 21일 (수) 오전 6:25, Michele Sciabarra <mi...@sciabarra.com>님이 작성:
>
> > I have a question: what if we just use UUID?
> > That would be easy to implement.
> > Unless there are other reasons because the id should be incremental
> > numbers.
> > I would like to try to get rid of this dependency because I really want
> to
> > update Kafka maybe using RedPanda or Kafka Kraft...
> >
> > --
> > Michele Sciabarra
> > michele@sciabarra.com
> >
> > ----- Original message -----
> > From: Brendan Doyle <bd...@gmail.com>
> > To: dev@openwhisk.apache.org
> > Subject: Re: Updating Kafka getting rid of Zookeeper
> > Date: Sunday, December 18, 2022 4:36 PM
> >
> > Yes this is one of the things I had in my list on slack of what I would
> > like to see accomplished in 2023 since the only zookeeper use is storing
> > the invoker id mappings. I think it should be doable, but the one tricky
> > part is the synchronization that the zookeeper counter provides. Whatever
> > new mechanism that is used, it must be guaranteed that two new invokers
> > starting at the same time wouldn’t have a race condition where they read
> > the current counter value and assign themselves the same id. Or just the
> > way invoker ids are represented could be changed entirely as well to be
> > more accommodating to auto scaling up and down too; since part of the
> > reason for the original incrementing integer invoker ids was the hashing
> > algorithm to schedule activations to invokers which now no longer
> exists. I
> > think an invoker node also no longer has to have a persisted id that is
> > reused on restart because there’s no longer an invoker activation kafka
> > topic with the invoker id where there may be activations left to still be
> > processed after restart.
> >
> > But of all the things work wise I have on my list, this is probably the
> > easiest value proposition to remove an entire dependency from they
> system.
> >
> > Brendan
> >
> > On Sun, Dec 18, 2022 at 8:36 AM Michele Sciabarra <michele@sciabarra.com
> >
> > wrote:
> >
> > > Hello,
> > >
> > > I was trying to run OpenWhisk with a newer version of Kafka and also
> with
> > > RedPanda. And I failed.
> > > I found there is a dependency on version 4 of curator that in turns
> > blocks
> > > zookeper to version 3.4 and newer version of Kafka uses 3.5 with a
> > > different protocol.
> > >
> > > In the code, the curator is only used here:
> > >
> > >
> > >
> >
> core/invoker/src/main/scala/org/apache/openwhisk/core/invoker/InstanceIdAssigner.scala
> > >
> > > what is exactly the purpose of this class?
> > >
> > > From what I can undestand, it is storing in zooker the id of the
> invoker
> > > for the purpose of guaranteeing
> > >
> > > Can we replace with something else that does not use zookeeper? We have
> > > now etcd for the scheduler if I am not wrong. Can we use etcd and get
> rid
> > > of the dependency on zookeepr 3.4 that holds back also the version of
> > > kafka, so we can experiment with Kafka Kraft and RedPanda (that does
> not
> > > have zookeeper at all)?
> > >
> > > --
> > > Michele Sciabarra
> > > michele@sciabarra.com
> > >
> >
>
Re: Updating Kafka getting rid of Zookeeper
Posted by Dominic Kim <st...@gmail.com>.
We can also consider ETCD.
ETCD supports a transaction API so we can make sure only one invoker can be
assigned an IP at a time.
(Though ETCD is currently only enabled when a scheduler is enabled.)
Aside from this, I also wanted to get rid of Kafka from the critical path
too as we can now use the scheduler(the new messaging queue).
Since Kafka is being used to deliver ContainerCreation messages and Ack +
Result messages, it would not be a simple task but we are already using
Akka-grpc and Akka-remote(cluster).
We can migrate to them.
-dom
[image: Mailtrack]
<https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&>
Sender
notified by
Mailtrack
<https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&>
22.
12. 21. 오전 09:45:27
2022년 12월 21일 (수) 오전 6:25, Michele Sciabarra <mi...@sciabarra.com>님이 작성:
> I have a question: what if we just use UUID?
> That would be easy to implement.
> Unless there are other reasons because the id should be incremental
> numbers.
> I would like to try to get rid of this dependency because I really want to
> update Kafka maybe using RedPanda or Kafka Kraft...
>
> --
> Michele Sciabarra
> michele@sciabarra.com
>
> ----- Original message -----
> From: Brendan Doyle <bd...@gmail.com>
> To: dev@openwhisk.apache.org
> Subject: Re: Updating Kafka getting rid of Zookeeper
> Date: Sunday, December 18, 2022 4:36 PM
>
> Yes this is one of the things I had in my list on slack of what I would
> like to see accomplished in 2023 since the only zookeeper use is storing
> the invoker id mappings. I think it should be doable, but the one tricky
> part is the synchronization that the zookeeper counter provides. Whatever
> new mechanism that is used, it must be guaranteed that two new invokers
> starting at the same time wouldn’t have a race condition where they read
> the current counter value and assign themselves the same id. Or just the
> way invoker ids are represented could be changed entirely as well to be
> more accommodating to auto scaling up and down too; since part of the
> reason for the original incrementing integer invoker ids was the hashing
> algorithm to schedule activations to invokers which now no longer exists. I
> think an invoker node also no longer has to have a persisted id that is
> reused on restart because there’s no longer an invoker activation kafka
> topic with the invoker id where there may be activations left to still be
> processed after restart.
>
> But of all the things work wise I have on my list, this is probably the
> easiest value proposition to remove an entire dependency from they system.
>
> Brendan
>
> On Sun, Dec 18, 2022 at 8:36 AM Michele Sciabarra <mi...@sciabarra.com>
> wrote:
>
> > Hello,
> >
> > I was trying to run OpenWhisk with a newer version of Kafka and also with
> > RedPanda. And I failed.
> > I found there is a dependency on version 4 of curator that in turns
> blocks
> > zookeper to version 3.4 and newer version of Kafka uses 3.5 with a
> > different protocol.
> >
> > In the code, the curator is only used here:
> >
> >
> >
> core/invoker/src/main/scala/org/apache/openwhisk/core/invoker/InstanceIdAssigner.scala
> >
> > what is exactly the purpose of this class?
> >
> > From what I can undestand, it is storing in zooker the id of the invoker
> > for the purpose of guaranteeing
> >
> > Can we replace with something else that does not use zookeeper? We have
> > now etcd for the scheduler if I am not wrong. Can we use etcd and get rid
> > of the dependency on zookeepr 3.4 that holds back also the version of
> > kafka, so we can experiment with Kafka Kraft and RedPanda (that does not
> > have zookeeper at all)?
> >
> > --
> > Michele Sciabarra
> > michele@sciabarra.com
> >
>
Re: Updating Kafka getting rid of Zookeeper
Posted by Michele Sciabarra <mi...@sciabarra.com>.
I have a question: what if we just use UUID?
That would be easy to implement.
Unless there are other reasons because the id should be incremental numbers.
I would like to try to get rid of this dependency because I really want to update Kafka maybe using RedPanda or Kafka Kraft...
--
Michele Sciabarra
michele@sciabarra.com
----- Original message -----
From: Brendan Doyle <bd...@gmail.com>
To: dev@openwhisk.apache.org
Subject: Re: Updating Kafka getting rid of Zookeeper
Date: Sunday, December 18, 2022 4:36 PM
Yes this is one of the things I had in my list on slack of what I would
like to see accomplished in 2023 since the only zookeeper use is storing
the invoker id mappings. I think it should be doable, but the one tricky
part is the synchronization that the zookeeper counter provides. Whatever
new mechanism that is used, it must be guaranteed that two new invokers
starting at the same time wouldn’t have a race condition where they read
the current counter value and assign themselves the same id. Or just the
way invoker ids are represented could be changed entirely as well to be
more accommodating to auto scaling up and down too; since part of the
reason for the original incrementing integer invoker ids was the hashing
algorithm to schedule activations to invokers which now no longer exists. I
think an invoker node also no longer has to have a persisted id that is
reused on restart because there’s no longer an invoker activation kafka
topic with the invoker id where there may be activations left to still be
processed after restart.
But of all the things work wise I have on my list, this is probably the
easiest value proposition to remove an entire dependency from they system.
Brendan
On Sun, Dec 18, 2022 at 8:36 AM Michele Sciabarra <mi...@sciabarra.com>
wrote:
> Hello,
>
> I was trying to run OpenWhisk with a newer version of Kafka and also with
> RedPanda. And I failed.
> I found there is a dependency on version 4 of curator that in turns blocks
> zookeper to version 3.4 and newer version of Kafka uses 3.5 with a
> different protocol.
>
> In the code, the curator is only used here:
>
>
> core/invoker/src/main/scala/org/apache/openwhisk/core/invoker/InstanceIdAssigner.scala
>
> what is exactly the purpose of this class?
>
> From what I can undestand, it is storing in zooker the id of the invoker
> for the purpose of guaranteeing
>
> Can we replace with something else that does not use zookeeper? We have
> now etcd for the scheduler if I am not wrong. Can we use etcd and get rid
> of the dependency on zookeepr 3.4 that holds back also the version of
> kafka, so we can experiment with Kafka Kraft and RedPanda (that does not
> have zookeeper at all)?
>
> --
> Michele Sciabarra
> michele@sciabarra.com
>
Re: Updating Kafka getting rid of Zookeeper
Posted by Brendan Doyle <bd...@gmail.com>.
Yes this is one of the things I had in my list on slack of what I would
like to see accomplished in 2023 since the only zookeeper use is storing
the invoker id mappings. I think it should be doable, but the one tricky
part is the synchronization that the zookeeper counter provides. Whatever
new mechanism that is used, it must be guaranteed that two new invokers
starting at the same time wouldn’t have a race condition where they read
the current counter value and assign themselves the same id. Or just the
way invoker ids are represented could be changed entirely as well to be
more accommodating to auto scaling up and down too; since part of the
reason for the original incrementing integer invoker ids was the hashing
algorithm to schedule activations to invokers which now no longer exists. I
think an invoker node also no longer has to have a persisted id that is
reused on restart because there’s no longer an invoker activation kafka
topic with the invoker id where there may be activations left to still be
processed after restart.
But of all the things work wise I have on my list, this is probably the
easiest value proposition to remove an entire dependency from they system.
Brendan
On Sun, Dec 18, 2022 at 8:36 AM Michele Sciabarra <mi...@sciabarra.com>
wrote:
> Hello,
>
> I was trying to run OpenWhisk with a newer version of Kafka and also with
> RedPanda. And I failed.
> I found there is a dependency on version 4 of curator that in turns blocks
> zookeper to version 3.4 and newer version of Kafka uses 3.5 with a
> different protocol.
>
> In the code, the curator is only used here:
>
>
> core/invoker/src/main/scala/org/apache/openwhisk/core/invoker/InstanceIdAssigner.scala
>
> what is exactly the purpose of this class?
>
> From what I can undestand, it is storing in zooker the id of the invoker
> for the purpose of guaranteeing
>
> Can we replace with something else that does not use zookeeper? We have
> now etcd for the scheduler if I am not wrong. Can we use etcd and get rid
> of the dependency on zookeepr 3.4 that holds back also the version of
> kafka, so we can experiment with Kafka Kraft and RedPanda (that does not
> have zookeeper at all)?
>
> --
> Michele Sciabarra
> michele@sciabarra.com
>