You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apisix.apache.org by Chao Zhang <to...@apache.org> on 2021/04/27 01:45:54 UTC
Multi Data Centers Support for APISIX
Hi, there
Below is the proposal about how to support multiple data centers for
Apache APISIX. Please review it and your advice will be appreciated.
Background
So far, APISIX only supports fetching configurations like Route,
Upstream from ETCD, although it’s capable of integrating a couple of
service registries (e.g. Consul, Nacos) to discover backend service
endpoints (aka upstream nodes) dynamically.
For some users, ETCD is not their optimal solution as they may already
have another similar data center or they don’t master the skill for
ETCD cluster maintenance. In such a case, introducing APISIX to their
platform will be stumbling, even letting them give up on using it.
So from this perspective, making APISIX decouple from ETCD is
important, it also enriches the ecosystem of APISIX.
Self Adjustment
The most intuitive way is evolving APISIX itself, letting it as the
client of the target data center, so no extra components will be
deployed, what administrators need to do is configuring the data
center in the configuration of APISIX, including address,
authentication and so on.
That’s good but the overheads to change APISIX is huge, firstly, the
protocol used in each data center might vary, considering the weak
ecosystem of OpenResty/Nginx, we may need to implement the data center
SDK manually, and it will be tracky for some multiplexing protocols
(due to the connection pool model of OpenResty); On the other hand,
The APISIX internal is tightly coupled with ETCD, it’s not easy to
decouple them from the code level, what’s more, these part of codes
now are very stable, changing them might influence the stability of
APISIX.
Agent Mode
> All Problems in Computer Science can be solved by another level of indirection.
Since it’s not easy to support this from APISIX itself, why not
introducing another component, it supports to communicate with any
data centers, and it mimics the ETCD V3 APIs and gRPC gateway, so that
it can pretend itself as the ETCD cluster from APISIX’s point of view.
With that a component, there is nothing that should be done in APISIX
itself (You can see the architecture figure in the attachments).
Agents can be deployed in any place as the networking is connected for
both the APISIX and data centers sides. It should be designed as
stateless, so APISIX can configure multiple agent instances and the
high availability can be sure. It can be even deployed for each APISIX
instance, that’s the famous sidecar mode.
Problems need to be addressed
The agent component has to solve a series of problems so it can
achieve its mission. Firstly, it should implement the server side of
ETCD V3 APIs. Fortunately, only minor parts of features are used by
APISIX, so the implementation won’t be too complicated, also, since
APISIX relis on the ETCD gRPC gateway, the agent should also support
it, in general, that’s not too difficult as the gRPC ecosystem is
strong.
Secondly, the internal design of the agent should be modular as soon
as possible, so once we need to support a new data center, the
development would be easy. Of course that’s another topic and will not
be detailed in this post.
Best regards
Chao Zhang
https://github.com/tokers
Re: Multi Data Centers Support for APISIX
Posted by Chao Zhang <zc...@gmail.com>.
Hi, Yang
Thanks for your rectification, that's really config center will be
more appropriate.
Best regards
Chao Zhang
https://github.com/tokers
On Tue, Apr 27, 2021 at 1:42 PM Li Yang <ya...@apache.org> wrote:
>
> The tile is misleading, it might be multi config center instead of
> multi data center.
>
> I'm trying to clarify a little first, so after this feature is developed,
> APISIX will be able to read configs not only from ETCD, but
> from any config server as long as we have created adaptor
> between config center protocol and APISIX protocol.
>
> It sounds very attractive since routes and upstreams can now be
> managed as configurations like any other configurations.
>
> On Tue, Apr 27, 2021 at 9:46 AM Chao Zhang <to...@apache.org> wrote:
> >
> > Hi, there
> >
> > Below is the proposal about how to support multiple data centers for
> > Apache APISIX. Please review it and your advice will be appreciated.
> >
> > Background
> >
> > So far, APISIX only supports fetching configurations like Route,
> > Upstream from ETCD, although it’s capable of integrating a couple of
> > service registries (e.g. Consul, Nacos) to discover backend service
> > endpoints (aka upstream nodes) dynamically.
> >
> > For some users, ETCD is not their optimal solution as they may already
> > have another similar data center or they don’t master the skill for
> > ETCD cluster maintenance. In such a case, introducing APISIX to their
> > platform will be stumbling, even letting them give up on using it.
> >
> > So from this perspective, making APISIX decouple from ETCD is
> > important, it also enriches the ecosystem of APISIX.
> >
> > Self Adjustment
> >
> > The most intuitive way is evolving APISIX itself, letting it as the
> > client of the target data center, so no extra components will be
> > deployed, what administrators need to do is configuring the data
> > center in the configuration of APISIX, including address,
> > authentication and so on.
> >
> > That’s good but the overheads to change APISIX is huge, firstly, the
> > protocol used in each data center might vary, considering the weak
> > ecosystem of OpenResty/Nginx, we may need to implement the data center
> > SDK manually, and it will be tracky for some multiplexing protocols
> > (due to the connection pool model of OpenResty); On the other hand,
> > The APISIX internal is tightly coupled with ETCD, it’s not easy to
> > decouple them from the code level, what’s more, these part of codes
> > now are very stable, changing them might influence the stability of
> > APISIX.
> >
> > Agent Mode
> >
> > > All Problems in Computer Science can be solved by another level of indirection.
> >
> > Since it’s not easy to support this from APISIX itself, why not
> > introducing another component, it supports to communicate with any
> > data centers, and it mimics the ETCD V3 APIs and gRPC gateway, so that
> > it can pretend itself as the ETCD cluster from APISIX’s point of view.
> > With that a component, there is nothing that should be done in APISIX
> > itself (You can see the architecture figure in the attachments).
> >
> > Agents can be deployed in any place as the networking is connected for
> > both the APISIX and data centers sides. It should be designed as
> > stateless, so APISIX can configure multiple agent instances and the
> > high availability can be sure. It can be even deployed for each APISIX
> > instance, that’s the famous sidecar mode.
> >
> > Problems need to be addressed
> >
> > The agent component has to solve a series of problems so it can
> > achieve its mission. Firstly, it should implement the server side of
> > ETCD V3 APIs. Fortunately, only minor parts of features are used by
> > APISIX, so the implementation won’t be too complicated, also, since
> > APISIX relis on the ETCD gRPC gateway, the agent should also support
> > it, in general, that’s not too difficult as the gRPC ecosystem is
> > strong.
> >
> > Secondly, the internal design of the agent should be modular as soon
> > as possible, so once we need to support a new data center, the
> > development would be easy. Of course that’s another topic and will not
> > be detailed in this post.
> >
> > Best regards
> > Chao Zhang
> >
> > https://github.com/tokers
Re: Multi Data Centers Support for APISIX
Posted by Li Yang <ya...@apache.org>.
The tile is misleading, it might be multi config center instead of
multi data center.
I'm trying to clarify a little first, so after this feature is developed,
APISIX will be able to read configs not only from ETCD, but
from any config server as long as we have created adaptor
between config center protocol and APISIX protocol.
It sounds very attractive since routes and upstreams can now be
managed as configurations like any other configurations.
On Tue, Apr 27, 2021 at 9:46 AM Chao Zhang <to...@apache.org> wrote:
>
> Hi, there
>
> Below is the proposal about how to support multiple data centers for
> Apache APISIX. Please review it and your advice will be appreciated.
>
> Background
>
> So far, APISIX only supports fetching configurations like Route,
> Upstream from ETCD, although it’s capable of integrating a couple of
> service registries (e.g. Consul, Nacos) to discover backend service
> endpoints (aka upstream nodes) dynamically.
>
> For some users, ETCD is not their optimal solution as they may already
> have another similar data center or they don’t master the skill for
> ETCD cluster maintenance. In such a case, introducing APISIX to their
> platform will be stumbling, even letting them give up on using it.
>
> So from this perspective, making APISIX decouple from ETCD is
> important, it also enriches the ecosystem of APISIX.
>
> Self Adjustment
>
> The most intuitive way is evolving APISIX itself, letting it as the
> client of the target data center, so no extra components will be
> deployed, what administrators need to do is configuring the data
> center in the configuration of APISIX, including address,
> authentication and so on.
>
> That’s good but the overheads to change APISIX is huge, firstly, the
> protocol used in each data center might vary, considering the weak
> ecosystem of OpenResty/Nginx, we may need to implement the data center
> SDK manually, and it will be tracky for some multiplexing protocols
> (due to the connection pool model of OpenResty); On the other hand,
> The APISIX internal is tightly coupled with ETCD, it’s not easy to
> decouple them from the code level, what’s more, these part of codes
> now are very stable, changing them might influence the stability of
> APISIX.
>
> Agent Mode
>
> > All Problems in Computer Science can be solved by another level of indirection.
>
> Since it’s not easy to support this from APISIX itself, why not
> introducing another component, it supports to communicate with any
> data centers, and it mimics the ETCD V3 APIs and gRPC gateway, so that
> it can pretend itself as the ETCD cluster from APISIX’s point of view.
> With that a component, there is nothing that should be done in APISIX
> itself (You can see the architecture figure in the attachments).
>
> Agents can be deployed in any place as the networking is connected for
> both the APISIX and data centers sides. It should be designed as
> stateless, so APISIX can configure multiple agent instances and the
> high availability can be sure. It can be even deployed for each APISIX
> instance, that’s the famous sidecar mode.
>
> Problems need to be addressed
>
> The agent component has to solve a series of problems so it can
> achieve its mission. Firstly, it should implement the server side of
> ETCD V3 APIs. Fortunately, only minor parts of features are used by
> APISIX, so the implementation won’t be too complicated, also, since
> APISIX relis on the ETCD gRPC gateway, the agent should also support
> it, in general, that’s not too difficult as the gRPC ecosystem is
> strong.
>
> Secondly, the internal design of the agent should be modular as soon
> as possible, so once we need to support a new data center, the
> development would be easy. Of course that’s another topic and will not
> be detailed in this post.
>
> Best regards
> Chao Zhang
>
> https://github.com/tokers
Re: Multi Data Centers Support for APISIX
Posted by Ming Wen <we...@apache.org>.
> But I was also thinking about a standalone mode of APISIX, all
> configurations are saved in the local file system.
Nice catch.
But users still need to make an agent by themselves to do the automatic
synchronization
from configuration centers to the YAML configuration file of APISIX.
Thanks,
Ming Wen, Apache APISIX PMC Chair
Twitter: _WenMing
Li Ling(Lien) <li...@apache.org> 于2021年4月27日周二 上午10:45写道：
> It seems like the proposal is very attractive, with an ETCD protocol
> compatible middleware, which brings more flexibility to APISIX eco-system.
>
> But I was also thinking about a standalone mode of APISIX, all
> configurations are saved in the local file system.
>
> which is easier for light-weight(less than 50 configrations) users to
> engage with APISIX.
> also easier for users to play around.
>
>
>
> Sheng Wu <wu...@gmail.com> 于2021年4月27日周二 上午9:58写道：
>
> > Basically, I like this idea, just suggest reconsidering whether keep on
> > etcd API, especially gRPC tech stack is already there.
> > APISIX works with its agent but using etcd protocol? This seems strange.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Chao Zhang <to...@apache.org> 于2021年4月27日周二 上午9:46写道：
> >
> > > Hi, there
> > >
> > > Below is the proposal about how to support multiple data centers for
> > > Apache APISIX. Please review it and your advice will be appreciated.
> > >
> > > Background
> > >
> > > So far, APISIX only supports fetching configurations like Route,
> > > Upstream from ETCD, although it’s capable of integrating a couple of
> > > service registries (e.g. Consul, Nacos) to discover backend service
> > > endpoints (aka upstream nodes) dynamically.
> > >
> > > For some users, ETCD is not their optimal solution as they may already
> > > have another similar data center or they don’t master the skill for
> > > ETCD cluster maintenance. In such a case, introducing APISIX to their
> > > platform will be stumbling, even letting them give up on using it.
> > >
> > > So from this perspective, making APISIX decouple from ETCD is
> > > important, it also enriches the ecosystem of APISIX.
> > >
> > > Self Adjustment
> > >
> > > The most intuitive way is evolving APISIX itself, letting it as the
> > > client of the target data center, so no extra components will be
> > > deployed, what administrators need to do is configuring the data
> > > center in the configuration of APISIX, including address,
> > > authentication and so on.
> > >
> > > That’s good but the overheads to change APISIX is huge, firstly, the
> > > protocol used in each data center might vary, considering the weak
> > > ecosystem of OpenResty/Nginx, we may need to implement the data center
> > > SDK manually, and it will be tracky for some multiplexing protocols
> > > (due to the connection pool model of OpenResty); On the other hand,
> > > The APISIX internal is tightly coupled with ETCD, it’s not easy to
> > > decouple them from the code level, what’s more, these part of codes
> > > now are very stable, changing them might influence the stability of
> > > APISIX.
> > >
> > > Agent Mode
> > >
> > > > All Problems in Computer Science can be solved by another level of
> > > indirection.
> > >
> > > Since it’s not easy to support this from APISIX itself, why not
> > > introducing another component, it supports to communicate with any
> > > data centers, and it mimics the ETCD V3 APIs and gRPC gateway, so that
> > > it can pretend itself as the ETCD cluster from APISIX’s point of view.
> > > With that a component, there is nothing that should be done in APISIX
> > > itself (You can see the architecture figure in the attachments).
> > >
> > > Agents can be deployed in any place as the networking is connected for
> > > both the APISIX and data centers sides. It should be designed as
> > > stateless, so APISIX can configure multiple agent instances and the
> > > high availability can be sure. It can be even deployed for each APISIX
> > > instance, that’s the famous sidecar mode.
> > >
> > > Problems need to be addressed
> > >
> > > The agent component has to solve a series of problems so it can
> > > achieve its mission. Firstly, it should implement the server side of
> > > ETCD V3 APIs. Fortunately, only minor parts of features are used by
> > > APISIX, so the implementation won’t be too complicated, also, since
> > > APISIX relis on the ETCD gRPC gateway, the agent should also support
> > > it, in general, that’s not too difficult as the gRPC ecosystem is
> > > strong.
> > >
> > > Secondly, the internal design of the agent should be modular as soon
> > > as possible, so once we need to support a new data center, the
> > > development would be easy. Of course that’s another topic and will not
> > > be detailed in this post.
> > >
> > > Best regards
> > > Chao Zhang
> > >
> > > https://github.com/tokers
> > >
> >
>
Re: Multi Data Centers Support for APISIX
Posted by "Li Ling(Lien)" <li...@apache.org>.
It seems like the proposal is very attractive, with an ETCD protocol
compatible middleware, which brings more flexibility to APISIX eco-system.
But I was also thinking about a standalone mode of APISIX, all
configurations are saved in the local file system.
which is easier for light-weight(less than 50 configrations) users to
engage with APISIX.
also easier for users to play around.
Sheng Wu <wu...@gmail.com> 于2021年4月27日周二 上午9:58写道：
> Basically, I like this idea, just suggest reconsidering whether keep on
> etcd API, especially gRPC tech stack is already there.
> APISIX works with its agent but using etcd protocol? This seems strange.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Chao Zhang <to...@apache.org> 于2021年4月27日周二 上午9:46写道：
>
> > Hi, there
> >
> > Below is the proposal about how to support multiple data centers for
> > Apache APISIX. Please review it and your advice will be appreciated.
> >
> > Background
> >
> > So far, APISIX only supports fetching configurations like Route,
> > Upstream from ETCD, although it’s capable of integrating a couple of
> > service registries (e.g. Consul, Nacos) to discover backend service
> > endpoints (aka upstream nodes) dynamically.
> >
> > For some users, ETCD is not their optimal solution as they may already
> > have another similar data center or they don’t master the skill for
> > ETCD cluster maintenance. In such a case, introducing APISIX to their
> > platform will be stumbling, even letting them give up on using it.
> >
> > So from this perspective, making APISIX decouple from ETCD is
> > important, it also enriches the ecosystem of APISIX.
> >
> > Self Adjustment
> >
> > The most intuitive way is evolving APISIX itself, letting it as the
> > client of the target data center, so no extra components will be
> > deployed, what administrators need to do is configuring the data
> > center in the configuration of APISIX, including address,
> > authentication and so on.
> >
> > That’s good but the overheads to change APISIX is huge, firstly, the
> > protocol used in each data center might vary, considering the weak
> > ecosystem of OpenResty/Nginx, we may need to implement the data center
> > SDK manually, and it will be tracky for some multiplexing protocols
> > (due to the connection pool model of OpenResty); On the other hand,
> > The APISIX internal is tightly coupled with ETCD, it’s not easy to
> > decouple them from the code level, what’s more, these part of codes
> > now are very stable, changing them might influence the stability of
> > APISIX.
> >
> > Agent Mode
> >
> > > All Problems in Computer Science can be solved by another level of
> > indirection.
> >
> > Since it’s not easy to support this from APISIX itself, why not
> > introducing another component, it supports to communicate with any
> > data centers, and it mimics the ETCD V3 APIs and gRPC gateway, so that
> > it can pretend itself as the ETCD cluster from APISIX’s point of view.
> > With that a component, there is nothing that should be done in APISIX
> > itself (You can see the architecture figure in the attachments).
> >
> > Agents can be deployed in any place as the networking is connected for
> > both the APISIX and data centers sides. It should be designed as
> > stateless, so APISIX can configure multiple agent instances and the
> > high availability can be sure. It can be even deployed for each APISIX
> > instance, that’s the famous sidecar mode.
> >
> > Problems need to be addressed
> >
> > The agent component has to solve a series of problems so it can
> > achieve its mission. Firstly, it should implement the server side of
> > ETCD V3 APIs. Fortunately, only minor parts of features are used by
> > APISIX, so the implementation won’t be too complicated, also, since
> > APISIX relis on the ETCD gRPC gateway, the agent should also support
> > it, in general, that’s not too difficult as the gRPC ecosystem is
> > strong.
> >
> > Secondly, the internal design of the agent should be modular as soon
> > as possible, so once we need to support a new data center, the
> > development would be easy. Of course that’s another topic and will not
> > be detailed in this post.
> >
> > Best regards
> > Chao Zhang
> >
> > https://github.com/tokers
> >
>
Re: Multi Data Centers Support for APISIX
Posted by Chao Zhang <zc...@gmail.com>.
Hi,
Thanks Ming and Sheng for your suggestion, and that's really terrible
if we don't have our own protocol but always try to incorporate
others. I will think about this.
As for the short term, we still can use ETCD v3 API and implement that
agent, after we really have a concrete proposal about the
"configuration discovery protocol",
we'll adopt it on this agent and try to break away from ETCD v3 API.
Best regards
Chao Zhang
https://github.com/tokers
On Wed, Apr 28, 2021 at 4:08 PM Ming Wen <we...@apache.org> wrote:
>
> I agreed with Sheng.
> In the long term, Apache APISIX should have an independent protocol,
> so that other data storage components can adapt to Apache APISIX,
> not only Apache APISIX to adapt others.
>
> Thanks,
> Ming Wen, Apache APISIX PMC Chair
> Twitter: _WenMing
>
>
> Sheng Wu <wu...@gmail.com> 于2021年4月27日周二 上午10:17写道：
>
> > I know this. So, I just said suggest reconsidering
> > This is a typical style of a top-level open source project doing.
> > Protocol is the key to a community, once you(APISIX) lead on both sides, it
> > is better the protocol is something we could make decisions when changes
> > are required in the future.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Chao Zhang <zc...@gmail.com> 于2021年4月27日周二 上午10:12写道：
> >
> > > Hi,
> > >
> > > We want to pretend the agent is an ETCD cluster so that we don't need
> > > any modifications on APISIX side, and actually etcd v3 API is based on
> > > gRPC.
> > >
> > > Best regards
> > > Chao Zhang
> > >
> > > https://github.com/tokers
> > >
> > > On Tue, Apr 27, 2021 at 9:58 AM Sheng Wu <wu...@gmail.com>
> > > wrote:
> > > >
> > > > Basically, I like this idea, just suggest reconsidering whether keep on
> > > > etcd API, especially gRPC tech stack is already there.
> > > > APISIX works with its agent but using etcd protocol? This seems
> > strange.
> > > >
> > > > Sheng Wu 吴晟
> > > > Twitter, wusheng1108
> > > >
> > > >
> > > > Chao Zhang <to...@apache.org> 于2021年4月27日周二 上午9:46写道：
> > > >
> > > > > Hi, there
> > > > >
> > > > > Below is the proposal about how to support multiple data centers for
> > > > > Apache APISIX. Please review it and your advice will be appreciated.
> > > > >
> > > > > Background
> > > > >
> > > > > So far, APISIX only supports fetching configurations like Route,
> > > > > Upstream from ETCD, although it’s capable of integrating a couple of
> > > > > service registries (e.g. Consul, Nacos) to discover backend service
> > > > > endpoints (aka upstream nodes) dynamically.
> > > > >
> > > > > For some users, ETCD is not their optimal solution as they may
> > already
> > > > > have another similar data center or they don’t master the skill for
> > > > > ETCD cluster maintenance. In such a case, introducing APISIX to their
> > > > > platform will be stumbling, even letting them give up on using it.
> > > > >
> > > > > So from this perspective, making APISIX decouple from ETCD is
> > > > > important, it also enriches the ecosystem of APISIX.
> > > > >
> > > > > Self Adjustment
> > > > >
> > > > > The most intuitive way is evolving APISIX itself, letting it as the
> > > > > client of the target data center, so no extra components will be
> > > > > deployed, what administrators need to do is configuring the data
> > > > > center in the configuration of APISIX, including address,
> > > > > authentication and so on.
> > > > >
> > > > > That’s good but the overheads to change APISIX is huge, firstly, the
> > > > > protocol used in each data center might vary, considering the weak
> > > > > ecosystem of OpenResty/Nginx, we may need to implement the data
> > center
> > > > > SDK manually, and it will be tracky for some multiplexing protocols
> > > > > (due to the connection pool model of OpenResty); On the other hand,
> > > > > The APISIX internal is tightly coupled with ETCD, it’s not easy to
> > > > > decouple them from the code level, what’s more, these part of codes
> > > > > now are very stable, changing them might influence the stability of
> > > > > APISIX.
> > > > >
> > > > > Agent Mode
> > > > >
> > > > > > All Problems in Computer Science can be solved by another level of
> > > > > indirection.
> > > > >
> > > > > Since it’s not easy to support this from APISIX itself, why not
> > > > > introducing another component, it supports to communicate with any
> > > > > data centers, and it mimics the ETCD V3 APIs and gRPC gateway, so
> > that
> > > > > it can pretend itself as the ETCD cluster from APISIX’s point of
> > view.
> > > > > With that a component, there is nothing that should be done in APISIX
> > > > > itself (You can see the architecture figure in the attachments).
> > > > >
> > > > > Agents can be deployed in any place as the networking is connected
> > for
> > > > > both the APISIX and data centers sides. It should be designed as
> > > > > stateless, so APISIX can configure multiple agent instances and the
> > > > > high availability can be sure. It can be even deployed for each
> > APISIX
> > > > > instance, that’s the famous sidecar mode.
> > > > >
> > > > > Problems need to be addressed
> > > > >
> > > > > The agent component has to solve a series of problems so it can
> > > > > achieve its mission. Firstly, it should implement the server side of
> > > > > ETCD V3 APIs. Fortunately, only minor parts of features are used by
> > > > > APISIX, so the implementation won’t be too complicated, also, since
> > > > > APISIX relis on the ETCD gRPC gateway, the agent should also support
> > > > > it, in general, that’s not too difficult as the gRPC ecosystem is
> > > > > strong.
> > > > >
> > > > > Secondly, the internal design of the agent should be modular as soon
> > > > > as possible, so once we need to support a new data center, the
> > > > > development would be easy. Of course that’s another topic and will
> > not
> > > > > be detailed in this post.
> > > > >
> > > > > Best regards
> > > > > Chao Zhang
> > > > >
> > > > > https://github.com/tokers
> > > > >
> > >
> >
Re: Multi Data Centers Support for APISIX
Posted by Chao Zhang <zc...@gmail.com>.
Hi, Fei
That would be great if APISIX itself can be "polymorphic" for the config center.
But What I want to say is we may first need to listen to the voice of
the community
about the most likely used config centers (like Consul, Nacos).
Best regards
Chao Zhang
https://github.com/tokers
On Wed, Apr 28, 2021 at 4:17 PM Fei Liang <bs...@gmail.com> wrote:
>
> I suggest APISIX abstract config-center interface. The APISIX
> implementation is based on this config-center.
>
>
> The config-center supports a variety of implementation methods. Which can
> be implemented by etcd, redis, or even MySQL(consider supporting DB that do
> not support push).
>
>
> In addition, because the implementation of APISIX itself is LUA, the basic
> components are relatively small. Combined with Chao Zhang’s proposal,
> APISIX can also consider design own’s protocal for the agent service. The
> agent connect the DB, which lua is difficultal connecting.
>
>
> Ming Wen <we...@apache.org> 于2021年4月28日周三 下午4:08写道：
>
> > I agreed with Sheng.
> > In the long term, Apache APISIX should have an independent protocol,
> > so that other data storage components can adapt to Apache APISIX,
> > not only Apache APISIX to adapt others.
> >
> > Thanks,
> > Ming Wen, Apache APISIX PMC Chair
> > Twitter: _WenMing
> >
> >
> > Sheng Wu <wu...@gmail.com> 于2021年4月27日周二 上午10:17写道：
> >
> > > I know this. So, I just said suggest reconsidering
> > > This is a typical style of a top-level open source project doing.
> > > Protocol is the key to a community, once you(APISIX) lead on both sides,
> > it
> > > is better the protocol is something we could make decisions when changes
> > > are required in the future.
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > Chao Zhang <zc...@gmail.com> 于2021年4月27日周二 上午10:12写道：
> > >
> > > > Hi,
> > > >
> > > > We want to pretend the agent is an ETCD cluster so that we don't need
> > > > any modifications on APISIX side, and actually etcd v3 API is based on
> > > > gRPC.
> > > >
> > > > Best regards
> > > > Chao Zhang
> > > >
> > > > https://github.com/tokers
> > > >
> > > > On Tue, Apr 27, 2021 at 9:58 AM Sheng Wu <wu...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Basically, I like this idea, just suggest reconsidering whether keep
> > on
> > > > > etcd API, especially gRPC tech stack is already there.
> > > > > APISIX works with its agent but using etcd protocol? This seems
> > > strange.
> > > > >
> > > > > Sheng Wu 吴晟
> > > > > Twitter, wusheng1108
> > > > >
> > > > >
> > > > > Chao Zhang <to...@apache.org> 于2021年4月27日周二 上午9:46写道：
> > > > >
> > > > > > Hi, there
> > > > > >
> > > > > > Below is the proposal about how to support multiple data centers
> > for
> > > > > > Apache APISIX. Please review it and your advice will be
> > appreciated.
> > > > > >
> > > > > > Background
> > > > > >
> > > > > > So far, APISIX only supports fetching configurations like Route,
> > > > > > Upstream from ETCD, although it’s capable of integrating a couple
> > of
> > > > > > service registries (e.g. Consul, Nacos) to discover backend service
> > > > > > endpoints (aka upstream nodes) dynamically.
> > > > > >
> > > > > > For some users, ETCD is not their optimal solution as they may
> > > already
> > > > > > have another similar data center or they don’t master the skill for
> > > > > > ETCD cluster maintenance. In such a case, introducing APISIX to
> > their
> > > > > > platform will be stumbling, even letting them give up on using it.
> > > > > >
> > > > > > So from this perspective, making APISIX decouple from ETCD is
> > > > > > important, it also enriches the ecosystem of APISIX.
> > > > > >
> > > > > > Self Adjustment
> > > > > >
> > > > > > The most intuitive way is evolving APISIX itself, letting it as the
> > > > > > client of the target data center, so no extra components will be
> > > > > > deployed, what administrators need to do is configuring the data
> > > > > > center in the configuration of APISIX, including address,
> > > > > > authentication and so on.
> > > > > >
> > > > > > That’s good but the overheads to change APISIX is huge, firstly,
> > the
> > > > > > protocol used in each data center might vary, considering the weak
> > > > > > ecosystem of OpenResty/Nginx, we may need to implement the data
> > > center
> > > > > > SDK manually, and it will be tracky for some multiplexing protocols
> > > > > > (due to the connection pool model of OpenResty); On the other hand,
> > > > > > The APISIX internal is tightly coupled with ETCD, it’s not easy to
> > > > > > decouple them from the code level, what’s more, these part of codes
> > > > > > now are very stable, changing them might influence the stability of
> > > > > > APISIX.
> > > > > >
> > > > > > Agent Mode
> > > > > >
> > > > > > > All Problems in Computer Science can be solved by another level
> > of
> > > > > > indirection.
> > > > > >
> > > > > > Since it’s not easy to support this from APISIX itself, why not
> > > > > > introducing another component, it supports to communicate with any
> > > > > > data centers, and it mimics the ETCD V3 APIs and gRPC gateway, so
> > > that
> > > > > > it can pretend itself as the ETCD cluster from APISIX’s point of
> > > view.
> > > > > > With that a component, there is nothing that should be done in
> > APISIX
> > > > > > itself (You can see the architecture figure in the attachments).
> > > > > >
> > > > > > Agents can be deployed in any place as the networking is connected
> > > for
> > > > > > both the APISIX and data centers sides. It should be designed as
> > > > > > stateless, so APISIX can configure multiple agent instances and the
> > > > > > high availability can be sure. It can be even deployed for each
> > > APISIX
> > > > > > instance, that’s the famous sidecar mode.
> > > > > >
> > > > > > Problems need to be addressed
> > > > > >
> > > > > > The agent component has to solve a series of problems so it can
> > > > > > achieve its mission. Firstly, it should implement the server side
> > of
> > > > > > ETCD V3 APIs. Fortunately, only minor parts of features are used by
> > > > > > APISIX, so the implementation won’t be too complicated, also, since
> > > > > > APISIX relis on the ETCD gRPC gateway, the agent should also
> > support
> > > > > > it, in general, that’s not too difficult as the gRPC ecosystem is
> > > > > > strong.
> > > > > >
> > > > > > Secondly, the internal design of the agent should be modular as
> > soon
> > > > > > as possible, so once we need to support a new data center, the
> > > > > > development would be easy. Of course that’s another topic and will
> > > not
> > > > > > be detailed in this post.
> > > > > >
> > > > > > Best regards
> > > > > > Chao Zhang
> > > > > >
> > > > > > https://github.com/tokers
> > > > > >
> > > >
> > >
> >
>
>
> --
Re: Multi Data Centers Support for APISIX
Posted by Fei Liang <bs...@gmail.com>.
I suggest APISIX abstract config-center interface. The APISIX
implementation is based on this config-center.
The config-center supports a variety of implementation methods. Which can
be implemented by etcd, redis, or even MySQL(consider supporting DB that do
not support push).
In addition, because the implementation of APISIX itself is LUA, the basic
components are relatively small. Combined with Chao Zhang’s proposal,
APISIX can also consider design own’s protocal for the agent service. The
agent connect the DB, which lua is difficultal connecting.
Ming Wen <we...@apache.org> 于2021年4月28日周三 下午4:08写道：
> I agreed with Sheng.
> In the long term, Apache APISIX should have an independent protocol,
> so that other data storage components can adapt to Apache APISIX,
> not only Apache APISIX to adapt others.
>
> Thanks,
> Ming Wen, Apache APISIX PMC Chair
> Twitter: _WenMing
>
>
> Sheng Wu <wu...@gmail.com> 于2021年4月27日周二 上午10:17写道：
>
> > I know this. So, I just said suggest reconsidering
> > This is a typical style of a top-level open source project doing.
> > Protocol is the key to a community, once you(APISIX) lead on both sides,
> it
> > is better the protocol is something we could make decisions when changes
> > are required in the future.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Chao Zhang <zc...@gmail.com> 于2021年4月27日周二 上午10:12写道：
> >
> > > Hi,
> > >
> > > We want to pretend the agent is an ETCD cluster so that we don't need
> > > any modifications on APISIX side, and actually etcd v3 API is based on
> > > gRPC.
> > >
> > > Best regards
> > > Chao Zhang
> > >
> > > https://github.com/tokers
> > >
> > > On Tue, Apr 27, 2021 at 9:58 AM Sheng Wu <wu...@gmail.com>
> > > wrote:
> > > >
> > > > Basically, I like this idea, just suggest reconsidering whether keep
> on
> > > > etcd API, especially gRPC tech stack is already there.
> > > > APISIX works with its agent but using etcd protocol? This seems
> > strange.
> > > >
> > > > Sheng Wu 吴晟
> > > > Twitter, wusheng1108
> > > >
> > > >
> > > > Chao Zhang <to...@apache.org> 于2021年4月27日周二 上午9:46写道：
> > > >
> > > > > Hi, there
> > > > >
> > > > > Below is the proposal about how to support multiple data centers
> for
> > > > > Apache APISIX. Please review it and your advice will be
> appreciated.
> > > > >
> > > > > Background
> > > > >
> > > > > So far, APISIX only supports fetching configurations like Route,
> > > > > Upstream from ETCD, although it’s capable of integrating a couple
> of
> > > > > service registries (e.g. Consul, Nacos) to discover backend service
> > > > > endpoints (aka upstream nodes) dynamically.
> > > > >
> > > > > For some users, ETCD is not their optimal solution as they may
> > already
> > > > > have another similar data center or they don’t master the skill for
> > > > > ETCD cluster maintenance. In such a case, introducing APISIX to
> their
> > > > > platform will be stumbling, even letting them give up on using it.
> > > > >
> > > > > So from this perspective, making APISIX decouple from ETCD is
> > > > > important, it also enriches the ecosystem of APISIX.
> > > > >
> > > > > Self Adjustment
> > > > >
> > > > > The most intuitive way is evolving APISIX itself, letting it as the
> > > > > client of the target data center, so no extra components will be
> > > > > deployed, what administrators need to do is configuring the data
> > > > > center in the configuration of APISIX, including address,
> > > > > authentication and so on.
> > > > >
> > > > > That’s good but the overheads to change APISIX is huge, firstly,
> the
> > > > > protocol used in each data center might vary, considering the weak
> > > > > ecosystem of OpenResty/Nginx, we may need to implement the data
> > center
> > > > > SDK manually, and it will be tracky for some multiplexing protocols
> > > > > (due to the connection pool model of OpenResty); On the other hand,
> > > > > The APISIX internal is tightly coupled with ETCD, it’s not easy to
> > > > > decouple them from the code level, what’s more, these part of codes
> > > > > now are very stable, changing them might influence the stability of
> > > > > APISIX.
> > > > >
> > > > > Agent Mode
> > > > >
> > > > > > All Problems in Computer Science can be solved by another level
> of
> > > > > indirection.
> > > > >
> > > > > Since it’s not easy to support this from APISIX itself, why not
> > > > > introducing another component, it supports to communicate with any
> > > > > data centers, and it mimics the ETCD V3 APIs and gRPC gateway, so
> > that
> > > > > it can pretend itself as the ETCD cluster from APISIX’s point of
> > view.
> > > > > With that a component, there is nothing that should be done in
> APISIX
> > > > > itself (You can see the architecture figure in the attachments).
> > > > >
> > > > > Agents can be deployed in any place as the networking is connected
> > for
> > > > > both the APISIX and data centers sides. It should be designed as
> > > > > stateless, so APISIX can configure multiple agent instances and the
> > > > > high availability can be sure. It can be even deployed for each
> > APISIX
> > > > > instance, that’s the famous sidecar mode.
> > > > >
> > > > > Problems need to be addressed
> > > > >
> > > > > The agent component has to solve a series of problems so it can
> > > > > achieve its mission. Firstly, it should implement the server side
> of
> > > > > ETCD V3 APIs. Fortunately, only minor parts of features are used by
> > > > > APISIX, so the implementation won’t be too complicated, also, since
> > > > > APISIX relis on the ETCD gRPC gateway, the agent should also
> support
> > > > > it, in general, that’s not too difficult as the gRPC ecosystem is
> > > > > strong.
> > > > >
> > > > > Secondly, the internal design of the agent should be modular as
> soon
> > > > > as possible, so once we need to support a new data center, the
> > > > > development would be easy. Of course that’s another topic and will
> > not
> > > > > be detailed in this post.
> > > > >
> > > > > Best regards
> > > > > Chao Zhang
> > > > >
> > > > > https://github.com/tokers
> > > > >
> > >
> >
>
--
Re: Multi Data Centers Support for APISIX
Posted by Ming Wen <we...@apache.org>.
I agreed with Sheng.
In the long term, Apache APISIX should have an independent protocol,
so that other data storage components can adapt to Apache APISIX,
not only Apache APISIX to adapt others.
Thanks,
Ming Wen, Apache APISIX PMC Chair
Twitter: _WenMing
Sheng Wu <wu...@gmail.com> 于2021年4月27日周二 上午10:17写道：
> I know this. So, I just said suggest reconsidering
> This is a typical style of a top-level open source project doing.
> Protocol is the key to a community, once you(APISIX) lead on both sides, it
> is better the protocol is something we could make decisions when changes
> are required in the future.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Chao Zhang <zc...@gmail.com> 于2021年4月27日周二 上午10:12写道：
>
> > Hi,
> >
> > We want to pretend the agent is an ETCD cluster so that we don't need
> > any modifications on APISIX side, and actually etcd v3 API is based on
> > gRPC.
> >
> > Best regards
> > Chao Zhang
> >
> > https://github.com/tokers
> >
> > On Tue, Apr 27, 2021 at 9:58 AM Sheng Wu <wu...@gmail.com>
> > wrote:
> > >
> > > Basically, I like this idea, just suggest reconsidering whether keep on
> > > etcd API, especially gRPC tech stack is already there.
> > > APISIX works with its agent but using etcd protocol? This seems
> strange.
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > Chao Zhang <to...@apache.org> 于2021年4月27日周二 上午9:46写道：
> > >
> > > > Hi, there
> > > >
> > > > Below is the proposal about how to support multiple data centers for
> > > > Apache APISIX. Please review it and your advice will be appreciated.
> > > >
> > > > Background
> > > >
> > > > So far, APISIX only supports fetching configurations like Route,
> > > > Upstream from ETCD, although it’s capable of integrating a couple of
> > > > service registries (e.g. Consul, Nacos) to discover backend service
> > > > endpoints (aka upstream nodes) dynamically.
> > > >
> > > > For some users, ETCD is not their optimal solution as they may
> already
> > > > have another similar data center or they don’t master the skill for
> > > > ETCD cluster maintenance. In such a case, introducing APISIX to their
> > > > platform will be stumbling, even letting them give up on using it.
> > > >
> > > > So from this perspective, making APISIX decouple from ETCD is
> > > > important, it also enriches the ecosystem of APISIX.
> > > >
> > > > Self Adjustment
> > > >
> > > > The most intuitive way is evolving APISIX itself, letting it as the
> > > > client of the target data center, so no extra components will be
> > > > deployed, what administrators need to do is configuring the data
> > > > center in the configuration of APISIX, including address,
> > > > authentication and so on.
> > > >
> > > > That’s good but the overheads to change APISIX is huge, firstly, the
> > > > protocol used in each data center might vary, considering the weak
> > > > ecosystem of OpenResty/Nginx, we may need to implement the data
> center
> > > > SDK manually, and it will be tracky for some multiplexing protocols
> > > > (due to the connection pool model of OpenResty); On the other hand,
> > > > The APISIX internal is tightly coupled with ETCD, it’s not easy to
> > > > decouple them from the code level, what’s more, these part of codes
> > > > now are very stable, changing them might influence the stability of
> > > > APISIX.
> > > >
> > > > Agent Mode
> > > >
> > > > > All Problems in Computer Science can be solved by another level of
> > > > indirection.
> > > >
> > > > Since it’s not easy to support this from APISIX itself, why not
> > > > introducing another component, it supports to communicate with any
> > > > data centers, and it mimics the ETCD V3 APIs and gRPC gateway, so
> that
> > > > it can pretend itself as the ETCD cluster from APISIX’s point of
> view.
> > > > With that a component, there is nothing that should be done in APISIX
> > > > itself (You can see the architecture figure in the attachments).
> > > >
> > > > Agents can be deployed in any place as the networking is connected
> for
> > > > both the APISIX and data centers sides. It should be designed as
> > > > stateless, so APISIX can configure multiple agent instances and the
> > > > high availability can be sure. It can be even deployed for each
> APISIX
> > > > instance, that’s the famous sidecar mode.
> > > >
> > > > Problems need to be addressed
> > > >
> > > > The agent component has to solve a series of problems so it can
> > > > achieve its mission. Firstly, it should implement the server side of
> > > > ETCD V3 APIs. Fortunately, only minor parts of features are used by
> > > > APISIX, so the implementation won’t be too complicated, also, since
> > > > APISIX relis on the ETCD gRPC gateway, the agent should also support
> > > > it, in general, that’s not too difficult as the gRPC ecosystem is
> > > > strong.
> > > >
> > > > Secondly, the internal design of the agent should be modular as soon
> > > > as possible, so once we need to support a new data center, the
> > > > development would be easy. Of course that’s another topic and will
> not
> > > > be detailed in this post.
> > > >
> > > > Best regards
> > > > Chao Zhang
> > > >
> > > > https://github.com/tokers
> > > >
> >
>
Re: Multi Data Centers Support for APISIX
Posted by Sheng Wu <wu...@gmail.com>.
I know this. So, I just said suggest reconsidering
This is a typical style of a top-level open source project doing.
Protocol is the key to a community, once you(APISIX) lead on both sides, it
is better the protocol is something we could make decisions when changes
are required in the future.
Sheng Wu 吴晟
Twitter, wusheng1108
Chao Zhang <zc...@gmail.com> 于2021年4月27日周二 上午10:12写道：
> Hi,
>
> We want to pretend the agent is an ETCD cluster so that we don't need
> any modifications on APISIX side, and actually etcd v3 API is based on
> gRPC.
>
> Best regards
> Chao Zhang
>
> https://github.com/tokers
>
> On Tue, Apr 27, 2021 at 9:58 AM Sheng Wu <wu...@gmail.com>
> wrote:
> >
> > Basically, I like this idea, just suggest reconsidering whether keep on
> > etcd API, especially gRPC tech stack is already there.
> > APISIX works with its agent but using etcd protocol? This seems strange.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Chao Zhang <to...@apache.org> 于2021年4月27日周二 上午9:46写道：
> >
> > > Hi, there
> > >
> > > Below is the proposal about how to support multiple data centers for
> > > Apache APISIX. Please review it and your advice will be appreciated.
> > >
> > > Background
> > >
> > > So far, APISIX only supports fetching configurations like Route,
> > > Upstream from ETCD, although it’s capable of integrating a couple of
> > > service registries (e.g. Consul, Nacos) to discover backend service
> > > endpoints (aka upstream nodes) dynamically.
> > >
> > > For some users, ETCD is not their optimal solution as they may already
> > > have another similar data center or they don’t master the skill for
> > > ETCD cluster maintenance. In such a case, introducing APISIX to their
> > > platform will be stumbling, even letting them give up on using it.
> > >
> > > So from this perspective, making APISIX decouple from ETCD is
> > > important, it also enriches the ecosystem of APISIX.
> > >
> > > Self Adjustment
> > >
> > > The most intuitive way is evolving APISIX itself, letting it as the
> > > client of the target data center, so no extra components will be
> > > deployed, what administrators need to do is configuring the data
> > > center in the configuration of APISIX, including address,
> > > authentication and so on.
> > >
> > > That’s good but the overheads to change APISIX is huge, firstly, the
> > > protocol used in each data center might vary, considering the weak
> > > ecosystem of OpenResty/Nginx, we may need to implement the data center
> > > SDK manually, and it will be tracky for some multiplexing protocols
> > > (due to the connection pool model of OpenResty); On the other hand,
> > > The APISIX internal is tightly coupled with ETCD, it’s not easy to
> > > decouple them from the code level, what’s more, these part of codes
> > > now are very stable, changing them might influence the stability of
> > > APISIX.
> > >
> > > Agent Mode
> > >
> > > > All Problems in Computer Science can be solved by another level of
> > > indirection.
> > >
> > > Since it’s not easy to support this from APISIX itself, why not
> > > introducing another component, it supports to communicate with any
> > > data centers, and it mimics the ETCD V3 APIs and gRPC gateway, so that
> > > it can pretend itself as the ETCD cluster from APISIX’s point of view.
> > > With that a component, there is nothing that should be done in APISIX
> > > itself (You can see the architecture figure in the attachments).
> > >
> > > Agents can be deployed in any place as the networking is connected for
> > > both the APISIX and data centers sides. It should be designed as
> > > stateless, so APISIX can configure multiple agent instances and the
> > > high availability can be sure. It can be even deployed for each APISIX
> > > instance, that’s the famous sidecar mode.
> > >
> > > Problems need to be addressed
> > >
> > > The agent component has to solve a series of problems so it can
> > > achieve its mission. Firstly, it should implement the server side of
> > > ETCD V3 APIs. Fortunately, only minor parts of features are used by
> > > APISIX, so the implementation won’t be too complicated, also, since
> > > APISIX relis on the ETCD gRPC gateway, the agent should also support
> > > it, in general, that’s not too difficult as the gRPC ecosystem is
> > > strong.
> > >
> > > Secondly, the internal design of the agent should be modular as soon
> > > as possible, so once we need to support a new data center, the
> > > development would be easy. Of course that’s another topic and will not
> > > be detailed in this post.
> > >
> > > Best regards
> > > Chao Zhang
> > >
> > > https://github.com/tokers
> > >
>
Re: Multi Data Centers Support for APISIX
Posted by Chao Zhang <zc...@gmail.com>.
Hi,
We want to pretend the agent is an ETCD cluster so that we don't need
any modifications on APISIX side, and actually etcd v3 API is based on gRPC.
Best regards
Chao Zhang
https://github.com/tokers
On Tue, Apr 27, 2021 at 9:58 AM Sheng Wu <wu...@gmail.com> wrote:
>
> Basically, I like this idea, just suggest reconsidering whether keep on
> etcd API, especially gRPC tech stack is already there.
> APISIX works with its agent but using etcd protocol? This seems strange.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Chao Zhang <to...@apache.org> 于2021年4月27日周二 上午9:46写道：
>
> > Hi, there
> >
> > Below is the proposal about how to support multiple data centers for
> > Apache APISIX. Please review it and your advice will be appreciated.
> >
> > Background
> >
> > So far, APISIX only supports fetching configurations like Route,
> > Upstream from ETCD, although it’s capable of integrating a couple of
> > service registries (e.g. Consul, Nacos) to discover backend service
> > endpoints (aka upstream nodes) dynamically.
> >
> > For some users, ETCD is not their optimal solution as they may already
> > have another similar data center or they don’t master the skill for
> > ETCD cluster maintenance. In such a case, introducing APISIX to their
> > platform will be stumbling, even letting them give up on using it.
> >
> > So from this perspective, making APISIX decouple from ETCD is
> > important, it also enriches the ecosystem of APISIX.
> >
> > Self Adjustment
> >
> > The most intuitive way is evolving APISIX itself, letting it as the
> > client of the target data center, so no extra components will be
> > deployed, what administrators need to do is configuring the data
> > center in the configuration of APISIX, including address,
> > authentication and so on.
> >
> > That’s good but the overheads to change APISIX is huge, firstly, the
> > protocol used in each data center might vary, considering the weak
> > ecosystem of OpenResty/Nginx, we may need to implement the data center
> > SDK manually, and it will be tracky for some multiplexing protocols
> > (due to the connection pool model of OpenResty); On the other hand,
> > The APISIX internal is tightly coupled with ETCD, it’s not easy to
> > decouple them from the code level, what’s more, these part of codes
> > now are very stable, changing them might influence the stability of
> > APISIX.
> >
> > Agent Mode
> >
> > > All Problems in Computer Science can be solved by another level of
> > indirection.
> >
> > Since it’s not easy to support this from APISIX itself, why not
> > introducing another component, it supports to communicate with any
> > data centers, and it mimics the ETCD V3 APIs and gRPC gateway, so that
> > it can pretend itself as the ETCD cluster from APISIX’s point of view.
> > With that a component, there is nothing that should be done in APISIX
> > itself (You can see the architecture figure in the attachments).
> >
> > Agents can be deployed in any place as the networking is connected for
> > both the APISIX and data centers sides. It should be designed as
> > stateless, so APISIX can configure multiple agent instances and the
> > high availability can be sure. It can be even deployed for each APISIX
> > instance, that’s the famous sidecar mode.
> >
> > Problems need to be addressed
> >
> > The agent component has to solve a series of problems so it can
> > achieve its mission. Firstly, it should implement the server side of
> > ETCD V3 APIs. Fortunately, only minor parts of features are used by
> > APISIX, so the implementation won’t be too complicated, also, since
> > APISIX relis on the ETCD gRPC gateway, the agent should also support
> > it, in general, that’s not too difficult as the gRPC ecosystem is
> > strong.
> >
> > Secondly, the internal design of the agent should be modular as soon
> > as possible, so once we need to support a new data center, the
> > development would be easy. Of course that’s another topic and will not
> > be detailed in this post.
> >
> > Best regards
> > Chao Zhang
> >
> > https://github.com/tokers
> >
Re: Multi Data Centers Support for APISIX
Posted by Sheng Wu <wu...@gmail.com>.
Basically, I like this idea, just suggest reconsidering whether keep on
etcd API, especially gRPC tech stack is already there.
APISIX works with its agent but using etcd protocol? This seems strange.
Sheng Wu 吴晟
Twitter, wusheng1108
Chao Zhang <to...@apache.org> 于2021年4月27日周二 上午9:46写道：
> Hi, there
>
> Below is the proposal about how to support multiple data centers for
> Apache APISIX. Please review it and your advice will be appreciated.
>
> Background
>
> So far, APISIX only supports fetching configurations like Route,
> Upstream from ETCD, although it’s capable of integrating a couple of
> service registries (e.g. Consul, Nacos) to discover backend service
> endpoints (aka upstream nodes) dynamically.
>
> For some users, ETCD is not their optimal solution as they may already
> have another similar data center or they don’t master the skill for
> ETCD cluster maintenance. In such a case, introducing APISIX to their
> platform will be stumbling, even letting them give up on using it.
>
> So from this perspective, making APISIX decouple from ETCD is
> important, it also enriches the ecosystem of APISIX.
>
> Self Adjustment
>
> The most intuitive way is evolving APISIX itself, letting it as the
> client of the target data center, so no extra components will be
> deployed, what administrators need to do is configuring the data
> center in the configuration of APISIX, including address,
> authentication and so on.
>
> That’s good but the overheads to change APISIX is huge, firstly, the
> protocol used in each data center might vary, considering the weak
> ecosystem of OpenResty/Nginx, we may need to implement the data center
> SDK manually, and it will be tracky for some multiplexing protocols
> (due to the connection pool model of OpenResty); On the other hand,
> The APISIX internal is tightly coupled with ETCD, it’s not easy to
> decouple them from the code level, what’s more, these part of codes
> now are very stable, changing them might influence the stability of
> APISIX.
>
> Agent Mode
>
> > All Problems in Computer Science can be solved by another level of
> indirection.
>
> Since it’s not easy to support this from APISIX itself, why not
> introducing another component, it supports to communicate with any
> data centers, and it mimics the ETCD V3 APIs and gRPC gateway, so that
> it can pretend itself as the ETCD cluster from APISIX’s point of view.
> With that a component, there is nothing that should be done in APISIX
> itself (You can see the architecture figure in the attachments).
>
> Agents can be deployed in any place as the networking is connected for
> both the APISIX and data centers sides. It should be designed as
> stateless, so APISIX can configure multiple agent instances and the
> high availability can be sure. It can be even deployed for each APISIX
> instance, that’s the famous sidecar mode.
>
> Problems need to be addressed
>
> The agent component has to solve a series of problems so it can
> achieve its mission. Firstly, it should implement the server side of
> ETCD V3 APIs. Fortunately, only minor parts of features are used by
> APISIX, so the implementation won’t be too complicated, also, since
> APISIX relis on the ETCD gRPC gateway, the agent should also support
> it, in general, that’s not too difficult as the gRPC ecosystem is
> strong.
>
> Secondly, the internal design of the agent should be modular as soon
> as possible, so once we need to support a new data center, the
> development would be easy. Of course that’s another topic and will not
> be detailed in this post.
>
> Best regards
> Chao Zhang
>
> https://github.com/tokers
>