You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@apisix.apache.org by LI Fabrice <yi...@hotmail.com> on 2023/08/03 15:33:51 UTC

回复: [DISCUSS] Proposal: APISIX: remove etcd grpc and conf server

I believe this proposal is excellent

First of all, there are a few questions that we need clarification:

1. Are we only removing the gRPC configuration of etcd in APISIX, or are we removing the entire gRPC feature of etcd in APISIX?

2. If it's the former, the impact and scope of changes are relatively small. If it's the latter, there will be significant pressure and work for regression testing due to the broad impact. How do we control the scope of these effects? Additionally, any merged PRs that fixed this feature for etcd gRPC until now will need to be extracted for a rollback.

3. Is the 'etcd-adapter' a configuration module? As I understand it, it's essentially a refactoring of the original etcd module, creating a new adapter layer that serves as an intermediary between the existing module and the database for interactions.

Our perspective is that maybe we should first hide the original gRPC functionality, or explicitly note that this feature may have blind spots. We also strongly hope that APISIX can interact with etcd through gRPC.

Best regards,

Fabriceli

________________________________
发件人: Jinhua Luo <lu...@gmail.com>
发送时间: 2023年8月3日 17:58
收件人: dev@apisix.apache.org <de...@apisix.apache.org>
主题: [DISCUSS] Proposal: APISIX: remove etcd grpc and conf server

## Background

### etcd grpc

In previous versions, apisix used HTTP to connect with etcd's
grpc-gateway. But it had the following disadvantages:
1. Each admin resource per worker process, e.g. routes, needs to use a
dedicated TCP connection, i.e. each worker process will have more than
10 TCP connections, which increases the load on the etcd server.
2. Some extra features only exist in etcd server, not in grpc-gateway.
3. etcd official may remove grpc-gateway at some date.

So we introduce grpc here to replace http.
However this module has too many bugs, meanwhile, we will introduce a
new conf server (maybe it's etcd-adapter) later, which will act as a
real etcd server but supports different db backends, e..g postgresql.

So we plan to remove etcd grpc in apisix.

On the other hand, HTTP optimizations have been merged into the apisix
master branch:
1. fix(etcd): reuse cli and enable keepalive
https://github.com/apache/apisix/pull/9420

2. feat(config_etcd): use a single long http connection to watch all resources
https://github.com/apache/apisix/pull/9456

After optimization, the HTTP way is as good as the grpc way.

### current conf server

This is an internal feature added earlier and is intended to be
database backend agnostic. However, the current implementation is a
simple pure proxy between DP/CP and etcd. Also, this makes debugging
via gdb difficult, since all handlers downstream and upstream will
always be triggered. It also obscures configuration.

It's worth noting that the decoupled mode doesn't work currently.

Since we'll be using the new unified external config server later,
we'd rather remove the currently useless dumb version.

And after refactoring, all deploy modes will work fine.

### github bug issues

https://github.com/apache/apisix/issues/9381
https://github.com/apache/apisix/issues/9347
https://github.com/apache/apisix/issues/8067
https://github.com/apache/apisix/issues/9212
https://github.com/apache/apisix/issues/9007
https://github.com/apache/apisix/issues/9336
https://github.com/apache/apisix/issues/9049
https://github.com/apache/apisix/issues/9582
https://github.com/apache/apisix/issues/9578
https://github.com/apache/apisix/issues/9528

## Conculsion

All in all, we wish to revert the apisix etcd access to 2.x behavior,
i.e. connect etcd directly via HTTP, but that doesn't mean we lose
grpc and config db variants, instead we will implement a separate conf
server to provide these features.

Re: [DISCUSS] Proposal: APISIX: remove etcd grpc and conf server

Posted by Jinhua Luo <lu...@gmail.com>.

1. We will disable grpc's access to etcd without removing the associated code.
More specifically, in config.yaml this is ignored:
deployment:
  etcd:
    use_grpc: true
In your words, yes, it will be hidden for a while.

2. The architecture of etcd-adapter (the naming is of course temporary) will be:
apex <-----http/https----> etcd adapter <------grpc------> etcd
                                                    \<-----------other
protocols--------> other db backends

Yes, etcd adapter is a separate optional component of apisix.
grpc is not abandoned, on the contrary, it will be useful in other
ways sooner or later.
Using grpc to access etcd is of course the most correct way, no doubt.

LI Fabrice <yi...@hotmail.com> 于2023年8月3日周四 23:34写道：
>
> I believe this proposal is excellent
>
> First of all, there are a few questions that we need clarification:
>
>   1.  Are we only removing the gRPC configuration of etcd in APISIX, or are we removing the entire gRPC feature of etcd in APISIX?
>
>   2.  If it's the former, the impact and scope of changes are relatively small. If it's the latter, there will be significant pressure and work for regression testing due to the broad impact. How do we control the scope of these effects? Additionally, any merged PRs that fixed this feature for etcd gRPC until now will need to be extracted for a rollback.
>
>   3.  Is the 'etcd-adapter' a configuration module? As I understand it, it's essentially a refactoring of the original etcd module, creating a new adapter layer that serves as an intermediary between the existing module and the database for interactions.
>
> Our perspective is that maybe we should first hide the original gRPC functionality, or explicitly note that this feature may have blind spots. We also strongly hope that APISIX can interact with etcd through gRPC.
>
> Best regards,
>
> Fabriceli
>
>
>
>
> ________________________________
> 发件人: Jinhua Luo <lu...@gmail.com>
> 发送时间: 2023年8月3日 17:58
> 收件人: dev@apisix.apache.org <de...@apisix.apache.org>
> 主题: [DISCUSS] Proposal: APISIX: remove etcd grpc and conf server
>
> ## Background
>
> ### etcd grpc
>
> In previous versions, apisix used HTTP to connect with etcd's
> grpc-gateway. But it had the following disadvantages:
> 1. Each admin resource per worker process, e.g. routes, needs to use a
> dedicated TCP connection, i.e. each worker process will have more than
> 10 TCP connections, which increases the load on the etcd server.
> 2. Some extra features only exist in etcd server, not in grpc-gateway.
> 3. etcd official may remove grpc-gateway at some date.
>
> So we introduce grpc here to replace http.
> However this module has too many bugs, meanwhile, we will introduce a
> new conf server (maybe it's etcd-adapter) later, which will act as a
> real etcd server but supports different db backends, e..g postgresql.
>
> So we plan to remove etcd grpc in apisix.
>
> On the other hand, HTTP optimizations have been merged into the apisix
> master branch:
> 1. fix(etcd): reuse cli and enable keepalive
> https://github.com/apache/apisix/pull/9420
>
> 2. feat(config_etcd): use a single long http connection to watch all resources
> https://github.com/apache/apisix/pull/9456
>
> After optimization, the HTTP way is as good as the grpc way.
>
> ### current conf server
>
> This is an internal feature added earlier and is intended to be
> database backend agnostic. However, the current implementation is a
> simple pure proxy between DP/CP and etcd. Also, this makes debugging
> via gdb difficult, since all handlers downstream and upstream will
> always be triggered. It also obscures configuration.
>
> It's worth noting that the decoupled mode doesn't work currently.
>
> Since we'll be using the new unified external config server later,
> we'd rather remove the currently useless dumb version.
>
> And after refactoring, all deploy modes will work fine.
>
> ### github bug issues
>
> https://github.com/apache/apisix/issues/9381
> https://github.com/apache/apisix/issues/9347
> https://github.com/apache/apisix/issues/8067
> https://github.com/apache/apisix/issues/9212
> https://github.com/apache/apisix/issues/9007
> https://github.com/apache/apisix/issues/9336
> https://github.com/apache/apisix/issues/9049
> https://github.com/apache/apisix/issues/9582
> https://github.com/apache/apisix/issues/9578
> https://github.com/apache/apisix/issues/9528
>
> ## Conculsion
>
> All in all, we wish to revert the apisix etcd access to 2.x behavior,
> i.e. connect etcd directly via HTTP, but that doesn't mean we lose
> grpc and config db variants, instead we will implement a separate conf
> server to provide these features.