[DISCUSS] Proposal: Supporting interacting with etcd via gRPC

[Zexuan Luo <sp...@apache.org>]

background

1. Problem to be solved

APISIX uses etcd as the configuration center. etcd has migrated the
interface to gRPC since v3. However, since there is no project in the
OpenResty ecosystem that supports gRPC, APISIX can only call the HTTP
interface of etcd. The HTTP interface of etcd is provided through
gRPC-gateway. Essentially, etcd runs an HTTP to gRPC proxy at the
server level, and external HTTP requests will be converted into gRPC
requests. In practice, we also found some problems with the
interaction between the HTTP API and the gRPC API. In fact, having a
gRPC-gateway does not mean that it can perfectly support HTTP access,
there are some subtle differences here.

2. The benefits of solving this problem

2.1 By supporting interacting with etcd via gRPC, APISIX can have
first-class support from etcd.
2.2 Since gRPC shares all the streams in a TCP connection, we can also
reduce the number of connections rapidly.
2.3 The connections can be made more secure by using gRPC directly

Solution

As the gRPC ability requires the grpc-client-nginx-module, we can't
enable gRPC support when using APISIX on vanilla OpenResty.
We interact with etcd in the scenes below:

Check and init in the cli mode

We can ship a grpcurl with APISIX, although this will make the packaging tough.

read/write keys in the Admin API

We need to write the gRPC version of core.etcd.set/delete/get/...

Sync conf during the startup

We need to make it possible to send unary call in init_worker_by_lua,
just like what we have done in the HTTP way (replacing cosocket with
luasocket). We can do this by introducing a separate blocking
implementation for unary call in the grpc-client-nginx-module.

Sync conf when running APISIX

There are two kinds of sync: full sync & incremental sync.
For full sync, we will reimplement readdir API in gRPC.
For incremental sync, current implementation is based on HTTP long
poll, so each watch operation will start a new request and have its
own timeout. When we switch to gRPC, we will use one connection at the
time. So we need to provide a longer default timeout for the gRPC. We
also need to reimplement the one-shot watchdir API with the recv call
in gRPC's server stream.

Re: [DISCUSS] Proposal: Supporting interacting with etcd via gRPC

[YuanSheng Wang <me...@apache.org>]

hi:

Love this new feature, Nginx has been relatively lacking in gRPC support,
and I am looking forward to this feature.


On Sun, Oct 9, 2022 at 10:42 AM Jintao Zhang <zh...@apache.org> wrote:

> SGTM!
>
> The processing of the entire request chain will be shorter.
> And we don't need to be limited by grpc_gateway
>
> Zexuan Luo <sp...@apache.org> 于2022年10月8日周六 19:54写道：
> >
> > background
> >
> > 1. Problem to be solved
> >
> > APISIX uses etcd as the configuration center. etcd has migrated the
> > interface to gRPC since v3. However, since there is no project in the
> > OpenResty ecosystem that supports gRPC, APISIX can only call the HTTP
> > interface of etcd. The HTTP interface of etcd is provided through
> > gRPC-gateway. Essentially, etcd runs an HTTP to gRPC proxy at the
> > server level, and external HTTP requests will be converted into gRPC
> > requests. In practice, we also found some problems with the
> > interaction between the HTTP API and the gRPC API. In fact, having a
> > gRPC-gateway does not mean that it can perfectly support HTTP access,
> > there are some subtle differences here.
> >
> > 2. The benefits of solving this problem
> >
> > 2.1 By supporting interacting with etcd via gRPC, APISIX can have
> > first-class support from etcd.
> > 2.2 Since gRPC shares all the streams in a TCP connection, we can also
> > reduce the number of connections rapidly.
> > 2.3 The connections can be made more secure by using gRPC directly
> >
> > Solution
> >
> > As the gRPC ability requires the grpc-client-nginx-module, we can't
> > enable gRPC support when using APISIX on vanilla OpenResty.
> > We interact with etcd in the scenes below:
> >
> > Check and init in the cli mode
> >
> > We can ship a grpcurl with APISIX, although this will make the packaging
> tough.
> >
> > read/write keys in the Admin API
> >
> > We need to write the gRPC version of core.etcd.set/delete/get/...
> >
> > Sync conf during the startup
> >
> > We need to make it possible to send unary call in init_worker_by_lua,
> > just like what we have done in the HTTP way (replacing cosocket with
> > luasocket). We can do this by introducing a separate blocking
> > implementation for unary call in the grpc-client-nginx-module.
> >
> > Sync conf when running APISIX
> >
> > There are two kinds of sync: full sync & incremental sync.
> > For full sync, we will reimplement readdir API in gRPC.
> > For incremental sync, current implementation is based on HTTP long
> > poll, so each watch operation will start a new request and have its
> > own timeout. When we switch to gRPC, we will use one connection at the
> > time. So we need to provide a longer default timeout for the gRPC. We
> > also need to reimplement the one-shot watchdir API with the recv call
> > in gRPC's server stream.
>


-- 

*MembPhis*
My GitHub: https://github.com/membphis
Apache APISIX: https://github.com/apache/apisix

Re: [DISCUSS] Proposal: Supporting interacting with etcd via gRPC

[Jintao Zhang <zh...@apache.org>]

SGTM!

The processing of the entire request chain will be shorter.
And we don't need to be limited by grpc_gateway

Zexuan Luo <sp...@apache.org> 于2022年10月8日周六 19:54写道：
>
> background
>
> 1. Problem to be solved
>
> APISIX uses etcd as the configuration center. etcd has migrated the
> interface to gRPC since v3. However, since there is no project in the
> OpenResty ecosystem that supports gRPC, APISIX can only call the HTTP
> interface of etcd. The HTTP interface of etcd is provided through
> gRPC-gateway. Essentially, etcd runs an HTTP to gRPC proxy at the
> server level, and external HTTP requests will be converted into gRPC
> requests. In practice, we also found some problems with the
> interaction between the HTTP API and the gRPC API. In fact, having a
> gRPC-gateway does not mean that it can perfectly support HTTP access,
> there are some subtle differences here.
>
> 2. The benefits of solving this problem
>
> 2.1 By supporting interacting with etcd via gRPC, APISIX can have
> first-class support from etcd.
> 2.2 Since gRPC shares all the streams in a TCP connection, we can also
> reduce the number of connections rapidly.
> 2.3 The connections can be made more secure by using gRPC directly
>
> Solution
>
> As the gRPC ability requires the grpc-client-nginx-module, we can't
> enable gRPC support when using APISIX on vanilla OpenResty.
> We interact with etcd in the scenes below:
>
> Check and init in the cli mode
>
> We can ship a grpcurl with APISIX, although this will make the packaging tough.
>
> read/write keys in the Admin API
>
> We need to write the gRPC version of core.etcd.set/delete/get/...
>
> Sync conf during the startup
>
> We need to make it possible to send unary call in init_worker_by_lua,
> just like what we have done in the HTTP way (replacing cosocket with
> luasocket). We can do this by introducing a separate blocking
> implementation for unary call in the grpc-client-nginx-module.
>
> Sync conf when running APISIX
>
> There are two kinds of sync: full sync & incremental sync.
> For full sync, we will reimplement readdir API in gRPC.
> For incremental sync, current implementation is based on HTTP long
> poll, so each watch operation will start a new request and have its
> own timeout. When we switch to gRPC, we will use one connection at the
> time. So we need to provide a longer default timeout for the gRPC. We
> also need to reimplement the one-shot watchdir API with the recv call
> in gRPC's server stream.