You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apisix.apache.org by Zexuan Luo <sp...@apache.org> on 2021/04/16 06:52:15 UTC
[Proposal] support using other languages to write plugin for APISIX
1. Background
APISIX currently only supports writing plugins in Lua. If other
languages are supported, it will greatly broaden the APISIX ecosystem
and user base.
2. Solution
Since WASM is not yet mature, we consider implementing it through local IPC.
For the sake of discussion, the following will assume that the plugin
is written in Java. However, in practice, our solution can be
interfaced with other languages.
2.1 Terminology
Several terms are defined here.
Plugin Runner: The service that runs the plugin, written in the same
language as the plugin. In the first version, we assume that there
will be only one Plugin Runner.
2.2 Plugin Runner lifecycle
To simplify user operation and reduce the difficulty of upgrading,
Plugin Runner is managed by APISIX.
APISIX starts the Plugin Runner when it starts and ends it when it
ends. if the Plugin Runner quits in the middle, APISIX will restart it
automatically.
2.3 Timing of APISIX communication with Plugin Runner
```
Router ----> Global Plugin (written in Lua) ---> Ext Plugin
(ext-plugin-pre-req) ----> Lua Plugin (Router)
---> Ext plugin (ext-plugin-post-req) ---> Upstream
```
Running the Ext Plugin in the Global Plugin is not supported at this
time, as the global logic can be executed uniformly in the Plugin
Runner.
Running Ext Plugin after getting an upstream response is not supported
at this time. We can support it later with a buffering response.
ext-plugin-pre runs before all non-global Lua plugins, and
ext-plugin-post runs after all non-global Lua plugins.
2.4 How APISIX communicates with Plugin Runner
APISIX communicates with Plugin Runner through a unix socket. The
communication protocol is as follows.
2.4.1 Communication format
```
1 byte of type + 3 bytes of length + data
```
The type can be 0 ~ 7, and the length can be [0, 8M). data length is
determined by length.
Since Ext Plugin usually does not exchange too much data, 8M should be
enough. The reason for taking 4 bytes is to keep the header small
enough to be read efficiently.
The current type takes the following values.
0 means error
1 means prepare_conf
2 means http_req_call
The data is serialized in capnproto, a binary serialization format.
capnproto is supported by many programming languages:
https://capnproto.org/otherlang.html
The advantages of choosing capnproto are.
1. focus on serialization performance
2. partial deserialization support, so that decode can be done only
when it is needed
2.4.2 Communication steps
Each ext plugin will have the following configuration.
```
{
"conf": [
{
"name": "configuration name",
"value": "configuration value"
}
],
"extra_info": [
...
]
}
```
conf can be used to set the execution configuration of the
plugin-related requests inside Plugin Runner.
The default data sent to Plugin Runner is only the most common
information. If you want additional information, you need to declare
it in extra_info beforehand.
To save communication costs, conf is sent separately.
1. APISIX will check if conf has a corresponding token in the local cache.
2. If not, APISIX sends a prepare_conf request to ask Plugin Runner
to cache the conf and return a token.
(Note that Plugin Runner's cache time needs to be longer than APISIX's
cache time.)
3. APISIX sends an http_req_call request to Plugin Runner.
4. Plugin Runner executes the request and returns a response to APISIX.
5. APISIX processes the request based on the response
2.4.3 proto
Refer to https://capnproto.org/language.html
The following is the proto for error
response
```
enum ErrorCode {
BAD_REQUEST @0; # Plugin Runner can't understand APISIX
SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the request
CONF_TOKEN_NOT_FOUND @2;
}
struct ErrorResp {
Code @0 :ErrorCode;
}
```
The following is the proto of prepare_conf
request
```
struct PrepareConfReq {
conf @0 :List(Pair);
}
```
Response
```
struct PrepareConfResp {
conf_token @0 :UInt32;
}
```
Here is the proto for http_req_call
request
```
struct Pair {
name @0 :Text;
value @1 :Text;
}
struct PairData {
name @0 :Text;
value @1 :Data;
}
enum Method {
GET @0;
...
}
struct HTTPReqCallReq {
id @0 :UInt32;
src_ip @1 :Data;
method @2 :Method;
path @3 :Text;
args @4 :List(Pair);
headers @5 :List(Pair);
conf_token @6 :UInt32;
extra_info @7 :List(PairData);
}
```
Response
```
struct HTTPReqCallResp {
id @0 :UInt32;
struct Stop {
status @0 :UInt16;
headers @1 :List(Pair);
body @2 :Data;
}
struct Rewrite {
path @0 :Text;
headers @1 :List(Pair);
# Note that args are modified in full.
# Either empty, meaning no args need to be moved
# or the entire modified args, not the incrementally changed parts
args @2 :List(Pair);
}
# What needs to be done when the response is received action
action :union {
# Do nothing
continue @1 :Void;
# Equivalent to core.response.exit(status, body), allowing
additional headers to be set
stop @2 :Stop;
# Rewrite the request
rewrite @3 :Rewrite;
}
}
```
2.4.4 Error handling
Logging and returning 503 error codes
2.4.5 Environment variables
APISIX configures the Plugin Runner with a number of environment
variables when it is started.
APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to listen to
APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for longer than this
Re: [Proposal] support using other languages to write plugin for APISIX
Posted by Sheng Wu <wu...@gmail.com>.
Let me give a pretty clear answer from a Java developer.
99% of Java developers today, who are writing business logic, are only
focusing on Spring framework.
Especially, we are talking about Spring Gateway. Who will write that's
filter w/o Spring libraries or core? You could try to find a real case, I
think it is hard to find.
What this proposal was discussed, no because Java VS LUA, it is about Java
ecosystem and Spring ecosystem.
Sheng Wu 吴晟
Twitter, wusheng1108
ZhengSong Tu <tz...@gmail.com> 于2021年4月29日周四 下午5:26写道：
> I have some questions about the Java plugin runner to discuss.
>
> Topic 1: Technical selection of communication protocols
> java did not provide support for unix domain sockets until jdk16, but
> obviously we will not be so aggressive in choosing jdk16, at this stage we
> focus on jdk 8, so in order to implement java's unix domain sockets under
> jdk8, we have two optional technology stacks
> Option 1 - junixsocket, for more information please refer to the github
> address: https://github.com/kohlschutter/junixsocket
> Option 2-netty: For more information, please refer to github at
> https://github.com/netty/netty
> Before I was inclined to go for junixsocket because it is more pure, now I
> am inclined to go for netty because it is more mature, I would like to know
> your votes on this.
>
> Topic 2: Does it depend on spring-boot?
> Do we need to fully rely on the spring-boot framework as a base to
> implement the java plugin runner in phase 1?
> In terms of audience, spring-boot has a broader audience and is easier to
> integrate with the components of the spring boot ecosystem, and it is
> faster to build on spring-boot. What do you think?
>
> Topic 3: Responsibilities of the SDK.
> We should provide an SDK, which defines some base class objects and
> function interfaces that users can use to write their own plugins based on
> the SDK, similar to the filter form. Users need to package their own code
> as a jar and declare the dependencies of their code in the configuration
> file. So that java's plugin runner can load the user's typed jar at
> startup. Is there anything that needs to be fixed in the above process?
>
>
> Sheng Wu <wu...@gmail.com> 于2021年4月27日周二 上午10:00写道：
>
> > OK, if it is a development kit, I want to know whether it is really a
> > runtime? Because eventually, whether debugging is friendly or not, rely
> on
> > this.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Daming <da...@apache.org> 于2021年4月27日周二 上午9:57写道：
> >
> > > PDK stands for Plugin Development Kit.
> > > It is a good choice to build on spring-boot in the initial stage.
> > > Because it is very friendly and easy to get started for Java
> > > developers(plugins developers).
> > >
> > >
> > > Sheng Wu <wu...@gmail.com> 于2021年4月27日周二 上午9:31写道：
> > > >
> > > > I can't see the detail from the graph.
> > > > But if we are speaking of Java and Spring gateway ecosystem, you
> should
> > > > know, that is not just interfaces.
> > > > Spring boot, annotation system, IoC, AOP are all going to be there.
> So,
> > > > basically, I would recommend you to boot the codes as all
> Spring-based
> > > > applications did.
> > > > BTW, I am not sure what PDK means.
> > > >
> > > > And about debugging, once your APISIX process could access the JVM
> > > through
> > > > a socket with manual localhost:port, it is fine for the Java
> developer
> > to
> > > > start this sidecar in the IDEA.
> > > >
> > > > Sheng Wu 吴晟
> > > > Twitter, wusheng1108
> > > >
> > > >
> > > > ZhengSong Tu <tz...@gmail.com> 于2021年4月26日周一 上午11:47写道：
> > > >
> > > > > I have sketched the flow of the java side of the plugin runner
> > > > > [image: APISIX 运行 Java 插件.png]
> > > > >
> > > > > And I have some questions to clarify.
> > > > >
> > > > > 1. How does the plugin runner load the client's code?
> > > > > It's up to each runner to figure it out according to its own
> > > > > language's ecology. Take java as an example,
> > > > > the user develops with the PDK provided by us, and packages the
> > custom
> > > > > plugin code into a jar and puts it in the specified location. When
> > the
> > > > > plugin runner starts, it loads the jar package in this location by
> > > means of
> > > > > a custom dynamic ClassLoader or something like that.
> > > > >
> > > > > 2. How do users develop?
> > > > > Just provide the filter chain like Spring Cloud Gateway. Users
> > can
> > > > > register their own filters. And the PDK is a jar, with some defined
> > > > > interfaces such as pre, post, etc., and some objects such as
> request,
> > > > > header, etc.
> > > > >
> > > > > 3. How to debug？
> > > > > Option 1: APISIX provides debugging mode for java plugins,
> this
> > > time
> > > > > it is not APISIX to start the plugin runner, but the user himself
> > > downloads
> > > > > the plugin runner code and runs the main method to start it, so as
> to
> > > > > construct a request to access APISIX, APISIX will pass the context
> of
> > > the
> > > > > request to the plugin runner, and then use it to debug java code.
> > > > > Option 2: java remote debug.
> > > > >
> > > > >
> > > > >
> > > > > Zexuan Luo <sp...@apache.org> 于2021年4月19日周一 下午6:18写道：
> > > > >
> > > > >> Here is the new flatbuffer schema:
> > > > >>
> > > > >>
> > >
> >
> https://github.com/spacewander/incubator-apisix/blob/step1/apisix/plugins/ext-plugin/ext-plugin.fbs
> > > > >>
> > > > >> Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午2:52写道：
> > > > >> >
> > > > >> > 1. Background
> > > > >> >
> > > > >> > APISIX currently only supports writing plugins in Lua. If other
> > > > >> > languages are supported, it will greatly broaden the APISIX
> > > ecosystem
> > > > >> > and user base.
> > > > >> >
> > > > >> > 2. Solution
> > > > >> >
> > > > >> > Since WASM is not yet mature, we consider implementing it
> through
> > > local
> > > > >> IPC.
> > > > >> >
> > > > >> > For the sake of discussion, the following will assume that the
> > > plugin
> > > > >> > is written in Java. However, in practice, our solution can be
> > > > >> > interfaced with other languages.
> > > > >> >
> > > > >> > 2.1 Terminology
> > > > >> >
> > > > >> > Several terms are defined here.
> > > > >> >
> > > > >> > Plugin Runner: The service that runs the plugin, written in the
> > same
> > > > >> > language as the plugin. In the first version, we assume that
> there
> > > > >> > will be only one Plugin Runner.
> > > > >> >
> > > > >> > 2.2 Plugin Runner lifecycle
> > > > >> >
> > > > >> > To simplify user operation and reduce the difficulty of
> upgrading,
> > > > >> > Plugin Runner is managed by APISIX.
> > > > >> >
> > > > >> > APISIX starts the Plugin Runner when it starts and ends it when
> it
> > > > >> > ends. if the Plugin Runner quits in the middle, APISIX will
> > restart
> > > it
> > > > >> > automatically.
> > > > >> >
> > > > >> > 2.3 Timing of APISIX communication with Plugin Runner
> > > > >> >
> > > > >> > ```
> > > > >> > Router ----> Global Plugin (written in Lua) ---> Ext Plugin
> > > > >> > (ext-plugin-pre-req) ----> Lua Plugin (Router)
> > > > >> > ---> Ext plugin (ext-plugin-post-req) ---> Upstream
> > > > >> > ```
> > > > >> >
> > > > >> > Running the Ext Plugin in the Global Plugin is not supported at
> > this
> > > > >> > time, as the global logic can be executed uniformly in the
> Plugin
> > > > >> > Runner.
> > > > >> >
> > > > >> > Running Ext Plugin after getting an upstream response is not
> > > supported
> > > > >> > at this time. We can support it later with a buffering response.
> > > > >> >
> > > > >> > ext-plugin-pre runs before all non-global Lua plugins, and
> > > > >> > ext-plugin-post runs after all non-global Lua plugins.
> > > > >> >
> > > > >> > 2.4 How APISIX communicates with Plugin Runner
> > > > >> >
> > > > >> > APISIX communicates with Plugin Runner through a unix socket.
> The
> > > > >> > communication protocol is as follows.
> > > > >> >
> > > > >> > 2.4.1 Communication format
> > > > >> >
> > > > >> > ```
> > > > >> > 1 byte of type + 3 bytes of length + data
> > > > >> > ```
> > > > >> >
> > > > >> > The type can be 0 ~ 7, and the length can be [0, 8M). data
> length
> > is
> > > > >> > determined by length.
> > > > >> > Since Ext Plugin usually does not exchange too much data, 8M
> > should
> > > be
> > > > >> > enough. The reason for taking 4 bytes is to keep the header
> small
> > > > >> > enough to be read efficiently.
> > > > >> >
> > > > >> > The current type takes the following values.
> > > > >> >
> > > > >> > 0 means error
> > > > >> > 1 means prepare_conf
> > > > >> > 2 means http_req_call
> > > > >> >
> > > > >> > The data is serialized in capnproto, a binary serialization
> > format.
> > > > >> >
> > > > >> > capnproto is supported by many programming languages:
> > > > >> > https://capnproto.org/otherlang.html
> > > > >> >
> > > > >> > The advantages of choosing capnproto are.
> > > > >> > 1. focus on serialization performance
> > > > >> > 2. partial deserialization support, so that decode can be done
> > only
> > > > >> > when it is needed
> > > > >> >
> > > > >> > 2.4.2 Communication steps
> > > > >> >
> > > > >> > Each ext plugin will have the following configuration.
> > > > >> >
> > > > >> > ```
> > > > >> > {
> > > > >> > "conf": [
> > > > >> > {
> > > > >> > "name": "configuration name",
> > > > >> > "value": "configuration value"
> > > > >> > }
> > > > >> > ],
> > > > >> > "extra_info": [
> > > > >> > ...
> > > > >> > ]
> > > > >> > }
> > > > >> > ```
> > > > >> >
> > > > >> > conf can be used to set the execution configuration of the
> > > > >> > plugin-related requests inside Plugin Runner.
> > > > >> >
> > > > >> > The default data sent to Plugin Runner is only the most common
> > > > >> > information. If you want additional information, you need to
> > declare
> > > > >> > it in extra_info beforehand.
> > > > >> >
> > > > >> > To save communication costs, conf is sent separately.
> > > > >> >
> > > > >> > 1. APISIX will check if conf has a corresponding token in the
> > local
> > > > >> cache.
> > > > >> > 2. If not, APISIX sends a prepare_conf request to ask Plugin
> > Runner
> > > > >> > to cache the conf and return a token.
> > > > >> > (Note that Plugin Runner's cache time needs to be longer than
> > > APISIX's
> > > > >> > cache time.)
> > > > >> > 3. APISIX sends an http_req_call request to Plugin Runner.
> > > > >> > 4. Plugin Runner executes the request and returns a response to
> > > APISIX.
> > > > >> > 5. APISIX processes the request based on the response
> > > > >> >
> > > > >> > 2.4.3 proto
> > > > >> >
> > > > >> > Refer to https://capnproto.org/language.html
> > > > >> >
> > > > >> > The following is the proto for error
> > > > >> >
> > > > >> > response
> > > > >> > ```
> > > > >> > enum ErrorCode {
> > > > >> > BAD_REQUEST @0; # Plugin Runner can't understand APISIX
> > > > >> > SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the
> > request
> > > > >> > CONF_TOKEN_NOT_FOUND @2;
> > > > >> > }
> > > > >> > struct ErrorResp {
> > > > >> > Code @0 :ErrorCode;
> > > > >> > }
> > > > >> > ```
> > > > >> >
> > > > >> > The following is the proto of prepare_conf
> > > > >> >
> > > > >> > request
> > > > >> > ```
> > > > >> > struct PrepareConfReq {
> > > > >> > conf @0 :List(Pair);
> > > > >> > }
> > > > >> > ```
> > > > >> >
> > > > >> > Response
> > > > >> > ```
> > > > >> > struct PrepareConfResp {
> > > > >> > conf_token @0 :UInt32;
> > > > >> > }
> > > > >> > ```
> > > > >> >
> > > > >> > Here is the proto for http_req_call
> > > > >> >
> > > > >> > request
> > > > >> > ```
> > > > >> > struct Pair {
> > > > >> > name @0 :Text;
> > > > >> > value @1 :Text;
> > > > >> > }
> > > > >> > struct PairData {
> > > > >> > name @0 :Text;
> > > > >> > value @1 :Data;
> > > > >> > }
> > > > >> > enum Method {
> > > > >> > GET @0;
> > > > >> > ...
> > > > >> > }
> > > > >> >
> > > > >> > struct HTTPReqCallReq {
> > > > >> > id @0 :UInt32;
> > > > >> > src_ip @1 :Data;
> > > > >> >
> > > > >> > method @2 :Method;
> > > > >> >
> > > > >> > path @3 :Text;
> > > > >> > args @4 :List(Pair);
> > > > >> > headers @5 :List(Pair);
> > > > >> >
> > > > >> > conf_token @6 :UInt32;
> > > > >> >
> > > > >> > extra_info @7 :List(PairData);
> > > > >> > }
> > > > >> > ```
> > > > >> >
> > > > >> > Response
> > > > >> > ```
> > > > >> > struct HTTPReqCallResp {
> > > > >> > id @0 :UInt32;
> > > > >> >
> > > > >> > struct Stop {
> > > > >> > status @0 :UInt16;
> > > > >> > headers @1 :List(Pair);
> > > > >> > body @2 :Data;
> > > > >> > }
> > > > >> > struct Rewrite {
> > > > >> > path @0 :Text;
> > > > >> > headers @1 :List(Pair);
> > > > >> > # Note that args are modified in full.
> > > > >> > # Either empty, meaning no args need to be moved
> > > > >> > # or the entire modified args, not the incrementally
> > changed
> > > > >> parts
> > > > >> > args @2 :List(Pair);
> > > > >> > }
> > > > >> >
> > > > >> > # What needs to be done when the response is received action
> > > > >> > action :union {
> > > > >> > # Do nothing
> > > > >> > continue @1 :Void;
> > > > >> > # Equivalent to core.response.exit(status, body),
> allowing
> > > > >> > additional headers to be set
> > > > >> > stop @2 :Stop;
> > > > >> > # Rewrite the request
> > > > >> > rewrite @3 :Rewrite;
> > > > >> > }
> > > > >> > }
> > > > >> > ```
> > > > >> >
> > > > >> > 2.4.4 Error handling
> > > > >> >
> > > > >> > Logging and returning 503 error codes
> > > > >> >
> > > > >> > 2.4.5 Environment variables
> > > > >> >
> > > > >> > APISIX configures the Plugin Runner with a number of environment
> > > > >> > variables when it is started.
> > > > >> >
> > > > >> > APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs
> to
> > > > >> listen to
> > > > >> > APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for
> > > longer
> > > > >> than this
> > > > >>
> > > > >
> > >
> >
>
Re: [Proposal] support using other languages to write plugin for APISIX
Posted by ZhengSong Tu <tz...@gmail.com>.
I have some questions about the Java plugin runner to discuss.
Topic 1: Technical selection of communication protocols
java did not provide support for unix domain sockets until jdk16, but
obviously we will not be so aggressive in choosing jdk16, at this stage we
focus on jdk 8, so in order to implement java's unix domain sockets under
jdk8, we have two optional technology stacks
Option 1 - junixsocket, for more information please refer to the github
address: https://github.com/kohlschutter/junixsocket
Option 2-netty: For more information, please refer to github at
https://github.com/netty/netty
Before I was inclined to go for junixsocket because it is more pure, now I
am inclined to go for netty because it is more mature, I would like to know
your votes on this.
Topic 2: Does it depend on spring-boot?
Do we need to fully rely on the spring-boot framework as a base to
implement the java plugin runner in phase 1?
In terms of audience, spring-boot has a broader audience and is easier to
integrate with the components of the spring boot ecosystem, and it is
faster to build on spring-boot. What do you think?
Topic 3: Responsibilities of the SDK.
We should provide an SDK, which defines some base class objects and
function interfaces that users can use to write their own plugins based on
the SDK, similar to the filter form. Users need to package their own code
as a jar and declare the dependencies of their code in the configuration
file. So that java's plugin runner can load the user's typed jar at
startup. Is there anything that needs to be fixed in the above process?
Sheng Wu <wu...@gmail.com> 于2021年4月27日周二 上午10:00写道：
> OK, if it is a development kit, I want to know whether it is really a
> runtime? Because eventually, whether debugging is friendly or not, rely on
> this.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Daming <da...@apache.org> 于2021年4月27日周二 上午9:57写道：
>
> > PDK stands for Plugin Development Kit.
> > It is a good choice to build on spring-boot in the initial stage.
> > Because it is very friendly and easy to get started for Java
> > developers(plugins developers).
> >
> >
> > Sheng Wu <wu...@gmail.com> 于2021年4月27日周二 上午9:31写道：
> > >
> > > I can't see the detail from the graph.
> > > But if we are speaking of Java and Spring gateway ecosystem, you should
> > > know, that is not just interfaces.
> > > Spring boot, annotation system, IoC, AOP are all going to be there. So,
> > > basically, I would recommend you to boot the codes as all Spring-based
> > > applications did.
> > > BTW, I am not sure what PDK means.
> > >
> > > And about debugging, once your APISIX process could access the JVM
> > through
> > > a socket with manual localhost:port, it is fine for the Java developer
> to
> > > start this sidecar in the IDEA.
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > ZhengSong Tu <tz...@gmail.com> 于2021年4月26日周一 上午11:47写道：
> > >
> > > > I have sketched the flow of the java side of the plugin runner
> > > > [image: APISIX 运行 Java 插件.png]
> > > >
> > > > And I have some questions to clarify.
> > > >
> > > > 1. How does the plugin runner load the client's code?
> > > > It's up to each runner to figure it out according to its own
> > > > language's ecology. Take java as an example,
> > > > the user develops with the PDK provided by us, and packages the
> custom
> > > > plugin code into a jar and puts it in the specified location. When
> the
> > > > plugin runner starts, it loads the jar package in this location by
> > means of
> > > > a custom dynamic ClassLoader or something like that.
> > > >
> > > > 2. How do users develop?
> > > > Just provide the filter chain like Spring Cloud Gateway. Users
> can
> > > > register their own filters. And the PDK is a jar, with some defined
> > > > interfaces such as pre, post, etc., and some objects such as request,
> > > > header, etc.
> > > >
> > > > 3. How to debug？
> > > > Option 1: APISIX provides debugging mode for java plugins, this
> > time
> > > > it is not APISIX to start the plugin runner, but the user himself
> > downloads
> > > > the plugin runner code and runs the main method to start it, so as to
> > > > construct a request to access APISIX, APISIX will pass the context of
> > the
> > > > request to the plugin runner, and then use it to debug java code.
> > > > Option 2: java remote debug.
> > > >
> > > >
> > > >
> > > > Zexuan Luo <sp...@apache.org> 于2021年4月19日周一 下午6:18写道：
> > > >
> > > >> Here is the new flatbuffer schema:
> > > >>
> > > >>
> >
> https://github.com/spacewander/incubator-apisix/blob/step1/apisix/plugins/ext-plugin/ext-plugin.fbs
> > > >>
> > > >> Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午2:52写道：
> > > >> >
> > > >> > 1. Background
> > > >> >
> > > >> > APISIX currently only supports writing plugins in Lua. If other
> > > >> > languages are supported, it will greatly broaden the APISIX
> > ecosystem
> > > >> > and user base.
> > > >> >
> > > >> > 2. Solution
> > > >> >
> > > >> > Since WASM is not yet mature, we consider implementing it through
> > local
> > > >> IPC.
> > > >> >
> > > >> > For the sake of discussion, the following will assume that the
> > plugin
> > > >> > is written in Java. However, in practice, our solution can be
> > > >> > interfaced with other languages.
> > > >> >
> > > >> > 2.1 Terminology
> > > >> >
> > > >> > Several terms are defined here.
> > > >> >
> > > >> > Plugin Runner: The service that runs the plugin, written in the
> same
> > > >> > language as the plugin. In the first version, we assume that there
> > > >> > will be only one Plugin Runner.
> > > >> >
> > > >> > 2.2 Plugin Runner lifecycle
> > > >> >
> > > >> > To simplify user operation and reduce the difficulty of upgrading,
> > > >> > Plugin Runner is managed by APISIX.
> > > >> >
> > > >> > APISIX starts the Plugin Runner when it starts and ends it when it
> > > >> > ends. if the Plugin Runner quits in the middle, APISIX will
> restart
> > it
> > > >> > automatically.
> > > >> >
> > > >> > 2.3 Timing of APISIX communication with Plugin Runner
> > > >> >
> > > >> > ```
> > > >> > Router ----> Global Plugin (written in Lua) ---> Ext Plugin
> > > >> > (ext-plugin-pre-req) ----> Lua Plugin (Router)
> > > >> > ---> Ext plugin (ext-plugin-post-req) ---> Upstream
> > > >> > ```
> > > >> >
> > > >> > Running the Ext Plugin in the Global Plugin is not supported at
> this
> > > >> > time, as the global logic can be executed uniformly in the Plugin
> > > >> > Runner.
> > > >> >
> > > >> > Running Ext Plugin after getting an upstream response is not
> > supported
> > > >> > at this time. We can support it later with a buffering response.
> > > >> >
> > > >> > ext-plugin-pre runs before all non-global Lua plugins, and
> > > >> > ext-plugin-post runs after all non-global Lua plugins.
> > > >> >
> > > >> > 2.4 How APISIX communicates with Plugin Runner
> > > >> >
> > > >> > APISIX communicates with Plugin Runner through a unix socket. The
> > > >> > communication protocol is as follows.
> > > >> >
> > > >> > 2.4.1 Communication format
> > > >> >
> > > >> > ```
> > > >> > 1 byte of type + 3 bytes of length + data
> > > >> > ```
> > > >> >
> > > >> > The type can be 0 ~ 7, and the length can be [0, 8M). data length
> is
> > > >> > determined by length.
> > > >> > Since Ext Plugin usually does not exchange too much data, 8M
> should
> > be
> > > >> > enough. The reason for taking 4 bytes is to keep the header small
> > > >> > enough to be read efficiently.
> > > >> >
> > > >> > The current type takes the following values.
> > > >> >
> > > >> > 0 means error
> > > >> > 1 means prepare_conf
> > > >> > 2 means http_req_call
> > > >> >
> > > >> > The data is serialized in capnproto, a binary serialization
> format.
> > > >> >
> > > >> > capnproto is supported by many programming languages:
> > > >> > https://capnproto.org/otherlang.html
> > > >> >
> > > >> > The advantages of choosing capnproto are.
> > > >> > 1. focus on serialization performance
> > > >> > 2. partial deserialization support, so that decode can be done
> only
> > > >> > when it is needed
> > > >> >
> > > >> > 2.4.2 Communication steps
> > > >> >
> > > >> > Each ext plugin will have the following configuration.
> > > >> >
> > > >> > ```
> > > >> > {
> > > >> > "conf": [
> > > >> > {
> > > >> > "name": "configuration name",
> > > >> > "value": "configuration value"
> > > >> > }
> > > >> > ],
> > > >> > "extra_info": [
> > > >> > ...
> > > >> > ]
> > > >> > }
> > > >> > ```
> > > >> >
> > > >> > conf can be used to set the execution configuration of the
> > > >> > plugin-related requests inside Plugin Runner.
> > > >> >
> > > >> > The default data sent to Plugin Runner is only the most common
> > > >> > information. If you want additional information, you need to
> declare
> > > >> > it in extra_info beforehand.
> > > >> >
> > > >> > To save communication costs, conf is sent separately.
> > > >> >
> > > >> > 1. APISIX will check if conf has a corresponding token in the
> local
> > > >> cache.
> > > >> > 2. If not, APISIX sends a prepare_conf request to ask Plugin
> Runner
> > > >> > to cache the conf and return a token.
> > > >> > (Note that Plugin Runner's cache time needs to be longer than
> > APISIX's
> > > >> > cache time.)
> > > >> > 3. APISIX sends an http_req_call request to Plugin Runner.
> > > >> > 4. Plugin Runner executes the request and returns a response to
> > APISIX.
> > > >> > 5. APISIX processes the request based on the response
> > > >> >
> > > >> > 2.4.3 proto
> > > >> >
> > > >> > Refer to https://capnproto.org/language.html
> > > >> >
> > > >> > The following is the proto for error
> > > >> >
> > > >> > response
> > > >> > ```
> > > >> > enum ErrorCode {
> > > >> > BAD_REQUEST @0; # Plugin Runner can't understand APISIX
> > > >> > SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the
> request
> > > >> > CONF_TOKEN_NOT_FOUND @2;
> > > >> > }
> > > >> > struct ErrorResp {
> > > >> > Code @0 :ErrorCode;
> > > >> > }
> > > >> > ```
> > > >> >
> > > >> > The following is the proto of prepare_conf
> > > >> >
> > > >> > request
> > > >> > ```
> > > >> > struct PrepareConfReq {
> > > >> > conf @0 :List(Pair);
> > > >> > }
> > > >> > ```
> > > >> >
> > > >> > Response
> > > >> > ```
> > > >> > struct PrepareConfResp {
> > > >> > conf_token @0 :UInt32;
> > > >> > }
> > > >> > ```
> > > >> >
> > > >> > Here is the proto for http_req_call
> > > >> >
> > > >> > request
> > > >> > ```
> > > >> > struct Pair {
> > > >> > name @0 :Text;
> > > >> > value @1 :Text;
> > > >> > }
> > > >> > struct PairData {
> > > >> > name @0 :Text;
> > > >> > value @1 :Data;
> > > >> > }
> > > >> > enum Method {
> > > >> > GET @0;
> > > >> > ...
> > > >> > }
> > > >> >
> > > >> > struct HTTPReqCallReq {
> > > >> > id @0 :UInt32;
> > > >> > src_ip @1 :Data;
> > > >> >
> > > >> > method @2 :Method;
> > > >> >
> > > >> > path @3 :Text;
> > > >> > args @4 :List(Pair);
> > > >> > headers @5 :List(Pair);
> > > >> >
> > > >> > conf_token @6 :UInt32;
> > > >> >
> > > >> > extra_info @7 :List(PairData);
> > > >> > }
> > > >> > ```
> > > >> >
> > > >> > Response
> > > >> > ```
> > > >> > struct HTTPReqCallResp {
> > > >> > id @0 :UInt32;
> > > >> >
> > > >> > struct Stop {
> > > >> > status @0 :UInt16;
> > > >> > headers @1 :List(Pair);
> > > >> > body @2 :Data;
> > > >> > }
> > > >> > struct Rewrite {
> > > >> > path @0 :Text;
> > > >> > headers @1 :List(Pair);
> > > >> > # Note that args are modified in full.
> > > >> > # Either empty, meaning no args need to be moved
> > > >> > # or the entire modified args, not the incrementally
> changed
> > > >> parts
> > > >> > args @2 :List(Pair);
> > > >> > }
> > > >> >
> > > >> > # What needs to be done when the response is received action
> > > >> > action :union {
> > > >> > # Do nothing
> > > >> > continue @1 :Void;
> > > >> > # Equivalent to core.response.exit(status, body), allowing
> > > >> > additional headers to be set
> > > >> > stop @2 :Stop;
> > > >> > # Rewrite the request
> > > >> > rewrite @3 :Rewrite;
> > > >> > }
> > > >> > }
> > > >> > ```
> > > >> >
> > > >> > 2.4.4 Error handling
> > > >> >
> > > >> > Logging and returning 503 error codes
> > > >> >
> > > >> > 2.4.5 Environment variables
> > > >> >
> > > >> > APISIX configures the Plugin Runner with a number of environment
> > > >> > variables when it is started.
> > > >> >
> > > >> > APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to
> > > >> listen to
> > > >> > APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for
> > longer
> > > >> than this
> > > >>
> > > >
> >
>
Re: [Proposal] support using other languages to write plugin for APISIX
Posted by Sheng Wu <wu...@gmail.com>.
OK, if it is a development kit, I want to know whether it is really a
runtime? Because eventually, whether debugging is friendly or not, rely on
this.
Sheng Wu 吴晟
Twitter, wusheng1108
Daming <da...@apache.org> 于2021年4月27日周二 上午9:57写道：
> PDK stands for Plugin Development Kit.
> It is a good choice to build on spring-boot in the initial stage.
> Because it is very friendly and easy to get started for Java
> developers(plugins developers).
>
>
> Sheng Wu <wu...@gmail.com> 于2021年4月27日周二 上午9:31写道：
> >
> > I can't see the detail from the graph.
> > But if we are speaking of Java and Spring gateway ecosystem, you should
> > know, that is not just interfaces.
> > Spring boot, annotation system, IoC, AOP are all going to be there. So,
> > basically, I would recommend you to boot the codes as all Spring-based
> > applications did.
> > BTW, I am not sure what PDK means.
> >
> > And about debugging, once your APISIX process could access the JVM
> through
> > a socket with manual localhost:port, it is fine for the Java developer to
> > start this sidecar in the IDEA.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > ZhengSong Tu <tz...@gmail.com> 于2021年4月26日周一 上午11:47写道：
> >
> > > I have sketched the flow of the java side of the plugin runner
> > > [image: APISIX 运行 Java 插件.png]
> > >
> > > And I have some questions to clarify.
> > >
> > > 1. How does the plugin runner load the client's code?
> > > It's up to each runner to figure it out according to its own
> > > language's ecology. Take java as an example,
> > > the user develops with the PDK provided by us, and packages the custom
> > > plugin code into a jar and puts it in the specified location. When the
> > > plugin runner starts, it loads the jar package in this location by
> means of
> > > a custom dynamic ClassLoader or something like that.
> > >
> > > 2. How do users develop?
> > > Just provide the filter chain like Spring Cloud Gateway. Users can
> > > register their own filters. And the PDK is a jar, with some defined
> > > interfaces such as pre, post, etc., and some objects such as request,
> > > header, etc.
> > >
> > > 3. How to debug？
> > > Option 1: APISIX provides debugging mode for java plugins, this
> time
> > > it is not APISIX to start the plugin runner, but the user himself
> downloads
> > > the plugin runner code and runs the main method to start it, so as to
> > > construct a request to access APISIX, APISIX will pass the context of
> the
> > > request to the plugin runner, and then use it to debug java code.
> > > Option 2: java remote debug.
> > >
> > >
> > >
> > > Zexuan Luo <sp...@apache.org> 于2021年4月19日周一 下午6:18写道：
> > >
> > >> Here is the new flatbuffer schema:
> > >>
> > >>
> https://github.com/spacewander/incubator-apisix/blob/step1/apisix/plugins/ext-plugin/ext-plugin.fbs
> > >>
> > >> Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午2:52写道：
> > >> >
> > >> > 1. Background
> > >> >
> > >> > APISIX currently only supports writing plugins in Lua. If other
> > >> > languages are supported, it will greatly broaden the APISIX
> ecosystem
> > >> > and user base.
> > >> >
> > >> > 2. Solution
> > >> >
> > >> > Since WASM is not yet mature, we consider implementing it through
> local
> > >> IPC.
> > >> >
> > >> > For the sake of discussion, the following will assume that the
> plugin
> > >> > is written in Java. However, in practice, our solution can be
> > >> > interfaced with other languages.
> > >> >
> > >> > 2.1 Terminology
> > >> >
> > >> > Several terms are defined here.
> > >> >
> > >> > Plugin Runner: The service that runs the plugin, written in the same
> > >> > language as the plugin. In the first version, we assume that there
> > >> > will be only one Plugin Runner.
> > >> >
> > >> > 2.2 Plugin Runner lifecycle
> > >> >
> > >> > To simplify user operation and reduce the difficulty of upgrading,
> > >> > Plugin Runner is managed by APISIX.
> > >> >
> > >> > APISIX starts the Plugin Runner when it starts and ends it when it
> > >> > ends. if the Plugin Runner quits in the middle, APISIX will restart
> it
> > >> > automatically.
> > >> >
> > >> > 2.3 Timing of APISIX communication with Plugin Runner
> > >> >
> > >> > ```
> > >> > Router ----> Global Plugin (written in Lua) ---> Ext Plugin
> > >> > (ext-plugin-pre-req) ----> Lua Plugin (Router)
> > >> > ---> Ext plugin (ext-plugin-post-req) ---> Upstream
> > >> > ```
> > >> >
> > >> > Running the Ext Plugin in the Global Plugin is not supported at this
> > >> > time, as the global logic can be executed uniformly in the Plugin
> > >> > Runner.
> > >> >
> > >> > Running Ext Plugin after getting an upstream response is not
> supported
> > >> > at this time. We can support it later with a buffering response.
> > >> >
> > >> > ext-plugin-pre runs before all non-global Lua plugins, and
> > >> > ext-plugin-post runs after all non-global Lua plugins.
> > >> >
> > >> > 2.4 How APISIX communicates with Plugin Runner
> > >> >
> > >> > APISIX communicates with Plugin Runner through a unix socket. The
> > >> > communication protocol is as follows.
> > >> >
> > >> > 2.4.1 Communication format
> > >> >
> > >> > ```
> > >> > 1 byte of type + 3 bytes of length + data
> > >> > ```
> > >> >
> > >> > The type can be 0 ~ 7, and the length can be [0, 8M). data length is
> > >> > determined by length.
> > >> > Since Ext Plugin usually does not exchange too much data, 8M should
> be
> > >> > enough. The reason for taking 4 bytes is to keep the header small
> > >> > enough to be read efficiently.
> > >> >
> > >> > The current type takes the following values.
> > >> >
> > >> > 0 means error
> > >> > 1 means prepare_conf
> > >> > 2 means http_req_call
> > >> >
> > >> > The data is serialized in capnproto, a binary serialization format.
> > >> >
> > >> > capnproto is supported by many programming languages:
> > >> > https://capnproto.org/otherlang.html
> > >> >
> > >> > The advantages of choosing capnproto are.
> > >> > 1. focus on serialization performance
> > >> > 2. partial deserialization support, so that decode can be done only
> > >> > when it is needed
> > >> >
> > >> > 2.4.2 Communication steps
> > >> >
> > >> > Each ext plugin will have the following configuration.
> > >> >
> > >> > ```
> > >> > {
> > >> > "conf": [
> > >> > {
> > >> > "name": "configuration name",
> > >> > "value": "configuration value"
> > >> > }
> > >> > ],
> > >> > "extra_info": [
> > >> > ...
> > >> > ]
> > >> > }
> > >> > ```
> > >> >
> > >> > conf can be used to set the execution configuration of the
> > >> > plugin-related requests inside Plugin Runner.
> > >> >
> > >> > The default data sent to Plugin Runner is only the most common
> > >> > information. If you want additional information, you need to declare
> > >> > it in extra_info beforehand.
> > >> >
> > >> > To save communication costs, conf is sent separately.
> > >> >
> > >> > 1. APISIX will check if conf has a corresponding token in the local
> > >> cache.
> > >> > 2. If not, APISIX sends a prepare_conf request to ask Plugin Runner
> > >> > to cache the conf and return a token.
> > >> > (Note that Plugin Runner's cache time needs to be longer than
> APISIX's
> > >> > cache time.)
> > >> > 3. APISIX sends an http_req_call request to Plugin Runner.
> > >> > 4. Plugin Runner executes the request and returns a response to
> APISIX.
> > >> > 5. APISIX processes the request based on the response
> > >> >
> > >> > 2.4.3 proto
> > >> >
> > >> > Refer to https://capnproto.org/language.html
> > >> >
> > >> > The following is the proto for error
> > >> >
> > >> > response
> > >> > ```
> > >> > enum ErrorCode {
> > >> > BAD_REQUEST @0; # Plugin Runner can't understand APISIX
> > >> > SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the request
> > >> > CONF_TOKEN_NOT_FOUND @2;
> > >> > }
> > >> > struct ErrorResp {
> > >> > Code @0 :ErrorCode;
> > >> > }
> > >> > ```
> > >> >
> > >> > The following is the proto of prepare_conf
> > >> >
> > >> > request
> > >> > ```
> > >> > struct PrepareConfReq {
> > >> > conf @0 :List(Pair);
> > >> > }
> > >> > ```
> > >> >
> > >> > Response
> > >> > ```
> > >> > struct PrepareConfResp {
> > >> > conf_token @0 :UInt32;
> > >> > }
> > >> > ```
> > >> >
> > >> > Here is the proto for http_req_call
> > >> >
> > >> > request
> > >> > ```
> > >> > struct Pair {
> > >> > name @0 :Text;
> > >> > value @1 :Text;
> > >> > }
> > >> > struct PairData {
> > >> > name @0 :Text;
> > >> > value @1 :Data;
> > >> > }
> > >> > enum Method {
> > >> > GET @0;
> > >> > ...
> > >> > }
> > >> >
> > >> > struct HTTPReqCallReq {
> > >> > id @0 :UInt32;
> > >> > src_ip @1 :Data;
> > >> >
> > >> > method @2 :Method;
> > >> >
> > >> > path @3 :Text;
> > >> > args @4 :List(Pair);
> > >> > headers @5 :List(Pair);
> > >> >
> > >> > conf_token @6 :UInt32;
> > >> >
> > >> > extra_info @7 :List(PairData);
> > >> > }
> > >> > ```
> > >> >
> > >> > Response
> > >> > ```
> > >> > struct HTTPReqCallResp {
> > >> > id @0 :UInt32;
> > >> >
> > >> > struct Stop {
> > >> > status @0 :UInt16;
> > >> > headers @1 :List(Pair);
> > >> > body @2 :Data;
> > >> > }
> > >> > struct Rewrite {
> > >> > path @0 :Text;
> > >> > headers @1 :List(Pair);
> > >> > # Note that args are modified in full.
> > >> > # Either empty, meaning no args need to be moved
> > >> > # or the entire modified args, not the incrementally changed
> > >> parts
> > >> > args @2 :List(Pair);
> > >> > }
> > >> >
> > >> > # What needs to be done when the response is received action
> > >> > action :union {
> > >> > # Do nothing
> > >> > continue @1 :Void;
> > >> > # Equivalent to core.response.exit(status, body), allowing
> > >> > additional headers to be set
> > >> > stop @2 :Stop;
> > >> > # Rewrite the request
> > >> > rewrite @3 :Rewrite;
> > >> > }
> > >> > }
> > >> > ```
> > >> >
> > >> > 2.4.4 Error handling
> > >> >
> > >> > Logging and returning 503 error codes
> > >> >
> > >> > 2.4.5 Environment variables
> > >> >
> > >> > APISIX configures the Plugin Runner with a number of environment
> > >> > variables when it is started.
> > >> >
> > >> > APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to
> > >> listen to
> > >> > APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for
> longer
> > >> than this
> > >>
> > >
>
Re: [Proposal] support using other languages to write plugin for APISIX
Posted by Daming <da...@apache.org>.
PDK stands for Plugin Development Kit.
It is a good choice to build on spring-boot in the initial stage.
Because it is very friendly and easy to get started for Java
developers(plugins developers).
Sheng Wu <wu...@gmail.com> 于2021年4月27日周二 上午9:31写道：
>
> I can't see the detail from the graph.
> But if we are speaking of Java and Spring gateway ecosystem, you should
> know, that is not just interfaces.
> Spring boot, annotation system, IoC, AOP are all going to be there. So,
> basically, I would recommend you to boot the codes as all Spring-based
> applications did.
> BTW, I am not sure what PDK means.
>
> And about debugging, once your APISIX process could access the JVM through
> a socket with manual localhost:port, it is fine for the Java developer to
> start this sidecar in the IDEA.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> ZhengSong Tu <tz...@gmail.com> 于2021年4月26日周一 上午11:47写道：
>
> > I have sketched the flow of the java side of the plugin runner
> > [image: APISIX 运行 Java 插件.png]
> >
> > And I have some questions to clarify.
> >
> > 1. How does the plugin runner load the client's code?
> > It's up to each runner to figure it out according to its own
> > language's ecology. Take java as an example,
> > the user develops with the PDK provided by us, and packages the custom
> > plugin code into a jar and puts it in the specified location. When the
> > plugin runner starts, it loads the jar package in this location by means of
> > a custom dynamic ClassLoader or something like that.
> >
> > 2. How do users develop?
> > Just provide the filter chain like Spring Cloud Gateway. Users can
> > register their own filters. And the PDK is a jar, with some defined
> > interfaces such as pre, post, etc., and some objects such as request,
> > header, etc.
> >
> > 3. How to debug？
> > Option 1: APISIX provides debugging mode for java plugins, this time
> > it is not APISIX to start the plugin runner, but the user himself downloads
> > the plugin runner code and runs the main method to start it, so as to
> > construct a request to access APISIX, APISIX will pass the context of the
> > request to the plugin runner, and then use it to debug java code.
> > Option 2: java remote debug.
> >
> >
> >
> > Zexuan Luo <sp...@apache.org> 于2021年4月19日周一 下午6:18写道：
> >
> >> Here is the new flatbuffer schema:
> >>
> >> https://github.com/spacewander/incubator-apisix/blob/step1/apisix/plugins/ext-plugin/ext-plugin.fbs
> >>
> >> Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午2:52写道：
> >> >
> >> > 1. Background
> >> >
> >> > APISIX currently only supports writing plugins in Lua. If other
> >> > languages are supported, it will greatly broaden the APISIX ecosystem
> >> > and user base.
> >> >
> >> > 2. Solution
> >> >
> >> > Since WASM is not yet mature, we consider implementing it through local
> >> IPC.
> >> >
> >> > For the sake of discussion, the following will assume that the plugin
> >> > is written in Java. However, in practice, our solution can be
> >> > interfaced with other languages.
> >> >
> >> > 2.1 Terminology
> >> >
> >> > Several terms are defined here.
> >> >
> >> > Plugin Runner: The service that runs the plugin, written in the same
> >> > language as the plugin. In the first version, we assume that there
> >> > will be only one Plugin Runner.
> >> >
> >> > 2.2 Plugin Runner lifecycle
> >> >
> >> > To simplify user operation and reduce the difficulty of upgrading,
> >> > Plugin Runner is managed by APISIX.
> >> >
> >> > APISIX starts the Plugin Runner when it starts and ends it when it
> >> > ends. if the Plugin Runner quits in the middle, APISIX will restart it
> >> > automatically.
> >> >
> >> > 2.3 Timing of APISIX communication with Plugin Runner
> >> >
> >> > ```
> >> > Router ----> Global Plugin (written in Lua) ---> Ext Plugin
> >> > (ext-plugin-pre-req) ----> Lua Plugin (Router)
> >> > ---> Ext plugin (ext-plugin-post-req) ---> Upstream
> >> > ```
> >> >
> >> > Running the Ext Plugin in the Global Plugin is not supported at this
> >> > time, as the global logic can be executed uniformly in the Plugin
> >> > Runner.
> >> >
> >> > Running Ext Plugin after getting an upstream response is not supported
> >> > at this time. We can support it later with a buffering response.
> >> >
> >> > ext-plugin-pre runs before all non-global Lua plugins, and
> >> > ext-plugin-post runs after all non-global Lua plugins.
> >> >
> >> > 2.4 How APISIX communicates with Plugin Runner
> >> >
> >> > APISIX communicates with Plugin Runner through a unix socket. The
> >> > communication protocol is as follows.
> >> >
> >> > 2.4.1 Communication format
> >> >
> >> > ```
> >> > 1 byte of type + 3 bytes of length + data
> >> > ```
> >> >
> >> > The type can be 0 ~ 7, and the length can be [0, 8M). data length is
> >> > determined by length.
> >> > Since Ext Plugin usually does not exchange too much data, 8M should be
> >> > enough. The reason for taking 4 bytes is to keep the header small
> >> > enough to be read efficiently.
> >> >
> >> > The current type takes the following values.
> >> >
> >> > 0 means error
> >> > 1 means prepare_conf
> >> > 2 means http_req_call
> >> >
> >> > The data is serialized in capnproto, a binary serialization format.
> >> >
> >> > capnproto is supported by many programming languages:
> >> > https://capnproto.org/otherlang.html
> >> >
> >> > The advantages of choosing capnproto are.
> >> > 1. focus on serialization performance
> >> > 2. partial deserialization support, so that decode can be done only
> >> > when it is needed
> >> >
> >> > 2.4.2 Communication steps
> >> >
> >> > Each ext plugin will have the following configuration.
> >> >
> >> > ```
> >> > {
> >> > "conf": [
> >> > {
> >> > "name": "configuration name",
> >> > "value": "configuration value"
> >> > }
> >> > ],
> >> > "extra_info": [
> >> > ...
> >> > ]
> >> > }
> >> > ```
> >> >
> >> > conf can be used to set the execution configuration of the
> >> > plugin-related requests inside Plugin Runner.
> >> >
> >> > The default data sent to Plugin Runner is only the most common
> >> > information. If you want additional information, you need to declare
> >> > it in extra_info beforehand.
> >> >
> >> > To save communication costs, conf is sent separately.
> >> >
> >> > 1. APISIX will check if conf has a corresponding token in the local
> >> cache.
> >> > 2. If not, APISIX sends a prepare_conf request to ask Plugin Runner
> >> > to cache the conf and return a token.
> >> > (Note that Plugin Runner's cache time needs to be longer than APISIX's
> >> > cache time.)
> >> > 3. APISIX sends an http_req_call request to Plugin Runner.
> >> > 4. Plugin Runner executes the request and returns a response to APISIX.
> >> > 5. APISIX processes the request based on the response
> >> >
> >> > 2.4.3 proto
> >> >
> >> > Refer to https://capnproto.org/language.html
> >> >
> >> > The following is the proto for error
> >> >
> >> > response
> >> > ```
> >> > enum ErrorCode {
> >> > BAD_REQUEST @0; # Plugin Runner can't understand APISIX
> >> > SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the request
> >> > CONF_TOKEN_NOT_FOUND @2;
> >> > }
> >> > struct ErrorResp {
> >> > Code @0 :ErrorCode;
> >> > }
> >> > ```
> >> >
> >> > The following is the proto of prepare_conf
> >> >
> >> > request
> >> > ```
> >> > struct PrepareConfReq {
> >> > conf @0 :List(Pair);
> >> > }
> >> > ```
> >> >
> >> > Response
> >> > ```
> >> > struct PrepareConfResp {
> >> > conf_token @0 :UInt32;
> >> > }
> >> > ```
> >> >
> >> > Here is the proto for http_req_call
> >> >
> >> > request
> >> > ```
> >> > struct Pair {
> >> > name @0 :Text;
> >> > value @1 :Text;
> >> > }
> >> > struct PairData {
> >> > name @0 :Text;
> >> > value @1 :Data;
> >> > }
> >> > enum Method {
> >> > GET @0;
> >> > ...
> >> > }
> >> >
> >> > struct HTTPReqCallReq {
> >> > id @0 :UInt32;
> >> > src_ip @1 :Data;
> >> >
> >> > method @2 :Method;
> >> >
> >> > path @3 :Text;
> >> > args @4 :List(Pair);
> >> > headers @5 :List(Pair);
> >> >
> >> > conf_token @6 :UInt32;
> >> >
> >> > extra_info @7 :List(PairData);
> >> > }
> >> > ```
> >> >
> >> > Response
> >> > ```
> >> > struct HTTPReqCallResp {
> >> > id @0 :UInt32;
> >> >
> >> > struct Stop {
> >> > status @0 :UInt16;
> >> > headers @1 :List(Pair);
> >> > body @2 :Data;
> >> > }
> >> > struct Rewrite {
> >> > path @0 :Text;
> >> > headers @1 :List(Pair);
> >> > # Note that args are modified in full.
> >> > # Either empty, meaning no args need to be moved
> >> > # or the entire modified args, not the incrementally changed
> >> parts
> >> > args @2 :List(Pair);
> >> > }
> >> >
> >> > # What needs to be done when the response is received action
> >> > action :union {
> >> > # Do nothing
> >> > continue @1 :Void;
> >> > # Equivalent to core.response.exit(status, body), allowing
> >> > additional headers to be set
> >> > stop @2 :Stop;
> >> > # Rewrite the request
> >> > rewrite @3 :Rewrite;
> >> > }
> >> > }
> >> > ```
> >> >
> >> > 2.4.4 Error handling
> >> >
> >> > Logging and returning 503 error codes
> >> >
> >> > 2.4.5 Environment variables
> >> >
> >> > APISIX configures the Plugin Runner with a number of environment
> >> > variables when it is started.
> >> >
> >> > APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to
> >> listen to
> >> > APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for longer
> >> than this
> >>
> >
Re: [Proposal] support using other languages to write plugin for APISIX
Posted by Sheng Wu <wu...@gmail.com>.
I can't see the detail from the graph.
But if we are speaking of Java and Spring gateway ecosystem, you should
know, that is not just interfaces.
Spring boot, annotation system, IoC, AOP are all going to be there. So,
basically, I would recommend you to boot the codes as all Spring-based
applications did.
BTW, I am not sure what PDK means.
And about debugging, once your APISIX process could access the JVM through
a socket with manual localhost:port, it is fine for the Java developer to
start this sidecar in the IDEA.
Sheng Wu 吴晟
Twitter, wusheng1108
ZhengSong Tu <tz...@gmail.com> 于2021年4月26日周一 上午11:47写道：
> I have sketched the flow of the java side of the plugin runner
> [image: APISIX 运行 Java 插件.png]
>
> And I have some questions to clarify.
>
> 1. How does the plugin runner load the client's code?
> It's up to each runner to figure it out according to its own
> language's ecology. Take java as an example,
> the user develops with the PDK provided by us, and packages the custom
> plugin code into a jar and puts it in the specified location. When the
> plugin runner starts, it loads the jar package in this location by means of
> a custom dynamic ClassLoader or something like that.
>
> 2. How do users develop?
> Just provide the filter chain like Spring Cloud Gateway. Users can
> register their own filters. And the PDK is a jar, with some defined
> interfaces such as pre, post, etc., and some objects such as request,
> header, etc.
>
> 3. How to debug？
> Option 1: APISIX provides debugging mode for java plugins, this time
> it is not APISIX to start the plugin runner, but the user himself downloads
> the plugin runner code and runs the main method to start it, so as to
> construct a request to access APISIX, APISIX will pass the context of the
> request to the plugin runner, and then use it to debug java code.
> Option 2: java remote debug.
>
>
>
> Zexuan Luo <sp...@apache.org> 于2021年4月19日周一 下午6:18写道：
>
>> Here is the new flatbuffer schema:
>>
>> https://github.com/spacewander/incubator-apisix/blob/step1/apisix/plugins/ext-plugin/ext-plugin.fbs
>>
>> Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午2:52写道：
>> >
>> > 1. Background
>> >
>> > APISIX currently only supports writing plugins in Lua. If other
>> > languages are supported, it will greatly broaden the APISIX ecosystem
>> > and user base.
>> >
>> > 2. Solution
>> >
>> > Since WASM is not yet mature, we consider implementing it through local
>> IPC.
>> >
>> > For the sake of discussion, the following will assume that the plugin
>> > is written in Java. However, in practice, our solution can be
>> > interfaced with other languages.
>> >
>> > 2.1 Terminology
>> >
>> > Several terms are defined here.
>> >
>> > Plugin Runner: The service that runs the plugin, written in the same
>> > language as the plugin. In the first version, we assume that there
>> > will be only one Plugin Runner.
>> >
>> > 2.2 Plugin Runner lifecycle
>> >
>> > To simplify user operation and reduce the difficulty of upgrading,
>> > Plugin Runner is managed by APISIX.
>> >
>> > APISIX starts the Plugin Runner when it starts and ends it when it
>> > ends. if the Plugin Runner quits in the middle, APISIX will restart it
>> > automatically.
>> >
>> > 2.3 Timing of APISIX communication with Plugin Runner
>> >
>> > ```
>> > Router ----> Global Plugin (written in Lua) ---> Ext Plugin
>> > (ext-plugin-pre-req) ----> Lua Plugin (Router)
>> > ---> Ext plugin (ext-plugin-post-req) ---> Upstream
>> > ```
>> >
>> > Running the Ext Plugin in the Global Plugin is not supported at this
>> > time, as the global logic can be executed uniformly in the Plugin
>> > Runner.
>> >
>> > Running Ext Plugin after getting an upstream response is not supported
>> > at this time. We can support it later with a buffering response.
>> >
>> > ext-plugin-pre runs before all non-global Lua plugins, and
>> > ext-plugin-post runs after all non-global Lua plugins.
>> >
>> > 2.4 How APISIX communicates with Plugin Runner
>> >
>> > APISIX communicates with Plugin Runner through a unix socket. The
>> > communication protocol is as follows.
>> >
>> > 2.4.1 Communication format
>> >
>> > ```
>> > 1 byte of type + 3 bytes of length + data
>> > ```
>> >
>> > The type can be 0 ~ 7, and the length can be [0, 8M). data length is
>> > determined by length.
>> > Since Ext Plugin usually does not exchange too much data, 8M should be
>> > enough. The reason for taking 4 bytes is to keep the header small
>> > enough to be read efficiently.
>> >
>> > The current type takes the following values.
>> >
>> > 0 means error
>> > 1 means prepare_conf
>> > 2 means http_req_call
>> >
>> > The data is serialized in capnproto, a binary serialization format.
>> >
>> > capnproto is supported by many programming languages:
>> > https://capnproto.org/otherlang.html
>> >
>> > The advantages of choosing capnproto are.
>> > 1. focus on serialization performance
>> > 2. partial deserialization support, so that decode can be done only
>> > when it is needed
>> >
>> > 2.4.2 Communication steps
>> >
>> > Each ext plugin will have the following configuration.
>> >
>> > ```
>> > {
>> > "conf": [
>> > {
>> > "name": "configuration name",
>> > "value": "configuration value"
>> > }
>> > ],
>> > "extra_info": [
>> > ...
>> > ]
>> > }
>> > ```
>> >
>> > conf can be used to set the execution configuration of the
>> > plugin-related requests inside Plugin Runner.
>> >
>> > The default data sent to Plugin Runner is only the most common
>> > information. If you want additional information, you need to declare
>> > it in extra_info beforehand.
>> >
>> > To save communication costs, conf is sent separately.
>> >
>> > 1. APISIX will check if conf has a corresponding token in the local
>> cache.
>> > 2. If not, APISIX sends a prepare_conf request to ask Plugin Runner
>> > to cache the conf and return a token.
>> > (Note that Plugin Runner's cache time needs to be longer than APISIX's
>> > cache time.)
>> > 3. APISIX sends an http_req_call request to Plugin Runner.
>> > 4. Plugin Runner executes the request and returns a response to APISIX.
>> > 5. APISIX processes the request based on the response
>> >
>> > 2.4.3 proto
>> >
>> > Refer to https://capnproto.org/language.html
>> >
>> > The following is the proto for error
>> >
>> > response
>> > ```
>> > enum ErrorCode {
>> > BAD_REQUEST @0; # Plugin Runner can't understand APISIX
>> > SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the request
>> > CONF_TOKEN_NOT_FOUND @2;
>> > }
>> > struct ErrorResp {
>> > Code @0 :ErrorCode;
>> > }
>> > ```
>> >
>> > The following is the proto of prepare_conf
>> >
>> > request
>> > ```
>> > struct PrepareConfReq {
>> > conf @0 :List(Pair);
>> > }
>> > ```
>> >
>> > Response
>> > ```
>> > struct PrepareConfResp {
>> > conf_token @0 :UInt32;
>> > }
>> > ```
>> >
>> > Here is the proto for http_req_call
>> >
>> > request
>> > ```
>> > struct Pair {
>> > name @0 :Text;
>> > value @1 :Text;
>> > }
>> > struct PairData {
>> > name @0 :Text;
>> > value @1 :Data;
>> > }
>> > enum Method {
>> > GET @0;
>> > ...
>> > }
>> >
>> > struct HTTPReqCallReq {
>> > id @0 :UInt32;
>> > src_ip @1 :Data;
>> >
>> > method @2 :Method;
>> >
>> > path @3 :Text;
>> > args @4 :List(Pair);
>> > headers @5 :List(Pair);
>> >
>> > conf_token @6 :UInt32;
>> >
>> > extra_info @7 :List(PairData);
>> > }
>> > ```
>> >
>> > Response
>> > ```
>> > struct HTTPReqCallResp {
>> > id @0 :UInt32;
>> >
>> > struct Stop {
>> > status @0 :UInt16;
>> > headers @1 :List(Pair);
>> > body @2 :Data;
>> > }
>> > struct Rewrite {
>> > path @0 :Text;
>> > headers @1 :List(Pair);
>> > # Note that args are modified in full.
>> > # Either empty, meaning no args need to be moved
>> > # or the entire modified args, not the incrementally changed
>> parts
>> > args @2 :List(Pair);
>> > }
>> >
>> > # What needs to be done when the response is received action
>> > action :union {
>> > # Do nothing
>> > continue @1 :Void;
>> > # Equivalent to core.response.exit(status, body), allowing
>> > additional headers to be set
>> > stop @2 :Stop;
>> > # Rewrite the request
>> > rewrite @3 :Rewrite;
>> > }
>> > }
>> > ```
>> >
>> > 2.4.4 Error handling
>> >
>> > Logging and returning 503 error codes
>> >
>> > 2.4.5 Environment variables
>> >
>> > APISIX configures the Plugin Runner with a number of environment
>> > variables when it is started.
>> >
>> > APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to
>> listen to
>> > APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for longer
>> than this
>>
>
Re: [Proposal] support using other languages to write plugin for APISIX
Posted by ZhengSong Tu <tz...@gmail.com>.
ok, I publish the image.
image link: https://www.processon.com/view/link/608651fa1e08537c0d341365
Zhiyuan Ju <ju...@apache.org> 于2021年4月26日周一 上午11:53写道：
> Hi, ZhengSong,
>
> You image is broken due to we could only use plain text in mailing list :)
>
> Best Regards!
> @ Zhiyuan Ju <https://github.com/juzhiyuan>
>
>
> ZhengSong Tu <tz...@gmail.com> 于2021年4月26日周一 上午11:47写道：
>
> > I have sketched the flow of the java side of the plugin runner
> > [image: APISIX 运行 Java 插件.png]
> >
> > And I have some questions to clarify.
> >
> > 1. How does the plugin runner load the client's code?
> > It's up to each runner to figure it out according to its own
> > language's ecology. Take java as an example,
> > the user develops with the PDK provided by us, and packages the custom
> > plugin code into a jar and puts it in the specified location. When the
> > plugin runner starts, it loads the jar package in this location by means
> of
> > a custom dynamic ClassLoader or something like that.
> >
> > 2. How do users develop?
> > Just provide the filter chain like Spring Cloud Gateway. Users can
> > register their own filters. And the PDK is a jar, with some defined
> > interfaces such as pre, post, etc., and some objects such as request,
> > header, etc.
> >
> > 3. How to debug？
> > Option 1: APISIX provides debugging mode for java plugins, this time
> > it is not APISIX to start the plugin runner, but the user himself
> downloads
> > the plugin runner code and runs the main method to start it, so as to
> > construct a request to access APISIX, APISIX will pass the context of the
> > request to the plugin runner, and then use it to debug java code.
> > Option 2: java remote debug.
> >
> >
> >
> > Zexuan Luo <sp...@apache.org> 于2021年4月19日周一 下午6:18写道：
> >
> >> Here is the new flatbuffer schema:
> >>
> >>
> https://github.com/spacewander/incubator-apisix/blob/step1/apisix/plugins/ext-plugin/ext-plugin.fbs
> >>
> >> Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午2:52写道：
> >> >
> >> > 1. Background
> >> >
> >> > APISIX currently only supports writing plugins in Lua. If other
> >> > languages are supported, it will greatly broaden the APISIX ecosystem
> >> > and user base.
> >> >
> >> > 2. Solution
> >> >
> >> > Since WASM is not yet mature, we consider implementing it through
> local
> >> IPC.
> >> >
> >> > For the sake of discussion, the following will assume that the plugin
> >> > is written in Java. However, in practice, our solution can be
> >> > interfaced with other languages.
> >> >
> >> > 2.1 Terminology
> >> >
> >> > Several terms are defined here.
> >> >
> >> > Plugin Runner: The service that runs the plugin, written in the same
> >> > language as the plugin. In the first version, we assume that there
> >> > will be only one Plugin Runner.
> >> >
> >> > 2.2 Plugin Runner lifecycle
> >> >
> >> > To simplify user operation and reduce the difficulty of upgrading,
> >> > Plugin Runner is managed by APISIX.
> >> >
> >> > APISIX starts the Plugin Runner when it starts and ends it when it
> >> > ends. if the Plugin Runner quits in the middle, APISIX will restart it
> >> > automatically.
> >> >
> >> > 2.3 Timing of APISIX communication with Plugin Runner
> >> >
> >> > ```
> >> > Router ----> Global Plugin (written in Lua) ---> Ext Plugin
> >> > (ext-plugin-pre-req) ----> Lua Plugin (Router)
> >> > ---> Ext plugin (ext-plugin-post-req) ---> Upstream
> >> > ```
> >> >
> >> > Running the Ext Plugin in the Global Plugin is not supported at this
> >> > time, as the global logic can be executed uniformly in the Plugin
> >> > Runner.
> >> >
> >> > Running Ext Plugin after getting an upstream response is not supported
> >> > at this time. We can support it later with a buffering response.
> >> >
> >> > ext-plugin-pre runs before all non-global Lua plugins, and
> >> > ext-plugin-post runs after all non-global Lua plugins.
> >> >
> >> > 2.4 How APISIX communicates with Plugin Runner
> >> >
> >> > APISIX communicates with Plugin Runner through a unix socket. The
> >> > communication protocol is as follows.
> >> >
> >> > 2.4.1 Communication format
> >> >
> >> > ```
> >> > 1 byte of type + 3 bytes of length + data
> >> > ```
> >> >
> >> > The type can be 0 ~ 7, and the length can be [0, 8M). data length is
> >> > determined by length.
> >> > Since Ext Plugin usually does not exchange too much data, 8M should be
> >> > enough. The reason for taking 4 bytes is to keep the header small
> >> > enough to be read efficiently.
> >> >
> >> > The current type takes the following values.
> >> >
> >> > 0 means error
> >> > 1 means prepare_conf
> >> > 2 means http_req_call
> >> >
> >> > The data is serialized in capnproto, a binary serialization format.
> >> >
> >> > capnproto is supported by many programming languages:
> >> > https://capnproto.org/otherlang.html
> >> >
> >> > The advantages of choosing capnproto are.
> >> > 1. focus on serialization performance
> >> > 2. partial deserialization support, so that decode can be done only
> >> > when it is needed
> >> >
> >> > 2.4.2 Communication steps
> >> >
> >> > Each ext plugin will have the following configuration.
> >> >
> >> > ```
> >> > {
> >> > "conf": [
> >> > {
> >> > "name": "configuration name",
> >> > "value": "configuration value"
> >> > }
> >> > ],
> >> > "extra_info": [
> >> > ...
> >> > ]
> >> > }
> >> > ```
> >> >
> >> > conf can be used to set the execution configuration of the
> >> > plugin-related requests inside Plugin Runner.
> >> >
> >> > The default data sent to Plugin Runner is only the most common
> >> > information. If you want additional information, you need to declare
> >> > it in extra_info beforehand.
> >> >
> >> > To save communication costs, conf is sent separately.
> >> >
> >> > 1. APISIX will check if conf has a corresponding token in the local
> >> cache.
> >> > 2. If not, APISIX sends a prepare_conf request to ask Plugin Runner
> >> > to cache the conf and return a token.
> >> > (Note that Plugin Runner's cache time needs to be longer than APISIX's
> >> > cache time.)
> >> > 3. APISIX sends an http_req_call request to Plugin Runner.
> >> > 4. Plugin Runner executes the request and returns a response to
> APISIX.
> >> > 5. APISIX processes the request based on the response
> >> >
> >> > 2.4.3 proto
> >> >
> >> > Refer to https://capnproto.org/language.html
> >> >
> >> > The following is the proto for error
> >> >
> >> > response
> >> > ```
> >> > enum ErrorCode {
> >> > BAD_REQUEST @0; # Plugin Runner can't understand APISIX
> >> > SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the request
> >> > CONF_TOKEN_NOT_FOUND @2;
> >> > }
> >> > struct ErrorResp {
> >> > Code @0 :ErrorCode;
> >> > }
> >> > ```
> >> >
> >> > The following is the proto of prepare_conf
> >> >
> >> > request
> >> > ```
> >> > struct PrepareConfReq {
> >> > conf @0 :List(Pair);
> >> > }
> >> > ```
> >> >
> >> > Response
> >> > ```
> >> > struct PrepareConfResp {
> >> > conf_token @0 :UInt32;
> >> > }
> >> > ```
> >> >
> >> > Here is the proto for http_req_call
> >> >
> >> > request
> >> > ```
> >> > struct Pair {
> >> > name @0 :Text;
> >> > value @1 :Text;
> >> > }
> >> > struct PairData {
> >> > name @0 :Text;
> >> > value @1 :Data;
> >> > }
> >> > enum Method {
> >> > GET @0;
> >> > ...
> >> > }
> >> >
> >> > struct HTTPReqCallReq {
> >> > id @0 :UInt32;
> >> > src_ip @1 :Data;
> >> >
> >> > method @2 :Method;
> >> >
> >> > path @3 :Text;
> >> > args @4 :List(Pair);
> >> > headers @5 :List(Pair);
> >> >
> >> > conf_token @6 :UInt32;
> >> >
> >> > extra_info @7 :List(PairData);
> >> > }
> >> > ```
> >> >
> >> > Response
> >> > ```
> >> > struct HTTPReqCallResp {
> >> > id @0 :UInt32;
> >> >
> >> > struct Stop {
> >> > status @0 :UInt16;
> >> > headers @1 :List(Pair);
> >> > body @2 :Data;
> >> > }
> >> > struct Rewrite {
> >> > path @0 :Text;
> >> > headers @1 :List(Pair);
> >> > # Note that args are modified in full.
> >> > # Either empty, meaning no args need to be moved
> >> > # or the entire modified args, not the incrementally changed
> >> parts
> >> > args @2 :List(Pair);
> >> > }
> >> >
> >> > # What needs to be done when the response is received action
> >> > action :union {
> >> > # Do nothing
> >> > continue @1 :Void;
> >> > # Equivalent to core.response.exit(status, body), allowing
> >> > additional headers to be set
> >> > stop @2 :Stop;
> >> > # Rewrite the request
> >> > rewrite @3 :Rewrite;
> >> > }
> >> > }
> >> > ```
> >> >
> >> > 2.4.4 Error handling
> >> >
> >> > Logging and returning 503 error codes
> >> >
> >> > 2.4.5 Environment variables
> >> >
> >> > APISIX configures the Plugin Runner with a number of environment
> >> > variables when it is started.
> >> >
> >> > APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to
> >> listen to
> >> > APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for longer
> >> than this
> >>
> >
>
Re: [Proposal] support using other languages to write plugin for APISIX
Posted by Zhiyuan Ju <ju...@apache.org>.
Hi, ZhengSong,
You image is broken due to we could only use plain text in mailing list :)
Best Regards!
@ Zhiyuan Ju <https://github.com/juzhiyuan>
ZhengSong Tu <tz...@gmail.com> 于2021年4月26日周一 上午11:47写道：
> I have sketched the flow of the java side of the plugin runner
> [image: APISIX 运行 Java 插件.png]
>
> And I have some questions to clarify.
>
> 1. How does the plugin runner load the client's code?
> It's up to each runner to figure it out according to its own
> language's ecology. Take java as an example,
> the user develops with the PDK provided by us, and packages the custom
> plugin code into a jar and puts it in the specified location. When the
> plugin runner starts, it loads the jar package in this location by means of
> a custom dynamic ClassLoader or something like that.
>
> 2. How do users develop?
> Just provide the filter chain like Spring Cloud Gateway. Users can
> register their own filters. And the PDK is a jar, with some defined
> interfaces such as pre, post, etc., and some objects such as request,
> header, etc.
>
> 3. How to debug？
> Option 1: APISIX provides debugging mode for java plugins, this time
> it is not APISIX to start the plugin runner, but the user himself downloads
> the plugin runner code and runs the main method to start it, so as to
> construct a request to access APISIX, APISIX will pass the context of the
> request to the plugin runner, and then use it to debug java code.
> Option 2: java remote debug.
>
>
>
> Zexuan Luo <sp...@apache.org> 于2021年4月19日周一 下午6:18写道：
>
>> Here is the new flatbuffer schema:
>>
>> https://github.com/spacewander/incubator-apisix/blob/step1/apisix/plugins/ext-plugin/ext-plugin.fbs
>>
>> Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午2:52写道：
>> >
>> > 1. Background
>> >
>> > APISIX currently only supports writing plugins in Lua. If other
>> > languages are supported, it will greatly broaden the APISIX ecosystem
>> > and user base.
>> >
>> > 2. Solution
>> >
>> > Since WASM is not yet mature, we consider implementing it through local
>> IPC.
>> >
>> > For the sake of discussion, the following will assume that the plugin
>> > is written in Java. However, in practice, our solution can be
>> > interfaced with other languages.
>> >
>> > 2.1 Terminology
>> >
>> > Several terms are defined here.
>> >
>> > Plugin Runner: The service that runs the plugin, written in the same
>> > language as the plugin. In the first version, we assume that there
>> > will be only one Plugin Runner.
>> >
>> > 2.2 Plugin Runner lifecycle
>> >
>> > To simplify user operation and reduce the difficulty of upgrading,
>> > Plugin Runner is managed by APISIX.
>> >
>> > APISIX starts the Plugin Runner when it starts and ends it when it
>> > ends. if the Plugin Runner quits in the middle, APISIX will restart it
>> > automatically.
>> >
>> > 2.3 Timing of APISIX communication with Plugin Runner
>> >
>> > ```
>> > Router ----> Global Plugin (written in Lua) ---> Ext Plugin
>> > (ext-plugin-pre-req) ----> Lua Plugin (Router)
>> > ---> Ext plugin (ext-plugin-post-req) ---> Upstream
>> > ```
>> >
>> > Running the Ext Plugin in the Global Plugin is not supported at this
>> > time, as the global logic can be executed uniformly in the Plugin
>> > Runner.
>> >
>> > Running Ext Plugin after getting an upstream response is not supported
>> > at this time. We can support it later with a buffering response.
>> >
>> > ext-plugin-pre runs before all non-global Lua plugins, and
>> > ext-plugin-post runs after all non-global Lua plugins.
>> >
>> > 2.4 How APISIX communicates with Plugin Runner
>> >
>> > APISIX communicates with Plugin Runner through a unix socket. The
>> > communication protocol is as follows.
>> >
>> > 2.4.1 Communication format
>> >
>> > ```
>> > 1 byte of type + 3 bytes of length + data
>> > ```
>> >
>> > The type can be 0 ~ 7, and the length can be [0, 8M). data length is
>> > determined by length.
>> > Since Ext Plugin usually does not exchange too much data, 8M should be
>> > enough. The reason for taking 4 bytes is to keep the header small
>> > enough to be read efficiently.
>> >
>> > The current type takes the following values.
>> >
>> > 0 means error
>> > 1 means prepare_conf
>> > 2 means http_req_call
>> >
>> > The data is serialized in capnproto, a binary serialization format.
>> >
>> > capnproto is supported by many programming languages:
>> > https://capnproto.org/otherlang.html
>> >
>> > The advantages of choosing capnproto are.
>> > 1. focus on serialization performance
>> > 2. partial deserialization support, so that decode can be done only
>> > when it is needed
>> >
>> > 2.4.2 Communication steps
>> >
>> > Each ext plugin will have the following configuration.
>> >
>> > ```
>> > {
>> > "conf": [
>> > {
>> > "name": "configuration name",
>> > "value": "configuration value"
>> > }
>> > ],
>> > "extra_info": [
>> > ...
>> > ]
>> > }
>> > ```
>> >
>> > conf can be used to set the execution configuration of the
>> > plugin-related requests inside Plugin Runner.
>> >
>> > The default data sent to Plugin Runner is only the most common
>> > information. If you want additional information, you need to declare
>> > it in extra_info beforehand.
>> >
>> > To save communication costs, conf is sent separately.
>> >
>> > 1. APISIX will check if conf has a corresponding token in the local
>> cache.
>> > 2. If not, APISIX sends a prepare_conf request to ask Plugin Runner
>> > to cache the conf and return a token.
>> > (Note that Plugin Runner's cache time needs to be longer than APISIX's
>> > cache time.)
>> > 3. APISIX sends an http_req_call request to Plugin Runner.
>> > 4. Plugin Runner executes the request and returns a response to APISIX.
>> > 5. APISIX processes the request based on the response
>> >
>> > 2.4.3 proto
>> >
>> > Refer to https://capnproto.org/language.html
>> >
>> > The following is the proto for error
>> >
>> > response
>> > ```
>> > enum ErrorCode {
>> > BAD_REQUEST @0; # Plugin Runner can't understand APISIX
>> > SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the request
>> > CONF_TOKEN_NOT_FOUND @2;
>> > }
>> > struct ErrorResp {
>> > Code @0 :ErrorCode;
>> > }
>> > ```
>> >
>> > The following is the proto of prepare_conf
>> >
>> > request
>> > ```
>> > struct PrepareConfReq {
>> > conf @0 :List(Pair);
>> > }
>> > ```
>> >
>> > Response
>> > ```
>> > struct PrepareConfResp {
>> > conf_token @0 :UInt32;
>> > }
>> > ```
>> >
>> > Here is the proto for http_req_call
>> >
>> > request
>> > ```
>> > struct Pair {
>> > name @0 :Text;
>> > value @1 :Text;
>> > }
>> > struct PairData {
>> > name @0 :Text;
>> > value @1 :Data;
>> > }
>> > enum Method {
>> > GET @0;
>> > ...
>> > }
>> >
>> > struct HTTPReqCallReq {
>> > id @0 :UInt32;
>> > src_ip @1 :Data;
>> >
>> > method @2 :Method;
>> >
>> > path @3 :Text;
>> > args @4 :List(Pair);
>> > headers @5 :List(Pair);
>> >
>> > conf_token @6 :UInt32;
>> >
>> > extra_info @7 :List(PairData);
>> > }
>> > ```
>> >
>> > Response
>> > ```
>> > struct HTTPReqCallResp {
>> > id @0 :UInt32;
>> >
>> > struct Stop {
>> > status @0 :UInt16;
>> > headers @1 :List(Pair);
>> > body @2 :Data;
>> > }
>> > struct Rewrite {
>> > path @0 :Text;
>> > headers @1 :List(Pair);
>> > # Note that args are modified in full.
>> > # Either empty, meaning no args need to be moved
>> > # or the entire modified args, not the incrementally changed
>> parts
>> > args @2 :List(Pair);
>> > }
>> >
>> > # What needs to be done when the response is received action
>> > action :union {
>> > # Do nothing
>> > continue @1 :Void;
>> > # Equivalent to core.response.exit(status, body), allowing
>> > additional headers to be set
>> > stop @2 :Stop;
>> > # Rewrite the request
>> > rewrite @3 :Rewrite;
>> > }
>> > }
>> > ```
>> >
>> > 2.4.4 Error handling
>> >
>> > Logging and returning 503 error codes
>> >
>> > 2.4.5 Environment variables
>> >
>> > APISIX configures the Plugin Runner with a number of environment
>> > variables when it is started.
>> >
>> > APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to
>> listen to
>> > APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for longer
>> than this
>>
>
Re: [Proposal] support using other languages to write plugin for APISIX
Posted by ZhengSong Tu <tz...@gmail.com>.
I have sketched the flow of the java side of the plugin runner
[image: APISIX 运行 Java 插件.png]
And I have some questions to clarify.
1. How does the plugin runner load the client's code?
It's up to each runner to figure it out according to its own language's
ecology. Take java as an example,
the user develops with the PDK provided by us, and packages the custom
plugin code into a jar and puts it in the specified location. When the
plugin runner starts, it loads the jar package in this location by means of
a custom dynamic ClassLoader or something like that.
2. How do users develop?
Just provide the filter chain like Spring Cloud Gateway. Users can
register their own filters. And the PDK is a jar, with some defined
interfaces such as pre, post, etc., and some objects such as request,
header, etc.
3. How to debug？
Option 1: APISIX provides debugging mode for java plugins, this time
it is not APISIX to start the plugin runner, but the user himself downloads
the plugin runner code and runs the main method to start it, so as to
construct a request to access APISIX, APISIX will pass the context of the
request to the plugin runner, and then use it to debug java code.
Option 2: java remote debug.
Zexuan Luo <sp...@apache.org> 于2021年4月19日周一 下午6:18写道：
> Here is the new flatbuffer schema:
>
> https://github.com/spacewander/incubator-apisix/blob/step1/apisix/plugins/ext-plugin/ext-plugin.fbs
>
> Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午2:52写道：
> >
> > 1. Background
> >
> > APISIX currently only supports writing plugins in Lua. If other
> > languages are supported, it will greatly broaden the APISIX ecosystem
> > and user base.
> >
> > 2. Solution
> >
> > Since WASM is not yet mature, we consider implementing it through local
> IPC.
> >
> > For the sake of discussion, the following will assume that the plugin
> > is written in Java. However, in practice, our solution can be
> > interfaced with other languages.
> >
> > 2.1 Terminology
> >
> > Several terms are defined here.
> >
> > Plugin Runner: The service that runs the plugin, written in the same
> > language as the plugin. In the first version, we assume that there
> > will be only one Plugin Runner.
> >
> > 2.2 Plugin Runner lifecycle
> >
> > To simplify user operation and reduce the difficulty of upgrading,
> > Plugin Runner is managed by APISIX.
> >
> > APISIX starts the Plugin Runner when it starts and ends it when it
> > ends. if the Plugin Runner quits in the middle, APISIX will restart it
> > automatically.
> >
> > 2.3 Timing of APISIX communication with Plugin Runner
> >
> > ```
> > Router ----> Global Plugin (written in Lua) ---> Ext Plugin
> > (ext-plugin-pre-req) ----> Lua Plugin (Router)
> > ---> Ext plugin (ext-plugin-post-req) ---> Upstream
> > ```
> >
> > Running the Ext Plugin in the Global Plugin is not supported at this
> > time, as the global logic can be executed uniformly in the Plugin
> > Runner.
> >
> > Running Ext Plugin after getting an upstream response is not supported
> > at this time. We can support it later with a buffering response.
> >
> > ext-plugin-pre runs before all non-global Lua plugins, and
> > ext-plugin-post runs after all non-global Lua plugins.
> >
> > 2.4 How APISIX communicates with Plugin Runner
> >
> > APISIX communicates with Plugin Runner through a unix socket. The
> > communication protocol is as follows.
> >
> > 2.4.1 Communication format
> >
> > ```
> > 1 byte of type + 3 bytes of length + data
> > ```
> >
> > The type can be 0 ~ 7, and the length can be [0, 8M). data length is
> > determined by length.
> > Since Ext Plugin usually does not exchange too much data, 8M should be
> > enough. The reason for taking 4 bytes is to keep the header small
> > enough to be read efficiently.
> >
> > The current type takes the following values.
> >
> > 0 means error
> > 1 means prepare_conf
> > 2 means http_req_call
> >
> > The data is serialized in capnproto, a binary serialization format.
> >
> > capnproto is supported by many programming languages:
> > https://capnproto.org/otherlang.html
> >
> > The advantages of choosing capnproto are.
> > 1. focus on serialization performance
> > 2. partial deserialization support, so that decode can be done only
> > when it is needed
> >
> > 2.4.2 Communication steps
> >
> > Each ext plugin will have the following configuration.
> >
> > ```
> > {
> > "conf": [
> > {
> > "name": "configuration name",
> > "value": "configuration value"
> > }
> > ],
> > "extra_info": [
> > ...
> > ]
> > }
> > ```
> >
> > conf can be used to set the execution configuration of the
> > plugin-related requests inside Plugin Runner.
> >
> > The default data sent to Plugin Runner is only the most common
> > information. If you want additional information, you need to declare
> > it in extra_info beforehand.
> >
> > To save communication costs, conf is sent separately.
> >
> > 1. APISIX will check if conf has a corresponding token in the local
> cache.
> > 2. If not, APISIX sends a prepare_conf request to ask Plugin Runner
> > to cache the conf and return a token.
> > (Note that Plugin Runner's cache time needs to be longer than APISIX's
> > cache time.)
> > 3. APISIX sends an http_req_call request to Plugin Runner.
> > 4. Plugin Runner executes the request and returns a response to APISIX.
> > 5. APISIX processes the request based on the response
> >
> > 2.4.3 proto
> >
> > Refer to https://capnproto.org/language.html
> >
> > The following is the proto for error
> >
> > response
> > ```
> > enum ErrorCode {
> > BAD_REQUEST @0; # Plugin Runner can't understand APISIX
> > SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the request
> > CONF_TOKEN_NOT_FOUND @2;
> > }
> > struct ErrorResp {
> > Code @0 :ErrorCode;
> > }
> > ```
> >
> > The following is the proto of prepare_conf
> >
> > request
> > ```
> > struct PrepareConfReq {
> > conf @0 :List(Pair);
> > }
> > ```
> >
> > Response
> > ```
> > struct PrepareConfResp {
> > conf_token @0 :UInt32;
> > }
> > ```
> >
> > Here is the proto for http_req_call
> >
> > request
> > ```
> > struct Pair {
> > name @0 :Text;
> > value @1 :Text;
> > }
> > struct PairData {
> > name @0 :Text;
> > value @1 :Data;
> > }
> > enum Method {
> > GET @0;
> > ...
> > }
> >
> > struct HTTPReqCallReq {
> > id @0 :UInt32;
> > src_ip @1 :Data;
> >
> > method @2 :Method;
> >
> > path @3 :Text;
> > args @4 :List(Pair);
> > headers @5 :List(Pair);
> >
> > conf_token @6 :UInt32;
> >
> > extra_info @7 :List(PairData);
> > }
> > ```
> >
> > Response
> > ```
> > struct HTTPReqCallResp {
> > id @0 :UInt32;
> >
> > struct Stop {
> > status @0 :UInt16;
> > headers @1 :List(Pair);
> > body @2 :Data;
> > }
> > struct Rewrite {
> > path @0 :Text;
> > headers @1 :List(Pair);
> > # Note that args are modified in full.
> > # Either empty, meaning no args need to be moved
> > # or the entire modified args, not the incrementally changed
> parts
> > args @2 :List(Pair);
> > }
> >
> > # What needs to be done when the response is received action
> > action :union {
> > # Do nothing
> > continue @1 :Void;
> > # Equivalent to core.response.exit(status, body), allowing
> > additional headers to be set
> > stop @2 :Stop;
> > # Rewrite the request
> > rewrite @3 :Rewrite;
> > }
> > }
> > ```
> >
> > 2.4.4 Error handling
> >
> > Logging and returning 503 error codes
> >
> > 2.4.5 Environment variables
> >
> > APISIX configures the Plugin Runner with a number of environment
> > variables when it is started.
> >
> > APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to
> listen to
> > APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for longer
> than this
>
Re: [Proposal] support using other languages to write plugin for APISIX
Posted by Zexuan Luo <sp...@apache.org>.
Here is the new flatbuffer schema:
https://github.com/spacewander/incubator-apisix/blob/step1/apisix/plugins/ext-plugin/ext-plugin.fbs
Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午2:52写道：
>
> 1. Background
>
> APISIX currently only supports writing plugins in Lua. If other
> languages are supported, it will greatly broaden the APISIX ecosystem
> and user base.
>
> 2. Solution
>
> Since WASM is not yet mature, we consider implementing it through local IPC.
>
> For the sake of discussion, the following will assume that the plugin
> is written in Java. However, in practice, our solution can be
> interfaced with other languages.
>
> 2.1 Terminology
>
> Several terms are defined here.
>
> Plugin Runner: The service that runs the plugin, written in the same
> language as the plugin. In the first version, we assume that there
> will be only one Plugin Runner.
>
> 2.2 Plugin Runner lifecycle
>
> To simplify user operation and reduce the difficulty of upgrading,
> Plugin Runner is managed by APISIX.
>
> APISIX starts the Plugin Runner when it starts and ends it when it
> ends. if the Plugin Runner quits in the middle, APISIX will restart it
> automatically.
>
> 2.3 Timing of APISIX communication with Plugin Runner
>
> ```
> Router ----> Global Plugin (written in Lua) ---> Ext Plugin
> (ext-plugin-pre-req) ----> Lua Plugin (Router)
> ---> Ext plugin (ext-plugin-post-req) ---> Upstream
> ```
>
> Running the Ext Plugin in the Global Plugin is not supported at this
> time, as the global logic can be executed uniformly in the Plugin
> Runner.
>
> Running Ext Plugin after getting an upstream response is not supported
> at this time. We can support it later with a buffering response.
>
> ext-plugin-pre runs before all non-global Lua plugins, and
> ext-plugin-post runs after all non-global Lua plugins.
>
> 2.4 How APISIX communicates with Plugin Runner
>
> APISIX communicates with Plugin Runner through a unix socket. The
> communication protocol is as follows.
>
> 2.4.1 Communication format
>
> ```
> 1 byte of type + 3 bytes of length + data
> ```
>
> The type can be 0 ~ 7, and the length can be [0, 8M). data length is
> determined by length.
> Since Ext Plugin usually does not exchange too much data, 8M should be
> enough. The reason for taking 4 bytes is to keep the header small
> enough to be read efficiently.
>
> The current type takes the following values.
>
> 0 means error
> 1 means prepare_conf
> 2 means http_req_call
>
> The data is serialized in capnproto, a binary serialization format.
>
> capnproto is supported by many programming languages:
> https://capnproto.org/otherlang.html
>
> The advantages of choosing capnproto are.
> 1. focus on serialization performance
> 2. partial deserialization support, so that decode can be done only
> when it is needed
>
> 2.4.2 Communication steps
>
> Each ext plugin will have the following configuration.
>
> ```
> {
> "conf": [
> {
> "name": "configuration name",
> "value": "configuration value"
> }
> ],
> "extra_info": [
> ...
> ]
> }
> ```
>
> conf can be used to set the execution configuration of the
> plugin-related requests inside Plugin Runner.
>
> The default data sent to Plugin Runner is only the most common
> information. If you want additional information, you need to declare
> it in extra_info beforehand.
>
> To save communication costs, conf is sent separately.
>
> 1. APISIX will check if conf has a corresponding token in the local cache.
> 2. If not, APISIX sends a prepare_conf request to ask Plugin Runner
> to cache the conf and return a token.
> (Note that Plugin Runner's cache time needs to be longer than APISIX's
> cache time.)
> 3. APISIX sends an http_req_call request to Plugin Runner.
> 4. Plugin Runner executes the request and returns a response to APISIX.
> 5. APISIX processes the request based on the response
>
> 2.4.3 proto
>
> Refer to https://capnproto.org/language.html
>
> The following is the proto for error
>
> response
> ```
> enum ErrorCode {
> BAD_REQUEST @0; # Plugin Runner can't understand APISIX
> SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the request
> CONF_TOKEN_NOT_FOUND @2;
> }
> struct ErrorResp {
> Code @0 :ErrorCode;
> }
> ```
>
> The following is the proto of prepare_conf
>
> request
> ```
> struct PrepareConfReq {
> conf @0 :List(Pair);
> }
> ```
>
> Response
> ```
> struct PrepareConfResp {
> conf_token @0 :UInt32;
> }
> ```
>
> Here is the proto for http_req_call
>
> request
> ```
> struct Pair {
> name @0 :Text;
> value @1 :Text;
> }
> struct PairData {
> name @0 :Text;
> value @1 :Data;
> }
> enum Method {
> GET @0;
> ...
> }
>
> struct HTTPReqCallReq {
> id @0 :UInt32;
> src_ip @1 :Data;
>
> method @2 :Method;
>
> path @3 :Text;
> args @4 :List(Pair);
> headers @5 :List(Pair);
>
> conf_token @6 :UInt32;
>
> extra_info @7 :List(PairData);
> }
> ```
>
> Response
> ```
> struct HTTPReqCallResp {
> id @0 :UInt32;
>
> struct Stop {
> status @0 :UInt16;
> headers @1 :List(Pair);
> body @2 :Data;
> }
> struct Rewrite {
> path @0 :Text;
> headers @1 :List(Pair);
> # Note that args are modified in full.
> # Either empty, meaning no args need to be moved
> # or the entire modified args, not the incrementally changed parts
> args @2 :List(Pair);
> }
>
> # What needs to be done when the response is received action
> action :union {
> # Do nothing
> continue @1 :Void;
> # Equivalent to core.response.exit(status, body), allowing
> additional headers to be set
> stop @2 :Stop;
> # Rewrite the request
> rewrite @3 :Rewrite;
> }
> }
> ```
>
> 2.4.4 Error handling
>
> Logging and returning 503 error codes
>
> 2.4.5 Environment variables
>
> APISIX configures the Plugin Runner with a number of environment
> variables when it is started.
>
> APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to listen to
> APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for longer than this
Re: [Proposal] support using other languages to write plugin for APISIX
Posted by YuanSheng Wang <me...@apache.org>.
It is an easier way to support more languages through out-of-process RPC
network communication.
Nice proposal, I think we can make a try.
^_^
On Sat, Apr 17, 2021 at 11:52 AM Zexuan Luo <sp...@apache.org> wrote:
> I am glad to hear that there is another option.
> We can use flatbuffers as the codec instead of capnproto.
>
> > as you are not using a standard RPC protocol, no
> possibility of internal tracing
>
> The protocol is extendable and tracing support can be added.
>
> Currently, I focus on performance most because performance is a good
> selling point.
>
> People will say, "APISIX + Java sidecar is faster than Spring Cloud
> Gateway, let's adopt it!".
> It is possible as Nginx is faster than Spring Cloud Gateway.
>
> People won't say, "APISIX + Java sidecar has better observability than
> Spring Cloud Gateway, let's adopt it!".
> As there are two processes than one, it is impossible to have better
> observability.
>
> If people don't even buy, there is no chance to compare observability.
>
> There is a trade-off between performance & observability. I want to
> improve the performance as much as we can, attract people to use it,
> then we can have a chance to improve the observability.
>
> Sheng Wu <wu...@gmail.com> 于2021年4月16日周五 下午8:22写道：
> >
> > I want to remind you, as you are not using a standard RPC protocol, no
> > possibility of internal tracing. So how should we collect
> > metrics/performance-data of this cross-processes communication?
> > And it is Java lib(maybe others have the same issue),
> > https://github.com/capnproto/capnproto-java/graphs/contributors, has
> > very little activities. I am not sure how healthy their community is.
> >
> > Have you taken a look at this? https://google.github.io/flatbuffers/
> > It is used in Istio, and SkyWalking used this in mesh scenarios. This
> > protocol has a very good performance especially when some fields are
> > not required to deserialize, which from my understanding, is a very
> > case when running the sidecar process.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> > Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午6:19写道：
> > >
> > > It will hold long connections.
> > >
> > > Need to clarify that the "long connection" doesn't conflict with the
> > > "request / response mode". For example, gRPC's unary mode can work on
> > > a long connection.
> > >
> > > bzp2010 <bz...@apache.org> 于2021年4月16日周五 下午6:04写道：
> > > >
> > > > Oh, I get it.
> > > > Will it use a long connection based on UNIX socket to transfer data?
> Or is
> > > > it a request / response mode similar to HTTP?
> > > >
> > > > Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午5:53写道：
> > > >
> > > > > This is why we choose capnproto rather than protobuf. The
> > > > > encode/decode performance is important, so we choose capnproto
> which
> > > > > has a better performance than protobuf.
> > > > >
> > > > > bzp2010 <bz...@apache.org> 于2021年4月16日周五 下午4:44写道：
> > > > > >
> > > > > > Hi,
> > > > > > I have some questions about this proposal.
> > > > > >
> > > > > > 1. If using similar C/S mode, whether the encoding and decoding
> process
> > > > > > will affect the performance.
> > > > > >
> > > > > > 2.Can we use protobuf, a more popular encoding and decoding
> toolkit?
> > > > > >
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > *发件人: *Zexuan Luo <sp...@apache.org>
> > > > > > *发送时间: *2021年4月16日 14:52
> > > > > > *收件人: *dev@apisix.apache.org
> > > > > > *主题: *[Proposal] support using other languages to write plugin
> for APISIX
> > > > > >
> > > > > >
> > > > > >
> > > > > > 1. Background
> > > > > >
> > > > > >
> > > > > >
> > > > > > APISIX currently only supports writing plugins in Lua. If other
> > > > > >
> > > > > > languages are supported, it will greatly broaden the APISIX
> ecosystem
> > > > > >
> > > > > > and user base.
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2. Solution
> > > > > >
> > > > > >
> > > > > >
> > > > > > Since WASM is not yet mature, we consider implementing it
> through local
> > > > > IPC.
> > > > > >
> > > > > >
> > > > > >
> > > > > > For the sake of discussion, the following will assume that the
> plugin
> > > > > >
> > > > > > is written in Java. However, in practice, our solution can be
> > > > > >
> > > > > > interfaced with other languages.
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2.1 Terminology
> > > > > >
> > > > > >
> > > > > >
> > > > > > Several terms are defined here.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Plugin Runner: The service that runs the plugin, written in the
> same
> > > > > >
> > > > > > language as the plugin. In the first version, we assume that
> there
> > > > > >
> > > > > > will be only one Plugin Runner.
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2.2 Plugin Runner lifecycle
> > > > > >
> > > > > >
> > > > > >
> > > > > > To simplify user operation and reduce the difficulty of
> upgrading,
> > > > > >
> > > > > > Plugin Runner is managed by APISIX.
> > > > > >
> > > > > >
> > > > > >
> > > > > > APISIX starts the Plugin Runner when it starts and ends it when
> it
> > > > > >
> > > > > > ends. if the Plugin Runner quits in the middle, APISIX will
> restart it
> > > > > >
> > > > > > automatically.
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2.3 Timing of APISIX communication with Plugin Runner
> > > > > >
> > > > > >
> > > > > >
> > > > > > ```
> > > > > >
> > > > > > Router ----> Global Plugin (written in Lua) ---> Ext Plugin
> > > > > >
> > > > > > (ext-plugin-pre-req) ----> Lua Plugin (Router)
> > > > > >
> > > > > > ---> Ext plugin (ext-plugin-post-req) ---> Upstream
> > > > > >
> > > > > > ```
> > > > > >
> > > > > >
> > > > > >
> > > > > > Running the Ext Plugin in the Global Plugin is not supported at
> this
> > > > > >
> > > > > > time, as the global logic can be executed uniformly in the Plugin
> > > > > >
> > > > > > Runner.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Running Ext Plugin after getting an upstream response is not
> supported
> > > > > >
> > > > > > at this time. We can support it later with a buffering response.
> > > > > >
> > > > > >
> > > > > >
> > > > > > ext-plugin-pre runs before all non-global Lua plugins, and
> > > > > >
> > > > > > ext-plugin-post runs after all non-global Lua plugins.
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2.4 How APISIX communicates with Plugin Runner
> > > > > >
> > > > > >
> > > > > >
> > > > > > APISIX communicates with Plugin Runner through a unix socket. The
> > > > > >
> > > > > > communication protocol is as follows.
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2.4.1 Communication format
> > > > > >
> > > > > >
> > > > > >
> > > > > > ```
> > > > > >
> > > > > > 1 byte of type + 3 bytes of length + data
> > > > > >
> > > > > > ```
> > > > > >
> > > > > >
> > > > > >
> > > > > > The type can be 0 ~ 7, and the length can be [0, 8M). data
> length is
> > > > > >
> > > > > > determined by length.
> > > > > >
> > > > > > Since Ext Plugin usually does not exchange too much data, 8M
> should be
> > > > > >
> > > > > > enough. The reason for taking 4 bytes is to keep the header small
> > > > > >
> > > > > > enough to be read efficiently.
> > > > > >
> > > > > >
> > > > > >
> > > > > > The current type takes the following values.
> > > > > >
> > > > > >
> > > > > >
> > > > > > 0 means error
> > > > > >
> > > > > > 1 means prepare_conf
> > > > > >
> > > > > > 2 means http_req_call
> > > > > >
> > > > > >
> > > > > >
> > > > > > The data is serialized in capnproto, a binary serialization
> format.
> > > > > >
> > > > > >
> > > > > >
> > > > > > capnproto is supported by many programming languages:
> > > > > >
> > > > > > https://capnproto.org/otherlang.html
> > > > > >
> > > > > >
> > > > > >
> > > > > > The advantages of choosing capnproto are.
> > > > > >
> > > > > > 1. focus on serialization performance
> > > > > >
> > > > > > 2. partial deserialization support, so that decode can be done
> only
> > > > > >
> > > > > > when it is needed
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2.4.2 Communication steps
> > > > > >
> > > > > >
> > > > > >
> > > > > > Each ext plugin will have the following configuration.
> > > > > >
> > > > > >
> > > > > >
> > > > > > ```
> > > > > >
> > > > > > {
> > > > > >
> > > > > > "conf": [
> > > > > >
> > > > > > {
> > > > > >
> > > > > > "name": "configuration name",
> > > > > >
> > > > > > "value": "configuration value"
> > > > > >
> > > > > > }
> > > > > >
> > > > > > ],
> > > > > >
> > > > > > "extra_info": [
> > > > > >
> > > > > > ...
> > > > > >
> > > > > > ]
> > > > > >
> > > > > > }
> > > > > >
> > > > > > ```
> > > > > >
> > > > > >
> > > > > >
> > > > > > conf can be used to set the execution configuration of the
> > > > > >
> > > > > > plugin-related requests inside Plugin Runner.
> > > > > >
> > > > > >
> > > > > >
> > > > > > The default data sent to Plugin Runner is only the most common
> > > > > >
> > > > > > information. If you want additional information, you need to
> declare
> > > > > >
> > > > > > it in extra_info beforehand.
> > > > > >
> > > > > >
> > > > > >
> > > > > > To save communication costs, conf is sent separately.
> > > > > >
> > > > > >
> > > > > >
> > > > > > 1. APISIX will check if conf has a corresponding token in the
> local
> > > > > cache.
> > > > > >
> > > > > > 2. If not, APISIX sends a prepare_conf request to ask Plugin
> Runner
> > > > > >
> > > > > > to cache the conf and return a token.
> > > > > >
> > > > > > (Note that Plugin Runner's cache time needs to be longer than
> APISIX's
> > > > > >
> > > > > > cache time.)
> > > > > >
> > > > > > 3. APISIX sends an http_req_call request to Plugin Runner.
> > > > > >
> > > > > > 4. Plugin Runner executes the request and returns a response to
> APISIX.
> > > > > >
> > > > > > 5. APISIX processes the request based on the response
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2.4.3 proto
> > > > > >
> > > > > >
> > > > > >
> > > > > > Refer to https://capnproto.org/language.html
> > > > > >
> > > > > >
> > > > > >
> > > > > > The following is the proto for error
> > > > > >
> > > > > >
> > > > > >
> > > > > > response
> > > > > >
> > > > > > ```
> > > > > >
> > > > > > enum ErrorCode {
> > > > > >
> > > > > > BAD_REQUEST @0; # Plugin Runner can't understand APISIX
> > > > > >
> > > > > > SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the
> request
> > > > > >
> > > > > > CONF_TOKEN_NOT_FOUND @2;
> > > > > >
> > > > > > }
> > > > > >
> > > > > > struct ErrorResp {
> > > > > >
> > > > > > Code @0 :ErrorCode;
> > > > > >
> > > > > > }
> > > > > >
> > > > > > ```
> > > > > >
> > > > > >
> > > > > >
> > > > > > The following is the proto of prepare_conf
> > > > > >
> > > > > >
> > > > > >
> > > > > > request
> > > > > >
> > > > > > ```
> > > > > >
> > > > > > struct PrepareConfReq {
> > > > > >
> > > > > > conf @0 :List(Pair);
> > > > > >
> > > > > > }
> > > > > >
> > > > > > ```
> > > > > >
> > > > > >
> > > > > >
> > > > > > Response
> > > > > >
> > > > > > ```
> > > > > >
> > > > > > struct PrepareConfResp {
> > > > > >
> > > > > > conf_token @0 :UInt32;
> > > > > >
> > > > > > }
> > > > > >
> > > > > > ```
> > > > > >
> > > > > >
> > > > > >
> > > > > > Here is the proto for http_req_call
> > > > > >
> > > > > >
> > > > > >
> > > > > > request
> > > > > >
> > > > > > ```
> > > > > >
> > > > > > struct Pair {
> > > > > >
> > > > > > name @0 :Text;
> > > > > >
> > > > > > value @1 :Text;
> > > > > >
> > > > > > }
> > > > > >
> > > > > > struct PairData {
> > > > > >
> > > > > > name @0 :Text;
> > > > > >
> > > > > > value @1 :Data;
> > > > > >
> > > > > > }
> > > > > >
> > > > > > enum Method {
> > > > > >
> > > > > > GET @0;
> > > > > >
> > > > > > ...
> > > > > >
> > > > > > }
> > > > > >
> > > > > >
> > > > > >
> > > > > > struct HTTPReqCallReq {
> > > > > >
> > > > > > id @0 :UInt32;
> > > > > >
> > > > > > src_ip @1 :Data;
> > > > > >
> > > > > >
> > > > > >
> > > > > > method @2 :Method;
> > > > > >
> > > > > >
> > > > > >
> > > > > > path @3 :Text;
> > > > > >
> > > > > > args @4 :List(Pair);
> > > > > >
> > > > > > headers @5 :List(Pair);
> > > > > >
> > > > > >
> > > > > >
> > > > > > conf_token @6 :UInt32;
> > > > > >
> > > > > >
> > > > > >
> > > > > > extra_info @7 :List(PairData);
> > > > > >
> > > > > > }
> > > > > >
> > > > > > ```
> > > > > >
> > > > > >
> > > > > >
> > > > > > Response
> > > > > >
> > > > > > ```
> > > > > >
> > > > > > struct HTTPReqCallResp {
> > > > > >
> > > > > > id @0 :UInt32;
> > > > > >
> > > > > >
> > > > > >
> > > > > > struct Stop {
> > > > > >
> > > > > > status @0 :UInt16;
> > > > > >
> > > > > > headers @1 :List(Pair);
> > > > > >
> > > > > > body @2 :Data;
> > > > > >
> > > > > > }
> > > > > >
> > > > > > struct Rewrite {
> > > > > >
> > > > > > path @0 :Text;
> > > > > >
> > > > > > headers @1 :List(Pair);
> > > > > >
> > > > > > # Note that args are modified in full.
> > > > > >
> > > > > > # Either empty, meaning no args need to be moved
> > > > > >
> > > > > > # or the entire modified args, not the incrementally
> changed
> > > > > parts
> > > > > >
> > > > > > args @2 :List(Pair);
> > > > > >
> > > > > > }
> > > > > >
> > > > > >
> > > > > >
> > > > > > # What needs to be done when the response is received action
> > > > > >
> > > > > > action :union {
> > > > > >
> > > > > > # Do nothing
> > > > > >
> > > > > > continue @1 :Void;
> > > > > >
> > > > > > # Equivalent to core.response.exit(status, body),
> allowing
> > > > > >
> > > > > > additional headers to be set
> > > > > >
> > > > > > stop @2 :Stop;
> > > > > >
> > > > > > # Rewrite the request
> > > > > >
> > > > > > rewrite @3 :Rewrite;
> > > > > >
> > > > > > }
> > > > > >
> > > > > > }
> > > > > >
> > > > > > ```
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2.4.4 Error handling
> > > > > >
> > > > > >
> > > > > >
> > > > > > Logging and returning 503 error codes
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2.4.5 Environment variables
> > > > > >
> > > > > >
> > > > > >
> > > > > > APISIX configures the Plugin Runner with a number of environment
> > > > > >
> > > > > > variables when it is started.
> > > > > >
> > > > > >
> > > > > >
> > > > > > APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs
> to
> > > > > listen to
> > > > > >
> > > > > > APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for
> longer
> > > > > than
> > > > > > this
> > > > >
>
--
*MembPhis*
My GitHub: https://github.com/membphis
Apache APISIX: https://github.com/apache/apisix
Re: [Proposal] support using other languages to write plugin for APISIX
Posted by Zexuan Luo <sp...@apache.org>.
I am glad to hear that there is another option.
We can use flatbuffers as the codec instead of capnproto.
> as you are not using a standard RPC protocol, no
possibility of internal tracing
The protocol is extendable and tracing support can be added.
Currently, I focus on performance most because performance is a good
selling point.
People will say, "APISIX + Java sidecar is faster than Spring Cloud
Gateway, let's adopt it!".
It is possible as Nginx is faster than Spring Cloud Gateway.
People won't say, "APISIX + Java sidecar has better observability than
Spring Cloud Gateway, let's adopt it!".
As there are two processes than one, it is impossible to have better
observability.
If people don't even buy, there is no chance to compare observability.
There is a trade-off between performance & observability. I want to
improve the performance as much as we can, attract people to use it,
then we can have a chance to improve the observability.
Sheng Wu <wu...@gmail.com> 于2021年4月16日周五 下午8:22写道：
>
> I want to remind you, as you are not using a standard RPC protocol, no
> possibility of internal tracing. So how should we collect
> metrics/performance-data of this cross-processes communication?
> And it is Java lib(maybe others have the same issue),
> https://github.com/capnproto/capnproto-java/graphs/contributors, has
> very little activities. I am not sure how healthy their community is.
>
> Have you taken a look at this? https://google.github.io/flatbuffers/
> It is used in Istio, and SkyWalking used this in mesh scenarios. This
> protocol has a very good performance especially when some fields are
> not required to deserialize, which from my understanding, is a very
> case when running the sidecar process.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
> Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午6:19写道：
> >
> > It will hold long connections.
> >
> > Need to clarify that the "long connection" doesn't conflict with the
> > "request / response mode". For example, gRPC's unary mode can work on
> > a long connection.
> >
> > bzp2010 <bz...@apache.org> 于2021年4月16日周五 下午6:04写道：
> > >
> > > Oh, I get it.
> > > Will it use a long connection based on UNIX socket to transfer data? Or is
> > > it a request / response mode similar to HTTP?
> > >
> > > Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午5:53写道：
> > >
> > > > This is why we choose capnproto rather than protobuf. The
> > > > encode/decode performance is important, so we choose capnproto which
> > > > has a better performance than protobuf.
> > > >
> > > > bzp2010 <bz...@apache.org> 于2021年4月16日周五 下午4:44写道：
> > > > >
> > > > > Hi,
> > > > > I have some questions about this proposal.
> > > > >
> > > > > 1. If using similar C/S mode, whether the encoding and decoding process
> > > > > will affect the performance.
> > > > >
> > > > > 2.Can we use protobuf, a more popular encoding and decoding toolkit?
> > > > >
> > > > >
> > > > > Thanks!
> > > > >
> > > > > *发件人: *Zexuan Luo <sp...@apache.org>
> > > > > *发送时间: *2021年4月16日 14:52
> > > > > *收件人: *dev@apisix.apache.org
> > > > > *主题: *[Proposal] support using other languages to write plugin for APISIX
> > > > >
> > > > >
> > > > >
> > > > > 1. Background
> > > > >
> > > > >
> > > > >
> > > > > APISIX currently only supports writing plugins in Lua. If other
> > > > >
> > > > > languages are supported, it will greatly broaden the APISIX ecosystem
> > > > >
> > > > > and user base.
> > > > >
> > > > >
> > > > >
> > > > > 2. Solution
> > > > >
> > > > >
> > > > >
> > > > > Since WASM is not yet mature, we consider implementing it through local
> > > > IPC.
> > > > >
> > > > >
> > > > >
> > > > > For the sake of discussion, the following will assume that the plugin
> > > > >
> > > > > is written in Java. However, in practice, our solution can be
> > > > >
> > > > > interfaced with other languages.
> > > > >
> > > > >
> > > > >
> > > > > 2.1 Terminology
> > > > >
> > > > >
> > > > >
> > > > > Several terms are defined here.
> > > > >
> > > > >
> > > > >
> > > > > Plugin Runner: The service that runs the plugin, written in the same
> > > > >
> > > > > language as the plugin. In the first version, we assume that there
> > > > >
> > > > > will be only one Plugin Runner.
> > > > >
> > > > >
> > > > >
> > > > > 2.2 Plugin Runner lifecycle
> > > > >
> > > > >
> > > > >
> > > > > To simplify user operation and reduce the difficulty of upgrading,
> > > > >
> > > > > Plugin Runner is managed by APISIX.
> > > > >
> > > > >
> > > > >
> > > > > APISIX starts the Plugin Runner when it starts and ends it when it
> > > > >
> > > > > ends. if the Plugin Runner quits in the middle, APISIX will restart it
> > > > >
> > > > > automatically.
> > > > >
> > > > >
> > > > >
> > > > > 2.3 Timing of APISIX communication with Plugin Runner
> > > > >
> > > > >
> > > > >
> > > > > ```
> > > > >
> > > > > Router ----> Global Plugin (written in Lua) ---> Ext Plugin
> > > > >
> > > > > (ext-plugin-pre-req) ----> Lua Plugin (Router)
> > > > >
> > > > > ---> Ext plugin (ext-plugin-post-req) ---> Upstream
> > > > >
> > > > > ```
> > > > >
> > > > >
> > > > >
> > > > > Running the Ext Plugin in the Global Plugin is not supported at this
> > > > >
> > > > > time, as the global logic can be executed uniformly in the Plugin
> > > > >
> > > > > Runner.
> > > > >
> > > > >
> > > > >
> > > > > Running Ext Plugin after getting an upstream response is not supported
> > > > >
> > > > > at this time. We can support it later with a buffering response.
> > > > >
> > > > >
> > > > >
> > > > > ext-plugin-pre runs before all non-global Lua plugins, and
> > > > >
> > > > > ext-plugin-post runs after all non-global Lua plugins.
> > > > >
> > > > >
> > > > >
> > > > > 2.4 How APISIX communicates with Plugin Runner
> > > > >
> > > > >
> > > > >
> > > > > APISIX communicates with Plugin Runner through a unix socket. The
> > > > >
> > > > > communication protocol is as follows.
> > > > >
> > > > >
> > > > >
> > > > > 2.4.1 Communication format
> > > > >
> > > > >
> > > > >
> > > > > ```
> > > > >
> > > > > 1 byte of type + 3 bytes of length + data
> > > > >
> > > > > ```
> > > > >
> > > > >
> > > > >
> > > > > The type can be 0 ~ 7, and the length can be [0, 8M). data length is
> > > > >
> > > > > determined by length.
> > > > >
> > > > > Since Ext Plugin usually does not exchange too much data, 8M should be
> > > > >
> > > > > enough. The reason for taking 4 bytes is to keep the header small
> > > > >
> > > > > enough to be read efficiently.
> > > > >
> > > > >
> > > > >
> > > > > The current type takes the following values.
> > > > >
> > > > >
> > > > >
> > > > > 0 means error
> > > > >
> > > > > 1 means prepare_conf
> > > > >
> > > > > 2 means http_req_call
> > > > >
> > > > >
> > > > >
> > > > > The data is serialized in capnproto, a binary serialization format.
> > > > >
> > > > >
> > > > >
> > > > > capnproto is supported by many programming languages:
> > > > >
> > > > > https://capnproto.org/otherlang.html
> > > > >
> > > > >
> > > > >
> > > > > The advantages of choosing capnproto are.
> > > > >
> > > > > 1. focus on serialization performance
> > > > >
> > > > > 2. partial deserialization support, so that decode can be done only
> > > > >
> > > > > when it is needed
> > > > >
> > > > >
> > > > >
> > > > > 2.4.2 Communication steps
> > > > >
> > > > >
> > > > >
> > > > > Each ext plugin will have the following configuration.
> > > > >
> > > > >
> > > > >
> > > > > ```
> > > > >
> > > > > {
> > > > >
> > > > > "conf": [
> > > > >
> > > > > {
> > > > >
> > > > > "name": "configuration name",
> > > > >
> > > > > "value": "configuration value"
> > > > >
> > > > > }
> > > > >
> > > > > ],
> > > > >
> > > > > "extra_info": [
> > > > >
> > > > > ...
> > > > >
> > > > > ]
> > > > >
> > > > > }
> > > > >
> > > > > ```
> > > > >
> > > > >
> > > > >
> > > > > conf can be used to set the execution configuration of the
> > > > >
> > > > > plugin-related requests inside Plugin Runner.
> > > > >
> > > > >
> > > > >
> > > > > The default data sent to Plugin Runner is only the most common
> > > > >
> > > > > information. If you want additional information, you need to declare
> > > > >
> > > > > it in extra_info beforehand.
> > > > >
> > > > >
> > > > >
> > > > > To save communication costs, conf is sent separately.
> > > > >
> > > > >
> > > > >
> > > > > 1. APISIX will check if conf has a corresponding token in the local
> > > > cache.
> > > > >
> > > > > 2. If not, APISIX sends a prepare_conf request to ask Plugin Runner
> > > > >
> > > > > to cache the conf and return a token.
> > > > >
> > > > > (Note that Plugin Runner's cache time needs to be longer than APISIX's
> > > > >
> > > > > cache time.)
> > > > >
> > > > > 3. APISIX sends an http_req_call request to Plugin Runner.
> > > > >
> > > > > 4. Plugin Runner executes the request and returns a response to APISIX.
> > > > >
> > > > > 5. APISIX processes the request based on the response
> > > > >
> > > > >
> > > > >
> > > > > 2.4.3 proto
> > > > >
> > > > >
> > > > >
> > > > > Refer to https://capnproto.org/language.html
> > > > >
> > > > >
> > > > >
> > > > > The following is the proto for error
> > > > >
> > > > >
> > > > >
> > > > > response
> > > > >
> > > > > ```
> > > > >
> > > > > enum ErrorCode {
> > > > >
> > > > > BAD_REQUEST @0; # Plugin Runner can't understand APISIX
> > > > >
> > > > > SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the request
> > > > >
> > > > > CONF_TOKEN_NOT_FOUND @2;
> > > > >
> > > > > }
> > > > >
> > > > > struct ErrorResp {
> > > > >
> > > > > Code @0 :ErrorCode;
> > > > >
> > > > > }
> > > > >
> > > > > ```
> > > > >
> > > > >
> > > > >
> > > > > The following is the proto of prepare_conf
> > > > >
> > > > >
> > > > >
> > > > > request
> > > > >
> > > > > ```
> > > > >
> > > > > struct PrepareConfReq {
> > > > >
> > > > > conf @0 :List(Pair);
> > > > >
> > > > > }
> > > > >
> > > > > ```
> > > > >
> > > > >
> > > > >
> > > > > Response
> > > > >
> > > > > ```
> > > > >
> > > > > struct PrepareConfResp {
> > > > >
> > > > > conf_token @0 :UInt32;
> > > > >
> > > > > }
> > > > >
> > > > > ```
> > > > >
> > > > >
> > > > >
> > > > > Here is the proto for http_req_call
> > > > >
> > > > >
> > > > >
> > > > > request
> > > > >
> > > > > ```
> > > > >
> > > > > struct Pair {
> > > > >
> > > > > name @0 :Text;
> > > > >
> > > > > value @1 :Text;
> > > > >
> > > > > }
> > > > >
> > > > > struct PairData {
> > > > >
> > > > > name @0 :Text;
> > > > >
> > > > > value @1 :Data;
> > > > >
> > > > > }
> > > > >
> > > > > enum Method {
> > > > >
> > > > > GET @0;
> > > > >
> > > > > ...
> > > > >
> > > > > }
> > > > >
> > > > >
> > > > >
> > > > > struct HTTPReqCallReq {
> > > > >
> > > > > id @0 :UInt32;
> > > > >
> > > > > src_ip @1 :Data;
> > > > >
> > > > >
> > > > >
> > > > > method @2 :Method;
> > > > >
> > > > >
> > > > >
> > > > > path @3 :Text;
> > > > >
> > > > > args @4 :List(Pair);
> > > > >
> > > > > headers @5 :List(Pair);
> > > > >
> > > > >
> > > > >
> > > > > conf_token @6 :UInt32;
> > > > >
> > > > >
> > > > >
> > > > > extra_info @7 :List(PairData);
> > > > >
> > > > > }
> > > > >
> > > > > ```
> > > > >
> > > > >
> > > > >
> > > > > Response
> > > > >
> > > > > ```
> > > > >
> > > > > struct HTTPReqCallResp {
> > > > >
> > > > > id @0 :UInt32;
> > > > >
> > > > >
> > > > >
> > > > > struct Stop {
> > > > >
> > > > > status @0 :UInt16;
> > > > >
> > > > > headers @1 :List(Pair);
> > > > >
> > > > > body @2 :Data;
> > > > >
> > > > > }
> > > > >
> > > > > struct Rewrite {
> > > > >
> > > > > path @0 :Text;
> > > > >
> > > > > headers @1 :List(Pair);
> > > > >
> > > > > # Note that args are modified in full.
> > > > >
> > > > > # Either empty, meaning no args need to be moved
> > > > >
> > > > > # or the entire modified args, not the incrementally changed
> > > > parts
> > > > >
> > > > > args @2 :List(Pair);
> > > > >
> > > > > }
> > > > >
> > > > >
> > > > >
> > > > > # What needs to be done when the response is received action
> > > > >
> > > > > action :union {
> > > > >
> > > > > # Do nothing
> > > > >
> > > > > continue @1 :Void;
> > > > >
> > > > > # Equivalent to core.response.exit(status, body), allowing
> > > > >
> > > > > additional headers to be set
> > > > >
> > > > > stop @2 :Stop;
> > > > >
> > > > > # Rewrite the request
> > > > >
> > > > > rewrite @3 :Rewrite;
> > > > >
> > > > > }
> > > > >
> > > > > }
> > > > >
> > > > > ```
> > > > >
> > > > >
> > > > >
> > > > > 2.4.4 Error handling
> > > > >
> > > > >
> > > > >
> > > > > Logging and returning 503 error codes
> > > > >
> > > > >
> > > > >
> > > > > 2.4.5 Environment variables
> > > > >
> > > > >
> > > > >
> > > > > APISIX configures the Plugin Runner with a number of environment
> > > > >
> > > > > variables when it is started.
> > > > >
> > > > >
> > > > >
> > > > > APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to
> > > > listen to
> > > > >
> > > > > APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for longer
> > > > than
> > > > > this
> > > >
Re: [Proposal] support using other languages to write plugin for APISIX
Posted by Sheng Wu <wu...@gmail.com>.
I want to remind you, as you are not using a standard RPC protocol, no
possibility of internal tracing. So how should we collect
metrics/performance-data of this cross-processes communication?
And it is Java lib(maybe others have the same issue),
https://github.com/capnproto/capnproto-java/graphs/contributors, has
very little activities. I am not sure how healthy their community is.
Have you taken a look at this? https://google.github.io/flatbuffers/
It is used in Istio, and SkyWalking used this in mesh scenarios. This
protocol has a very good performance especially when some fields are
not required to deserialize, which from my understanding, is a very
case when running the sidecar process.
Sheng Wu 吴晟
Twitter, wusheng1108
Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午6:19写道：
>
> It will hold long connections.
>
> Need to clarify that the "long connection" doesn't conflict with the
> "request / response mode". For example, gRPC's unary mode can work on
> a long connection.
>
> bzp2010 <bz...@apache.org> 于2021年4月16日周五 下午6:04写道：
> >
> > Oh, I get it.
> > Will it use a long connection based on UNIX socket to transfer data? Or is
> > it a request / response mode similar to HTTP?
> >
> > Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午5:53写道：
> >
> > > This is why we choose capnproto rather than protobuf. The
> > > encode/decode performance is important, so we choose capnproto which
> > > has a better performance than protobuf.
> > >
> > > bzp2010 <bz...@apache.org> 于2021年4月16日周五 下午4:44写道：
> > > >
> > > > Hi,
> > > > I have some questions about this proposal.
> > > >
> > > > 1. If using similar C/S mode, whether the encoding and decoding process
> > > > will affect the performance.
> > > >
> > > > 2.Can we use protobuf, a more popular encoding and decoding toolkit?
> > > >
> > > >
> > > > Thanks!
> > > >
> > > > *发件人: *Zexuan Luo <sp...@apache.org>
> > > > *发送时间: *2021年4月16日 14:52
> > > > *收件人: *dev@apisix.apache.org
> > > > *主题: *[Proposal] support using other languages to write plugin for APISIX
> > > >
> > > >
> > > >
> > > > 1. Background
> > > >
> > > >
> > > >
> > > > APISIX currently only supports writing plugins in Lua. If other
> > > >
> > > > languages are supported, it will greatly broaden the APISIX ecosystem
> > > >
> > > > and user base.
> > > >
> > > >
> > > >
> > > > 2. Solution
> > > >
> > > >
> > > >
> > > > Since WASM is not yet mature, we consider implementing it through local
> > > IPC.
> > > >
> > > >
> > > >
> > > > For the sake of discussion, the following will assume that the plugin
> > > >
> > > > is written in Java. However, in practice, our solution can be
> > > >
> > > > interfaced with other languages.
> > > >
> > > >
> > > >
> > > > 2.1 Terminology
> > > >
> > > >
> > > >
> > > > Several terms are defined here.
> > > >
> > > >
> > > >
> > > > Plugin Runner: The service that runs the plugin, written in the same
> > > >
> > > > language as the plugin. In the first version, we assume that there
> > > >
> > > > will be only one Plugin Runner.
> > > >
> > > >
> > > >
> > > > 2.2 Plugin Runner lifecycle
> > > >
> > > >
> > > >
> > > > To simplify user operation and reduce the difficulty of upgrading,
> > > >
> > > > Plugin Runner is managed by APISIX.
> > > >
> > > >
> > > >
> > > > APISIX starts the Plugin Runner when it starts and ends it when it
> > > >
> > > > ends. if the Plugin Runner quits in the middle, APISIX will restart it
> > > >
> > > > automatically.
> > > >
> > > >
> > > >
> > > > 2.3 Timing of APISIX communication with Plugin Runner
> > > >
> > > >
> > > >
> > > > ```
> > > >
> > > > Router ----> Global Plugin (written in Lua) ---> Ext Plugin
> > > >
> > > > (ext-plugin-pre-req) ----> Lua Plugin (Router)
> > > >
> > > > ---> Ext plugin (ext-plugin-post-req) ---> Upstream
> > > >
> > > > ```
> > > >
> > > >
> > > >
> > > > Running the Ext Plugin in the Global Plugin is not supported at this
> > > >
> > > > time, as the global logic can be executed uniformly in the Plugin
> > > >
> > > > Runner.
> > > >
> > > >
> > > >
> > > > Running Ext Plugin after getting an upstream response is not supported
> > > >
> > > > at this time. We can support it later with a buffering response.
> > > >
> > > >
> > > >
> > > > ext-plugin-pre runs before all non-global Lua plugins, and
> > > >
> > > > ext-plugin-post runs after all non-global Lua plugins.
> > > >
> > > >
> > > >
> > > > 2.4 How APISIX communicates with Plugin Runner
> > > >
> > > >
> > > >
> > > > APISIX communicates with Plugin Runner through a unix socket. The
> > > >
> > > > communication protocol is as follows.
> > > >
> > > >
> > > >
> > > > 2.4.1 Communication format
> > > >
> > > >
> > > >
> > > > ```
> > > >
> > > > 1 byte of type + 3 bytes of length + data
> > > >
> > > > ```
> > > >
> > > >
> > > >
> > > > The type can be 0 ~ 7, and the length can be [0, 8M). data length is
> > > >
> > > > determined by length.
> > > >
> > > > Since Ext Plugin usually does not exchange too much data, 8M should be
> > > >
> > > > enough. The reason for taking 4 bytes is to keep the header small
> > > >
> > > > enough to be read efficiently.
> > > >
> > > >
> > > >
> > > > The current type takes the following values.
> > > >
> > > >
> > > >
> > > > 0 means error
> > > >
> > > > 1 means prepare_conf
> > > >
> > > > 2 means http_req_call
> > > >
> > > >
> > > >
> > > > The data is serialized in capnproto, a binary serialization format.
> > > >
> > > >
> > > >
> > > > capnproto is supported by many programming languages:
> > > >
> > > > https://capnproto.org/otherlang.html
> > > >
> > > >
> > > >
> > > > The advantages of choosing capnproto are.
> > > >
> > > > 1. focus on serialization performance
> > > >
> > > > 2. partial deserialization support, so that decode can be done only
> > > >
> > > > when it is needed
> > > >
> > > >
> > > >
> > > > 2.4.2 Communication steps
> > > >
> > > >
> > > >
> > > > Each ext plugin will have the following configuration.
> > > >
> > > >
> > > >
> > > > ```
> > > >
> > > > {
> > > >
> > > > "conf": [
> > > >
> > > > {
> > > >
> > > > "name": "configuration name",
> > > >
> > > > "value": "configuration value"
> > > >
> > > > }
> > > >
> > > > ],
> > > >
> > > > "extra_info": [
> > > >
> > > > ...
> > > >
> > > > ]
> > > >
> > > > }
> > > >
> > > > ```
> > > >
> > > >
> > > >
> > > > conf can be used to set the execution configuration of the
> > > >
> > > > plugin-related requests inside Plugin Runner.
> > > >
> > > >
> > > >
> > > > The default data sent to Plugin Runner is only the most common
> > > >
> > > > information. If you want additional information, you need to declare
> > > >
> > > > it in extra_info beforehand.
> > > >
> > > >
> > > >
> > > > To save communication costs, conf is sent separately.
> > > >
> > > >
> > > >
> > > > 1. APISIX will check if conf has a corresponding token in the local
> > > cache.
> > > >
> > > > 2. If not, APISIX sends a prepare_conf request to ask Plugin Runner
> > > >
> > > > to cache the conf and return a token.
> > > >
> > > > (Note that Plugin Runner's cache time needs to be longer than APISIX's
> > > >
> > > > cache time.)
> > > >
> > > > 3. APISIX sends an http_req_call request to Plugin Runner.
> > > >
> > > > 4. Plugin Runner executes the request and returns a response to APISIX.
> > > >
> > > > 5. APISIX processes the request based on the response
> > > >
> > > >
> > > >
> > > > 2.4.3 proto
> > > >
> > > >
> > > >
> > > > Refer to https://capnproto.org/language.html
> > > >
> > > >
> > > >
> > > > The following is the proto for error
> > > >
> > > >
> > > >
> > > > response
> > > >
> > > > ```
> > > >
> > > > enum ErrorCode {
> > > >
> > > > BAD_REQUEST @0; # Plugin Runner can't understand APISIX
> > > >
> > > > SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the request
> > > >
> > > > CONF_TOKEN_NOT_FOUND @2;
> > > >
> > > > }
> > > >
> > > > struct ErrorResp {
> > > >
> > > > Code @0 :ErrorCode;
> > > >
> > > > }
> > > >
> > > > ```
> > > >
> > > >
> > > >
> > > > The following is the proto of prepare_conf
> > > >
> > > >
> > > >
> > > > request
> > > >
> > > > ```
> > > >
> > > > struct PrepareConfReq {
> > > >
> > > > conf @0 :List(Pair);
> > > >
> > > > }
> > > >
> > > > ```
> > > >
> > > >
> > > >
> > > > Response
> > > >
> > > > ```
> > > >
> > > > struct PrepareConfResp {
> > > >
> > > > conf_token @0 :UInt32;
> > > >
> > > > }
> > > >
> > > > ```
> > > >
> > > >
> > > >
> > > > Here is the proto for http_req_call
> > > >
> > > >
> > > >
> > > > request
> > > >
> > > > ```
> > > >
> > > > struct Pair {
> > > >
> > > > name @0 :Text;
> > > >
> > > > value @1 :Text;
> > > >
> > > > }
> > > >
> > > > struct PairData {
> > > >
> > > > name @0 :Text;
> > > >
> > > > value @1 :Data;
> > > >
> > > > }
> > > >
> > > > enum Method {
> > > >
> > > > GET @0;
> > > >
> > > > ...
> > > >
> > > > }
> > > >
> > > >
> > > >
> > > > struct HTTPReqCallReq {
> > > >
> > > > id @0 :UInt32;
> > > >
> > > > src_ip @1 :Data;
> > > >
> > > >
> > > >
> > > > method @2 :Method;
> > > >
> > > >
> > > >
> > > > path @3 :Text;
> > > >
> > > > args @4 :List(Pair);
> > > >
> > > > headers @5 :List(Pair);
> > > >
> > > >
> > > >
> > > > conf_token @6 :UInt32;
> > > >
> > > >
> > > >
> > > > extra_info @7 :List(PairData);
> > > >
> > > > }
> > > >
> > > > ```
> > > >
> > > >
> > > >
> > > > Response
> > > >
> > > > ```
> > > >
> > > > struct HTTPReqCallResp {
> > > >
> > > > id @0 :UInt32;
> > > >
> > > >
> > > >
> > > > struct Stop {
> > > >
> > > > status @0 :UInt16;
> > > >
> > > > headers @1 :List(Pair);
> > > >
> > > > body @2 :Data;
> > > >
> > > > }
> > > >
> > > > struct Rewrite {
> > > >
> > > > path @0 :Text;
> > > >
> > > > headers @1 :List(Pair);
> > > >
> > > > # Note that args are modified in full.
> > > >
> > > > # Either empty, meaning no args need to be moved
> > > >
> > > > # or the entire modified args, not the incrementally changed
> > > parts
> > > >
> > > > args @2 :List(Pair);
> > > >
> > > > }
> > > >
> > > >
> > > >
> > > > # What needs to be done when the response is received action
> > > >
> > > > action :union {
> > > >
> > > > # Do nothing
> > > >
> > > > continue @1 :Void;
> > > >
> > > > # Equivalent to core.response.exit(status, body), allowing
> > > >
> > > > additional headers to be set
> > > >
> > > > stop @2 :Stop;
> > > >
> > > > # Rewrite the request
> > > >
> > > > rewrite @3 :Rewrite;
> > > >
> > > > }
> > > >
> > > > }
> > > >
> > > > ```
> > > >
> > > >
> > > >
> > > > 2.4.4 Error handling
> > > >
> > > >
> > > >
> > > > Logging and returning 503 error codes
> > > >
> > > >
> > > >
> > > > 2.4.5 Environment variables
> > > >
> > > >
> > > >
> > > > APISIX configures the Plugin Runner with a number of environment
> > > >
> > > > variables when it is started.
> > > >
> > > >
> > > >
> > > > APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to
> > > listen to
> > > >
> > > > APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for longer
> > > than
> > > > this
> > >
Re: [Proposal] support using other languages to write plugin for APISIX
Posted by Zexuan Luo <sp...@apache.org>.
It will hold long connections.
Need to clarify that the "long connection" doesn't conflict with the
"request / response mode". For example, gRPC's unary mode can work on
a long connection.
bzp2010 <bz...@apache.org> 于2021年4月16日周五 下午6:04写道：
>
> Oh, I get it.
> Will it use a long connection based on UNIX socket to transfer data? Or is
> it a request / response mode similar to HTTP?
>
> Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午5:53写道：
>
> > This is why we choose capnproto rather than protobuf. The
> > encode/decode performance is important, so we choose capnproto which
> > has a better performance than protobuf.
> >
> > bzp2010 <bz...@apache.org> 于2021年4月16日周五 下午4:44写道：
> > >
> > > Hi,
> > > I have some questions about this proposal.
> > >
> > > 1. If using similar C/S mode, whether the encoding and decoding process
> > > will affect the performance.
> > >
> > > 2.Can we use protobuf, a more popular encoding and decoding toolkit?
> > >
> > >
> > > Thanks!
> > >
> > > *发件人: *Zexuan Luo <sp...@apache.org>
> > > *发送时间: *2021年4月16日 14:52
> > > *收件人: *dev@apisix.apache.org
> > > *主题: *[Proposal] support using other languages to write plugin for APISIX
> > >
> > >
> > >
> > > 1. Background
> > >
> > >
> > >
> > > APISIX currently only supports writing plugins in Lua. If other
> > >
> > > languages are supported, it will greatly broaden the APISIX ecosystem
> > >
> > > and user base.
> > >
> > >
> > >
> > > 2. Solution
> > >
> > >
> > >
> > > Since WASM is not yet mature, we consider implementing it through local
> > IPC.
> > >
> > >
> > >
> > > For the sake of discussion, the following will assume that the plugin
> > >
> > > is written in Java. However, in practice, our solution can be
> > >
> > > interfaced with other languages.
> > >
> > >
> > >
> > > 2.1 Terminology
> > >
> > >
> > >
> > > Several terms are defined here.
> > >
> > >
> > >
> > > Plugin Runner: The service that runs the plugin, written in the same
> > >
> > > language as the plugin. In the first version, we assume that there
> > >
> > > will be only one Plugin Runner.
> > >
> > >
> > >
> > > 2.2 Plugin Runner lifecycle
> > >
> > >
> > >
> > > To simplify user operation and reduce the difficulty of upgrading,
> > >
> > > Plugin Runner is managed by APISIX.
> > >
> > >
> > >
> > > APISIX starts the Plugin Runner when it starts and ends it when it
> > >
> > > ends. if the Plugin Runner quits in the middle, APISIX will restart it
> > >
> > > automatically.
> > >
> > >
> > >
> > > 2.3 Timing of APISIX communication with Plugin Runner
> > >
> > >
> > >
> > > ```
> > >
> > > Router ----> Global Plugin (written in Lua) ---> Ext Plugin
> > >
> > > (ext-plugin-pre-req) ----> Lua Plugin (Router)
> > >
> > > ---> Ext plugin (ext-plugin-post-req) ---> Upstream
> > >
> > > ```
> > >
> > >
> > >
> > > Running the Ext Plugin in the Global Plugin is not supported at this
> > >
> > > time, as the global logic can be executed uniformly in the Plugin
> > >
> > > Runner.
> > >
> > >
> > >
> > > Running Ext Plugin after getting an upstream response is not supported
> > >
> > > at this time. We can support it later with a buffering response.
> > >
> > >
> > >
> > > ext-plugin-pre runs before all non-global Lua plugins, and
> > >
> > > ext-plugin-post runs after all non-global Lua plugins.
> > >
> > >
> > >
> > > 2.4 How APISIX communicates with Plugin Runner
> > >
> > >
> > >
> > > APISIX communicates with Plugin Runner through a unix socket. The
> > >
> > > communication protocol is as follows.
> > >
> > >
> > >
> > > 2.4.1 Communication format
> > >
> > >
> > >
> > > ```
> > >
> > > 1 byte of type + 3 bytes of length + data
> > >
> > > ```
> > >
> > >
> > >
> > > The type can be 0 ~ 7, and the length can be [0, 8M). data length is
> > >
> > > determined by length.
> > >
> > > Since Ext Plugin usually does not exchange too much data, 8M should be
> > >
> > > enough. The reason for taking 4 bytes is to keep the header small
> > >
> > > enough to be read efficiently.
> > >
> > >
> > >
> > > The current type takes the following values.
> > >
> > >
> > >
> > > 0 means error
> > >
> > > 1 means prepare_conf
> > >
> > > 2 means http_req_call
> > >
> > >
> > >
> > > The data is serialized in capnproto, a binary serialization format.
> > >
> > >
> > >
> > > capnproto is supported by many programming languages:
> > >
> > > https://capnproto.org/otherlang.html
> > >
> > >
> > >
> > > The advantages of choosing capnproto are.
> > >
> > > 1. focus on serialization performance
> > >
> > > 2. partial deserialization support, so that decode can be done only
> > >
> > > when it is needed
> > >
> > >
> > >
> > > 2.4.2 Communication steps
> > >
> > >
> > >
> > > Each ext plugin will have the following configuration.
> > >
> > >
> > >
> > > ```
> > >
> > > {
> > >
> > > "conf": [
> > >
> > > {
> > >
> > > "name": "configuration name",
> > >
> > > "value": "configuration value"
> > >
> > > }
> > >
> > > ],
> > >
> > > "extra_info": [
> > >
> > > ...
> > >
> > > ]
> > >
> > > }
> > >
> > > ```
> > >
> > >
> > >
> > > conf can be used to set the execution configuration of the
> > >
> > > plugin-related requests inside Plugin Runner.
> > >
> > >
> > >
> > > The default data sent to Plugin Runner is only the most common
> > >
> > > information. If you want additional information, you need to declare
> > >
> > > it in extra_info beforehand.
> > >
> > >
> > >
> > > To save communication costs, conf is sent separately.
> > >
> > >
> > >
> > > 1. APISIX will check if conf has a corresponding token in the local
> > cache.
> > >
> > > 2. If not, APISIX sends a prepare_conf request to ask Plugin Runner
> > >
> > > to cache the conf and return a token.
> > >
> > > (Note that Plugin Runner's cache time needs to be longer than APISIX's
> > >
> > > cache time.)
> > >
> > > 3. APISIX sends an http_req_call request to Plugin Runner.
> > >
> > > 4. Plugin Runner executes the request and returns a response to APISIX.
> > >
> > > 5. APISIX processes the request based on the response
> > >
> > >
> > >
> > > 2.4.3 proto
> > >
> > >
> > >
> > > Refer to https://capnproto.org/language.html
> > >
> > >
> > >
> > > The following is the proto for error
> > >
> > >
> > >
> > > response
> > >
> > > ```
> > >
> > > enum ErrorCode {
> > >
> > > BAD_REQUEST @0; # Plugin Runner can't understand APISIX
> > >
> > > SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the request
> > >
> > > CONF_TOKEN_NOT_FOUND @2;
> > >
> > > }
> > >
> > > struct ErrorResp {
> > >
> > > Code @0 :ErrorCode;
> > >
> > > }
> > >
> > > ```
> > >
> > >
> > >
> > > The following is the proto of prepare_conf
> > >
> > >
> > >
> > > request
> > >
> > > ```
> > >
> > > struct PrepareConfReq {
> > >
> > > conf @0 :List(Pair);
> > >
> > > }
> > >
> > > ```
> > >
> > >
> > >
> > > Response
> > >
> > > ```
> > >
> > > struct PrepareConfResp {
> > >
> > > conf_token @0 :UInt32;
> > >
> > > }
> > >
> > > ```
> > >
> > >
> > >
> > > Here is the proto for http_req_call
> > >
> > >
> > >
> > > request
> > >
> > > ```
> > >
> > > struct Pair {
> > >
> > > name @0 :Text;
> > >
> > > value @1 :Text;
> > >
> > > }
> > >
> > > struct PairData {
> > >
> > > name @0 :Text;
> > >
> > > value @1 :Data;
> > >
> > > }
> > >
> > > enum Method {
> > >
> > > GET @0;
> > >
> > > ...
> > >
> > > }
> > >
> > >
> > >
> > > struct HTTPReqCallReq {
> > >
> > > id @0 :UInt32;
> > >
> > > src_ip @1 :Data;
> > >
> > >
> > >
> > > method @2 :Method;
> > >
> > >
> > >
> > > path @3 :Text;
> > >
> > > args @4 :List(Pair);
> > >
> > > headers @5 :List(Pair);
> > >
> > >
> > >
> > > conf_token @6 :UInt32;
> > >
> > >
> > >
> > > extra_info @7 :List(PairData);
> > >
> > > }
> > >
> > > ```
> > >
> > >
> > >
> > > Response
> > >
> > > ```
> > >
> > > struct HTTPReqCallResp {
> > >
> > > id @0 :UInt32;
> > >
> > >
> > >
> > > struct Stop {
> > >
> > > status @0 :UInt16;
> > >
> > > headers @1 :List(Pair);
> > >
> > > body @2 :Data;
> > >
> > > }
> > >
> > > struct Rewrite {
> > >
> > > path @0 :Text;
> > >
> > > headers @1 :List(Pair);
> > >
> > > # Note that args are modified in full.
> > >
> > > # Either empty, meaning no args need to be moved
> > >
> > > # or the entire modified args, not the incrementally changed
> > parts
> > >
> > > args @2 :List(Pair);
> > >
> > > }
> > >
> > >
> > >
> > > # What needs to be done when the response is received action
> > >
> > > action :union {
> > >
> > > # Do nothing
> > >
> > > continue @1 :Void;
> > >
> > > # Equivalent to core.response.exit(status, body), allowing
> > >
> > > additional headers to be set
> > >
> > > stop @2 :Stop;
> > >
> > > # Rewrite the request
> > >
> > > rewrite @3 :Rewrite;
> > >
> > > }
> > >
> > > }
> > >
> > > ```
> > >
> > >
> > >
> > > 2.4.4 Error handling
> > >
> > >
> > >
> > > Logging and returning 503 error codes
> > >
> > >
> > >
> > > 2.4.5 Environment variables
> > >
> > >
> > >
> > > APISIX configures the Plugin Runner with a number of environment
> > >
> > > variables when it is started.
> > >
> > >
> > >
> > > APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to
> > listen to
> > >
> > > APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for longer
> > than
> > > this
> >
Re: [Proposal] support using other languages to write plugin for APISIX
Posted by bzp2010 <bz...@apache.org>.
Oh, I get it.
Will it use a long connection based on UNIX socket to transfer data? Or is
it a request / response mode similar to HTTP?
Zexuan Luo <sp...@apache.org> 于2021年4月16日周五 下午5:53写道：
> This is why we choose capnproto rather than protobuf. The
> encode/decode performance is important, so we choose capnproto which
> has a better performance than protobuf.
>
> bzp2010 <bz...@apache.org> 于2021年4月16日周五 下午4:44写道：
> >
> > Hi,
> > I have some questions about this proposal.
> >
> > 1. If using similar C/S mode, whether the encoding and decoding process
> > will affect the performance.
> >
> > 2.Can we use protobuf, a more popular encoding and decoding toolkit?
> >
> >
> > Thanks!
> >
> > *发件人: *Zexuan Luo <sp...@apache.org>
> > *发送时间: *2021年4月16日 14:52
> > *收件人: *dev@apisix.apache.org
> > *主题: *[Proposal] support using other languages to write plugin for APISIX
> >
> >
> >
> > 1. Background
> >
> >
> >
> > APISIX currently only supports writing plugins in Lua. If other
> >
> > languages are supported, it will greatly broaden the APISIX ecosystem
> >
> > and user base.
> >
> >
> >
> > 2. Solution
> >
> >
> >
> > Since WASM is not yet mature, we consider implementing it through local
> IPC.
> >
> >
> >
> > For the sake of discussion, the following will assume that the plugin
> >
> > is written in Java. However, in practice, our solution can be
> >
> > interfaced with other languages.
> >
> >
> >
> > 2.1 Terminology
> >
> >
> >
> > Several terms are defined here.
> >
> >
> >
> > Plugin Runner: The service that runs the plugin, written in the same
> >
> > language as the plugin. In the first version, we assume that there
> >
> > will be only one Plugin Runner.
> >
> >
> >
> > 2.2 Plugin Runner lifecycle
> >
> >
> >
> > To simplify user operation and reduce the difficulty of upgrading,
> >
> > Plugin Runner is managed by APISIX.
> >
> >
> >
> > APISIX starts the Plugin Runner when it starts and ends it when it
> >
> > ends. if the Plugin Runner quits in the middle, APISIX will restart it
> >
> > automatically.
> >
> >
> >
> > 2.3 Timing of APISIX communication with Plugin Runner
> >
> >
> >
> > ```
> >
> > Router ----> Global Plugin (written in Lua) ---> Ext Plugin
> >
> > (ext-plugin-pre-req) ----> Lua Plugin (Router)
> >
> > ---> Ext plugin (ext-plugin-post-req) ---> Upstream
> >
> > ```
> >
> >
> >
> > Running the Ext Plugin in the Global Plugin is not supported at this
> >
> > time, as the global logic can be executed uniformly in the Plugin
> >
> > Runner.
> >
> >
> >
> > Running Ext Plugin after getting an upstream response is not supported
> >
> > at this time. We can support it later with a buffering response.
> >
> >
> >
> > ext-plugin-pre runs before all non-global Lua plugins, and
> >
> > ext-plugin-post runs after all non-global Lua plugins.
> >
> >
> >
> > 2.4 How APISIX communicates with Plugin Runner
> >
> >
> >
> > APISIX communicates with Plugin Runner through a unix socket. The
> >
> > communication protocol is as follows.
> >
> >
> >
> > 2.4.1 Communication format
> >
> >
> >
> > ```
> >
> > 1 byte of type + 3 bytes of length + data
> >
> > ```
> >
> >
> >
> > The type can be 0 ~ 7, and the length can be [0, 8M). data length is
> >
> > determined by length.
> >
> > Since Ext Plugin usually does not exchange too much data, 8M should be
> >
> > enough. The reason for taking 4 bytes is to keep the header small
> >
> > enough to be read efficiently.
> >
> >
> >
> > The current type takes the following values.
> >
> >
> >
> > 0 means error
> >
> > 1 means prepare_conf
> >
> > 2 means http_req_call
> >
> >
> >
> > The data is serialized in capnproto, a binary serialization format.
> >
> >
> >
> > capnproto is supported by many programming languages:
> >
> > https://capnproto.org/otherlang.html
> >
> >
> >
> > The advantages of choosing capnproto are.
> >
> > 1. focus on serialization performance
> >
> > 2. partial deserialization support, so that decode can be done only
> >
> > when it is needed
> >
> >
> >
> > 2.4.2 Communication steps
> >
> >
> >
> > Each ext plugin will have the following configuration.
> >
> >
> >
> > ```
> >
> > {
> >
> > "conf": [
> >
> > {
> >
> > "name": "configuration name",
> >
> > "value": "configuration value"
> >
> > }
> >
> > ],
> >
> > "extra_info": [
> >
> > ...
> >
> > ]
> >
> > }
> >
> > ```
> >
> >
> >
> > conf can be used to set the execution configuration of the
> >
> > plugin-related requests inside Plugin Runner.
> >
> >
> >
> > The default data sent to Plugin Runner is only the most common
> >
> > information. If you want additional information, you need to declare
> >
> > it in extra_info beforehand.
> >
> >
> >
> > To save communication costs, conf is sent separately.
> >
> >
> >
> > 1. APISIX will check if conf has a corresponding token in the local
> cache.
> >
> > 2. If not, APISIX sends a prepare_conf request to ask Plugin Runner
> >
> > to cache the conf and return a token.
> >
> > (Note that Plugin Runner's cache time needs to be longer than APISIX's
> >
> > cache time.)
> >
> > 3. APISIX sends an http_req_call request to Plugin Runner.
> >
> > 4. Plugin Runner executes the request and returns a response to APISIX.
> >
> > 5. APISIX processes the request based on the response
> >
> >
> >
> > 2.4.3 proto
> >
> >
> >
> > Refer to https://capnproto.org/language.html
> >
> >
> >
> > The following is the proto for error
> >
> >
> >
> > response
> >
> > ```
> >
> > enum ErrorCode {
> >
> > BAD_REQUEST @0; # Plugin Runner can't understand APISIX
> >
> > SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the request
> >
> > CONF_TOKEN_NOT_FOUND @2;
> >
> > }
> >
> > struct ErrorResp {
> >
> > Code @0 :ErrorCode;
> >
> > }
> >
> > ```
> >
> >
> >
> > The following is the proto of prepare_conf
> >
> >
> >
> > request
> >
> > ```
> >
> > struct PrepareConfReq {
> >
> > conf @0 :List(Pair);
> >
> > }
> >
> > ```
> >
> >
> >
> > Response
> >
> > ```
> >
> > struct PrepareConfResp {
> >
> > conf_token @0 :UInt32;
> >
> > }
> >
> > ```
> >
> >
> >
> > Here is the proto for http_req_call
> >
> >
> >
> > request
> >
> > ```
> >
> > struct Pair {
> >
> > name @0 :Text;
> >
> > value @1 :Text;
> >
> > }
> >
> > struct PairData {
> >
> > name @0 :Text;
> >
> > value @1 :Data;
> >
> > }
> >
> > enum Method {
> >
> > GET @0;
> >
> > ...
> >
> > }
> >
> >
> >
> > struct HTTPReqCallReq {
> >
> > id @0 :UInt32;
> >
> > src_ip @1 :Data;
> >
> >
> >
> > method @2 :Method;
> >
> >
> >
> > path @3 :Text;
> >
> > args @4 :List(Pair);
> >
> > headers @5 :List(Pair);
> >
> >
> >
> > conf_token @6 :UInt32;
> >
> >
> >
> > extra_info @7 :List(PairData);
> >
> > }
> >
> > ```
> >
> >
> >
> > Response
> >
> > ```
> >
> > struct HTTPReqCallResp {
> >
> > id @0 :UInt32;
> >
> >
> >
> > struct Stop {
> >
> > status @0 :UInt16;
> >
> > headers @1 :List(Pair);
> >
> > body @2 :Data;
> >
> > }
> >
> > struct Rewrite {
> >
> > path @0 :Text;
> >
> > headers @1 :List(Pair);
> >
> > # Note that args are modified in full.
> >
> > # Either empty, meaning no args need to be moved
> >
> > # or the entire modified args, not the incrementally changed
> parts
> >
> > args @2 :List(Pair);
> >
> > }
> >
> >
> >
> > # What needs to be done when the response is received action
> >
> > action :union {
> >
> > # Do nothing
> >
> > continue @1 :Void;
> >
> > # Equivalent to core.response.exit(status, body), allowing
> >
> > additional headers to be set
> >
> > stop @2 :Stop;
> >
> > # Rewrite the request
> >
> > rewrite @3 :Rewrite;
> >
> > }
> >
> > }
> >
> > ```
> >
> >
> >
> > 2.4.4 Error handling
> >
> >
> >
> > Logging and returning 503 error codes
> >
> >
> >
> > 2.4.5 Environment variables
> >
> >
> >
> > APISIX configures the Plugin Runner with a number of environment
> >
> > variables when it is started.
> >
> >
> >
> > APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to
> listen to
> >
> > APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for longer
> than
> > this
>
Re: [Proposal] support using other languages to write plugin for APISIX
Posted by Zexuan Luo <sp...@apache.org>.
This is why we choose capnproto rather than protobuf. The
encode/decode performance is important, so we choose capnproto which
has a better performance than protobuf.
bzp2010 <bz...@apache.org> 于2021年4月16日周五 下午4:44写道：
>
> Hi,
> I have some questions about this proposal.
>
> 1. If using similar C/S mode, whether the encoding and decoding process
> will affect the performance.
>
> 2.Can we use protobuf, a more popular encoding and decoding toolkit?
>
>
> Thanks!
>
> *发件人: *Zexuan Luo <sp...@apache.org>
> *发送时间: *2021年4月16日 14:52
> *收件人: *dev@apisix.apache.org
> *主题: *[Proposal] support using other languages to write plugin for APISIX
>
>
>
> 1. Background
>
>
>
> APISIX currently only supports writing plugins in Lua. If other
>
> languages are supported, it will greatly broaden the APISIX ecosystem
>
> and user base.
>
>
>
> 2. Solution
>
>
>
> Since WASM is not yet mature, we consider implementing it through local IPC.
>
>
>
> For the sake of discussion, the following will assume that the plugin
>
> is written in Java. However, in practice, our solution can be
>
> interfaced with other languages.
>
>
>
> 2.1 Terminology
>
>
>
> Several terms are defined here.
>
>
>
> Plugin Runner: The service that runs the plugin, written in the same
>
> language as the plugin. In the first version, we assume that there
>
> will be only one Plugin Runner.
>
>
>
> 2.2 Plugin Runner lifecycle
>
>
>
> To simplify user operation and reduce the difficulty of upgrading,
>
> Plugin Runner is managed by APISIX.
>
>
>
> APISIX starts the Plugin Runner when it starts and ends it when it
>
> ends. if the Plugin Runner quits in the middle, APISIX will restart it
>
> automatically.
>
>
>
> 2.3 Timing of APISIX communication with Plugin Runner
>
>
>
> ```
>
> Router ----> Global Plugin (written in Lua) ---> Ext Plugin
>
> (ext-plugin-pre-req) ----> Lua Plugin (Router)
>
> ---> Ext plugin (ext-plugin-post-req) ---> Upstream
>
> ```
>
>
>
> Running the Ext Plugin in the Global Plugin is not supported at this
>
> time, as the global logic can be executed uniformly in the Plugin
>
> Runner.
>
>
>
> Running Ext Plugin after getting an upstream response is not supported
>
> at this time. We can support it later with a buffering response.
>
>
>
> ext-plugin-pre runs before all non-global Lua plugins, and
>
> ext-plugin-post runs after all non-global Lua plugins.
>
>
>
> 2.4 How APISIX communicates with Plugin Runner
>
>
>
> APISIX communicates with Plugin Runner through a unix socket. The
>
> communication protocol is as follows.
>
>
>
> 2.4.1 Communication format
>
>
>
> ```
>
> 1 byte of type + 3 bytes of length + data
>
> ```
>
>
>
> The type can be 0 ~ 7, and the length can be [0, 8M). data length is
>
> determined by length.
>
> Since Ext Plugin usually does not exchange too much data, 8M should be
>
> enough. The reason for taking 4 bytes is to keep the header small
>
> enough to be read efficiently.
>
>
>
> The current type takes the following values.
>
>
>
> 0 means error
>
> 1 means prepare_conf
>
> 2 means http_req_call
>
>
>
> The data is serialized in capnproto, a binary serialization format.
>
>
>
> capnproto is supported by many programming languages:
>
> https://capnproto.org/otherlang.html
>
>
>
> The advantages of choosing capnproto are.
>
> 1. focus on serialization performance
>
> 2. partial deserialization support, so that decode can be done only
>
> when it is needed
>
>
>
> 2.4.2 Communication steps
>
>
>
> Each ext plugin will have the following configuration.
>
>
>
> ```
>
> {
>
> "conf": [
>
> {
>
> "name": "configuration name",
>
> "value": "configuration value"
>
> }
>
> ],
>
> "extra_info": [
>
> ...
>
> ]
>
> }
>
> ```
>
>
>
> conf can be used to set the execution configuration of the
>
> plugin-related requests inside Plugin Runner.
>
>
>
> The default data sent to Plugin Runner is only the most common
>
> information. If you want additional information, you need to declare
>
> it in extra_info beforehand.
>
>
>
> To save communication costs, conf is sent separately.
>
>
>
> 1. APISIX will check if conf has a corresponding token in the local cache.
>
> 2. If not, APISIX sends a prepare_conf request to ask Plugin Runner
>
> to cache the conf and return a token.
>
> (Note that Plugin Runner's cache time needs to be longer than APISIX's
>
> cache time.)
>
> 3. APISIX sends an http_req_call request to Plugin Runner.
>
> 4. Plugin Runner executes the request and returns a response to APISIX.
>
> 5. APISIX processes the request based on the response
>
>
>
> 2.4.3 proto
>
>
>
> Refer to https://capnproto.org/language.html
>
>
>
> The following is the proto for error
>
>
>
> response
>
> ```
>
> enum ErrorCode {
>
> BAD_REQUEST @0; # Plugin Runner can't understand APISIX
>
> SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the request
>
> CONF_TOKEN_NOT_FOUND @2;
>
> }
>
> struct ErrorResp {
>
> Code @0 :ErrorCode;
>
> }
>
> ```
>
>
>
> The following is the proto of prepare_conf
>
>
>
> request
>
> ```
>
> struct PrepareConfReq {
>
> conf @0 :List(Pair);
>
> }
>
> ```
>
>
>
> Response
>
> ```
>
> struct PrepareConfResp {
>
> conf_token @0 :UInt32;
>
> }
>
> ```
>
>
>
> Here is the proto for http_req_call
>
>
>
> request
>
> ```
>
> struct Pair {
>
> name @0 :Text;
>
> value @1 :Text;
>
> }
>
> struct PairData {
>
> name @0 :Text;
>
> value @1 :Data;
>
> }
>
> enum Method {
>
> GET @0;
>
> ...
>
> }
>
>
>
> struct HTTPReqCallReq {
>
> id @0 :UInt32;
>
> src_ip @1 :Data;
>
>
>
> method @2 :Method;
>
>
>
> path @3 :Text;
>
> args @4 :List(Pair);
>
> headers @5 :List(Pair);
>
>
>
> conf_token @6 :UInt32;
>
>
>
> extra_info @7 :List(PairData);
>
> }
>
> ```
>
>
>
> Response
>
> ```
>
> struct HTTPReqCallResp {
>
> id @0 :UInt32;
>
>
>
> struct Stop {
>
> status @0 :UInt16;
>
> headers @1 :List(Pair);
>
> body @2 :Data;
>
> }
>
> struct Rewrite {
>
> path @0 :Text;
>
> headers @1 :List(Pair);
>
> # Note that args are modified in full.
>
> # Either empty, meaning no args need to be moved
>
> # or the entire modified args, not the incrementally changed parts
>
> args @2 :List(Pair);
>
> }
>
>
>
> # What needs to be done when the response is received action
>
> action :union {
>
> # Do nothing
>
> continue @1 :Void;
>
> # Equivalent to core.response.exit(status, body), allowing
>
> additional headers to be set
>
> stop @2 :Stop;
>
> # Rewrite the request
>
> rewrite @3 :Rewrite;
>
> }
>
> }
>
> ```
>
>
>
> 2.4.4 Error handling
>
>
>
> Logging and returning 503 error codes
>
>
>
> 2.4.5 Environment variables
>
>
>
> APISIX configures the Plugin Runner with a number of environment
>
> variables when it is started.
>
>
>
> APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to listen to
>
> APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for longer than
> this
回复: [Proposal] support using other languages to write plugin for APISIX
Posted by bzp2010 <bz...@apache.org>.
Hi,
I have some questions about this proposal.
1. If using similar C/S mode, whether the encoding and decoding process
will affect the performance.
2.Can we use protobuf, a more popular encoding and decoding toolkit?
Thanks!
*发件人: *Zexuan Luo <sp...@apache.org>
*发送时间: *2021年4月16日 14:52
*收件人: *dev@apisix.apache.org
*主题: *[Proposal] support using other languages to write plugin for APISIX
1. Background
APISIX currently only supports writing plugins in Lua. If other
languages are supported, it will greatly broaden the APISIX ecosystem
and user base.
2. Solution
Since WASM is not yet mature, we consider implementing it through local IPC.
For the sake of discussion, the following will assume that the plugin
is written in Java. However, in practice, our solution can be
interfaced with other languages.
2.1 Terminology
Several terms are defined here.
Plugin Runner: The service that runs the plugin, written in the same
language as the plugin. In the first version, we assume that there
will be only one Plugin Runner.
2.2 Plugin Runner lifecycle
To simplify user operation and reduce the difficulty of upgrading,
Plugin Runner is managed by APISIX.
APISIX starts the Plugin Runner when it starts and ends it when it
ends. if the Plugin Runner quits in the middle, APISIX will restart it
automatically.
2.3 Timing of APISIX communication with Plugin Runner
```
Router ----> Global Plugin (written in Lua) ---> Ext Plugin
(ext-plugin-pre-req) ----> Lua Plugin (Router)
---> Ext plugin (ext-plugin-post-req) ---> Upstream
```
Running the Ext Plugin in the Global Plugin is not supported at this
time, as the global logic can be executed uniformly in the Plugin
Runner.
Running Ext Plugin after getting an upstream response is not supported
at this time. We can support it later with a buffering response.
ext-plugin-pre runs before all non-global Lua plugins, and
ext-plugin-post runs after all non-global Lua plugins.
2.4 How APISIX communicates with Plugin Runner
APISIX communicates with Plugin Runner through a unix socket. The
communication protocol is as follows.
2.4.1 Communication format
```
1 byte of type + 3 bytes of length + data
```
The type can be 0 ~ 7, and the length can be [0, 8M). data length is
determined by length.
Since Ext Plugin usually does not exchange too much data, 8M should be
enough. The reason for taking 4 bytes is to keep the header small
enough to be read efficiently.
The current type takes the following values.
0 means error
1 means prepare_conf
2 means http_req_call
The data is serialized in capnproto, a binary serialization format.
capnproto is supported by many programming languages:
https://capnproto.org/otherlang.html
The advantages of choosing capnproto are.
1. focus on serialization performance
2. partial deserialization support, so that decode can be done only
when it is needed
2.4.2 Communication steps
Each ext plugin will have the following configuration.
```
{
"conf": [
{
"name": "configuration name",
"value": "configuration value"
}
],
"extra_info": [
...
]
}
```
conf can be used to set the execution configuration of the
plugin-related requests inside Plugin Runner.
The default data sent to Plugin Runner is only the most common
information. If you want additional information, you need to declare
it in extra_info beforehand.
To save communication costs, conf is sent separately.
1. APISIX will check if conf has a corresponding token in the local cache.
2. If not, APISIX sends a prepare_conf request to ask Plugin Runner
to cache the conf and return a token.
(Note that Plugin Runner's cache time needs to be longer than APISIX's
cache time.)
3. APISIX sends an http_req_call request to Plugin Runner.
4. Plugin Runner executes the request and returns a response to APISIX.
5. APISIX processes the request based on the response
2.4.3 proto
Refer to https://capnproto.org/language.html
The following is the proto for error
response
```
enum ErrorCode {
BAD_REQUEST @0; # Plugin Runner can't understand APISIX
SERVICE_UNAVAILABLE @1; # Plugin Runner can't handle the request
CONF_TOKEN_NOT_FOUND @2;
}
struct ErrorResp {
Code @0 :ErrorCode;
}
```
The following is the proto of prepare_conf
request
```
struct PrepareConfReq {
conf @0 :List(Pair);
}
```
Response
```
struct PrepareConfResp {
conf_token @0 :UInt32;
}
```
Here is the proto for http_req_call
request
```
struct Pair {
name @0 :Text;
value @1 :Text;
}
struct PairData {
name @0 :Text;
value @1 :Data;
}
enum Method {
GET @0;
...
}
struct HTTPReqCallReq {
id @0 :UInt32;
src_ip @1 :Data;
method @2 :Method;
path @3 :Text;
args @4 :List(Pair);
headers @5 :List(Pair);
conf_token @6 :UInt32;
extra_info @7 :List(PairData);
}
```
Response
```
struct HTTPReqCallResp {
id @0 :UInt32;
struct Stop {
status @0 :UInt16;
headers @1 :List(Pair);
body @2 :Data;
}
struct Rewrite {
path @0 :Text;
headers @1 :List(Pair);
# Note that args are modified in full.
# Either empty, meaning no args need to be moved
# or the entire modified args, not the incrementally changed parts
args @2 :List(Pair);
}
# What needs to be done when the response is received action
action :union {
# Do nothing
continue @1 :Void;
# Equivalent to core.response.exit(status, body), allowing
additional headers to be set
stop @2 :Stop;
# Rewrite the request
rewrite @3 :Rewrite;
}
}
```
2.4.4 Error handling
Logging and returning 503 error codes
2.4.5 Environment variables
APISIX configures the Plugin Runner with a number of environment
variables when it is started.
APISIX_LISTEN_ADDRESS: the address that the Plugin Runner needs to listen to
APISIX_CONF_EXPIRE_TIME: Plugin Runner needs to cache conf for longer than
this