You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apisix.apache.org by Zexuan Luo <sp...@apache.org> on 2021/08/03 12:53:46 UTC

[DISCUSS] Introduce a way to get extra request information in the external plugin

Sometimes we need to use some information that can't be predicted
before running the extern plugin. For example, access the
`ngx.var.request_start_time` in the extern plugin.

Hence, we add a pair of messages: ExtraInfoReq & ExtraInfoResp.

Now a request from APISIX to Plugin Runner will be like this:

```
APISIX sends HTTPReqCallReq
while reply from Runner isn't HTTPReqCallResp:
    APISIX handle ExtraInfoReq
    APISIX sends ExtraInfoResp
APISIX handle HTTPReqCallResp
```

The ExtraInfo schema is:

```
namespace A6.ExtraInfo;

table Var {
    name:string;
}

table ReqBody {
}

union Info {
    // Get the `ngx.var.name`
    Var,
    // Get the request body
    ReqBody,
}

table Req {
    info:Info;
}

table Resp {
    result:[ubyte];
}
```


In some situations, user may want to process request body that is
greater than 16M.
Although handling a large body in the extern plugin is discourage, we
still need to way to adapt it instead of just raise an error.

To solve this problem, let's extend the current rpc header.

Normally we have a header like this:

1 byte type + 3 byte length + body

Since body can't be zero in all known types, we can introduce an
extension with zero length placeholder.

Now we can accept such a header to represent a body larger than 16M:

1 byte type + 3 byte zero + 4 byte length + body.

Body larger than 4GB is unsupported since we will buffer the whole
body both in APISIX and in Runner.

Re: [DISCUSS] Introduce a way to get extra request information in the external plugin

Posted by Zexuan Luo <sp...@apache.org>.
I already talked about this issue at the beginning. Let me  repeat my word here:
> Although handling a large body in the extern plugin is discourage, we
> still need to way to adapt it instead of just raise an error.

> Since body can't be zero in all known types, we can introduce an
extension with zero length placeholder.

Daming <ha...@gmail.com> 于2021年8月4日周三 上午11:43写道：
>
>
> First, I am considering whether we need to take cover the huge request body. In usually, that should upload scenario. And this scenario, it is not a good resolution that is handled by the ext-plugin.
>
> And then, do we need to consider about compatible?
>
> ————
> Haochao Zhuang @dmsolr
>
>
> > 2021年8月4日 上午11:11，ZhengSong Tu <tz...@gmail.com> 写道：
> >
> >>
> >> APISIX sends HTTPReqCallReq
> >> while reply from Runner isn't HTTPReqCallResp:
> >>
> >    APISIX handle ExtraInfoReq
> >
> >
> > I feel there is some ambiguity here.
> > The reply from Runner to APISIX isn't HTTPReqCallResp = Runner wants
> > ExtraInfo, so why not just tell APISIX that it wants ExtraInfo. Or a
> > response that specifically identifies ExtraInfo.
> >
> > I have two suggestions:
> >
> >   1. declare ExtraInfo in conf (has this been discussed before? I can't
> >   remember, sorry), for example:
> >
> > ```
> > "ext-plugin-pre-req": {
> >    "conf": [
> >        {
> >            "name":"foo",
> >            "value":"bar",
> >            "extra_info": {
> >                             "ngx.var.request_start_time": "1454670920"
> >                                 }
> >        },
> >    ]
> > }
> > ```
> >
> > 2. Before processing the HTTPReqCallResp, the Runner adds a pre-processing
> > step: checking the required ExtraInfo, and returns a Required ExtraInfo
> > true/false response.(This is to add to the points I was wondering about)
> >
> > I prefer the first one. I think developers know exactly what they want from
> > the ExtraInfo when they write the Runner plugin, so the way it's declared
> > is clearer.
> >
> >
> >
> >
> > In some situations, user may want to process request body that is greater
> >> than 16M.
> >
> >
> > Does this also include handling the body of the POST request? I have
> > encountered this problem before.
> >
> >
> >
> > 1 byte type + 3 byte zero + 4 byte length + body.
> >
> >
> > This is an upgrade to the protocol between Runner and APISIX? Sounds good
> > to me. This allows Runner to use some common TCP unpacking functions.
> >
> >
> > *ZhengSong Tu*
> > My GitHub: https://github.com/tzssangglass <https://github.com/membphis>
> > Apache APISIX: https://github.com/apache/apisix
> >
> >
> > Zexuan Luo <sp...@apache.org> 于2021年8月3日周二 下午8:54写道：
> >
> >> Sometimes we need to use some information that can't be predicted
> >> before running the extern plugin. For example, access the
> >> `ngx.var.request_start_time` in the extern plugin.
> >>
> >> Hence, we add a pair of messages: ExtraInfoReq & ExtraInfoResp.
> >>
> >> Now a request from APISIX to Plugin Runner will be like this:
> >>
> >> ```
> >> APISIX sends HTTPReqCallReq
> >> while reply from Runner isn't HTTPReqCallResp:
> >>    APISIX handle ExtraInfoReq
> >>    APISIX sends ExtraInfoResp
> >> APISIX handle HTTPReqCallResp
> >> ```
> >>
> >> The ExtraInfo schema is:
> >>
> >> ```
> >> namespace A6.ExtraInfo;
> >>
> >> table Var {
> >>    name:string;
> >> }
> >>
> >> table ReqBody {
> >> }
> >>
> >> union Info {
> >>    // Get the `ngx.var.name`
> >>    Var,
> >>    // Get the request body
> >>    ReqBody,
> >> }
> >>
> >> table Req {
> >>    info:Info;
> >> }
> >>
> >> table Resp {
> >>    result:[ubyte];
> >> }
> >> ```
> >>
> >>
> >> In some situations, user may want to process request body that is
> >> greater than 16M.
> >> Although handling a large body in the extern plugin is discourage, we
> >> still need to way to adapt it instead of just raise an error.
> >>
> >> To solve this problem, let's extend the current rpc header.
> >>
> >> Normally we have a header like this:
> >>
> >> 1 byte type + 3 byte length + body
> >>
> >> Since body can't be zero in all known types, we can introduce an
> >> extension with zero length placeholder.
> >>
> >> Now we can accept such a header to represent a body larger than 16M:
> >>
> >> 1 byte type + 3 byte zero + 4 byte length + body.
> >>
> >> Body larger than 4GB is unsupported since we will buffer the whole
> >> body both in APISIX and in Runner.
> >>
>

Re: [DISCUSS] Introduce a way to get extra request information in the external plugin

Posted by Daming <ha...@gmail.com>.
First, I am considering whether we need to take cover the huge request body. In usually, that should upload scenario. And this scenario, it is not a good resolution that is handled by the ext-plugin.

And then, do we need to consider about compatible?

————
Haochao Zhuang @dmsolr


> 2021年8月4日 上午11:11，ZhengSong Tu <tz...@gmail.com> 写道：
> 
>> 
>> APISIX sends HTTPReqCallReq
>> while reply from Runner isn't HTTPReqCallResp:
>> 
>    APISIX handle ExtraInfoReq
> 
> 
> I feel there is some ambiguity here.
> The reply from Runner to APISIX isn't HTTPReqCallResp = Runner wants
> ExtraInfo, so why not just tell APISIX that it wants ExtraInfo. Or a
> response that specifically identifies ExtraInfo.
> 
> I have two suggestions:
> 
>   1. declare ExtraInfo in conf (has this been discussed before? I can't
>   remember, sorry), for example:
> 
> ```
> "ext-plugin-pre-req": {
>    "conf": [
>        {
>            "name":"foo",
>            "value":"bar",
>            "extra_info": {
>                             "ngx.var.request_start_time": "1454670920"
>                                 }
>        },
>    ]
> }
> ```
> 
> 2. Before processing the HTTPReqCallResp, the Runner adds a pre-processing
> step: checking the required ExtraInfo, and returns a Required ExtraInfo
> true/false response.(This is to add to the points I was wondering about)
> 
> I prefer the first one. I think developers know exactly what they want from
> the ExtraInfo when they write the Runner plugin, so the way it's declared
> is clearer.
> 
> 
> 
> 
> In some situations, user may want to process request body that is greater
>> than 16M.
> 
> 
> Does this also include handling the body of the POST request? I have
> encountered this problem before.
> 
> 
> 
> 1 byte type + 3 byte zero + 4 byte length + body.
> 
> 
> This is an upgrade to the protocol between Runner and APISIX? Sounds good
> to me. This allows Runner to use some common TCP unpacking functions.
> 
> 
> *ZhengSong Tu*
> My GitHub: https://github.com/tzssangglass <https://github.com/membphis>
> Apache APISIX: https://github.com/apache/apisix
> 
> 
> Zexuan Luo <sp...@apache.org> 于2021年8月3日周二 下午8:54写道：
> 
>> Sometimes we need to use some information that can't be predicted
>> before running the extern plugin. For example, access the
>> `ngx.var.request_start_time` in the extern plugin.
>> 
>> Hence, we add a pair of messages: ExtraInfoReq & ExtraInfoResp.
>> 
>> Now a request from APISIX to Plugin Runner will be like this:
>> 
>> ```
>> APISIX sends HTTPReqCallReq
>> while reply from Runner isn't HTTPReqCallResp:
>>    APISIX handle ExtraInfoReq
>>    APISIX sends ExtraInfoResp
>> APISIX handle HTTPReqCallResp
>> ```
>> 
>> The ExtraInfo schema is:
>> 
>> ```
>> namespace A6.ExtraInfo;
>> 
>> table Var {
>>    name:string;
>> }
>> 
>> table ReqBody {
>> }
>> 
>> union Info {
>>    // Get the `ngx.var.name`
>>    Var,
>>    // Get the request body
>>    ReqBody,
>> }
>> 
>> table Req {
>>    info:Info;
>> }
>> 
>> table Resp {
>>    result:[ubyte];
>> }
>> ```
>> 
>> 
>> In some situations, user may want to process request body that is
>> greater than 16M.
>> Although handling a large body in the extern plugin is discourage, we
>> still need to way to adapt it instead of just raise an error.
>> 
>> To solve this problem, let's extend the current rpc header.
>> 
>> Normally we have a header like this:
>> 
>> 1 byte type + 3 byte length + body
>> 
>> Since body can't be zero in all known types, we can introduce an
>> extension with zero length placeholder.
>> 
>> Now we can accept such a header to represent a body larger than 16M:
>> 
>> 1 byte type + 3 byte zero + 4 byte length + body.
>> 
>> Body larger than 4GB is unsupported since we will buffer the whole
>> body both in APISIX and in Runner.
>>

Re: [DISCUSS] Introduce a way to get extra request information in the external plugin

Posted by ZhengSong Tu <tz...@gmail.com>.
agree +1
*ZhengSong Tu*
My GitHub: https://github.com/tzssangglass <https://github.com/membphis>
Apache APISIX: https://github.com/apache/apisix


Zexuan Luo <sp...@apache.org> 于2021年8月5日周四 上午9:43写道：

> Yes.
> The ExtraInfoReq already contains type and name (if needed).
>
> ZhengSong Tu <tz...@gmail.com> 于2021年8月5日周四 上午9:36写道：
> >
> > One more question: How does APISIX know which ExtraInfo the Runner
> > wants? Declare it in ExtraInfoReq？
> >
> > *ZhengSong Tu*
> > My GitHub: https://github.com/tzssangglass <https://github.com/membphis>
> > Apache APISIX: https://github.com/apache/apisix
> >
> >
> > Zexuan Luo <sp...@apache.org> 于2021年8月5日周四 上午9:01写道：
> >
> > > > 1. declare ExtraInfo in conf
> > >
> > > I used to solve it with this. But this way is too cumbersome. It let
> > > me think about Java's check exception. You have to add a new exception
> > > handler when the function you called has changed. There will be the
> > > same with the ExtraInfo
> > >
> > > > Does this also include handling the body of the POST request? I have
> > > encountered this problem before.
> > >
> > > Yes
> > >
> > > ZhengSong Tu <tz...@gmail.com> 于2021年8月4日周三 上午11:11写道：
> > > >
> > > > >
> > > > > APISIX sends HTTPReqCallReq
> > > > > while reply from Runner isn't HTTPReqCallResp:
> > > > >
> > > >     APISIX handle ExtraInfoReq
> > > >
> > > >
> > > > I feel there is some ambiguity here.
> > > > The reply from Runner to APISIX isn't HTTPReqCallResp = Runner wants
> > > > ExtraInfo, so why not just tell APISIX that it wants ExtraInfo. Or a
> > > > response that specifically identifies ExtraInfo.
> > > >
> > > > I have two suggestions:
> > > >
> > > >    1. declare ExtraInfo in conf (has this been discussed before? I
> can't
> > > >    remember, sorry), for example:
> > > >
> > > > ```
> > > > "ext-plugin-pre-req": {
> > > >     "conf": [
> > > >         {
> > > >             "name":"foo",
> > > >             "value":"bar",
> > > >             "extra_info": {
> > > >                              "ngx.var.request_start_time":
> "1454670920"
> > > >                                  }
> > > >         },
> > > >     ]
> > > > }
> > > > ```
> > > >
> > > > 2. Before processing the HTTPReqCallResp, the Runner adds a
> > > pre-processing
> > > > step: checking the required ExtraInfo, and returns a Required
> ExtraInfo
> > > > true/false response.(This is to add to the points I was wondering
> about)
> > > >
> > > > I prefer the first one. I think developers know exactly what they
> want
> > > from
> > > > the ExtraInfo when they write the Runner plugin, so the way it's
> declared
> > > > is clearer.
> > > >
> > > >
> > > >
> > > >
> > > > In some situations, user may want to process request body that is
> greater
> > > > > than 16M.
> > > >
> > > >
> > > > Does this also include handling the body of the POST request? I have
> > > > encountered this problem before.
> > > >
> > > >
> > > >
> > > > 1 byte type + 3 byte zero + 4 byte length + body.
> > > >
> > > >
> > > > This is an upgrade to the protocol between Runner and APISIX? Sounds
> good
> > > > to me. This allows Runner to use some common TCP unpacking functions.
> > > >
> > > >
> > > > *ZhengSong Tu*
> > > > My GitHub: https://github.com/tzssangglass <
> https://github.com/membphis>
> > > > Apache APISIX: https://github.com/apache/apisix
> > > >
> > > >
> > > > Zexuan Luo <sp...@apache.org> 于2021年8月3日周二 下午8:54写道：
> > > >
> > > > > Sometimes we need to use some information that can't be predicted
> > > > > before running the extern plugin. For example, access the
> > > > > `ngx.var.request_start_time` in the extern plugin.
> > > > >
> > > > > Hence, we add a pair of messages: ExtraInfoReq & ExtraInfoResp.
> > > > >
> > > > > Now a request from APISIX to Plugin Runner will be like this:
> > > > >
> > > > > ```
> > > > > APISIX sends HTTPReqCallReq
> > > > > while reply from Runner isn't HTTPReqCallResp:
> > > > >     APISIX handle ExtraInfoReq
> > > > >     APISIX sends ExtraInfoResp
> > > > > APISIX handle HTTPReqCallResp
> > > > > ```
> > > > >
> > > > > The ExtraInfo schema is:
> > > > >
> > > > > ```
> > > > > namespace A6.ExtraInfo;
> > > > >
> > > > > table Var {
> > > > >     name:string;
> > > > > }
> > > > >
> > > > > table ReqBody {
> > > > > }
> > > > >
> > > > > union Info {
> > > > >     // Get the `ngx.var.name`
> > > > >     Var,
> > > > >     // Get the request body
> > > > >     ReqBody,
> > > > > }
> > > > >
> > > > > table Req {
> > > > >     info:Info;
> > > > > }
> > > > >
> > > > > table Resp {
> > > > >     result:[ubyte];
> > > > > }
> > > > > ```
> > > > >
> > > > >
> > > > > In some situations, user may want to process request body that is
> > > > > greater than 16M.
> > > > > Although handling a large body in the extern plugin is discourage,
> we
> > > > > still need to way to adapt it instead of just raise an error.
> > > > >
> > > > > To solve this problem, let's extend the current rpc header.
> > > > >
> > > > > Normally we have a header like this:
> > > > >
> > > > > 1 byte type + 3 byte length + body
> > > > >
> > > > > Since body can't be zero in all known types, we can introduce an
> > > > > extension with zero length placeholder.
> > > > >
> > > > > Now we can accept such a header to represent a body larger than
> 16M:
> > > > >
> > > > > 1 byte type + 3 byte zero + 4 byte length + body.
> > > > >
> > > > > Body larger than 4GB is unsupported since we will buffer the whole
> > > > > body both in APISIX and in Runner.
> > > > >
> > >
>

Re: [DISCUSS] Introduce a way to get extra request information in the external plugin

Posted by Zexuan Luo <sp...@apache.org>.
Yes.
The ExtraInfoReq already contains type and name (if needed).

ZhengSong Tu <tz...@gmail.com> 于2021年8月5日周四 上午9:36写道：
>
> One more question: How does APISIX know which ExtraInfo the Runner
> wants? Declare it in ExtraInfoReq？
>
> *ZhengSong Tu*
> My GitHub: https://github.com/tzssangglass <https://github.com/membphis>
> Apache APISIX: https://github.com/apache/apisix
>
>
> Zexuan Luo <sp...@apache.org> 于2021年8月5日周四 上午9:01写道：
>
> > > 1. declare ExtraInfo in conf
> >
> > I used to solve it with this. But this way is too cumbersome. It let
> > me think about Java's check exception. You have to add a new exception
> > handler when the function you called has changed. There will be the
> > same with the ExtraInfo
> >
> > > Does this also include handling the body of the POST request? I have
> > encountered this problem before.
> >
> > Yes
> >
> > ZhengSong Tu <tz...@gmail.com> 于2021年8月4日周三 上午11:11写道：
> > >
> > > >
> > > > APISIX sends HTTPReqCallReq
> > > > while reply from Runner isn't HTTPReqCallResp:
> > > >
> > >     APISIX handle ExtraInfoReq
> > >
> > >
> > > I feel there is some ambiguity here.
> > > The reply from Runner to APISIX isn't HTTPReqCallResp = Runner wants
> > > ExtraInfo, so why not just tell APISIX that it wants ExtraInfo. Or a
> > > response that specifically identifies ExtraInfo.
> > >
> > > I have two suggestions:
> > >
> > >    1. declare ExtraInfo in conf (has this been discussed before? I can't
> > >    remember, sorry), for example:
> > >
> > > ```
> > > "ext-plugin-pre-req": {
> > >     "conf": [
> > >         {
> > >             "name":"foo",
> > >             "value":"bar",
> > >             "extra_info": {
> > >                              "ngx.var.request_start_time": "1454670920"
> > >                                  }
> > >         },
> > >     ]
> > > }
> > > ```
> > >
> > > 2. Before processing the HTTPReqCallResp, the Runner adds a
> > pre-processing
> > > step: checking the required ExtraInfo, and returns a Required ExtraInfo
> > > true/false response.(This is to add to the points I was wondering about)
> > >
> > > I prefer the first one. I think developers know exactly what they want
> > from
> > > the ExtraInfo when they write the Runner plugin, so the way it's declared
> > > is clearer.
> > >
> > >
> > >
> > >
> > > In some situations, user may want to process request body that is greater
> > > > than 16M.
> > >
> > >
> > > Does this also include handling the body of the POST request? I have
> > > encountered this problem before.
> > >
> > >
> > >
> > > 1 byte type + 3 byte zero + 4 byte length + body.
> > >
> > >
> > > This is an upgrade to the protocol between Runner and APISIX? Sounds good
> > > to me. This allows Runner to use some common TCP unpacking functions.
> > >
> > >
> > > *ZhengSong Tu*
> > > My GitHub: https://github.com/tzssangglass <https://github.com/membphis>
> > > Apache APISIX: https://github.com/apache/apisix
> > >
> > >
> > > Zexuan Luo <sp...@apache.org> 于2021年8月3日周二 下午8:54写道：
> > >
> > > > Sometimes we need to use some information that can't be predicted
> > > > before running the extern plugin. For example, access the
> > > > `ngx.var.request_start_time` in the extern plugin.
> > > >
> > > > Hence, we add a pair of messages: ExtraInfoReq & ExtraInfoResp.
> > > >
> > > > Now a request from APISIX to Plugin Runner will be like this:
> > > >
> > > > ```
> > > > APISIX sends HTTPReqCallReq
> > > > while reply from Runner isn't HTTPReqCallResp:
> > > >     APISIX handle ExtraInfoReq
> > > >     APISIX sends ExtraInfoResp
> > > > APISIX handle HTTPReqCallResp
> > > > ```
> > > >
> > > > The ExtraInfo schema is:
> > > >
> > > > ```
> > > > namespace A6.ExtraInfo;
> > > >
> > > > table Var {
> > > >     name:string;
> > > > }
> > > >
> > > > table ReqBody {
> > > > }
> > > >
> > > > union Info {
> > > >     // Get the `ngx.var.name`
> > > >     Var,
> > > >     // Get the request body
> > > >     ReqBody,
> > > > }
> > > >
> > > > table Req {
> > > >     info:Info;
> > > > }
> > > >
> > > > table Resp {
> > > >     result:[ubyte];
> > > > }
> > > > ```
> > > >
> > > >
> > > > In some situations, user may want to process request body that is
> > > > greater than 16M.
> > > > Although handling a large body in the extern plugin is discourage, we
> > > > still need to way to adapt it instead of just raise an error.
> > > >
> > > > To solve this problem, let's extend the current rpc header.
> > > >
> > > > Normally we have a header like this:
> > > >
> > > > 1 byte type + 3 byte length + body
> > > >
> > > > Since body can't be zero in all known types, we can introduce an
> > > > extension with zero length placeholder.
> > > >
> > > > Now we can accept such a header to represent a body larger than 16M:
> > > >
> > > > 1 byte type + 3 byte zero + 4 byte length + body.
> > > >
> > > > Body larger than 4GB is unsupported since we will buffer the whole
> > > > body both in APISIX and in Runner.
> > > >
> >

Re: [DISCUSS] Introduce a way to get extra request information in the external plugin

Posted by ZhengSong Tu <tz...@gmail.com>.
One more question: How does APISIX know which ExtraInfo the Runner
wants? Declare it in ExtraInfoReq？

*ZhengSong Tu*
My GitHub: https://github.com/tzssangglass <https://github.com/membphis>
Apache APISIX: https://github.com/apache/apisix


Zexuan Luo <sp...@apache.org> 于2021年8月5日周四 上午9:01写道：

> > 1. declare ExtraInfo in conf
>
> I used to solve it with this. But this way is too cumbersome. It let
> me think about Java's check exception. You have to add a new exception
> handler when the function you called has changed. There will be the
> same with the ExtraInfo
>
> > Does this also include handling the body of the POST request? I have
> encountered this problem before.
>
> Yes
>
> ZhengSong Tu <tz...@gmail.com> 于2021年8月4日周三 上午11:11写道：
> >
> > >
> > > APISIX sends HTTPReqCallReq
> > > while reply from Runner isn't HTTPReqCallResp:
> > >
> >     APISIX handle ExtraInfoReq
> >
> >
> > I feel there is some ambiguity here.
> > The reply from Runner to APISIX isn't HTTPReqCallResp = Runner wants
> > ExtraInfo, so why not just tell APISIX that it wants ExtraInfo. Or a
> > response that specifically identifies ExtraInfo.
> >
> > I have two suggestions:
> >
> >    1. declare ExtraInfo in conf (has this been discussed before? I can't
> >    remember, sorry), for example:
> >
> > ```
> > "ext-plugin-pre-req": {
> >     "conf": [
> >         {
> >             "name":"foo",
> >             "value":"bar",
> >             "extra_info": {
> >                              "ngx.var.request_start_time": "1454670920"
> >                                  }
> >         },
> >     ]
> > }
> > ```
> >
> > 2. Before processing the HTTPReqCallResp, the Runner adds a
> pre-processing
> > step: checking the required ExtraInfo, and returns a Required ExtraInfo
> > true/false response.(This is to add to the points I was wondering about)
> >
> > I prefer the first one. I think developers know exactly what they want
> from
> > the ExtraInfo when they write the Runner plugin, so the way it's declared
> > is clearer.
> >
> >
> >
> >
> > In some situations, user may want to process request body that is greater
> > > than 16M.
> >
> >
> > Does this also include handling the body of the POST request? I have
> > encountered this problem before.
> >
> >
> >
> > 1 byte type + 3 byte zero + 4 byte length + body.
> >
> >
> > This is an upgrade to the protocol between Runner and APISIX? Sounds good
> > to me. This allows Runner to use some common TCP unpacking functions.
> >
> >
> > *ZhengSong Tu*
> > My GitHub: https://github.com/tzssangglass <https://github.com/membphis>
> > Apache APISIX: https://github.com/apache/apisix
> >
> >
> > Zexuan Luo <sp...@apache.org> 于2021年8月3日周二 下午8:54写道：
> >
> > > Sometimes we need to use some information that can't be predicted
> > > before running the extern plugin. For example, access the
> > > `ngx.var.request_start_time` in the extern plugin.
> > >
> > > Hence, we add a pair of messages: ExtraInfoReq & ExtraInfoResp.
> > >
> > > Now a request from APISIX to Plugin Runner will be like this:
> > >
> > > ```
> > > APISIX sends HTTPReqCallReq
> > > while reply from Runner isn't HTTPReqCallResp:
> > >     APISIX handle ExtraInfoReq
> > >     APISIX sends ExtraInfoResp
> > > APISIX handle HTTPReqCallResp
> > > ```
> > >
> > > The ExtraInfo schema is:
> > >
> > > ```
> > > namespace A6.ExtraInfo;
> > >
> > > table Var {
> > >     name:string;
> > > }
> > >
> > > table ReqBody {
> > > }
> > >
> > > union Info {
> > >     // Get the `ngx.var.name`
> > >     Var,
> > >     // Get the request body
> > >     ReqBody,
> > > }
> > >
> > > table Req {
> > >     info:Info;
> > > }
> > >
> > > table Resp {
> > >     result:[ubyte];
> > > }
> > > ```
> > >
> > >
> > > In some situations, user may want to process request body that is
> > > greater than 16M.
> > > Although handling a large body in the extern plugin is discourage, we
> > > still need to way to adapt it instead of just raise an error.
> > >
> > > To solve this problem, let's extend the current rpc header.
> > >
> > > Normally we have a header like this:
> > >
> > > 1 byte type + 3 byte length + body
> > >
> > > Since body can't be zero in all known types, we can introduce an
> > > extension with zero length placeholder.
> > >
> > > Now we can accept such a header to represent a body larger than 16M:
> > >
> > > 1 byte type + 3 byte zero + 4 byte length + body.
> > >
> > > Body larger than 4GB is unsupported since we will buffer the whole
> > > body both in APISIX and in Runner.
> > >
>

Re: [DISCUSS] Introduce a way to get extra request information in the external plugin

Posted by Zexuan Luo <sp...@apache.org>.
> 1. declare ExtraInfo in conf

I used to solve it with this. But this way is too cumbersome. It let
me think about Java's check exception. You have to add a new exception
handler when the function you called has changed. There will be the
same with the ExtraInfo

> Does this also include handling the body of the POST request? I have
encountered this problem before.

Yes

ZhengSong Tu <tz...@gmail.com> 于2021年8月4日周三 上午11:11写道：
>
> >
> > APISIX sends HTTPReqCallReq
> > while reply from Runner isn't HTTPReqCallResp:
> >
>     APISIX handle ExtraInfoReq
>
>
> I feel there is some ambiguity here.
> The reply from Runner to APISIX isn't HTTPReqCallResp = Runner wants
> ExtraInfo, so why not just tell APISIX that it wants ExtraInfo. Or a
> response that specifically identifies ExtraInfo.
>
> I have two suggestions:
>
>    1. declare ExtraInfo in conf (has this been discussed before? I can't
>    remember, sorry), for example:
>
> ```
> "ext-plugin-pre-req": {
>     "conf": [
>         {
>             "name":"foo",
>             "value":"bar",
>             "extra_info": {
>                              "ngx.var.request_start_time": "1454670920"
>                                  }
>         },
>     ]
> }
> ```
>
> 2. Before processing the HTTPReqCallResp, the Runner adds a pre-processing
> step: checking the required ExtraInfo, and returns a Required ExtraInfo
> true/false response.(This is to add to the points I was wondering about)
>
> I prefer the first one. I think developers know exactly what they want from
> the ExtraInfo when they write the Runner plugin, so the way it's declared
> is clearer.
>
>
>
>
> In some situations, user may want to process request body that is greater
> > than 16M.
>
>
> Does this also include handling the body of the POST request? I have
> encountered this problem before.
>
>
>
> 1 byte type + 3 byte zero + 4 byte length + body.
>
>
> This is an upgrade to the protocol between Runner and APISIX? Sounds good
> to me. This allows Runner to use some common TCP unpacking functions.
>
>
> *ZhengSong Tu*
> My GitHub: https://github.com/tzssangglass <https://github.com/membphis>
> Apache APISIX: https://github.com/apache/apisix
>
>
> Zexuan Luo <sp...@apache.org> 于2021年8月3日周二 下午8:54写道：
>
> > Sometimes we need to use some information that can't be predicted
> > before running the extern plugin. For example, access the
> > `ngx.var.request_start_time` in the extern plugin.
> >
> > Hence, we add a pair of messages: ExtraInfoReq & ExtraInfoResp.
> >
> > Now a request from APISIX to Plugin Runner will be like this:
> >
> > ```
> > APISIX sends HTTPReqCallReq
> > while reply from Runner isn't HTTPReqCallResp:
> >     APISIX handle ExtraInfoReq
> >     APISIX sends ExtraInfoResp
> > APISIX handle HTTPReqCallResp
> > ```
> >
> > The ExtraInfo schema is:
> >
> > ```
> > namespace A6.ExtraInfo;
> >
> > table Var {
> >     name:string;
> > }
> >
> > table ReqBody {
> > }
> >
> > union Info {
> >     // Get the `ngx.var.name`
> >     Var,
> >     // Get the request body
> >     ReqBody,
> > }
> >
> > table Req {
> >     info:Info;
> > }
> >
> > table Resp {
> >     result:[ubyte];
> > }
> > ```
> >
> >
> > In some situations, user may want to process request body that is
> > greater than 16M.
> > Although handling a large body in the extern plugin is discourage, we
> > still need to way to adapt it instead of just raise an error.
> >
> > To solve this problem, let's extend the current rpc header.
> >
> > Normally we have a header like this:
> >
> > 1 byte type + 3 byte length + body
> >
> > Since body can't be zero in all known types, we can introduce an
> > extension with zero length placeholder.
> >
> > Now we can accept such a header to represent a body larger than 16M:
> >
> > 1 byte type + 3 byte zero + 4 byte length + body.
> >
> > Body larger than 4GB is unsupported since we will buffer the whole
> > body both in APISIX and in Runner.
> >

Re: [DISCUSS] Introduce a way to get extra request information in the external plugin

Posted by ZhengSong Tu <tz...@gmail.com>.
>
> APISIX sends HTTPReqCallReq
> while reply from Runner isn't HTTPReqCallResp:
>
    APISIX handle ExtraInfoReq


I feel there is some ambiguity here.
The reply from Runner to APISIX isn't HTTPReqCallResp = Runner wants
ExtraInfo, so why not just tell APISIX that it wants ExtraInfo. Or a
response that specifically identifies ExtraInfo.

I have two suggestions:

   1. declare ExtraInfo in conf (has this been discussed before? I can't
   remember, sorry), for example:

```
"ext-plugin-pre-req": {
    "conf": [
        {
            "name":"foo",
            "value":"bar",
            "extra_info": {
                             "ngx.var.request_start_time": "1454670920"
                                 }
        },
    ]
}
```

2. Before processing the HTTPReqCallResp, the Runner adds a pre-processing
step: checking the required ExtraInfo, and returns a Required ExtraInfo
true/false response.(This is to add to the points I was wondering about)

I prefer the first one. I think developers know exactly what they want from
the ExtraInfo when they write the Runner plugin, so the way it's declared
is clearer.




In some situations, user may want to process request body that is greater
> than 16M.


Does this also include handling the body of the POST request? I have
encountered this problem before.



1 byte type + 3 byte zero + 4 byte length + body.


This is an upgrade to the protocol between Runner and APISIX? Sounds good
to me. This allows Runner to use some common TCP unpacking functions.


*ZhengSong Tu*
My GitHub: https://github.com/tzssangglass <https://github.com/membphis>
Apache APISIX: https://github.com/apache/apisix


Zexuan Luo <sp...@apache.org> 于2021年8月3日周二 下午8:54写道：

> Sometimes we need to use some information that can't be predicted
> before running the extern plugin. For example, access the
> `ngx.var.request_start_time` in the extern plugin.
>
> Hence, we add a pair of messages: ExtraInfoReq & ExtraInfoResp.
>
> Now a request from APISIX to Plugin Runner will be like this:
>
> ```
> APISIX sends HTTPReqCallReq
> while reply from Runner isn't HTTPReqCallResp:
>     APISIX handle ExtraInfoReq
>     APISIX sends ExtraInfoResp
> APISIX handle HTTPReqCallResp
> ```
>
> The ExtraInfo schema is:
>
> ```
> namespace A6.ExtraInfo;
>
> table Var {
>     name:string;
> }
>
> table ReqBody {
> }
>
> union Info {
>     // Get the `ngx.var.name`
>     Var,
>     // Get the request body
>     ReqBody,
> }
>
> table Req {
>     info:Info;
> }
>
> table Resp {
>     result:[ubyte];
> }
> ```
>
>
> In some situations, user may want to process request body that is
> greater than 16M.
> Although handling a large body in the extern plugin is discourage, we
> still need to way to adapt it instead of just raise an error.
>
> To solve this problem, let's extend the current rpc header.
>
> Normally we have a header like this:
>
> 1 byte type + 3 byte length + body
>
> Since body can't be zero in all known types, we can introduce an
> extension with zero length placeholder.
>
> Now we can accept such a header to represent a body larger than 16M:
>
> 1 byte type + 3 byte zero + 4 byte length + body.
>
> Body larger than 4GB is unsupported since we will buffer the whole
> body both in APISIX and in Runner.
>