You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/11/11 03:17:44 UTC
[GitHub] [apisix] crazyMonkey1995 opened a new issue, #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
crazyMonkey1995 opened a new issue, #8311:
URL: https://github.com/apache/apisix/issues/8311
### Description
Recently, we deployed apisix on different clouds and used the feature of kubernetes service discovery. The problem is that on different clouds, the proxy layer (LB) in front of apiserver has different idle timeouts. However, in apisix's kubernetes service discovery, The time of a watch is fixed, which will cause a problem: when there is no endpoints event in the cluster for a long time, the server will time out instead of the client, and then the service discovery will restart the list-watch after a fixed 40 seconds , so if you can add some configuration items, such as the duration of a watch, retry time or strategy, etc., thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #8311:
URL: https://github.com/apache/apisix/issues/8311#issuecomment-1323138072
> ```yaml
> kubernetes:
> service: ...
> client: ...
> retry_interval: 30
> min_watch: 1800
> max_watch: 2000
> ```
what about
```yaml
kubernetes:
service: ...
client: ...
retry_interval: 30
watch:
connect:
send:
read:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhixiongdu027 commented on issue #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
Posted by GitBox <gi...@apache.org>.
zhixiongdu027 commented on issue #8311:
URL: https://github.com/apache/apisix/issues/8311#issuecomment-1316691707
@crazyMonkey1995 @tokers @tzssangglass
I would like to make a PR for "support configuration watchSeconds and retryInterval" latter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] crazyMonkey1995 commented on issue #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
Posted by GitBox <gi...@apache.org>.
crazyMonkey1995 commented on issue #8311:
URL: https://github.com/apache/apisix/issues/8311#issuecomment-1316184399
> Too short watchSeconds value will produce many "re list-watch" Too long watchSeconds value will cause the proxy to terminate the connection early
>
> do we have to use an proxy before apiserver ?
In actual usage scenarios, such as Alibaba Cloud, AWS, Azure, etc., the apiserver will have a proxy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhixiongdu027 commented on issue #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
Posted by GitBox <gi...@apache.org>.
zhixiongdu027 commented on issue #8311:
URL: https://github.com/apache/apisix/issues/8311#issuecomment-1316159475
Too short watchSeconds value will produce many "re list-watch"
Too long watchSeconds value will cause the proxy to terminate the connection early
do we have to use an proxy before apiserver ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhixiongdu027 commented on issue #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
Posted by GitBox <gi...@apache.org>.
zhixiongdu027 commented on issue #8311:
URL: https://github.com/apache/apisix/issues/8311#issuecomment-1322959352
> I would like to make a PR for "support configuration watchSeconds and retryInterval" latter
I tend to use a config in the following format, or any other suggestions ?
```yaml
kubernetes:
service: ...
client: ...
retry_interval: 30
min_watch: 1800
max_watch: 2000
```
@crazyMonkey1995 @tokers @tzssangglass @spacewander
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhixiongdu027 commented on issue #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
Posted by GitBox <gi...@apache.org>.
zhixiongdu027 commented on issue #8311:
URL: https://github.com/apache/apisix/issues/8311#issuecomment-1316117745
My take on this issue:
1. "40s" is just a retry interval, it has nothing to do with this problem
2. The root of the problem is in the following network topology:
discovery ---(1)---> proxy ---(2)----> apiserver
(1) does not match the timeout policy of (2),
This timeout is determined by watchSeconds.
3. Too short watchSeconds value will produce many "re list-watch"
A too long watchSeconds value will cause the proxy to terminate the connection early
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #8311:
URL: https://github.com/apache/apisix/issues/8311#issuecomment-1316189583
> Too short watchSeconds value will produce many "re list-watch" Too long watchSeconds value will cause the proxy to terminate the connection early
>
> do we have to use an proxy before apiserver ?
In fact, if you use `resty.http` or `ngx.tcp.socket`, even if you don't set the `timeout`, there will be a default `timeout`, which is 60 s as I remember.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #8311:
URL: https://github.com/apache/apisix/issues/8311#issuecomment-1312694818
> I think the goal is to avoid "re list-watch". and that's not what "40 seconds" brings
Any suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhixiongdu027 commented on issue #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
Posted by GitBox <gi...@apache.org>.
zhixiongdu027 commented on issue #8311:
URL: https://github.com/apache/apisix/issues/8311#issuecomment-1313168892
In order to solve the problem,
Maybe we can make events via mock endpoints to keep tcp active
@crazyMonkey1995 @tokers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhixiongdu027 commented on issue #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
Posted by GitBox <gi...@apache.org>.
zhixiongdu027 commented on issue #8311:
URL: https://github.com/apache/apisix/issues/8311#issuecomment-1311519911
I think the goal is to avoid "re list-watch".
and that's not what "40 seconds" brings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] crazyMonkey1995 commented on issue #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
Posted by GitBox <gi...@apache.org>.
crazyMonkey1995 commented on issue #8311:
URL: https://github.com/apache/apisix/issues/8311#issuecomment-1313222553
> In order to solve the problem, Maybe we can make events via mock endpoints change in a specific namespace to keep tcp active @crazyMonkey1995 @tokers
How about making `timeout` a configurable parameter? Because the user himself knows what the timeout of the target apiserver (or its proxy) is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ro4i7 commented on issue #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
Posted by "ro4i7 (via GitHub)" <gi...@apache.org>.
ro4i7 commented on issue #8311:
URL: https://github.com/apache/apisix/issues/8311#issuecomment-1465225620
Hello @spacewander @tokers @tzssangglass @crazyMonkey1995
if this issue is still open, please assign it to me:
please give the feedback on following solution:
To solve this issue, we can add some configuration items to the Kubernetes service discovery such as the duration of a watch, retry time, or strategy, as shown below:
```
service:
client:
retry_interval: 30
watch:
duration: 60
retry_strategy: exponential_backoff
```
In this configuration, the duration of a watch is set to 60 seconds, and the retry strategy is set to exponential backoff. The retry interval is set to 30 seconds, which means that the client will retry connecting to the service after 30 seconds if the initial connection attempt fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #8311:
URL: https://github.com/apache/apisix/issues/8311#issuecomment-1316786360
> The problem is that in a network topology like the following
> discovery --(1)--> proxy --(2)--> apiserver
we can make `2000, 3000, http_seconds * 1000 ` in the code `httpc:set_timeouts(2000, 3000, http_seconds * 1000)` be configurabled by the user.
> How about making `timeout` a configurable parameter? Because the user himself knows what the timeout of the target apiserver (or its proxy) is.
As described here, the user needs to configure the timeout to be smaller than the proxy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhixiongdu027 commented on issue #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
Posted by GitBox <gi...@apache.org>.
zhixiongdu027 commented on issue #8311:
URL: https://github.com/apache/apisix/issues/8311#issuecomment-1316199283
> In fact, if you use `resty.http` or `ngx.tcp.socket`, even if you don't set the `timeout`, there will be a default `timeout`, which is 60 s as I remember.
The problem is not here, and in the code it is already set
https.set_timeouts
https://github.com/apache/apisix/blob/288708cbe0098fd3f62fadd725490804d5d0a3db/apisix/discovery/kubernetes/informer_factory.lua#L199-L206
The problem is that in a network topology like the following
discovery --(1)--> proxy --(2)--> apiserver
Position(1) does not match timeout policy for Position(2)
@tzssangglass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #8311: feat: As a user, I want kubernetes service discovery to support more configuration items
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #8311:
URL: https://github.com/apache/apisix/issues/8311#issuecomment-1311457513
The current watch timeout is hard coded with a built-in sample algorithm. I think we can add a new field for users to configure the watch timeout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org