You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/01/18 10:37:00 UTC
[GitHub] [apisix] anjia0532 opened a new issue #6143: feature: Support config merge/overwrite by profile(like spring boot profile style)
anjia0532 opened a new issue #6143:
URL: https://github.com/apache/apisix/issues/6143
### Issue description
目前官方已经支持了基于环境的配置文件的区分，文档: [基于环境变量进行配置文件切换](https://apisix.apache.org/zh/docs/apisix/profile)
看文档描述初衷是为了应对变化，但是个人感觉实际使用中，更多的是80%-90%的文件不变，少量是地址，密码，用户这种的变更。简单翻看了下源码 [apisix/core/profile.lua](https://github.com/apache/apisix/blob/master/apisix/core/profile.lua) ，应该只是基于 `APISIX_PROFILE ` 来读取不同的文件（类似加后缀）,意味着要维护两份绝大多数一样，少部分不一样的配置文件。好处是隔离风险，坏处是，久了容易导致配置割裂越来越严重。
出于最小化改动，不引入破坏性更新，及摒弃无用功能引入的想法，参考但不是照搬spring boot profile，提出以下提议：
以 `conf/config.yaml` 为例
```yaml
apisix:
node_listen: 8000 # APISIX listening port
etcd:
host: "http://foo:2379" # etcd address
```
`conf/config-dev.yaml`
```yaml
etcd:
host: "http://dev:2379" # dev etcd address
```
`conf/config-prod.yaml`
```yaml
etcd:
host: "http://prod:2379" # prod etcd address
```
`APISIX_PROFILE=prod`
最终的apisix读取到的配置为
```yaml
apisix:
node_listen: 8000 # APISIX listening port
etcd:
host: "http://prod:2379" # prod etcd address
```
而不是现在的
```yaml
etcd:
host: "http://prod:2379" # prod etcd address
```
基于此可以实现动静分离，比如，把upstreams这种常见易变配置摘出来，配置成 `config/apisix-dynamic.yaml`,加上静态配置 `config/apisix.yaml`
技术难点就是lua table merge 参考 [how-to-merge-two-tables-overwriting-the-elements-which-are-in-both](https://stackoverflow.com/questions/1283388/how-to-merge-two-tables-overwriting-the-elements-which-are-in-both)
没必要跟spring boot profile 那样支持表达式(SpEL),多种数据文件(yaml,properties),多文件混织 `spring.profiles.active=foo,bar,dev`(同时读取 application.yml,application-foo.yml,application-bar.yml,application-dev.yml)
### Environment
- apisix version (cmd: `apisix version`):
- OS (cmd: `uname -a`):
- OpenResty / Nginx version (cmd: `nginx -V` or `openresty -V`):
- etcd version, if have (cmd: run `curl http://127.0.0.1:9090/v1/server_info` to get the info from server-info API):
- apisix-dashboard version, if have:
- the plugin runner version, if the issue is about a plugin runner (cmd: depended on the kind of runner):
- luarocks version, if the issue is about installation (cmd: `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #6143: discuss: Support config merge/overwrite by profile(like spring boot profile style)
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #6143:
URL: https://github.com/apache/apisix/issues/6143#issuecomment-1019646729
Kong is migrating its dbless backend to LMDB. For effectiveness, multiple files can't win an embedded database.
But multiple files may be a good solution for middle-scale configuration.
What about using a directory to configure, and merging all files into one?
For example,
```
conf-dir/
routes-xxx.yml # we don't need `apisix-` prefix if we have a directory
routes-yyy.yml
upstreams-xxx.yml
...
```
```
# routes-xxx.yml
routes:
-
uri: /hello
upstream:
nodes:
"127.0.0.1:1980": 1
type: roundrobin
...
```
We can keep the list in memory and update the data if one of the files changes.
It is only my idea. Since this feature is not trivial, let's discuss it for several days before coding. Ideas are welcome!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] anjia0532 commented on issue #6143: discuss: Support config merge/overwrite by profile(like spring boot profile style)
Posted by GitBox <gi...@apache.org>.
anjia0532 commented on issue #6143:
URL: https://github.com/apache/apisix/issues/6143#issuecomment-1019668875
> @anjia0532
>
> We are developing [etcd-adaptor](https://github.com/api7/etcd-adapter) as the alternative to standalone mode.
>
> Currently, etcd-adapter already supports MySQL as the backend.
>
> We can introduce SQLite or BoltDB as the backend of etcd-adapter.
>
> It would be great if you can join the development with us and use the adapter in your company 😄
Yes, It's so Cool! I want to give it a try. 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] anjia0532 commented on issue #6143: feature: Support config merge/overwrite by profile(like spring boot profile style)
Posted by GitBox <gi...@apache.org>.
anjia0532 commented on issue #6143:
URL: https://github.com/apache/apisix/issues/6143#issuecomment-1015994674
如果可以，可以考虑把apisix.yml进行拆分，易于阅读，审计，备份，同时对于大文件的变更检查肯定不如小文件来的高效
比如原来配置
```yaml
routes:{}
```
```yaml
routes_from_file:/path/to/routes.yaml
```
是读取文件，二者只能有一个，同时存在报错。其他项类似(`upstreams,upstreams_from_file`)
这样像是`plugins`，`ssl`，`services`等不常变更部分，检查ctime也行，检查文件md5也行，没变化跳过，有变化只读这一块就行了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] anjia0532 commented on issue #6143: discuss: Support config merge/overwrite by profile(like spring boot profile style)
Posted by GitBox <gi...@apache.org>.
anjia0532 commented on issue #6143:
URL: https://github.com/apache/apisix/issues/6143#issuecomment-1019666576
ping @spacewander
我试图遵循社区礼仪，使用英文交流，但是机翻的效果并不是很好，为了不引起歧义，仍用中文，请谅解。
> Kong is migrating its dbless backend to LMDB. For effectiveness, multiple files can't win an embedded database.
通过在社群(比如微信群组和qq群组)etcd的运维问题仍然会时常被提及，说明etcd作为k8s的常见组件，关于其运维能力仍然没有被绝大多数人所掌握，甚至群内仍有时不时反馈，是否会在将来使用mysql作为存储数据库的建议。
鉴于此，可能 [LMDB](https://lmdb.readthedocs.io/en/release/) 是个很棒的提议，但是请考虑是否易于维护，易于理解，开箱即用。反之，mysql,redis,pgsql一类的方案，或许不够酷，不够高效，但是对于一些中小型团队，效率可能不那么重要，稳定且易于维护，是否更重要？(PS: 正在编辑这个回复时，看到您另外一条回复提及的 [etcd-adaptor](https://github.com/api7/etcd-adapter) ，可替换存储后端，是个很酷的想法)
> But multiple files may be a good solution for middle-scale configuration.
> What about using a directory to configure, and merging all files into one?
会不会因为优先级问题导致歧义？
比如
`a.yaml ,b.yaml ,c.yaml ` 三个文件中都配置了 `etcd.host` 以哪个为准？如果简单按照文件名排序，类似`map`或者 `lua table` 的最后写入即为最终的话，可能会有歧义（基于文件管理，并没有可视化操作那么容易及时发现，尤其涉及到多文件管理时，当然稍微会点linux的，可以用 `grep 'etcd' -r .` 但是，yaml的结构决定了，对grep并不友好，并且在设计的时候，我们应当认为用户可能并不会用 `grep 'etcd' -r .`，来降低心智负担，您认可吗？ ）
扫目录过于灵活，当然用户可以用 0-xxx, 1-xxx这样来强制指定顺序
我的提议中，为了降低歧义且让代码最小化改动，故而提出的草案是，类似 spring boot profile 的方案，
1. 通过 `APISIX_PROFILE=prod` 来指定唯一动态/覆写配置文件 即: `base+prod`,同key以`prod`优先级更高，
2. 如果要支持多文件指定,类似 `APISIX_PROFILE=prod,log`,即: `base+prod+log` 的方案 ，同key以最后一个优先级最高，这种方案与直接扫描目录的区别在于，用户指定，意为对此了解且负责，不会产生扫目录这种不确定性
> 如果可以，可以考虑把apisix.yml进行拆分，易于阅读，审计，备份，同时对于大文件的变更检查肯定不如小文件来的高效
>
> 比如原来配置
>
> ```yaml
> routes:{}
> ```
>
> ```yaml
> routes_from_file:/path/to/routes.yaml
> ```
>
> 是读取文件，二者只能有一个，同时存在报错。其他项类似(`upstreams,upstreams_from_file`)
>
> 这样像是`plugins`，`ssl`，`services`等不常变更部分，检查`ctime`也行，检查文件`checksum`也行，没变化跳过，有变化只读这一块就行了
我的这个方案的初始假设是，证书(ssl)，路由(routes)，upstream不会同时需要变更，其中路由或者upstream变更会比较频繁（比如引入了注册中心，并且没有用官方的nacos/eureka等插件，而是用的外部/自研方案）,这样来看，不变的部分，不需要刷新，变更部分比如， `upstreams_from_file:/path/to/upstreams.yaml` 是一个上千行的大文件，还是几十/几百个小文件，对于审计(比如依托git进行审计或者归档)没有太大区别，并且多个小文件容易因为某个upstream被下掉而没有删除文件导致，该upstream持续存在的问题
```yaml
routes:{}
routes_from_file:/path/to/routes.yaml
upstreams: {}
upstreams_from_file:/path/to/upstreams.yaml
services: {}
services_from_file:/path/to/services.yaml
// 忽略其余plugins等
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #6143: discuss: Support config merge/overwrite by profile(like spring boot profile style)
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #6143:
URL: https://github.com/apache/apisix/issues/6143#issuecomment-1019659414
@anjia0532
We are developing [etcd-adaptor](https://github.com/api7/etcd-adapter) as the alternative to standalone mode.
Currently, etcd-adapter already supports MySQL as the backend.
We can introduce SQLite or BoltDB as the backend of etcd-adapter.
It would be great if you can join the development with us and use the adapter in your company 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] anjia0532 commented on issue #6143: discuss: Support config merge/overwrite by profile(like spring boot profile style)
Posted by GitBox <gi...@apache.org>.
anjia0532 commented on issue #6143:
URL: https://github.com/apache/apisix/issues/6143#issuecomment-1019492541
有任何进展吗？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] anjia0532 edited a comment on issue #6143: feature: Support config merge/overwrite by profile(like spring boot profile style)
Posted by GitBox <gi...@apache.org>.
anjia0532 edited a comment on issue #6143:
URL: https://github.com/apache/apisix/issues/6143#issuecomment-1015994674
如果可以，可以考虑把apisix.yml进行拆分，易于阅读，审计，备份，同时对于大文件的变更检查肯定不如小文件来的高效
比如原来配置
```yaml
routes:{}
```
```yaml
routes_from_file:/path/to/routes.yaml
```
是读取文件，二者只能有一个，同时存在报错。其他项类似(`upstreams,upstreams_from_file`)
这样像是`plugins`，`ssl`，`services`等不常变更部分，检查`ctime`也行，检查文件`checksum`也行，没变化跳过，有变化只读这一块就行了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org