You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/03/20 17:21:02 UTC
[GitHub] [apisix] dkrantsberg opened a new issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
dkrantsberg opened a new issue #6666:
URL: https://github.com/apache/apisix/issues/6666
### Current Behavior
I've enabled `openid-connect` plugin on a route but it when making a request to this route causes nginx process to crash (see debug level logs below). I'm running it on Mac M1 (arm64) so this issue could be specific to arm64.
It looks like it executes successfully up to this point:
https://github.com/zmartzone/lua-resty-openidc/blob/master/lib/resty/openidc.lua#L567
Then [client.lua](https://github.com/apache/apisix/blob/master/apisix/core/dns/client.lua) and [resolver.lua](https://github.com/apache/apisix/blob/master/apisix/core/resolver.lua) somehow get involved and it results in `nginx` crashing:
```
worker process 48 exited on signal 11
```
This could be specific to arm64. I haven't tried it with x64.
### Expected Behavior
Expected normal oidc flow
### Error Logs
2022/03/20 16:18:19 [info] 48#48: *1893 [lua] radixtree.lua:346: pre_insert_route(): path: /get operator: =, client: 172.26.0.1, server: _, request: "GET /get HTTP/1.1", host: "localhost:9080"
2022/03/20 16:18:19 [info] 48#48: *1893 [lua] init.lua:398: http_access_phase(): matched route: {"orig_modifiedIndex":779,"createdIndex":178,"clean_handlers":"table: 0x1cadece16508","value":{"create_time":1647628837,"id":"5","plugins":{"openid-connect":{"bearer_only":true,"discovery":"https:\/\/auth.my-auth-server.org\/_api\/auth\/mytenant\/.well-known\/openid-configuration","timeout":3,"access_token_in_authorization_header":true,"ssl_verify":false,"set_userinfo_header":true,"set_access_token_header":true,"realm":"apisix","client_id":"mytenant-ui","scope":"openid","client_secret":"9180987f-bc65-6482-9300-812d3719faa6","logout_path":"\/logout","set_id_token_header":true,"introspection_endpoint_auth_method":"client_secret_basic"}},"update_time":1647654404,"uri":"\/get","status":1,"priority":0,"upstream":{"hash_on":"vars","type":"roundrobin","parent":{"orig_modifiedIndex":779,"createdIndex":178,"clean_handlers":{},"value":"table: 0x1cadecfab320","key":"\/apisix\/routes\/5","update_co
unt":0,"modifiedIndex":779,"has_domain":true},"scheme":"http","pass_host":"pass","nodes":[{"weight":1,"host":"httpbin.org","port":80}]}},"key":"\/apisix\/routes\/5","update_count":0,"modifiedIndex":779,"has_domain":true}, client: 172.26.0.1, server: _, request: "GET /get HTTP/1.1", host: "localhost:9080"
2022/03/20 16:18:19 [debug] 48#48: *1893 [lua] openidc.lua:565: openidc_discover(): openidc_discover: URL is: https://auth.my-auth-server.org/_api/auth/mytenant/.well-known/openid-configuration
2022/03/20 16:18:19 [debug] 48#48: *1893 [lua] openidc.lua:571: openidc_discover(): discovery data not in cache, making call to discovery endpoint
2022/03/20 16:18:19 [debug] 48#48: *1893 [lua] openidc.lua:408: openidc_configure_proxy(): openidc_configure_proxy : don't use http proxy
2022/03/20 16:18:19 [info] 48#48: *1893 [lua] client.lua:126: dns_parse(): dns resolve auth.my-auth-server.org, result: {"name":"auth.my-auth-server.org","class":1,"address":"38.134.56.123","ttl":4502,"section":1,"type":1}, client: 172.26.0.1, server: _, request: "GET /get HTTP/1.1", host: "localhost:9080"
2022/03/20 16:18:19 [info] 48#48: *1893 [lua] resolver.lua:39: parse_domain(): parse addr: {"name":"auth.my-auth-server.org","class":1,"type":1,"section":1,"address":"38.134.56.123","ttl":4502}, client: 172.26.0.1, server: _, request: "GET /get HTTP/1.1", host: "localhost:9080"
2022/03/20 16:18:19 [info] 48#48: *1893 [lua] resolver.lua:40: parse_domain(): resolver: ["127.0.0.11"], client: 172.26.0.1, server: _, request: "GET /get HTTP/1.1", host: "localhost:9080"
2022/03/20 16:18:19 [info] 48#48: *1893 [lua] resolver.lua:41: parse_domain(): host: auth.my-auth-server.org, client: 172.26.0.1, server: _, request: "GET /get HTTP/1.1", host: "localhost:9080"
2022/03/20 16:18:19 [info] 48#48: *1893 [lua] resolver.lua:43: parse_domain(): dns resolver domain: auth.my-auth-server.org to 38.134.56.123, client: 172.26.0.1, server: _, request: "GET /get HTTP/1.1", host: "localhost:9080"
2022/03/20 16:18:19 [info] 52#52: *1916 [lua] timers.lua:39: run timer[plugin#server-info], context: ngx.timer
2022/03/20 16:18:19 [notice] 1#1: signal 17 (SIGCHLD) received from 48
2022/03/20 16:18:19 [alert] 1#1: worker process 48 exited on signal 11
2022/03/20 16:18:19 [notice] 1#1: start worker process 59
2022/03/20 16:18:19 [notice] 1#1: signal 29 (SIGIO) received
2022/03/20 16:18:19 [notice] 59#59: sched_setaffinity(): using cpu #3
### Steps to Reproduce
1. Run `docker-compose` with this file: https://github.com/apache/apisix-docker/blob/master/example/docker-compose-arm64.yml
1. Enable `openid-connect` plugin:
```
curl http://127.0.0.1:9080/apisix/admin/routes/5 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri": "/get",
"plugins": {
"openid-connect": {
"client_id": "my-client",
"client_secret": "XXXX-XXXX-XXX",
"discovery": "https://my-auth-server/myapi/.well-known/openid-configuration",
"access_token_in_authorization_header": true,
"bearer_only": true
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"httpbin.org:80": 1
}
}
}'
```
3. Make a request to the route:
```
curl -i -X GET http://127.0.0.1:9080/get -H "Authorization: Bearer #####token####"
curl: (52) Empty reply from server
```
4. Request results in no reply since nginx crashes
### Environment
- Host OS: `macOS 12.2.1`
- Docker version: `20.10.13`
- APISIX version: `2.12.1`
- Operating system: `Linux dc3c486f70f2 5.10.104-linuxkit #1 SMP PREEMPT Wed Mar 9 19:01:25 UTC 2022 aarch64 Linux`
- OpenResty / Nginx version: `openresty/1.19.9.1`
- etcd version: `3.4.16`
- Plugin runner version: not sure how to get it
- LuaRocks version: `3.8.0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] moonming commented on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
moonming commented on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1075754655
> I used the same test environment as you, but I didn't reproduce it. And a difference is the `discovery` endpoint. So if you can open the service of auth.my-auth-server.org(38.134.56.123), I will try again. It would be nice if you could provide core dump file about this alert
@soulbird Did you use the arm64 local environment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] moonming commented on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
moonming commented on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1074628158
I think Apache APISIX should add ARM64 to CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird edited a comment on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
soulbird edited a comment on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1075862043
When the process exits unexpectedly, you can use the following method to get coredump file
steup 1: adjust core file size
```bash
ulimit -c
unlimited
# If 0, use the following commands to lift the limit or set a custom size.
$ sudo ulimit -c unlimited
```
steup 2: change the default core file directory
```bash
$ sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%t
$ sysctl -p /etc/syslctl.conf
```
You will see the coredump file in the `/tmp` directory, when a worker process exits unexpectedly.
```bash
$ ll /tmp
rw------- 1 root root 280 MiB Mon Mar 21 14:11:22 2022 core-openresty.29770.1647843081
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird commented on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
soulbird commented on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1075803333
> I have used MacOS with M1 and docker to reproduce it, but did not get the expected result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird edited a comment on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
soulbird edited a comment on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1075862043
When the process exits unexpectedly, you can use the following method to get coredump file
steup 1: adjust apisix mirror startup parameters
```yaml
apisix:
image: apache/apisix:2.12.1-alpine
restart: always
privileged: true
volumes:
- ./apisix_log:/usr/local/apisix/logs
- ./apisix_conf/config.yaml:/usr/local/apisix/conf/config.yaml:ro
depends_on:
- etcd
ports:
- "9080:9080/tcp"
- "9091:9091/tcp"
- "9443:9443/tcp"
- "9092:9092/tcp"
networks:
apisix:
```
steup 2: adjust core file size
```bash
ulimit -c
unlimited
# If 0, use the following commands to lift the limit or set a custom size.
$ sudo ulimit -c unlimited
```
steup 3: change the default core file directory
```bash
$ sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%t
$ sysctl -p /etc/syslctl.conf
```
You will see the coredump file in the `/tmp` directory, when a worker process exits unexpectedly.
```bash
$ ll /tmp
rw------- 1 root root 280 MiB Mon Mar 21 14:11:22 2022 core-openresty.29770.1647843081
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird edited a comment on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
soulbird edited a comment on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1075862043
When the process exits unexpectedly, you can use the following method to get coredump file
steup 1: adjust apisix mirror startup parameters
```yaml
apisix:
image: apache/apisix:2.12.1-alpine
restart: always
privileged: true
volumes:
- ./apisix_log:/usr/local/apisix/logs
- ./apisix_conf/config.yaml:/usr/local/apisix/conf/config.yaml:ro
depends_on:
- etcd
ports:
- "9080:9080/tcp"
- "9091:9091/tcp"
- "9443:9443/tcp"
- "9092:9092/tcp"
networks:
apisix:
```
steup 2: adjust core file size
```bash
ulimit -c
unlimited
# If 0, use the following commands to lift the limit or set a custom size.
$ ulimit -c unlimited
```
steup 3: change the default core file directory
```bash
$ sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%t
$ sysctl -p /etc/syslctl.conf
```
You will see the coredump file in the `/tmp` directory, when a worker process exits unexpectedly.
```bash
$ ll /tmp
rw------- 1 root root 280 MiB Mon Mar 21 14:11:22 2022 core-openresty.29770.1647843081
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird edited a comment on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
soulbird edited a comment on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1075862043
When the process exits unexpectedly, you can use the following method to get coredump file
steup 1: adjust apisix mirror startup parameters
```yaml
apisix:
image: apache/apisix:2.12.1-alpine
restart: always
privileged: true
volumes:
- ./apisix_log:/usr/local/apisix/logs
- ./apisix_conf/config.yaml:/usr/local/apisix/conf/config.yaml:ro
depends_on:
- etcd
ports:
- "9080:9080/tcp"
- "9091:9091/tcp"
- "9443:9443/tcp"
- "9092:9092/tcp"
networks:
apisix:
```
steup 1: adjust core file size
```bash
ulimit -c
unlimited
# If 0, use the following commands to lift the limit or set a custom size.
$ sudo ulimit -c unlimited
```
steup 2: change the default core file directory
```bash
$ sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%t
$ sysctl -p /etc/syslctl.conf
```
You will see the coredump file in the `/tmp` directory, when a worker process exits unexpectedly.
```bash
$ ll /tmp
rw------- 1 root root 280 MiB Mon Mar 21 14:11:22 2022 core-openresty.29770.1647843081
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird commented on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
soulbird commented on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1075862043
When the process exits unexpectedly, you can use the following method to get coredump file
steup 1: adjust core file size
```bash
ulimit -c
unlimited
# If 0, use the following commands to lift the limit or set a custom size.
$ sudo ulimit -c unlimited
```
steup 2: change the default core file directory
```bash
$ sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%t
$ sysctl -p /etc/syslog.conf
```
You will see the coredump file in the `/tmp` directory, when a worker process exits unexpectedly.
```bash
$ ll /tmp
rw------- 1 root root 280 MiB Mon Mar 21 14:11:22 2022 core-openresty.29770.1647843081
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1074623811
@soulbird @shuaijinchao pls take a look this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird removed a comment on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
soulbird removed a comment on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1075803333
> I have used MacOS with M1 and docker to reproduce it, but did not get the expected result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1073456326
> Update: Looks like the issue is specific to Alpine Linux for ARM 64. I've switched to Centos based docker image (apache/apisix:2.12.1-centos) and the error is gone.
Yes, support on M1 is not perfect at the moment. It would be nice if you could provide core dump file about this `alert `.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird edited a comment on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
soulbird edited a comment on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1075862043
When the process exits unexpectedly, you can use the following method to get coredump file
steup 1: adjust apisix mirror startup parameters
```yaml
apisix:
image: apache/apisix:2.12.1-alpine
restart: always
privileged: true
volumes:
- ./apisix_log:/usr/local/apisix/logs
- ./apisix_conf/config.yaml:/usr/local/apisix/conf/config.yaml:ro
depends_on:
- etcd
ports:
- "9080:9080/tcp"
- "9091:9091/tcp"
- "9443:9443/tcp"
- "9092:9092/tcp"
networks:
apisix:
```
steup 2: adjust core file size
```bash
ulimit -c
unlimited
# If 0, use the following commands to lift the limit or set a custom size.
$ ulimit -c unlimited
```
steup 3: change the default core file directory
```bash
$ sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%t
$ sysctl -p /etc/sysctl.conf
```
You will see the coredump file in the `/tmp` directory, when a worker process exits unexpectedly.
```bash
$ ll /tmp
rw------- 1 root root 280 MiB Mon Mar 21 14:11:22 2022 core-openresty.29770.1647843081
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird commented on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
soulbird commented on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1075194243
I used the same test environment as you, but I didn't reproduce it. And a difference is the `discovery` endpoint. So if you can open the service of auth.my-auth-server.org(38.134.56.123), I will try again.
It would be nice if you could provide core dump file about this alert
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dkrantsberg commented on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
dkrantsberg commented on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1073369308
Update: Looks like the issue is specific to Alpine Linux for ARM 64. I've switched to Centos based docker image (apache/apisix:2.12.1-centos) and the error is gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird commented on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
soulbird commented on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1074625316
I'll take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird commented on issue #6666: bug: openid-connect plugin causes nginx to crash on ARM64
Posted by GitBox <gi...@apache.org>.
soulbird commented on issue #6666:
URL: https://github.com/apache/apisix/issues/6666#issuecomment-1075803601
> > I used the same test environment as you, but I didn't reproduce it. And a difference is the `discovery` endpoint. So if you can open the service of auth.my-auth-server.org(38.134.56.123), I will try again. It would be nice if you could provide core dump file about this alert
>
> @soulbird Did you use the arm64 local environment?
I used MacOS with M1 and docker to reproduce it, but did not get the expected result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org