You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2020/12/03 09:32:49 UTC
[GitHub] [apisix] AloneFire opened a new issue #2950: bug: Admin 接口并发创建 Router ID 重复
AloneFire opened a new issue #2950:
URL: https://github.com/apache/apisix/issues/2950
### Issue description
Admin 接口并发创建 Router 得到的 ID 重复
### Environment
* apisix version (cmd: `apisix version`): 2.0
* OS: linux
### Minimal test code / Steps to reproduce the issue
1.
2.
3.
### What's the actual result? (including assertion message & call stack if applicable)
### What's the expected result?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] AloneFire commented on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
AloneFire commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-738531523
我测试过老版本，etcd使用v2的接口是没有这个问题的
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] AloneFire commented on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
AloneFire commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-737813654
原因是我想通过apisix做一个权限网关，routes 配置 是从另一个系统批量同步过来的，同步过程使用了多线程，同步结束后发现数据不一致有几条数据创建返回的ID是重复的
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] idbeta edited a comment on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
idbeta edited a comment on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-737805510
@AloneFire Please describe your operation steps and tools in detail. but why do you want to create routes concurrently?
BTW: We hope to use English in public communication channels
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] oldthreefeng commented on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
oldthreefeng commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-738625580
> 我测试过老版本，etcd使用v2的接口是没有这个问题的
能给出测试过程的脚本么. @AloneFire
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] oldthreefeng commented on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
oldthreefeng commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-737880478
> 为什么不同步etcd的数据
@Hcreak 怎么同步, 感觉etcd同步的文档很少见.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] oldthreefeng commented on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
oldthreefeng commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-738703490
> If you use put, there is no problem specifying a different id each time.
do `put` ~ this is a lot helpful for me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] AloneFire removed a comment on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
AloneFire removed a comment on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-738582685
yes,you're right
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander closed issue #2950: bug: Admin API concurrently created Router ID duplicate
Posted by GitBox <gi...@apache.org>.
spacewander closed issue #2950:
URL: https://github.com/apache/apisix/issues/2950
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis edited a comment on issue #2950: bug: Admin API concurrently created Router ID duplicate
Posted by GitBox <gi...@apache.org>.
membphis edited a comment on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-740518850
@spacewander I use this script reproduce this issue. It seems to be a bug of APISIX. please take a look at this issue
```py
$ cat test.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import grequests
#并发设置route
payloads = {}
for id in range(100):
payload = {
"uri": "/test_%s"%id,
"upstream": {
"type": "roundrobin",
"nodes": {
"127.0.0.1:7000": 1
}
}
}
payloads[id]=payload
unsent_request = []
header = {"X-API-KEY":"edd1c9f034335f136f87ad84b625c8f1"}
for id in payloads:
print(id)
unsent_request.append(grequests.post("http://127.0.0.1:9080/apisix/admin/routes/" ,json=payloads[id], headers = header))
responses = grequests.map(unsent_request)
for response in responses:
print(response.content)
response.close()
```
Same UUID:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] AloneFire edited a comment on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
AloneFire edited a comment on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-738518941
> 为什么不同步etcd的数据 或者使用同一个etcd集群呢
是这样，数据源不是etcd ，同步解决方案呢当然也可以通过操作etcd或者直接就避免并发同步， 但是Admin 接口在并发状态下会出现问题我觉得这就是一个bug
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Hcreak commented on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
Hcreak commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-737855803
> 原因是我想通过apisix做一个权限网关，routes 配置 是从另一个系统批量同步过来的，同步过程使用了多线程，同步结束后发现数据不一致有几条数据创建返回的ID是重复的
为什么不同步etcd的数据 或者使用同一个etcd集群呢
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] AloneFire commented on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
AloneFire commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-739691723
> pls provide the steps which can reproduce this issue
只是简单的并发调用 Admin POST接口
```python
from gevent import monkey
import gevent
monkey.patch_all()
from pprint import pprint
import requests
APIKEY = "****************************"
APISIX_HOST = "http://127.0.0.1"
def create_route():
api = f"{APISIX_HOST}/apisix/admin/routes"
resp = requests.post(api, json={
"uri": "*",
"upstream_id": "one upstream_id"
}, headers={
"X-API-KEY": APIKEY
})
return resp.json().get("node", {}).get("key")
if __name__ == '__main__':
N = 10
tasks = [gevent.spawn(create_route) for n in range(N)]
gevent.joinall(tasks)
keys = [t.value for t in tasks]
pprint(keys)
assert (set([t.value for t in tasks]).__len__() == N)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-738897270
pls provide the steps which can reproduce this issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] AloneFire commented on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
AloneFire commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-738642272
@oldthreefeng 这个脚本不太好单独抽出来，就是同样的多线程批量插入，apisix 1.x 不会出现这个问题
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Yiyiyimu commented on issue #2950: bug: Admin API concurrently created Router ID duplicate
Posted by GitBox <gi...@apache.org>.
Yiyiyimu commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-741509781
Yes the bug is imported in v3. The "manual suffix" in etcd v2 is added on the etcd side to make sure the duplication will not happen. I didn't consider concurrency at that time.
Agree with @spacewander
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] AloneFire commented on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
AloneFire commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-738518941
> 为什么不同步etcd的数据 或者使用同一个etcd集群呢
是这样，数据源不是etcd ，同步解决方案呢当然也可以通过操作etcd或者直接就避免并发同步， 但是Admin 接口在并发状态下会出现问题我觉得这就是一个bug
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] idbeta commented on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
idbeta commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-737805510
@AloneFire Please describe your operation steps and tools in detail. but why do you want to create routes concurrently?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] idbeta edited a comment on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
idbeta edited a comment on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-738539648
This problem does exist if use `post` concurrently.
If you use `put`, there is no problem specifying a different id each time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] AloneFire commented on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
AloneFire commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-738582685
yes,you're right
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #2950: bug: Admin API concurrently created Router ID duplicate
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-740533052
https://github.com/apache/apisix/blob/32c7df44cacc28d33abf5c1179b0fc8afa934036/apisix/core/etcd.lua#L277-L287
I think there is a race between read and set.
Maybe we should introduce a `/apisix/_gen_seq` as the id generator.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #2950: bug: Admin API concurrently created Router ID duplicate
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-740518850
@spacewander I use this script reproduce this issue. It seems to be a bug of APISIX. please take a look at this issue
```py
$ cat test.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import grequests
#并发设置route
payloads = {}
for id in range(100):
payload = {
"uri": "/test_%s"%id,
"upstream": {
"type": "roundrobin",
"nodes": {
"127.0.0.1:7000": 1
}
}
}
payloads[id]=payload
unsent_request = []
header = {"X-API-KEY":"edd1c9f034335f136f87ad84b625c8f1"}
for id in payloads:
print(id)
unsent_request.append(grequests.post("http://127.0.0.1:9080/apisix/admin/routes/" ,json=payloads[id], headers = header))
responses = grequests.map(unsent_request) # post创建路由不用指定id，否则报错{"error_msg":"wrong route id, do not need it"}
for response in responses:
print(response.content)
response.close()
```
Same UUID:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #2950: bug: Admin API concurrently created Router ID duplicate
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-740685003
only etcd v3 have this issue, is right? @nic-chen @Yiyiyimu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #2950: bug: Admin API concurrently created Router ID duplicate
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-740506410
@AloneFire need more information:
1. how did you install the apisix?
2. how did you start the apisix?
3. did you modify the apisix source code?
4. what is the content of file `conf/config.yaml`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] AloneFire commented on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
AloneFire commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-738582839
> This problem does exist if use `post` concurrently.
> If you use `put`, there is no problem specifying a different id each time.
yes,you're right
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis edited a comment on issue #2950: bug: Admin API concurrently created Router ID duplicate
Posted by GitBox <gi...@apache.org>.
membphis edited a comment on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-740518850
@spacewander I use this script to reproduce this issue. It seems to be a bug of APISIX. please take a look at this issue
```py
$ cat test.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import grequests
# add route
payloads = {}
for id in range(100):
payload = {
"uri": "/test_%s"%id,
"upstream": {
"type": "roundrobin",
"nodes": {
"127.0.0.1:7000": 1
}
}
}
payloads[id]=payload
unsent_request = []
header = {"X-API-KEY":"edd1c9f034335f136f87ad84b625c8f1"}
for id in payloads:
print(id)
unsent_request.append(grequests.post("http://127.0.0.1:9080/apisix/admin/routes/" ,json=payloads[id], headers = header))
responses = grequests.map(unsent_request)
for response in responses:
print(response.content)
response.close()
```
Same UUID:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] idbeta commented on issue #2950: bug: Admin 接口并发创建 Router ID 重复
Posted by GitBox <gi...@apache.org>.
idbeta commented on issue #2950:
URL: https://github.com/apache/apisix/issues/2950#issuecomment-738539648
This problem does exist if use `post` concurrently.
If you use `put`, there is no problem specifying a different id.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org