You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/07/07 15:33:10 UTC
[GitHub] [apisix] dickens7 opened a new pull request #4559: feat Request-ID plugin add snowflake algorithm
dickens7 opened a new pull request #4559:
URL: https://github.com/apache/apisix/pull/4559
### What this PR does / why we need it:
#4209
### Pre-submission checklist:
* [x] Did you explain what problem does this PR solve? Or what new features have been added?
* [x] Have you added corresponding test cases?
* [ ] Have you modified the corresponding document?
* [x] Is this PR backward compatible? **If it is not backward compatible, please discuss on the [mailing list](https://github.com/apache/apisix/tree/master#community) first**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r667342787
##########
File path: apisix/plugins/request-id.lua
##########
@@ -14,36 +14,163 @@
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-local core = require("apisix.core")
-local plugin_name = "request-id"
-local ngx = ngx
-local uuid = require("resty.jit-uuid")
+
+local ngx = ngx
+local bit = require("bit")
+local core = require("apisix.core")
+local snowflake = require("snowflake")
+local uuid = require("resty.jit-uuid")
+local process = require("ngx.process")
+local tostring = tostring
+local math_pow = math.pow
+
+local plugin_name = "request-id"
+
+local worker_number = nil
+local snowflake_init = nil
+
+local attr = nil
local schema = {
type = "object",
properties = {
header_name = {type = "string", default = "X-Request-Id"},
- include_in_response = {type = "boolean", default = true}
+ include_in_response = {type = "boolean", default = true},
+ algorithm = {type = "string", enum = {"uuid", "snowflake"}, default = "uuid"}
}
}
+local attr_schema = {
+ type = "object",
+ properties = {
+ snowflake = {
+ type = "object",
+ properties = {
+ enable = {type = "boolean"},
+ snowflake_epoc = {type = "integer", minimum = 1, default = 1609459200000},
+ node_id_bits = {type = "integer", minimum = 1, default = 5},
+ sequence_bits = {type = "integer", minimum = 1, default = 10},
+ datacenter_id_bits = {type = "integer", minimum = 1, default = 5},
Review comment:
<s> `default=5` is the setting of standard snowflake algorithm. Setting `default=7` may be confusing for people who are not familiar with the snowflake algorithm. The Snowflake configuration is off by default, so we can introduce the settings in document clearly. </s>
The configuration items were adjusted to merge `datacenter_id_bits` and `node_id_bits` into `data_machine_bits`.
```yaml
plugin_attr:
request-id:
snowflake:
enable: true
snowflake_epoc: 1609459200000 # the starting timestamp is expressed in milliseconds
data_machine_bits: 12 # data machine bit
sequence_bits: 10 # each machine generates a maximum of (1 << sequence_bits) serial numbers per millisecond
worker_number_ttl: 30 # live time for worker number in etcd (unit: second)
worker_number_interval: 10 # lease renewal interval in etcd (unit: second)
```
The PR https://github.com/api7/lua-snowflake/pull/3, commit the code after processing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r669234243
##########
File path: docs/zh/latest/plugins/request-id.md
##########
@@ -71,6 +72,99 @@ X-Request-Id: fe32076a-d0a5-49a6-a361-6c244c1df956
......
```
+### 使用 snowflake 算法
Review comment:
Let's update the English doc too. We can use an online translator for help.
##########
File path: apisix/plugins/request-id.lua
##########
@@ -14,25 +14,58 @@
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-local core = require("apisix.core")
-local plugin_name = "request-id"
-local ngx = ngx
-local uuid = require("resty.jit-uuid")
+
+local ngx = ngx
+local bit = require("bit")
+local core = require("apisix.core")
+local snowflake = require("snowflake")
+local uuid = require("resty.jit-uuid")
+local process = require("ngx.process")
+local timers = require("apisix.timers")
+local tostring = tostring
+local math_pow = math.pow
+local math_ceil = math.ceil
+local math_floor = math.floor
+
+local plugin_name = "request-id"
+
+local data_machine = nil
+local snowflake_inited = nil
+
+local attr = nil
local schema = {
type = "object",
properties = {
header_name = {type = "string", default = "X-Request-Id"},
- include_in_response = {type = "boolean", default = true}
+ include_in_response = {type = "boolean", default = true},
+ algorithm = {type = "string", enum = {"uuid", "snowflake"}, default = "uuid"}
}
}
+local attr_schema = {
+ type = "object",
+ properties = {
+ snowflake = {
+ type = "object",
+ properties = {
+ enable = {type = "boolean"},
+ snowflake_epoc = {type = "integer", minimum = 1, default = 1609459200000},
+ data_machine_bits = {type = "integer", minimum = 1, maximum = 31, default = 12},
+ sequence_bits = {type = "integer", minimum = 1, default = 10},
+ delta_offset = {type = "integer", default = 1, enum = {1, 10, 100, 1000}},
Review comment:
I think we can move delta_offset to the next PR. This one is big enough. Note that it is not easy to catch people to review PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
tokers commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r677904297
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +75,144 @@ function _M.check_schema(conf)
end
+-- Generates the current process data machine
+local function gen_data_machine(max_number)
+ if data_machine == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ ::continue::
+ while (id <= max_number) do
+ local res, err = etcd_cli:grant(attr.snowflake.data_machine_ttl)
+ if err then
+ id = id + 1
+ core.log.error("Etcd grant failure, err: ".. err)
+ goto continue
+ end
+
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
Review comment:
So we have to **write** ETCD so that we can write the uuid?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r674107543
##########
File path: apisix/plugins/request-id.lua
##########
@@ -245,5 +245,8 @@ function _M.init()
end
end
+function _M.destroy()
+ timers.unregister_timer("plugin#request-id")
Review comment:
Is it necessary to do this? I have not seen similar examples elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r669241158
##########
File path: docs/zh/latest/plugins/request-id.md
##########
@@ -71,6 +72,99 @@ X-Request-Id: fe32076a-d0a5-49a6-a361-6c244c1df956
......
```
+### 使用 snowflake 算法
Review comment:
First confirm that the document is ok, I am translating to English
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r674479828
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +74,142 @@ function _M.check_schema(conf)
end
+-- Generates the current process data machine
+local function gen_data_machine(max_number)
+ if data_machine == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ ::continue::
+ while (id <= max_number) do
+ local res, err = etcd_cli:grant(attr.snowflake.data_machine_ttl)
+ if err then
+ core.log.error("Etcd grant failure, err: ".. err)
+ end
+
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("data_machine " .. id .. " is not available")
+ id = id + 1
+ else
+ data_machine = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
Review comment:
Now `etcd_cli:grant` will exit the current loop when the return value `err` is `true`, do I need to check `res` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] starsz commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
starsz commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r677052926
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +74,142 @@ function _M.check_schema(conf)
end
+-- Generates the current process data machine
+local function gen_data_machine(max_number)
+ if data_machine == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ ::continue::
+ while (id <= max_number) do
+ local res, err = etcd_cli:grant(attr.snowflake.data_machine_ttl)
+ if err then
+ core.log.error("Etcd grant failure, err: ".. err)
+ end
+
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("data_machine " .. id .. " is not available")
+ id = id + 1
+ else
+ data_machine = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
Review comment:
Ok. You don't need to check the res here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Yiyiyimu edited a comment on pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
Yiyiyimu edited a comment on pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#issuecomment-876595542
> CI help
Rerun to see how it goes
---
Good to go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Yiyiyimu commented on pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
Yiyiyimu commented on pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#issuecomment-876613294
Hi @dickens7 do you mind also add documentations of this feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r667356228
##########
File path: apisix/plugins/request-id.lua
##########
@@ -14,36 +14,163 @@
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-local core = require("apisix.core")
-local plugin_name = "request-id"
-local ngx = ngx
-local uuid = require("resty.jit-uuid")
+
+local ngx = ngx
+local bit = require("bit")
+local core = require("apisix.core")
+local snowflake = require("snowflake")
+local uuid = require("resty.jit-uuid")
+local process = require("ngx.process")
+local tostring = tostring
+local math_pow = math.pow
+
+local plugin_name = "request-id"
+
+local worker_number = nil
+local snowflake_init = nil
+
+local attr = nil
local schema = {
type = "object",
properties = {
header_name = {type = "string", default = "X-Request-Id"},
- include_in_response = {type = "boolean", default = true}
+ include_in_response = {type = "boolean", default = true},
+ algorithm = {type = "string", enum = {"uuid", "snowflake"}, default = "uuid"}
}
}
+local attr_schema = {
+ type = "object",
+ properties = {
+ snowflake = {
+ type = "object",
+ properties = {
+ enable = {type = "boolean"},
+ snowflake_epoc = {type = "integer", minimum = 1, default = 1609459200000},
+ node_id_bits = {type = "integer", minimum = 1, default = 5},
+ sequence_bits = {type = "integer", minimum = 1, default = 10},
+ datacenter_id_bits = {type = "integer", minimum = 1, default = 5},
+ worker_number_ttl = {type = "integer", minimum = 1, default = 30},
+ worker_number_interval = {type = "integer", minimum = 1, default = 10}
+ }
+ }
+ }
+}
local _M = {
version = 0.1,
priority = 11010,
name = plugin_name,
- schema = schema,
+ schema = schema
}
-
function _M.check_schema(conf)
return core.schema.check(schema, conf)
end
+local function gen_worker_number(max_number)
+ if worker_number == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local res, _ = etcd_cli:grant(attr.snowflake.worker_number_ttl)
+
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ while (id <= max_number) do
+ ::continue::
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("worker_number " .. id .. " is not available")
+ id = id + 1
+ else
+ worker_number = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
+ }
+ )
+
+ if err3 then
+ id = id + 1
+ etcd_cli:delete(prefix .. tostring(id))
+ core.log.error("set worker_number " .. id .. " lease error: " .. err3)
+ goto continue
+ end
+
+ local handler = function(premature, etcd_cli, lease_id)
+ local _, err4 = etcd_cli:keepalive(lease_id)
+ if err4 then
+ snowflake_init = nil
+ worker_number = nil
+ core.log.error("snowflake worker_number lease faild.")
+ end
+ core.log.info("snowflake worker_number lease success.")
+ end
+ ngx.timer.every(attr.snowflake.worker_number_interval,
Review comment:
> Hi @dickens7 do you mind also add documentations of this feature
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r673223211
##########
File path: apisix/plugins/request-id.lua
##########
@@ -14,25 +14,58 @@
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-local core = require("apisix.core")
-local plugin_name = "request-id"
-local ngx = ngx
-local uuid = require("resty.jit-uuid")
+
+local ngx = ngx
+local bit = require("bit")
+local core = require("apisix.core")
+local snowflake = require("snowflake")
+local uuid = require("resty.jit-uuid")
+local process = require("ngx.process")
+local timers = require("apisix.timers")
+local tostring = tostring
+local math_pow = math.pow
+local math_ceil = math.ceil
+local math_floor = math.floor
+
+local plugin_name = "request-id"
+
+local data_machine = nil
+local snowflake_inited = nil
+
+local attr = nil
local schema = {
type = "object",
properties = {
header_name = {type = "string", default = "X-Request-Id"},
- include_in_response = {type = "boolean", default = true}
+ include_in_response = {type = "boolean", default = true},
+ algorithm = {type = "string", enum = {"uuid", "snowflake"}, default = "uuid"}
}
}
+local attr_schema = {
+ type = "object",
+ properties = {
+ snowflake = {
+ type = "object",
+ properties = {
+ enable = {type = "boolean"},
+ snowflake_epoc = {type = "integer", minimum = 1, default = 1609459200000},
+ data_machine_bits = {type = "integer", minimum = 1, maximum = 31, default = 12},
+ sequence_bits = {type = "integer", minimum = 1, default = 10},
+ delta_offset = {type = "integer", default = 1, enum = {1, 10, 100, 1000}},
Review comment:
delta_offset has been removed from this PR
> Only the configuration items in the configuration list are removed, and the code is compatible so there is no need to remove them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
spacewander commented on pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#issuecomment-886309375
Please solve the conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#issuecomment-891466062
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
tokers commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r667579814
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +72,134 @@ function _M.check_schema(conf)
end
+-- Generates the current process worker number
+local function gen_worker_number(max_number)
+ if worker_number == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ while (id <= max_number) do
+ ::continue::
+ local res, _ = etcd_cli:grant(attr.snowflake.worker_number_ttl)
Review comment:
I am afraid that we shouldn't ignore this error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r667342787
##########
File path: apisix/plugins/request-id.lua
##########
@@ -14,36 +14,163 @@
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-local core = require("apisix.core")
-local plugin_name = "request-id"
-local ngx = ngx
-local uuid = require("resty.jit-uuid")
+
+local ngx = ngx
+local bit = require("bit")
+local core = require("apisix.core")
+local snowflake = require("snowflake")
+local uuid = require("resty.jit-uuid")
+local process = require("ngx.process")
+local tostring = tostring
+local math_pow = math.pow
+
+local plugin_name = "request-id"
+
+local worker_number = nil
+local snowflake_init = nil
+
+local attr = nil
local schema = {
type = "object",
properties = {
header_name = {type = "string", default = "X-Request-Id"},
- include_in_response = {type = "boolean", default = true}
+ include_in_response = {type = "boolean", default = true},
+ algorithm = {type = "string", enum = {"uuid", "snowflake"}, default = "uuid"}
}
}
+local attr_schema = {
+ type = "object",
+ properties = {
+ snowflake = {
+ type = "object",
+ properties = {
+ enable = {type = "boolean"},
+ snowflake_epoc = {type = "integer", minimum = 1, default = 1609459200000},
+ node_id_bits = {type = "integer", minimum = 1, default = 5},
+ sequence_bits = {type = "integer", minimum = 1, default = 10},
+ datacenter_id_bits = {type = "integer", minimum = 1, default = 5},
Review comment:
`default=5` is the setting of standard snowflake algorithm. Setting `default=7` may be confusing for people who are not familiar with the snowflake algorithm. The Snowflake configuration is off by default, so we can introduce the settings in document clearly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r684299882
##########
File path: docs/en/latest/plugins/request-id.md
##########
@@ -72,6 +73,60 @@ X-Request-Id: fe32076a-d0a5-49a6-a361-6c244c1df956
......
```
+### Use the snowflake algorithm to generate an ID
+
+> supports using the Snowflake algorithm to generate ID.
+> read the documentation first before deciding to use snowflake. Because once the configuration information is enabled, you can not arbitrarily adjust the configuration information. Failure to do so may result in duplicate ID being generated.
+
+The Snowflake algorithm is not enabled by default and needs to be configured in 'conf/config.yaml'.
+
+```yaml
+plugin_attr:
+ request-id:
+ snowflake:
+ enable: true
+ snowflake_epoc: 1609459200000
+ data_machine_bits: 12
+ sequence_bits: 10
+ data_machine_ttl: 30
+ data_machine_interval: 10
+```
+
+#### Configuration parameters
+
+| Name | Type | Requirement | Default | Valid | Description |
+| ------------------- | ------- | ------------- | -------------- | ------- | ------------------------------ |
+| enable | boolean | optional | false | | When set it to true, enable the snowflake algorithm. |
+| snowflake_epoc | integer | optional | 1609459200000 | | Start timestamp (in milliseconds) |
+| data_machine_bits | integer | optional | 12 | | Maximum number of supported machines (processes) `1 << data_machine_bits` |
+| sequence_bits | integer | optional | 10 | | Maximum number of generated ID per millisecond per node `1 << sequence_bits` |
+| data_machine_ttl | integer | optional | 30 | | Valid time of registration of 'data_machine' in 'etcd' (unit: seconds) |
+| data_machine_interval | integer | optional | 10 | | Time between 'data_machine' renewal in 'etcd' (unit: seconds) |
+
+- `snowflake_epoc` default start time is `2021-01-01T00:00:00Z`, and it can support `69 year` approximately to `2090-09-0715:47:35Z` according to the default configuration
+- `data_machine_bits` corresponds to the set of workIDs and datacEnteridd in the snowflake definition. The plug-in aslocates a unique ID to each process. Maximum number of supported processes is `pow(2, data_machine_bits)`. The default number of `12 bits` is up to `4096`.
+- `sequence_bits` defaults to `10 bits` and each process generates up to `1024` ID per second
Review comment:
This is the definition of two parts of the snowflake algorithm. Millisecond is the definition of the timestamp part, which means that the timestamp part is in milliseconds. Second is the definition of the sequence number part, which means that the number of id can be generated per second
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r669242926
##########
File path: docs/zh/latest/plugins/request-id.md
##########
@@ -71,6 +72,99 @@ X-Request-Id: fe32076a-d0a5-49a6-a361-6c244c1df956
......
```
+### 使用 snowflake 算法
Review comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r668898785
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +72,134 @@ function _M.check_schema(conf)
end
+-- Generates the current process worker number
+local function gen_worker_number(max_number)
+ if worker_number == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ while (id <= max_number) do
+ ::continue::
+ local res, _ = etcd_cli:grant(attr.snowflake.worker_number_ttl)
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("worker_number " .. id .. " is not available")
+ id = id + 1
+ else
+ worker_number = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
+ }
+ )
+
+ if err3 then
+ id = id + 1
+ etcd_cli:delete(prefix .. tostring(id))
+ core.log.error("set worker_number " .. id .. " lease error: " .. err3)
+ goto continue
+ end
+
+ local lease_id = res.body.ID
+ local start_at = ngx.time()
+ local handler = function()
+ local now = ngx.time()
+ if now - start_at < attr.snowflake.worker_number_interval then
+ return
+ end
+
+ local _, err4 = etcd_cli:keepalive(lease_id)
+ if err4 then
+ snowflake_inited = nil
+ worker_number = nil
+ core.log.error("snowflake worker_number: " .. id .." lease faild.")
+ end
+ start_at = now
+ core.log.info("snowflake worker_number: " .. id .." lease success.")
+ end
+
+ timers.register_timer("plugin#request-id", handler)
+ core.log.info(
+ "timer created to lease snowflake algorithm worker number, interval: ",
+ attr.snowflake.worker_number_interval)
+ core.log.notice("lease snowflake worker_number: " .. id)
+ break
+ end
+ end
+
+ if worker_number == nil then
+ core.log.error("No worker_number is not available")
+ return nil
+ end
+ end
+ return worker_number
+end
+
+
+-- Split 'Worker Number' into 'Worker ID' and 'datacenter ID'
+local function split_worker_number(worker_number, node_id_bits, datacenter_id_bits)
+ local num = bit.tobit(worker_number)
+ local worker_id = bit.band(num, math_pow(2, node_id_bits) - 1) + 1
+ num = bit.rshift(num, node_id_bits)
+ local datacenter_id = bit.band(num, math_pow(2, datacenter_id_bits) - 1) + 1
+ return worker_id, datacenter_id
Review comment:
It's going to be a bug to start at 1 here, let me fix that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r668906633
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +72,134 @@ function _M.check_schema(conf)
end
+-- Generates the current process worker number
+local function gen_worker_number(max_number)
+ if worker_number == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ while (id <= max_number) do
+ ::continue::
+ local res, _ = etcd_cli:grant(attr.snowflake.worker_number_ttl)
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("worker_number " .. id .. " is not available")
+ id = id + 1
+ else
+ worker_number = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
+ }
+ )
+
+ if err3 then
+ id = id + 1
+ etcd_cli:delete(prefix .. tostring(id))
+ core.log.error("set worker_number " .. id .. " lease error: " .. err3)
+ goto continue
+ end
+
+ local lease_id = res.body.ID
+ local start_at = ngx.time()
+ local handler = function()
+ local now = ngx.time()
+ if now - start_at < attr.snowflake.worker_number_interval then
+ return
+ end
+
+ local _, err4 = etcd_cli:keepalive(lease_id)
+ if err4 then
+ snowflake_inited = nil
+ worker_number = nil
+ core.log.error("snowflake worker_number: " .. id .." lease faild.")
+ end
+ start_at = now
+ core.log.info("snowflake worker_number: " .. id .." lease success.")
+ end
+
+ timers.register_timer("plugin#request-id", handler)
Review comment:
When does `_M. destroy()` triggered? I don't think there should be a need to `timers.unregister_timer` in `_M. destroy()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r672756697
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +74,142 @@ function _M.check_schema(conf)
end
+-- Generates the current process data machine
+local function gen_data_machine(max_number)
+ if data_machine == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ ::continue::
+ while (id <= max_number) do
+ local res, err = etcd_cli:grant(attr.snowflake.data_machine_ttl)
+ if err then
+ core.log.error("Etcd grant failure, err: ".. err)
+ end
+
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("data_machine " .. id .. " is not available")
+ id = id + 1
+ else
+ data_machine = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
+ }
+ )
+
+ if err3 then
+ id = id + 1
+ etcd_cli:delete(prefix .. tostring(id))
+ core.log.error("set data_machine " .. id .. " lease error: " .. err3)
+ goto continue
+ end
+
+ local lease_id = res.body.ID
+ local start_at = ngx.time()
+ local handler = function()
+ local now = ngx.time()
+ if now - start_at < attr.snowflake.data_machine_interval then
+ return
+ end
+
+ local _, err4 = etcd_cli:keepalive(lease_id)
+ if err4 then
+ snowflake_inited = nil
+ data_machine = nil
+ core.log.error("snowflake data_machine: " .. id .." lease faild.")
+ end
+ start_at = now
+ core.log.info("snowflake data_machine: " .. id .." lease success.")
+ end
+
+ timers.register_timer("plugin#request-id", handler)
+ core.log.info(
+ "timer created to lease snowflake algorithm data_machine, interval: ",
+ attr.snowflake.data_machine_interval)
+ core.log.notice("lease snowflake data_machine: " .. id)
+ break
+ end
+ end
+
+ if data_machine == nil then
+ core.log.error("No data_machine is not available")
+ return nil
+ end
+ end
+ return data_machine
+end
+
+
+-- Split 'Data Machine' into 'Worker ID' and 'datacenter ID'
+local function split_data_machine(data_machine, node_id_bits, datacenter_id_bits)
+ local num = bit.tobit(data_machine)
+ local worker_id = bit.band(num, math_pow(2, node_id_bits) - 1)
+ num = bit.rshift(num, node_id_bits)
+ local datacenter_id = bit.band(num, math_pow(2, datacenter_id_bits) - 1)
+ return worker_id, datacenter_id
+end
+
+
+-- Initialize the snowflake algorithm
+local function snowflake_init()
+ if snowflake_inited == nil then
+ local max_number = math_pow(2, (attr.snowflake.data_machine_bits))
+ local datacenter_id_bits = math_floor(attr.snowflake.data_machine_bits / 2)
+ local node_id_bits = math_ceil(attr.snowflake.data_machine_bits / 2)
+ data_machine = gen_data_machine(max_number)
+ if data_machine == nil then
+ return ""
+ end
+
+ local worker_id, datacenter_id = split_data_machine(data_machine,
+ node_id_bits, datacenter_id_bits)
+
+ core.log.info("snowflake init datacenter_id: " ..
+ datacenter_id .. " worker_id: " .. worker_id)
+ snowflake.init(
+ datacenter_id,
+ worker_id,
+ attr.snowflake.snowflake_epoc,
+ node_id_bits,
+ datacenter_id_bits,
+ attr.snowflake.sequence_bits,
+ attr.delta_offset
+ )
+ snowflake_inited = true
+ end
+end
+
+
+-- generate snowflake id
+local function next_id()
+ if snowflake_inited == nil then
+ snowflake_init()
+ end
+ return snowflake:next_id()
+end
+
Review comment:
Let's implement this in the next pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r669204115
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +72,134 @@ function _M.check_schema(conf)
end
+-- Generates the current process worker number
+local function gen_worker_number(max_number)
+ if worker_number == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ while (id <= max_number) do
+ ::continue::
+ local res, _ = etcd_cli:grant(attr.snowflake.worker_number_ttl)
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("worker_number " .. id .. " is not available")
+ id = id + 1
+ else
+ worker_number = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
+ }
+ )
+
+ if err3 then
+ id = id + 1
+ etcd_cli:delete(prefix .. tostring(id))
+ core.log.error("set worker_number " .. id .. " lease error: " .. err3)
+ goto continue
+ end
+
+ local lease_id = res.body.ID
+ local start_at = ngx.time()
+ local handler = function()
+ local now = ngx.time()
+ if now - start_at < attr.snowflake.worker_number_interval then
+ return
+ end
+
+ local _, err4 = etcd_cli:keepalive(lease_id)
+ if err4 then
+ snowflake_inited = nil
+ worker_number = nil
+ core.log.error("snowflake worker_number: " .. id .." lease faild.")
+ end
+ start_at = now
+ core.log.info("snowflake worker_number: " .. id .." lease success.")
+ end
+
+ timers.register_timer("plugin#request-id", handler)
Review comment:
When the plugin is hot disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r677951766
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +75,144 @@ function _M.check_schema(conf)
end
+-- Generates the current process data machine
+local function gen_data_machine(max_number)
+ if data_machine == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ ::continue::
+ while (id <= max_number) do
+ local res, err = etcd_cli:grant(attr.snowflake.data_machine_ttl)
+ if err then
+ id = id + 1
+ core.log.error("Etcd grant failure, err: ".. err)
+ goto continue
+ end
+
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
Review comment:
The purpose of writing etcd here is to generate `DataMachineId` for snowflake. The value is uuid because `etcd_cli:setnx` still returns `succeeded` even if the key exists. UUID is used for verification after `etcd_cli:setnx`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zuiyangqingzhou commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
zuiyangqingzhou commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r671948175
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +74,142 @@ function _M.check_schema(conf)
end
+-- Generates the current process data machine
+local function gen_data_machine(max_number)
+ if data_machine == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ ::continue::
+ while (id <= max_number) do
+ local res, err = etcd_cli:grant(attr.snowflake.data_machine_ttl)
+ if err then
+ core.log.error("Etcd grant failure, err: ".. err)
+ end
+
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("data_machine " .. id .. " is not available")
+ id = id + 1
+ else
+ data_machine = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
+ }
+ )
+
+ if err3 then
+ id = id + 1
+ etcd_cli:delete(prefix .. tostring(id))
+ core.log.error("set data_machine " .. id .. " lease error: " .. err3)
+ goto continue
+ end
+
+ local lease_id = res.body.ID
+ local start_at = ngx.time()
+ local handler = function()
+ local now = ngx.time()
+ if now - start_at < attr.snowflake.data_machine_interval then
+ return
+ end
+
+ local _, err4 = etcd_cli:keepalive(lease_id)
+ if err4 then
+ snowflake_inited = nil
+ data_machine = nil
+ core.log.error("snowflake data_machine: " .. id .." lease faild.")
+ end
+ start_at = now
+ core.log.info("snowflake data_machine: " .. id .." lease success.")
+ end
+
+ timers.register_timer("plugin#request-id", handler)
+ core.log.info(
+ "timer created to lease snowflake algorithm data_machine, interval: ",
+ attr.snowflake.data_machine_interval)
+ core.log.notice("lease snowflake data_machine: " .. id)
+ break
+ end
+ end
+
+ if data_machine == nil then
+ core.log.error("No data_machine is not available")
+ return nil
+ end
+ end
+ return data_machine
+end
+
+
+-- Split 'Data Machine' into 'Worker ID' and 'datacenter ID'
+local function split_data_machine(data_machine, node_id_bits, datacenter_id_bits)
+ local num = bit.tobit(data_machine)
+ local worker_id = bit.band(num, math_pow(2, node_id_bits) - 1)
+ num = bit.rshift(num, node_id_bits)
+ local datacenter_id = bit.band(num, math_pow(2, datacenter_id_bits) - 1)
+ return worker_id, datacenter_id
+end
+
+
+-- Initialize the snowflake algorithm
+local function snowflake_init()
+ if snowflake_inited == nil then
+ local max_number = math_pow(2, (attr.snowflake.data_machine_bits))
+ local datacenter_id_bits = math_floor(attr.snowflake.data_machine_bits / 2)
+ local node_id_bits = math_ceil(attr.snowflake.data_machine_bits / 2)
+ data_machine = gen_data_machine(max_number)
+ if data_machine == nil then
+ return ""
+ end
+
+ local worker_id, datacenter_id = split_data_machine(data_machine,
+ node_id_bits, datacenter_id_bits)
+
+ core.log.info("snowflake init datacenter_id: " ..
+ datacenter_id .. " worker_id: " .. worker_id)
+ snowflake.init(
+ datacenter_id,
+ worker_id,
+ attr.snowflake.snowflake_epoc,
+ node_id_bits,
+ datacenter_id_bits,
+ attr.snowflake.sequence_bits,
+ attr.delta_offset
+ )
+ snowflake_inited = true
+ end
+end
+
+
+-- generate snowflake id
+local function next_id()
+ if snowflake_inited == nil then
+ snowflake_init()
+ end
+ return snowflake:next_id()
+end
+
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r674478629
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +74,142 @@ function _M.check_schema(conf)
end
+-- Generates the current process data machine
+local function gen_data_machine(max_number)
+ if data_machine == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ ::continue::
+ while (id <= max_number) do
+ local res, err = etcd_cli:grant(attr.snowflake.data_machine_ttl)
+ if err then
+ core.log.error("Etcd grant failure, err: ".. err)
Review comment:
Should not continue here.
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#issuecomment-879531464
TODO: upgrade api7/lua-snowflake to 2.1-1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r673222267
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +72,134 @@ function _M.check_schema(conf)
end
+-- Generates the current process worker number
+local function gen_worker_number(max_number)
+ if worker_number == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ while (id <= max_number) do
+ ::continue::
+ local res, _ = etcd_cli:grant(attr.snowflake.worker_number_ttl)
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("worker_number " .. id .. " is not available")
+ id = id + 1
+ else
+ worker_number = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
+ }
+ )
+
+ if err3 then
+ id = id + 1
+ etcd_cli:delete(prefix .. tostring(id))
+ core.log.error("set worker_number " .. id .. " lease error: " .. err3)
+ goto continue
+ end
+
+ local lease_id = res.body.ID
+ local start_at = ngx.time()
+ local handler = function()
+ local now = ngx.time()
+ if now - start_at < attr.snowflake.worker_number_interval then
+ return
+ end
+
+ local _, err4 = etcd_cli:keepalive(lease_id)
+ if err4 then
+ snowflake_inited = nil
+ worker_number = nil
+ core.log.error("snowflake worker_number: " .. id .." lease faild.")
+ end
+ start_at = now
+ core.log.info("snowflake worker_number: " .. id .." lease success.")
+ end
+
+ timers.register_timer("plugin#request-id", handler)
Review comment:
resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r677499541
##########
File path: apisix/plugins/request-id.lua
##########
@@ -14,25 +14,58 @@
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-local core = require("apisix.core")
-local plugin_name = "request-id"
-local ngx = ngx
-local uuid = require("resty.jit-uuid")
+
+local ngx = ngx
+local bit = require("bit")
+local core = require("apisix.core")
+local snowflake = require("snowflake")
+local uuid = require("resty.jit-uuid")
+local process = require("ngx.process")
+local timers = require("apisix.timers")
+local tostring = tostring
+local math_pow = math.pow
+local math_ceil = math.ceil
+local math_floor = math.floor
+
+local plugin_name = "request-id"
+
+local data_machine = nil
+local snowflake_inited = nil
+
+local attr = nil
local schema = {
type = "object",
properties = {
header_name = {type = "string", default = "X-Request-Id"},
- include_in_response = {type = "boolean", default = true}
+ include_in_response = {type = "boolean", default = true},
+ algorithm = {type = "string", enum = {"uuid", "snowflake"}, default = "uuid"}
}
}
+local attr_schema = {
+ type = "object",
+ properties = {
+ snowflake = {
+ type = "object",
+ properties = {
+ enable = {type = "boolean"},
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r673220978
##########
File path: docs/zh/latest/plugins/request-id.md
##########
@@ -71,6 +72,99 @@ X-Request-Id: fe32076a-d0a5-49a6-a361-6c244c1df956
......
```
+### 使用 snowflake 算法
Review comment:
The English document has been added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r668900783
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +72,134 @@ function _M.check_schema(conf)
end
+-- Generates the current process worker number
+local function gen_worker_number(max_number)
+ if worker_number == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ while (id <= max_number) do
+ ::continue::
+ local res, _ = etcd_cli:grant(attr.snowflake.worker_number_ttl)
Review comment:
Okay, I'm going to add a check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r674435513
##########
File path: apisix/plugins/request-id.lua
##########
@@ -245,5 +245,8 @@ function _M.init()
end
end
+function _M.destroy()
+ timers.unregister_timer("plugin#request-id")
Review comment:
If we don't register a timer, why unregister it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r669241399
##########
File path: apisix/plugins/request-id.lua
##########
@@ -14,25 +14,58 @@
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-local core = require("apisix.core")
-local plugin_name = "request-id"
-local ngx = ngx
-local uuid = require("resty.jit-uuid")
+
+local ngx = ngx
+local bit = require("bit")
+local core = require("apisix.core")
+local snowflake = require("snowflake")
+local uuid = require("resty.jit-uuid")
+local process = require("ngx.process")
+local timers = require("apisix.timers")
+local tostring = tostring
+local math_pow = math.pow
+local math_ceil = math.ceil
+local math_floor = math.floor
+
+local plugin_name = "request-id"
+
+local data_machine = nil
+local snowflake_inited = nil
+
+local attr = nil
local schema = {
type = "object",
properties = {
header_name = {type = "string", default = "X-Request-Id"},
- include_in_response = {type = "boolean", default = true}
+ include_in_response = {type = "boolean", default = true},
+ algorithm = {type = "string", enum = {"uuid", "snowflake"}, default = "uuid"}
}
}
+local attr_schema = {
+ type = "object",
+ properties = {
+ snowflake = {
+ type = "object",
+ properties = {
+ enable = {type = "boolean"},
+ snowflake_epoc = {type = "integer", minimum = 1, default = 1609459200000},
+ data_machine_bits = {type = "integer", minimum = 1, maximum = 31, default = 12},
+ sequence_bits = {type = "integer", minimum = 1, default = 10},
+ delta_offset = {type = "integer", default = 1, enum = {1, 10, 100, 1000}},
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r666608780
##########
File path: apisix/plugins/request-id.lua
##########
@@ -14,36 +14,163 @@
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-local core = require("apisix.core")
-local plugin_name = "request-id"
-local ngx = ngx
-local uuid = require("resty.jit-uuid")
+
+local ngx = ngx
+local bit = require("bit")
+local core = require("apisix.core")
+local snowflake = require("snowflake")
+local uuid = require("resty.jit-uuid")
+local process = require("ngx.process")
+local tostring = tostring
+local math_pow = math.pow
+
+local plugin_name = "request-id"
+
+local worker_number = nil
+local snowflake_init = nil
+
+local attr = nil
local schema = {
type = "object",
properties = {
header_name = {type = "string", default = "X-Request-Id"},
- include_in_response = {type = "boolean", default = true}
+ include_in_response = {type = "boolean", default = true},
+ algorithm = {type = "string", enum = {"uuid", "snowflake"}, default = "uuid"}
}
}
+local attr_schema = {
+ type = "object",
+ properties = {
+ snowflake = {
+ type = "object",
+ properties = {
+ enable = {type = "boolean"},
+ snowflake_epoc = {type = "integer", minimum = 1, default = 1609459200000},
+ node_id_bits = {type = "integer", minimum = 1, default = 5},
+ sequence_bits = {type = "integer", minimum = 1, default = 10},
+ datacenter_id_bits = {type = "integer", minimum = 1, default = 5},
Review comment:
Better to use a user-friendly name, like max_worker_number or ids_per_second
##########
File path: apisix/plugins/request-id.lua
##########
@@ -65,4 +191,47 @@ function _M.header_filter(conf, ctx)
end
end
+function _M.init()
+ local local_conf = core.config.local_conf()
+ attr = core.table.try_read_attr(local_conf, "plugin_attr", plugin_name)
+ local ok, err = core.schema.check(attr_schema, attr)
+ if not ok then
+ core.log.error("failed to check the plugin_attr[", plugin_name, "]", ": ", err)
+ return
+ end
+ if attr.snowflake.enable then
+ if process.type() == "worker" then
+ ngx.timer.at(0, next_id)
+ end
+ end
+end
+
+function _M.api()
Review comment:
Why do we need to expose the uuid? This will leak the info about the mechanism that we use to generate the uuid.
##########
File path: apisix/plugins/request-id.lua
##########
@@ -14,36 +14,163 @@
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-local core = require("apisix.core")
-local plugin_name = "request-id"
-local ngx = ngx
-local uuid = require("resty.jit-uuid")
+
+local ngx = ngx
+local bit = require("bit")
+local core = require("apisix.core")
+local snowflake = require("snowflake")
+local uuid = require("resty.jit-uuid")
+local process = require("ngx.process")
+local tostring = tostring
+local math_pow = math.pow
+
+local plugin_name = "request-id"
+
+local worker_number = nil
+local snowflake_init = nil
+
+local attr = nil
local schema = {
type = "object",
properties = {
header_name = {type = "string", default = "X-Request-Id"},
- include_in_response = {type = "boolean", default = true}
+ include_in_response = {type = "boolean", default = true},
+ algorithm = {type = "string", enum = {"uuid", "snowflake"}, default = "uuid"}
}
}
+local attr_schema = {
+ type = "object",
+ properties = {
+ snowflake = {
+ type = "object",
+ properties = {
+ enable = {type = "boolean"},
+ snowflake_epoc = {type = "integer", minimum = 1, default = 1609459200000},
+ node_id_bits = {type = "integer", minimum = 1, default = 5},
+ sequence_bits = {type = "integer", minimum = 1, default = 10},
+ datacenter_id_bits = {type = "integer", minimum = 1, default = 5},
+ worker_number_ttl = {type = "integer", minimum = 1, default = 30},
+ worker_number_interval = {type = "integer", minimum = 1, default = 10}
+ }
+ }
+ }
+}
local _M = {
version = 0.1,
priority = 11010,
name = plugin_name,
- schema = schema,
+ schema = schema
}
-
function _M.check_schema(conf)
return core.schema.check(schema, conf)
end
+local function gen_worker_number(max_number)
+ if worker_number == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local res, _ = etcd_cli:grant(attr.snowflake.worker_number_ttl)
+
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ while (id <= max_number) do
+ ::continue::
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("worker_number " .. id .. " is not available")
+ id = id + 1
+ else
+ worker_number = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
+ }
+ )
+
+ if err3 then
+ id = id + 1
+ etcd_cli:delete(prefix .. tostring(id))
+ core.log.error("set worker_number " .. id .. " lease error: " .. err3)
+ goto continue
+ end
+
+ local handler = function(premature, etcd_cli, lease_id)
+ local _, err4 = etcd_cli:keepalive(lease_id)
+ if err4 then
+ snowflake_init = nil
+ worker_number = nil
+ core.log.error("snowflake worker_number lease faild.")
+ end
+ core.log.info("snowflake worker_number lease success.")
+ end
+ ngx.timer.every(attr.snowflake.worker_number_interval,
+ handler, etcd_cli, res.body.ID)
+
+ core.log.notice("snowflake worker_number: " .. id)
+ break
+ end
+ end
+
+ if worker_number == nil then
+ core.log.error("No worker_number is not available")
+ return nil
+ end
+ end
+ return worker_number
+end
+
+local function split_worker_number(worker_number, node_id_bits, datacenter_id_bits)
+ local num = bit.tobit(worker_number)
+ local worker_id = bit.band(num, math_pow(2, node_id_bits) - 1) + 1
+ num = bit.rshift(num, node_id_bits)
+ local datacenter_id = bit.band(num, math_pow(2, datacenter_id_bits) - 1) + 1
+ return worker_id, datacenter_id
+end
+
+local function next_id()
+ if snowflake_init == nil then
+ local max_number = math_pow(2, (attr.snowflake.node_id_bits +
+ attr.snowflake.datacenter_id_bits))
+ worker_number = gen_worker_number(max_number)
+ if worker_number == nil then
+ return ""
+ end
+ local worker_id, datacenter_id = split_worker_number(worker_number,
+ attr.snowflake.node_id_bits, attr.snowflake.datacenter_id_bits)
+ core.log.notice("snowflake init datacenter_id: " ..
+ datacenter_id .. " worker_id: " .. worker_id)
+ snowflake.init(
+ worker_id,
+ datacenter_id,
+ attr.snowflake.snowflake_epoc,
+ attr.snowflake.node_id_bits,
+ attr.snowflake.datacenter_id_bits,
+ attr.snowflake.sequence_bits
+ )
+ snowflake_init = true
+ end
+ return snowflake:next_id()
+end
function _M.rewrite(conf, ctx)
local headers = ngx.req.get_headers()
- local uuid_val = uuid()
+ local uuid_val
Review comment:
Better to wrap it into a function
##########
File path: apisix/plugins/request-id.lua
##########
@@ -65,4 +191,47 @@ function _M.header_filter(conf, ctx)
end
end
+function _M.init()
+ local local_conf = core.config.local_conf()
+ attr = core.table.try_read_attr(local_conf, "plugin_attr", plugin_name)
+ local ok, err = core.schema.check(attr_schema, attr)
+ if not ok then
+ core.log.error("failed to check the plugin_attr[", plugin_name, "]", ": ", err)
+ return
+ end
+ if attr.snowflake.enable then
+ if process.type() == "worker" then
+ ngx.timer.at(0, next_id)
Review comment:
What about splitting the next_id into two functions, one for the init?
##########
File path: apisix/plugins/request-id.lua
##########
@@ -14,36 +14,163 @@
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-local core = require("apisix.core")
-local plugin_name = "request-id"
-local ngx = ngx
-local uuid = require("resty.jit-uuid")
+
+local ngx = ngx
+local bit = require("bit")
+local core = require("apisix.core")
+local snowflake = require("snowflake")
+local uuid = require("resty.jit-uuid")
+local process = require("ngx.process")
+local tostring = tostring
+local math_pow = math.pow
+
+local plugin_name = "request-id"
+
+local worker_number = nil
+local snowflake_init = nil
+
+local attr = nil
local schema = {
type = "object",
properties = {
header_name = {type = "string", default = "X-Request-Id"},
- include_in_response = {type = "boolean", default = true}
+ include_in_response = {type = "boolean", default = true},
+ algorithm = {type = "string", enum = {"uuid", "snowflake"}, default = "uuid"}
}
}
+local attr_schema = {
+ type = "object",
+ properties = {
+ snowflake = {
+ type = "object",
+ properties = {
+ enable = {type = "boolean"},
+ snowflake_epoc = {type = "integer", minimum = 1, default = 1609459200000},
+ node_id_bits = {type = "integer", minimum = 1, default = 5},
+ sequence_bits = {type = "integer", minimum = 1, default = 10},
+ datacenter_id_bits = {type = "integer", minimum = 1, default = 5},
+ worker_number_ttl = {type = "integer", minimum = 1, default = 30},
+ worker_number_interval = {type = "integer", minimum = 1, default = 10}
+ }
+ }
+ }
+}
local _M = {
version = 0.1,
priority = 11010,
name = plugin_name,
- schema = schema,
+ schema = schema
}
-
function _M.check_schema(conf)
return core.schema.check(schema, conf)
end
+local function gen_worker_number(max_number)
+ if worker_number == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local res, _ = etcd_cli:grant(attr.snowflake.worker_number_ttl)
+
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ while (id <= max_number) do
+ ::continue::
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("worker_number " .. id .. " is not available")
+ id = id + 1
+ else
+ worker_number = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
+ }
+ )
+
+ if err3 then
+ id = id + 1
+ etcd_cli:delete(prefix .. tostring(id))
+ core.log.error("set worker_number " .. id .. " lease error: " .. err3)
+ goto continue
+ end
+
+ local handler = function(premature, etcd_cli, lease_id)
+ local _, err4 = etcd_cli:keepalive(lease_id)
+ if err4 then
+ snowflake_init = nil
+ worker_number = nil
+ core.log.error("snowflake worker_number lease faild.")
+ end
+ core.log.info("snowflake worker_number lease success.")
+ end
+ ngx.timer.every(attr.snowflake.worker_number_interval,
Review comment:
Can we write it via: https://github.com/apache/apisix/blob/89765ba188834c1b4c9f55088a28db25a959f39e/apisix/plugins/error-log-logger.lua#L198
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Yiyiyimu commented on pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
Yiyiyimu commented on pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#issuecomment-876595542
> CI help
Rerun to see how it goes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander merged pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
spacewander merged pull request #4559:
URL: https://github.com/apache/apisix/pull/4559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r677936999
##########
File path: docs/en/latest/plugins/request-id.md
##########
@@ -72,6 +72,60 @@ X-Request-Id: fe32076a-d0a5-49a6-a361-6c244c1df956
......
```
+### Use the snowflake algorithm to generate an ID
+
+> supports using the Snowflake algorithm to generate ID.
+> read the documentation first before deciding to use snowflake. Because once the configuration information is enabled, you cannot arbitrarily adjust the configuration information. Failure to do so may result in duplicate ID being generated.
+
+The Snowflake algorithm is not enabled by default and needs to be configured in 'conf/config.yaml'.
+
+```yaml
+plugin_attr:
+ request-id:
+ snowflake:
+ enable: true
+ snowflake_epoc: 1609459200000
+ data_machine_bits: 12
+ sequence_bits: 10
+ data_machine_ttl: 30
+ data_machine_interval: 10
+```
+
+#### Configuration parameters
+
+| Name | Type | Requirement | Default | Valid | Description |
+| ------------------- | ------- | ------------- | -------------- | ------- | ------------------------------ |
+| enable | boolean | required | false | | When set it to true, enable the snowflake algorithm. |
Review comment:
Got it. I'll make it optional
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r667565558
##########
File path: apisix/plugins/request-id.lua
##########
@@ -65,4 +191,47 @@ function _M.header_filter(conf, ctx)
end
end
+function _M.init()
+ local local_conf = core.config.local_conf()
+ attr = core.table.try_read_attr(local_conf, "plugin_attr", plugin_name)
+ local ok, err = core.schema.check(attr_schema, attr)
+ if not ok then
+ core.log.error("failed to check the plugin_attr[", plugin_name, "]", ": ", err)
+ return
+ end
+ if attr.snowflake.enable then
+ if process.type() == "worker" then
+ ngx.timer.at(0, next_id)
+ end
+ end
+end
+
+function _M.api()
Review comment:
Let's make it optional and submit it in the next PR. This one is already big enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r677076966
##########
File path: apisix/plugins/request-id.lua
##########
@@ -14,25 +14,58 @@
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-local core = require("apisix.core")
-local plugin_name = "request-id"
-local ngx = ngx
-local uuid = require("resty.jit-uuid")
+
+local ngx = ngx
+local bit = require("bit")
+local core = require("apisix.core")
+local snowflake = require("snowflake")
+local uuid = require("resty.jit-uuid")
+local process = require("ngx.process")
+local timers = require("apisix.timers")
+local tostring = tostring
+local math_pow = math.pow
+local math_ceil = math.ceil
+local math_floor = math.floor
+
+local plugin_name = "request-id"
+
+local data_machine = nil
+local snowflake_inited = nil
+
+local attr = nil
local schema = {
type = "object",
properties = {
header_name = {type = "string", default = "X-Request-Id"},
- include_in_response = {type = "boolean", default = true}
+ include_in_response = {type = "boolean", default = true},
+ algorithm = {type = "string", enum = {"uuid", "snowflake"}, default = "uuid"}
}
}
+local attr_schema = {
+ type = "object",
+ properties = {
+ snowflake = {
+ type = "object",
+ properties = {
+ enable = {type = "boolean"},
Review comment:
Better to set a default value for the optional field
##########
File path: docs/en/latest/plugins/request-id.md
##########
@@ -72,6 +72,60 @@ X-Request-Id: fe32076a-d0a5-49a6-a361-6c244c1df956
......
```
+### Use the snowflake algorithm to generate an ID
+
+> supports using the Snowflake algorithm to generate ID.
+> read the documentation first before deciding to use snowflake. Because once the configuration information is enabled, you cannot arbitrarily adjust the configuration information. Failure to do so may result in duplicate ID being generated.
+
+The Snowflake algorithm is not enabled by default and needs to be configured in 'conf/config.yaml'.
+
+```yaml
+plugin_attr:
+ request-id:
+ snowflake:
+ enable: true
+ snowflake_epoc: 1609459200000
+ data_machine_bits: 12
+ sequence_bits: 10
+ data_machine_ttl: 30
+ data_machine_interval: 10
+```
+
+#### Configuration parameters
+
+| Name | Type | Requirement | Default | Valid | Description |
+| ------------------- | ------- | ------------- | -------------- | ------- | ------------------------------ |
+| enable | boolean | required | false | | When set it to true, enable the snowflake algorithm. |
+| snowflake_epoc | integer | required | 1609459200000 | | Start timestamp (in milliseconds) |
+| data_machine_bits | integer | deprecated | 12 | | Maximum number of supported machines (processes) `1 << data_machine_bits` |
Review comment:
Why 'deprecated'?
##########
File path: docs/en/latest/plugins/request-id.md
##########
@@ -72,6 +72,60 @@ X-Request-Id: fe32076a-d0a5-49a6-a361-6c244c1df956
......
```
+### Use the snowflake algorithm to generate an ID
+
+> supports using the Snowflake algorithm to generate ID.
+> read the documentation first before deciding to use snowflake. Because once the configuration information is enabled, you cannot arbitrarily adjust the configuration information. Failure to do so may result in duplicate ID being generated.
+
+The Snowflake algorithm is not enabled by default and needs to be configured in 'conf/config.yaml'.
+
+```yaml
+plugin_attr:
+ request-id:
+ snowflake:
+ enable: true
+ snowflake_epoc: 1609459200000
+ data_machine_bits: 12
+ sequence_bits: 10
+ data_machine_ttl: 30
+ data_machine_interval: 10
+```
+
+#### Configuration parameters
+
+| Name | Type | Requirement | Default | Valid | Description |
+| ------------------- | ------- | ------------- | -------------- | ------- | ------------------------------ |
+| enable | boolean | required | false | | When set it to true, enable the snowflake algorithm. |
Review comment:
```suggestion
| enable | boolean | optional | false | | When set it to true, enable the snowflake algorithm. |
```
##########
File path: docs/zh/latest/plugins/request-id.md
##########
@@ -71,9 +72,63 @@ X-Request-Id: fe32076a-d0a5-49a6-a361-6c244c1df956
......
```
+### 使用 snowflake 算法生成ID
+
+> 支持使用 snowflake 算法来生成ID。
+> 在决定使用snowflake时，请优先阅读一下文档。因为一旦启用配置信息则不可随意调整配置信息。否则可能会导致生成重复ID。
+
+snowflake 算法默认是不启用的，需要在 `conf/config.yaml` 中开启配置。
+
+```yaml
+plugin_attr:
+ request-id:
+ snowflake:
+ enable: true
+ snowflake_epoc: 1609459200000
+ data_machine_bits: 12
+ sequence_bits: 10
+ data_machine_ttl: 30
+ data_machine_interval: 10
+```
+
+#### 配置参数
+
+| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 |
+| ------------------- | ------- | -------- | -------------- | ------ | ------------------------------ |
+| enable | boolean | 可选 | false | | 当设置为true时， 启用snowflake算法。 |
+| snowflake_epoc | integer | 可选 | 1609459200000 | | 起始时间戳（单位： 毫秒） |
+| data_machine_bits | integer | 可选 | 12 | | 最多支持机器（进程）数量 `1 << data_machine_bits` |
+| sequence_bits | integer | 可选 | 10 | | 每个节点每毫秒内最多产生ID数量 `1 << sequence_bits` |
+| data_machine_ttl | integer | 可选 | 30 | | `etcd` 中 `data_machine` 注册有效时间（单位： 秒）|
+| data_machine_interval | integer | 可选 | 10 | | `etcd` 中 `data_machine` 续约间隔时间（单位： 秒）|
+
+- snowflake_epoc 默认起始时间为 `2021-01-01T00:00:00Z`, 按默认配置可以支持 `69年` 大约可以使用到 `2090-09-07 15:47:35Z`
+- data_machine_bits 对应的是 snowflake 定义中的 WorkerID 和 DatacenterIDd的集合，插件会为每一个进程分配一个唯一ID，最大支持进程数为 `pow(2, data_machine_bits)`。默认占 `12 bits` 最多支持 `4096` 个进程。
Review comment:
```suggestion
- data_machine_bits 对应的是 snowflake 定义中的 WorkerID 和 DatacenterID 的集合，插件会为每一个进程分配一个唯一ID，最大支持进程数为 `pow(2, data_machine_bits)`。默认占 `12 bits` 最多支持 `4096` 个进程。
```
##########
File path: docs/en/latest/plugins/request-id.md
##########
@@ -72,6 +72,60 @@ X-Request-Id: fe32076a-d0a5-49a6-a361-6c244c1df956
......
```
+### Use the snowflake algorithm to generate an ID
+
+> supports using the Snowflake algorithm to generate ID.
+> read the documentation first before deciding to use snowflake. Because once the configuration information is enabled, you cannot arbitrarily adjust the configuration information. Failure to do so may result in duplicate ID being generated.
Review comment:
```suggestion
> read the documentation first before deciding to use snowflake. Because once the configuration information is enabled, you can not arbitrarily adjust the configuration information. Failure to do so may result in duplicate ID being generated.
```
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +74,143 @@ function _M.check_schema(conf)
end
+-- Generates the current process data machine
+local function gen_data_machine(max_number)
+ if data_machine == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ ::continue::
+ while (id <= max_number) do
+ local res, err = etcd_cli:grant(attr.snowflake.data_machine_ttl)
+ if err then
+ core.log.error("Etcd grant failure, err: ".. err)
+ goto continue
Review comment:
There is a chance to cause an infinite loop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r668900193
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +72,134 @@ function _M.check_schema(conf)
end
+-- Generates the current process worker number
+local function gen_worker_number(max_number)
+ if worker_number == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ while (id <= max_number) do
+ ::continue::
Review comment:
Indeed, I will fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r667570485
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +72,134 @@ function _M.check_schema(conf)
end
+-- Generates the current process worker number
+local function gen_worker_number(max_number)
+ if worker_number == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ while (id <= max_number) do
+ ::continue::
+ local res, _ = etcd_cli:grant(attr.snowflake.worker_number_ttl)
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("worker_number " .. id .. " is not available")
+ id = id + 1
+ else
+ worker_number = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
+ }
+ )
+
+ if err3 then
+ id = id + 1
+ etcd_cli:delete(prefix .. tostring(id))
+ core.log.error("set worker_number " .. id .. " lease error: " .. err3)
+ goto continue
+ end
+
+ local lease_id = res.body.ID
+ local start_at = ngx.time()
+ local handler = function()
+ local now = ngx.time()
+ if now - start_at < attr.snowflake.worker_number_interval then
+ return
+ end
+
+ local _, err4 = etcd_cli:keepalive(lease_id)
+ if err4 then
+ snowflake_inited = nil
+ worker_number = nil
+ core.log.error("snowflake worker_number: " .. id .." lease faild.")
+ end
+ start_at = now
+ core.log.info("snowflake worker_number: " .. id .." lease success.")
+ end
+
+ timers.register_timer("plugin#request-id", handler)
Review comment:
Also need to cancel it like this:
https://github.com/apache/apisix/blob/32f9d699a986864612963868d7d23408e3860550/apisix/plugins/error-log-logger.lua#L202-L205
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +72,134 @@ function _M.check_schema(conf)
end
+-- Generates the current process worker number
+local function gen_worker_number(max_number)
+ if worker_number == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ while (id <= max_number) do
+ ::continue::
+ local res, _ = etcd_cli:grant(attr.snowflake.worker_number_ttl)
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("worker_number " .. id .. " is not available")
+ id = id + 1
+ else
+ worker_number = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
+ }
+ )
+
+ if err3 then
+ id = id + 1
+ etcd_cli:delete(prefix .. tostring(id))
+ core.log.error("set worker_number " .. id .. " lease error: " .. err3)
+ goto continue
+ end
+
+ local lease_id = res.body.ID
+ local start_at = ngx.time()
+ local handler = function()
+ local now = ngx.time()
+ if now - start_at < attr.snowflake.worker_number_interval then
+ return
+ end
+
+ local _, err4 = etcd_cli:keepalive(lease_id)
+ if err4 then
+ snowflake_inited = nil
+ worker_number = nil
+ core.log.error("snowflake worker_number: " .. id .." lease faild.")
+ end
+ start_at = now
+ core.log.info("snowflake worker_number: " .. id .." lease success.")
+ end
+
+ timers.register_timer("plugin#request-id", handler)
+ core.log.info(
+ "timer created to lease snowflake algorithm worker number, interval: ",
+ attr.snowflake.worker_number_interval)
+ core.log.notice("lease snowflake worker_number: " .. id)
+ break
+ end
+ end
+
+ if worker_number == nil then
+ core.log.error("No worker_number is not available")
+ return nil
+ end
+ end
+ return worker_number
+end
+
+
+-- Split 'Worker Number' into 'Worker ID' and 'datacenter ID'
+local function split_worker_number(worker_number, node_id_bits, datacenter_id_bits)
+ local num = bit.tobit(worker_number)
+ local worker_id = bit.band(num, math_pow(2, node_id_bits) - 1) + 1
+ num = bit.rshift(num, node_id_bits)
+ local datacenter_id = bit.band(num, math_pow(2, datacenter_id_bits) - 1) + 1
+ return worker_id, datacenter_id
Review comment:
Why start the id from 1? It seems 0 is also valid.
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +72,134 @@ function _M.check_schema(conf)
end
+-- Generates the current process worker number
+local function gen_worker_number(max_number)
+ if worker_number == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ while (id <= max_number) do
+ ::continue::
Review comment:
continue back may causes the id > max_number. For instance:
```
local id = 0
while id <= 3 do
::continue::
if id == 3 then
print(id)
id = id + 1
goto continue
end
print(id)
id = id + 1
end
-- 0, ..., 4
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r672756697
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +74,142 @@ function _M.check_schema(conf)
end
+-- Generates the current process data machine
+local function gen_data_machine(max_number)
+ if data_machine == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ ::continue::
+ while (id <= max_number) do
+ local res, err = etcd_cli:grant(attr.snowflake.data_machine_ttl)
+ if err then
+ core.log.error("Etcd grant failure, err: ".. err)
+ end
+
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("data_machine " .. id .. " is not available")
+ id = id + 1
+ else
+ data_machine = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
+ }
+ )
+
+ if err3 then
+ id = id + 1
+ etcd_cli:delete(prefix .. tostring(id))
+ core.log.error("set data_machine " .. id .. " lease error: " .. err3)
+ goto continue
+ end
+
+ local lease_id = res.body.ID
+ local start_at = ngx.time()
+ local handler = function()
+ local now = ngx.time()
+ if now - start_at < attr.snowflake.data_machine_interval then
+ return
+ end
+
+ local _, err4 = etcd_cli:keepalive(lease_id)
+ if err4 then
+ snowflake_inited = nil
+ data_machine = nil
+ core.log.error("snowflake data_machine: " .. id .." lease faild.")
+ end
+ start_at = now
+ core.log.info("snowflake data_machine: " .. id .." lease success.")
+ end
+
+ timers.register_timer("plugin#request-id", handler)
+ core.log.info(
+ "timer created to lease snowflake algorithm data_machine, interval: ",
+ attr.snowflake.data_machine_interval)
+ core.log.notice("lease snowflake data_machine: " .. id)
+ break
+ end
+ end
+
+ if data_machine == nil then
+ core.log.error("No data_machine is not available")
+ return nil
+ end
+ end
+ return data_machine
+end
+
+
+-- Split 'Data Machine' into 'Worker ID' and 'datacenter ID'
+local function split_data_machine(data_machine, node_id_bits, datacenter_id_bits)
+ local num = bit.tobit(data_machine)
+ local worker_id = bit.band(num, math_pow(2, node_id_bits) - 1)
+ num = bit.rshift(num, node_id_bits)
+ local datacenter_id = bit.band(num, math_pow(2, datacenter_id_bits) - 1)
+ return worker_id, datacenter_id
+end
+
+
+-- Initialize the snowflake algorithm
+local function snowflake_init()
+ if snowflake_inited == nil then
+ local max_number = math_pow(2, (attr.snowflake.data_machine_bits))
+ local datacenter_id_bits = math_floor(attr.snowflake.data_machine_bits / 2)
+ local node_id_bits = math_ceil(attr.snowflake.data_machine_bits / 2)
+ data_machine = gen_data_machine(max_number)
+ if data_machine == nil then
+ return ""
+ end
+
+ local worker_id, datacenter_id = split_data_machine(data_machine,
+ node_id_bits, datacenter_id_bits)
+
+ core.log.info("snowflake init datacenter_id: " ..
+ datacenter_id .. " worker_id: " .. worker_id)
+ snowflake.init(
+ datacenter_id,
+ worker_id,
+ attr.snowflake.snowflake_epoc,
+ node_id_bits,
+ datacenter_id_bits,
+ attr.snowflake.sequence_bits,
+ attr.delta_offset
+ )
+ snowflake_inited = true
+ end
+end
+
+
+-- generate snowflake id
+local function next_id()
+ if snowflake_inited == nil then
+ snowflake_init()
+ end
+ return snowflake:next_id()
+end
+
Review comment:
Let's implement this in the next pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r677502003
##########
File path: docs/en/latest/plugins/request-id.md
##########
@@ -72,6 +72,60 @@ X-Request-Id: fe32076a-d0a5-49a6-a361-6c244c1df956
......
```
+### Use the snowflake algorithm to generate an ID
+
+> supports using the Snowflake algorithm to generate ID.
+> read the documentation first before deciding to use snowflake. Because once the configuration information is enabled, you cannot arbitrarily adjust the configuration information. Failure to do so may result in duplicate ID being generated.
+
+The Snowflake algorithm is not enabled by default and needs to be configured in 'conf/config.yaml'.
+
+```yaml
+plugin_attr:
+ request-id:
+ snowflake:
+ enable: true
+ snowflake_epoc: 1609459200000
+ data_machine_bits: 12
+ sequence_bits: 10
+ data_machine_ttl: 30
+ data_machine_interval: 10
+```
+
+#### Configuration parameters
+
+| Name | Type | Requirement | Default | Valid | Description |
+| ------------------- | ------- | ------------- | -------------- | ------- | ------------------------------ |
+| enable | boolean | required | false | | When set it to true, enable the snowflake algorithm. |
Review comment:
Modified jsonSchema `required = {"enable", "snowflake_epoc "}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r673584829
##########
File path: apisix/plugins/request-id.lua
##########
@@ -245,5 +245,8 @@ function _M.init()
end
end
+function _M.destroy()
+ timers.unregister_timer("plugin#request-id")
Review comment:
Better to check if the timer is registered?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#issuecomment-876310306
CI help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r681179162
##########
File path: docs/en/latest/plugins/request-id.md
##########
@@ -72,6 +73,60 @@ X-Request-Id: fe32076a-d0a5-49a6-a361-6c244c1df956
......
```
+### Use the snowflake algorithm to generate an ID
+
+> supports using the Snowflake algorithm to generate ID.
+> read the documentation first before deciding to use snowflake. Because once the configuration information is enabled, you can not arbitrarily adjust the configuration information. Failure to do so may result in duplicate ID being generated.
+
+The Snowflake algorithm is not enabled by default and needs to be configured in 'conf/config.yaml'.
+
+```yaml
+plugin_attr:
+ request-id:
+ snowflake:
+ enable: true
+ snowflake_epoc: 1609459200000
+ data_machine_bits: 12
+ sequence_bits: 10
+ data_machine_ttl: 30
+ data_machine_interval: 10
+```
+
+#### Configuration parameters
+
+| Name | Type | Requirement | Default | Valid | Description |
+| ------------------- | ------- | ------------- | -------------- | ------- | ------------------------------ |
+| enable | boolean | optional | false | | When set it to true, enable the snowflake algorithm. |
+| snowflake_epoc | integer | optional | 1609459200000 | | Start timestamp (in milliseconds) |
+| data_machine_bits | integer | optional | 12 | | Maximum number of supported machines (processes) `1 << data_machine_bits` |
+| sequence_bits | integer | optional | 10 | | Maximum number of generated ID per millisecond per node `1 << sequence_bits` |
+| data_machine_ttl | integer | optional | 30 | | Valid time of registration of 'data_machine' in 'etcd' (unit: seconds) |
+| data_machine_interval | integer | optional | 10 | | Time between 'data_machine' renewal in 'etcd' (unit: seconds) |
+
+- `snowflake_epoc` default start time is `2021-01-01T00:00:00Z`, and it can support `69 year` approximately to `2090-09-0715:47:35Z` according to the default configuration
+- `data_machine_bits` corresponds to the set of workIDs and datacEnteridd in the snowflake definition. The plug-in aslocates a unique ID to each process. Maximum number of supported processes is `pow(2, data_machine_bits)`. The default number of `12 bits` is up to `4096`.
+- `sequence_bits` defaults to `10 bits` and each process generates up to `1024` ID per second
Review comment:
The `per second` here and the `per millisecond` above confuse me
##########
File path: conf/config-default.yaml
##########
@@ -348,3 +348,12 @@ plugin_attr:
report_ttl: 3600 # live time for server info in etcd (unit: second)
dubbo-proxy:
upstream_multiplex_count: 32
+ request-id:
+ snowflake:
+ enable: false
+ snowflake_epoc: 1609459200000 # the starting timestamp is expressed in milliseconds
+ data_machine_bits: 12 # data machine bit, maximum 31, because Lua cannot do bit operations greater than 31
Review comment:
should check `data_machine_bits` + `sequence_bits` = 22 always? may the test case need to cover it.
##########
File path: t/plugin/request-id.t
##########
@@ -470,3 +470,268 @@ GET /t
X-Request-Id and Custom-Header-Name are different
--- no_error_log
[error]
+
+
+
+=== TEST 12: check for snowflake id
+--- yaml_config
+plugins:
+ - request-id
+plugin_attr:
+ request-id:
+ snowflake:
+ enable: true
+ snowflake_epoc: 1609459200000
+ data_machine_bits: 10
+ sequence_bits: 10
+ data_machine_ttl: 30
+ data_machine_interval: 10
+--- config
+location /t {
+ content_by_lua_block {
+ ngx.sleep(3)
+ local core = require("apisix.core")
+ local key = "/plugins/request-id/snowflake/1"
+ local res, err = core.etcd.get(key)
+ if err ~= nil then
+ ngx.status = 500
+ ngx.say(err)
+ return
+ end
+ if res.body.node.key ~= "/apisix/plugins/request-id/snowflake/1" then
+ ngx.say(core.json.encode(res.body.node))
+ end
+ ngx.say("ok")
+ }
+}
+--- request
+GET /t
+--- response_body
+ok
+--- no_error_log
+[error]
+
+
+
+=== TEST 13: wrong type
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.request-id")
+ local ok, err = plugin.check_schema({algorithm = "bad_algorithm"})
+ if not ok then
+ ngx.say(err)
+ end
+ ngx.say("done")
+ }
+ }
+--- request
+GET /t
+--- response_body
+property "algorithm" validation failed: matches none of the enum values
+done
+--- no_error_log
+[error]
+
+
+
+=== TEST 14: add plugin with algorithm snowflake (default uuid)
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "request-id": {
+ "algorithm": "snowflake"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "node": {
+ "value": {
+ "plugins": {
+ "request-id": {
+ "algorithm": "snowflake"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ },
+ "key": "/apisix/routes/1"
+ },
+ "action": "set"
+ }]]
+ )
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 15: check for snowflake id
+--- yaml_config
+plugins:
+ - request-id
+plugin_attr:
+ request-id:
+ snowflake:
+ enable: true
+--- config
+ location /t {
+ content_by_lua_block {
+ local http = require "resty.http"
+ local t = {}
+ local ids = {}
+ for i = 1, 180 do
+ local th = assert(ngx.thread.spawn(function()
+ local httpc = http.new()
+ local uri = "http://127.0.0.1:" .. ngx.var.server_port .. "/opentracing"
+ local res, err = httpc:request_uri(uri,
+ {
+ method = "GET",
+ headers = {
+ ["Content-Type"] = "application/json",
+ }
+ }
+ )
+ if not res then
+ ngx.log(ngx.ERR, err)
+ return
+ end
+ local id = res.headers["X-Request-Id"]
+ if not id then
+ return -- ignore if the data is not synced yet.
+ end
+ if ids[id] == true then
+ ngx.say("ids not unique")
+ return
+ end
+ ids[id] = true
Review comment:
Is there any way to check that the length of the `ids` has reached a fixed value? such as 100, or at least 2. I am worried that there is no data in the ids due to sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r674454259
##########
File path: apisix/plugins/request-id.lua
##########
@@ -245,5 +245,8 @@ function _M.init()
end
end
+function _M.destroy()
+ timers.unregister_timer("plugin#request-id")
Review comment:
You have a point, I added the logic of judgment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r666933079
##########
File path: apisix/plugins/request-id.lua
##########
@@ -14,36 +14,163 @@
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-local core = require("apisix.core")
-local plugin_name = "request-id"
-local ngx = ngx
-local uuid = require("resty.jit-uuid")
+
+local ngx = ngx
+local bit = require("bit")
+local core = require("apisix.core")
+local snowflake = require("snowflake")
+local uuid = require("resty.jit-uuid")
+local process = require("ngx.process")
+local tostring = tostring
+local math_pow = math.pow
+
+local plugin_name = "request-id"
+
+local worker_number = nil
+local snowflake_init = nil
+
+local attr = nil
local schema = {
type = "object",
properties = {
header_name = {type = "string", default = "X-Request-Id"},
- include_in_response = {type = "boolean", default = true}
+ include_in_response = {type = "boolean", default = true},
+ algorithm = {type = "string", enum = {"uuid", "snowflake"}, default = "uuid"}
}
}
+local attr_schema = {
+ type = "object",
+ properties = {
+ snowflake = {
+ type = "object",
+ properties = {
+ enable = {type = "boolean"},
+ snowflake_epoc = {type = "integer", minimum = 1, default = 1609459200000},
+ node_id_bits = {type = "integer", minimum = 1, default = 5},
+ sequence_bits = {type = "integer", minimum = 1, default = 10},
+ datacenter_id_bits = {type = "integer", minimum = 1, default = 5},
Review comment:
Maybe "default = 7" is better? One of the largest APISIX user have more than 2000 workers. They don't want to be in trouble when using the default value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] starsz commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
starsz commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r674457576
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +74,142 @@ function _M.check_schema(conf)
end
+-- Generates the current process data machine
+local function gen_data_machine(max_number)
+ if data_machine == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ ::continue::
+ while (id <= max_number) do
+ local res, err = etcd_cli:grant(attr.snowflake.data_machine_ttl)
+ if err then
+ core.log.error("Etcd grant failure, err: ".. err)
Review comment:
Do here need to go to continue?
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +74,142 @@ function _M.check_schema(conf)
end
+-- Generates the current process data machine
+local function gen_data_machine(max_number)
+ if data_machine == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ ::continue::
+ while (id <= max_number) do
+ local res, err = etcd_cli:grant(attr.snowflake.data_machine_ttl)
+ if err then
+ core.log.error("Etcd grant failure, err: ".. err)
+ end
+
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("data_machine " .. id .. " is not available")
+ id = id + 1
+ else
+ data_machine = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
Review comment:
Need to check the `res`, it will be nil if `etcd_cli:grant` failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r666608780
##########
File path: apisix/plugins/request-id.lua
##########
@@ -14,36 +14,163 @@
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-local core = require("apisix.core")
-local plugin_name = "request-id"
-local ngx = ngx
-local uuid = require("resty.jit-uuid")
+
+local ngx = ngx
+local bit = require("bit")
+local core = require("apisix.core")
+local snowflake = require("snowflake")
+local uuid = require("resty.jit-uuid")
+local process = require("ngx.process")
+local tostring = tostring
+local math_pow = math.pow
+
+local plugin_name = "request-id"
+
+local worker_number = nil
+local snowflake_init = nil
+
+local attr = nil
local schema = {
type = "object",
properties = {
header_name = {type = "string", default = "X-Request-Id"},
- include_in_response = {type = "boolean", default = true}
+ include_in_response = {type = "boolean", default = true},
+ algorithm = {type = "string", enum = {"uuid", "snowflake"}, default = "uuid"}
}
}
+local attr_schema = {
+ type = "object",
+ properties = {
+ snowflake = {
+ type = "object",
+ properties = {
+ enable = {type = "boolean"},
+ snowflake_epoc = {type = "integer", minimum = 1, default = 1609459200000},
+ node_id_bits = {type = "integer", minimum = 1, default = 5},
+ sequence_bits = {type = "integer", minimum = 1, default = 10},
+ datacenter_id_bits = {type = "integer", minimum = 1, default = 5},
Review comment:
~~Better to use a user-friendly name, like max_worker_number or ids_per_second~~
The current name is fine. But we should clarify in the doc that the xxx_bits are for snowflake. There is nothing relative to node or datacenter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 edited a comment on pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 edited a comment on pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#issuecomment-879531464
<s>TODO: upgrade api7/lua-snowflake to 2.1-1<s>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r672756697
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +74,142 @@ function _M.check_schema(conf)
end
+-- Generates the current process data machine
+local function gen_data_machine(max_number)
+ if data_machine == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ ::continue::
+ while (id <= max_number) do
+ local res, err = etcd_cli:grant(attr.snowflake.data_machine_ttl)
+ if err then
+ core.log.error("Etcd grant failure, err: ".. err)
+ end
+
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("data_machine " .. id .. " is not available")
+ id = id + 1
+ else
+ data_machine = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
+ }
+ )
+
+ if err3 then
+ id = id + 1
+ etcd_cli:delete(prefix .. tostring(id))
+ core.log.error("set data_machine " .. id .. " lease error: " .. err3)
+ goto continue
+ end
+
+ local lease_id = res.body.ID
+ local start_at = ngx.time()
+ local handler = function()
+ local now = ngx.time()
+ if now - start_at < attr.snowflake.data_machine_interval then
+ return
+ end
+
+ local _, err4 = etcd_cli:keepalive(lease_id)
+ if err4 then
+ snowflake_inited = nil
+ data_machine = nil
+ core.log.error("snowflake data_machine: " .. id .." lease faild.")
+ end
+ start_at = now
+ core.log.info("snowflake data_machine: " .. id .." lease success.")
+ end
+
+ timers.register_timer("plugin#request-id", handler)
+ core.log.info(
+ "timer created to lease snowflake algorithm data_machine, interval: ",
+ attr.snowflake.data_machine_interval)
+ core.log.notice("lease snowflake data_machine: " .. id)
+ break
+ end
+ end
+
+ if data_machine == nil then
+ core.log.error("No data_machine is not available")
+ return nil
+ end
+ end
+ return data_machine
+end
+
+
+-- Split 'Data Machine' into 'Worker ID' and 'datacenter ID'
+local function split_data_machine(data_machine, node_id_bits, datacenter_id_bits)
+ local num = bit.tobit(data_machine)
+ local worker_id = bit.band(num, math_pow(2, node_id_bits) - 1)
+ num = bit.rshift(num, node_id_bits)
+ local datacenter_id = bit.band(num, math_pow(2, datacenter_id_bits) - 1)
+ return worker_id, datacenter_id
+end
+
+
+-- Initialize the snowflake algorithm
+local function snowflake_init()
+ if snowflake_inited == nil then
+ local max_number = math_pow(2, (attr.snowflake.data_machine_bits))
+ local datacenter_id_bits = math_floor(attr.snowflake.data_machine_bits / 2)
+ local node_id_bits = math_ceil(attr.snowflake.data_machine_bits / 2)
+ data_machine = gen_data_machine(max_number)
+ if data_machine == nil then
+ return ""
+ end
+
+ local worker_id, datacenter_id = split_data_machine(data_machine,
+ node_id_bits, datacenter_id_bits)
+
+ core.log.info("snowflake init datacenter_id: " ..
+ datacenter_id .. " worker_id: " .. worker_id)
+ snowflake.init(
+ datacenter_id,
+ worker_id,
+ attr.snowflake.snowflake_epoc,
+ node_id_bits,
+ datacenter_id_bits,
+ attr.snowflake.sequence_bits,
+ attr.delta_offset
+ )
+ snowflake_inited = true
+ end
+end
+
+
+-- generate snowflake id
+local function next_id()
+ if snowflake_inited == nil then
+ snowflake_init()
+ end
+ return snowflake:next_id()
+end
+
Review comment:
Let's implement this in the next pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r684296559
##########
File path: conf/config-default.yaml
##########
@@ -348,3 +348,12 @@ plugin_attr:
report_ttl: 3600 # live time for server info in etcd (unit: second)
dubbo-proxy:
upstream_multiplex_count: 32
+ request-id:
+ snowflake:
+ enable: false
+ snowflake_epoc: 1609459200000 # the starting timestamp is expressed in milliseconds
+ data_machine_bits: 12 # data machine bit, maximum 31, because Lua cannot do bit operations greater than 31
Review comment:
`data_machine_bits` and `sequence_bits` are not fixed and can be configured according to different requirements
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r677901240
##########
File path: docs/en/latest/plugins/request-id.md
##########
@@ -72,6 +72,60 @@ X-Request-Id: fe32076a-d0a5-49a6-a361-6c244c1df956
......
```
+### Use the snowflake algorithm to generate an ID
+
+> supports using the Snowflake algorithm to generate ID.
+> read the documentation first before deciding to use snowflake. Because once the configuration information is enabled, you cannot arbitrarily adjust the configuration information. Failure to do so may result in duplicate ID being generated.
+
+The Snowflake algorithm is not enabled by default and needs to be configured in 'conf/config.yaml'.
+
+```yaml
+plugin_attr:
+ request-id:
+ snowflake:
+ enable: true
+ snowflake_epoc: 1609459200000
+ data_machine_bits: 12
+ sequence_bits: 10
+ data_machine_ttl: 30
+ data_machine_interval: 10
+```
+
+#### Configuration parameters
+
+| Name | Type | Requirement | Default | Valid | Description |
+| ------------------- | ------- | ------------- | -------------- | ------- | ------------------------------ |
+| enable | boolean | required | false | | When set it to true, enable the snowflake algorithm. |
Review comment:
Once you provide a default value, the field is actually optional, as users don't need to specify their values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
spacewander commented on pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#issuecomment-876069932
Need to make the CI pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r673223211
##########
File path: apisix/plugins/request-id.lua
##########
@@ -14,25 +14,58 @@
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-local core = require("apisix.core")
-local plugin_name = "request-id"
-local ngx = ngx
-local uuid = require("resty.jit-uuid")
+
+local ngx = ngx
+local bit = require("bit")
+local core = require("apisix.core")
+local snowflake = require("snowflake")
+local uuid = require("resty.jit-uuid")
+local process = require("ngx.process")
+local timers = require("apisix.timers")
+local tostring = tostring
+local math_pow = math.pow
+local math_ceil = math.ceil
+local math_floor = math.floor
+
+local plugin_name = "request-id"
+
+local data_machine = nil
+local snowflake_inited = nil
+
+local attr = nil
local schema = {
type = "object",
properties = {
header_name = {type = "string", default = "X-Request-Id"},
- include_in_response = {type = "boolean", default = true}
+ include_in_response = {type = "boolean", default = true},
+ algorithm = {type = "string", enum = {"uuid", "snowflake"}, default = "uuid"}
}
}
+local attr_schema = {
+ type = "object",
+ properties = {
+ snowflake = {
+ type = "object",
+ properties = {
+ enable = {type = "boolean"},
+ snowflake_epoc = {type = "integer", minimum = 1, default = 1609459200000},
+ data_machine_bits = {type = "integer", minimum = 1, maximum = 31, default = 12},
+ sequence_bits = {type = "integer", minimum = 1, default = 10},
+ delta_offset = {type = "integer", default = 1, enum = {1, 10, 100, 1000}},
Review comment:
delta_offset has been removed from this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] nic-chen commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
nic-chen commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r669427292
##########
File path: apisix/plugins/request-id.lua
##########
@@ -41,9 +74,142 @@ function _M.check_schema(conf)
end
+-- Generates the current process data machine
+local function gen_data_machine(max_number)
+ if data_machine == nil then
+ local etcd_cli, prefix = core.etcd.new()
+ local prefix = prefix .. "/plugins/request-id/snowflake/"
+ local uuid = uuid.generate_v4()
+ local id = 1
+ ::continue::
+ while (id <= max_number) do
+ local res, err = etcd_cli:grant(attr.snowflake.data_machine_ttl)
+ if err then
+ core.log.error("Etcd grant failure, err: ".. err)
+ end
+
+ local _, err1 = etcd_cli:setnx(prefix .. tostring(id), uuid)
+ local res2, err2 = etcd_cli:get(prefix .. tostring(id))
+
+ if err1 or err2 or res2.body.kvs[1].value ~= uuid then
+ core.log.notice("data_machine " .. id .. " is not available")
+ id = id + 1
+ else
+ data_machine = id
+
+ local _, err3 =
+ etcd_cli:set(
+ prefix .. tostring(id),
+ uuid,
+ {
+ prev_kv = true,
+ lease = res.body.ID
+ }
+ )
+
+ if err3 then
+ id = id + 1
+ etcd_cli:delete(prefix .. tostring(id))
+ core.log.error("set data_machine " .. id .. " lease error: " .. err3)
+ goto continue
+ end
+
+ local lease_id = res.body.ID
+ local start_at = ngx.time()
+ local handler = function()
+ local now = ngx.time()
+ if now - start_at < attr.snowflake.data_machine_interval then
+ return
+ end
+
+ local _, err4 = etcd_cli:keepalive(lease_id)
+ if err4 then
+ snowflake_inited = nil
+ data_machine = nil
+ core.log.error("snowflake data_machine: " .. id .." lease faild.")
+ end
+ start_at = now
+ core.log.info("snowflake data_machine: " .. id .." lease success.")
+ end
+
+ timers.register_timer("plugin#request-id", handler)
+ core.log.info(
+ "timer created to lease snowflake algorithm data_machine, interval: ",
+ attr.snowflake.data_machine_interval)
+ core.log.notice("lease snowflake data_machine: " .. id)
+ break
+ end
+ end
+
+ if data_machine == nil then
+ core.log.error("No data_machine is not available")
+ return nil
+ end
+ end
+ return data_machine
+end
+
+
+-- Split 'Data Machine' into 'Worker ID' and 'datacenter ID'
+local function split_data_machine(data_machine, node_id_bits, datacenter_id_bits)
+ local num = bit.tobit(data_machine)
+ local worker_id = bit.band(num, math_pow(2, node_id_bits) - 1)
+ num = bit.rshift(num, node_id_bits)
+ local datacenter_id = bit.band(num, math_pow(2, datacenter_id_bits) - 1)
+ return worker_id, datacenter_id
+end
+
+
+-- Initialize the snowflake algorithm
+local function snowflake_init()
+ if snowflake_inited == nil then
+ local max_number = math_pow(2, (attr.snowflake.data_machine_bits))
+ local datacenter_id_bits = math_floor(attr.snowflake.data_machine_bits / 2)
+ local node_id_bits = math_ceil(attr.snowflake.data_machine_bits / 2)
+ data_machine = gen_data_machine(max_number)
+ if data_machine == nil then
+ return ""
+ end
+
+ local worker_id, datacenter_id = split_data_machine(data_machine,
+ node_id_bits, datacenter_id_bits)
+
+ core.log.info("snowflake init datacenter_id: " ..
+ datacenter_id .. " worker_id: " .. worker_id)
+ snowflake.init(
+ datacenter_id,
+ worker_id,
+ attr.snowflake.snowflake_epoc,
+ node_id_bits,
+ datacenter_id_bits,
+ attr.snowflake.sequence_bits,
+ attr.delta_offset
+ )
+ snowflake_inited = true
+ end
+end
+
+
+-- generate snowflake id
+local function next_id()
+ if snowflake_inited == nil then
+ snowflake_init()
+ end
+ return snowflake:next_id()
+end
+
Review comment:
Could we put them under `utils`? Snowflake ID generating should also be used elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dickens7 commented on a change in pull request #4559: feat: Request-ID plugin add snowflake algorithm
Posted by GitBox <gi...@apache.org>.
dickens7 commented on a change in pull request #4559:
URL: https://github.com/apache/apisix/pull/4559#discussion_r667345377
##########
File path: apisix/plugins/request-id.lua
##########
@@ -65,4 +191,47 @@ function _M.header_filter(conf, ctx)
end
end
+function _M.init()
+ local local_conf = core.config.local_conf()
+ attr = core.table.try_read_attr(local_conf, "plugin_attr", plugin_name)
+ local ok, err = core.schema.check(attr_schema, attr)
+ if not ok then
+ core.log.error("failed to check the plugin_attr[", plugin_name, "]", ": ", err)
+ return
+ end
+ if attr.snowflake.enable then
+ if process.type() == "worker" then
+ ngx.timer.at(0, next_id)
+ end
+ end
+end
+
+function _M.api()
Review comment:
The API is exposed to make it a global ID generator. The real applicatipon may require multiple ID while the request-id can only be passed to one proxy application.If there is a concern about the exposing of the algorithm mechanism, I can implement it in other way, such as, registering a specific route and configure the request-id plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org