You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/08/02 08:42:41 UTC
[GitHub] [apisix] ychensha opened a new pull request, #7593: [WIP]support Tencent Cloud Log Service
ychensha opened a new pull request, #7593:
URL: https://github.com/apache/apisix/pull/7593
### Description
[product doc](https://cloud.tencent.com/document/product/614)
apisix log util is more powerfull than nginx access log, and it's useful for debugging and replaying later.
Fixes # (7592)
### Checklist
- [ ] I have explained the need for this PR and the problem it solves
- [ ] I have explained the changes or the new features added to this PR
- [ ] I have added tests corresponding to this change
- [ ] I have updated the documentation to reflect this change
- [ ] I have verified that this change is backward compatible (If not, please discuss on the [APISIX mailing list](https://github.com/apache/apisix/tree/master#community) first)
<!--
Note
1. Mark the PR as draft until it's ready to be reviewed.
2. Always add/update tests for any changes unless you have a good reason.
3. Always update the documentation to reflect the changes made in the PR.
4. Make a new commit to resolve conversations instead of `push -f`.
5. To resolve merge conflicts, merge master instead of rebasing.
6. Use "request review" to notify the reviewer after making changes.
7. Only a reviewer can mark a conversation as resolved.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
soulbird commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r947486624
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,308 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local protoc = require("protoc").new()
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+local setmetatable = setmetatable
+local pcall = pcall
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
Review Comment:
Is this value immutable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
soulbird commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1203385980
Thanks for your contribution. But please add a license to each file and add test cases first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1217370858
https://github.com/apache/apisix/runs/7861163309?check_suite_focus=true
Let's add the `tencent-cloud-cls`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ychensha commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
ychensha commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r938379695
##########
apisix/plugins/tencent-cloud-cls.lua:
##########
@@ -0,0 +1,94 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local log_util = require("apisix.utils.log-util")
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local cls_sdk = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+local random = math.random
+math.randomseed(ngx.time() + ngx.worker.pid())
+local ngx = ngx
+local pairs = pairs
+
+local plugin_name = "tencent-cloud-cls"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+local schema = {
+ type = "object",
+ properties = {
+ cls_host = { type = "string" },
+ cls_topic = { type = "string" },
+ -- https://console.cloud.tencent.com/capi
+ secret_id = { type = "string" },
+ secret_key = { type = "string" },
+ sample_rate = { type = "integer", minimum = 1, maximum = 100, default = 100 },
+ include_req_body = { type = "boolean", default = false },
+ include_resp_body = { type = "boolean", default = false },
+ global_tag = { type = "object" },
+ },
+ required = { "cls_host", "cls_topic", "secret_id", "secret_key" }
+}
+
+local _M = {
+ version = 0.1,
+ priority = 397,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+}
+
+function _M.check_schema(conf)
+ return core.schema.check(schema, conf)
+end
+
+function _M.body_filter(conf, ctx)
+ -- sample if set
Review Comment:
OK. I will do in that way..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ychensha commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
ychensha commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1211569874
>
If hook is ok for now, I will add test code like otel plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r951052416
##########
t/plugin/tencent-cloud-cls.t:
##########
@@ -0,0 +1,304 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+use t::APISIX 'no_plan';
+
+log_level('debug');
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "tencent-cloud-cls body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "tencent-cloud-cls headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+ server {
+ listen 10421;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.exit(500)
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests;
+
+__DATA__
+
+=== TEST 1: schema check
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ secret_key = "secret_key",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+done
+
+
+
+=== TEST 2: cls config missing
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+property "secret_key" is required
+done
+
+
+
+=== TEST 3: add plugin for incorrect server
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "tencent-cloud-cls": {
+ "cls_host": "127.0.0.1:10421",
+ "cls_topic": "143b5d70-139b-4aec-b54e-bb97756916de",
+ "secret_id": "secret_id",
+ "secret_key": "secret_key",
+ "batch_max_size": 1,
+ "max_retry_count": 1,
+ "retry_delay": 2,
+ "buffer_duration": 2,
+ "inactive_timeout": 2
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 4: incorrect server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] failed to process entries [1/1]: got wrong status: 500
+--- wait: 0.5
+
+
+
+=== TEST 5: add plugin
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "tencent-cloud-cls": {
+ "cls_host": "127.0.0.1:10420",
+ "cls_topic": "143b5d70-139b-4aec-b54e-bb97756916de",
+ "secret_id": "secret_id",
+ "secret_key": "secret_key",
+ "batch_max_size": 1,
+ "max_retry_count": 1,
+ "retry_delay": 2,
+ "buffer_duration": 2,
+ "inactive_timeout": 2
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 6: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] successfully processed the entries
+--- wait: 0.5
+
+
+
+=== TEST 7: verify request
+--- extra_init_by_lua
+ local cls = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+ cls.send_to_cls = function(self, logs)
+ if (#logs ~= 1) then
+ ngx.log(ngx.ERR, "unexpected logs length: ", #logs)
+ return
+ end
+ return true
+ end
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] successfully processed the entries
+--- wait: 0.5
+
+
+
+=== TEST 8: verify cls api request
+--- extra_init_by_lua
+ local cls = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+ cls.send_cls_request = function(self, pb_obj)
+ if (#pb_obj.logGroupList ~= 1) then
+ ngx.log(ngx.ERR, "unexpected logGroupList length: ", #pb_obj.logGroupList)
+ return false
+ end
+ local log_group = pb_obj.logGroupList[1]
+ if #log_group.logs ~= 1 then
+ ngx.log(ngx.ERR, "unexpected logs length: ", #log_group.logs)
+ return false
+ end
+ local log = log_group.logs[1]
+ if #log.contents == 0 then
+ ngx.log(ngx.ERR, "unexpected contents length: ", #log.contents)
+ return false
+ end
+ return true
+ end
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- no_error_log
+[error]
Review Comment:
We don't need to set `no_error_log` again as we already set at the top of this file:
```
if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
$block->set_value("no_error_log", "[error]");
}
```
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,315 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local protoc = require("protoc").new()
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+local setmetatable = setmetatable
+local pcall = pcall
+
+-- api doc https://www.tencentcloud.com/document/product/614/16873
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
+local auth_expire_time = 60
+local cls_conn_timeout = 1000
+local cls_read_timeout = 10000
+local cls_send_timeout = 10000
+
+local headers_cache = {}
+local params_cache = {
+ ssl_verify = false,
+ headers = headers_cache,
+}
+
+
+local function get_ip(hostname)
+ local _, resolved = socket.dns.toip(hostname)
+ local ip_list = {}
+ for _, v in ipairs(resolved.ip) do
+ insert_tab(ip_list, v)
+ end
+ return ip_list
+end
+
+local host_ip = tostring(unpack(get_ip(core_gethostname())))
+local log_group_list = {}
+local log_group_list_pb = {
+ logGroupList = log_group_list,
+}
+
+
+local function sha1(msg)
+ return str_util.to_hex(ngx_sha1_bin(msg))
+end
+
+
+local function sha1_hmac(key, msg)
+ return str_util.to_hex(ngx_hmac_sha1(key, msg))
+end
+
+
+-- sign algorithm https://cloud.tencent.com/document/product/614/12445
+local function sign(secret_id, secret_key)
+ local method = "post"
+ local format_params = ""
+ local format_headers = ""
+ local sign_algorithm = "sha1"
+ local http_request_info = fmt("%s\n%s\n%s\n%s\n",
+ method, cls_api_path, format_params, format_headers)
+ local cur_time = ngx_time()
+ local sign_time = fmt("%d;%d", cur_time, cur_time + auth_expire_time)
+ local string_to_sign = fmt("%s\n%s\n%s\n", sign_algorithm, sign_time, sha1(http_request_info))
+
+ local sign_key = sha1_hmac(secret_key, sign_time)
+ local signature = sha1_hmac(sign_key, string_to_sign)
+
+ local arr = {
+ "q-sign-algorithm=sha1",
+ "q-ak=" .. secret_id,
+ "q-sign-time=" .. sign_time,
+ "q-key-time=" .. sign_time,
+ "q-header-list=",
+ "q-url-param-list=",
+ "q-signature=" .. signature,
+ }
+
+ return concat_tab(arr, '&')
+end
+
+
+-- normalized log data for CLS API
+local function normalize_log(log)
+ local normalized_log = {}
+ local log_size = 4 -- empty obj alignment
+ for k, v in pairs(log) do
+ local v_type = type(v)
+ local field = { key = k, value = "" }
+ if v_type == "string" then
+ field["value"] = v
+ elseif v_type == "number" then
+ field["value"] = tostring(v)
+ elseif v_type == "table" then
+ field["value"] = json_encode(v)
+ else
+ field["value"] = tostring(v)
+ core.log.warn("unexpected type " .. v_type .. " for field " .. k)
+ end
+ if #field.value > MAX_SINGLE_VALUE_SIZE then
+ core.log.warn(field.key, " value size over ", MAX_SINGLE_VALUE_SIZE, " , truncated")
+ field.value = field.value:sub(1, MAX_SINGLE_VALUE_SIZE)
+ end
+ insert_tab(normalized_log, field)
+ log_size = log_size + #field.key + #field.value
+ end
+ return normalized_log, log_size
+end
+
+
+local _M = { version = 0.1 }
+local mt = { __index = _M }
+
+local pb_state
+local function init_pb_state()
+ local old_pb_state = pb.state(nil)
+ protoc.reload()
+ local cls_sdk_protoc = protoc.new()
+ -- proto file in https://www.tencentcloud.com/document/product/614/42787
+ local ok, err = pcall(cls_sdk_protoc.load, cls_sdk_protoc, [[
+package cls;
+
+message Log
+{
+ message Content
+ {
+ required string key = 1; // Key of each field group
+ required string value = 2; // Value of each field group
+ }
+ required int64 time = 1; // Unix timestamp
+ repeated Content contents = 2; // Multiple key-value pairs in one log
+}
+
+message LogTag
+{
+ required string key = 1;
+ required string value = 2;
+}
+
+message LogGroup
+{
+ repeated Log logs = 1; // Log array consisting of multiple logs
+ optional string contextFlow = 2; // This parameter does not take effect currently
+ optional string filename = 3; // Log filename
+ optional string source = 4; // Log source, which is generally the machine IP
+ repeated LogTag logTags = 5;
+}
+
+message LogGroupList
+{
+ repeated LogGroup logGroupList = 1; // Log group list
+}
+ ]], "tencent-cloud-cls/cls.proto")
+ if not ok then
+ cls_sdk_protoc:reset()
+ pb.state(old_pb_state)
+ return "failed to load cls.proto: ".. err
+ end
+ pb_state = pb.state(old_pb_state)
+end
+
+
+function _M.new(host, topic, secret_id, secret_key)
+ if not pb_state then
+ local err = init_pb_state()
+ if err then
+ return nil, err
+ end
+ end
+ local self = {
+ host = host,
+ topic = topic,
+ secret_id = secret_id,
+ secret_key = secret_key,
+ }
+ return setmetatable(self, mt)
+end
+
+
+local function do_request_uri(uri, params)
+ local client = http:new()
+ client:set_timeouts(cls_conn_timeout, cls_send_timeout, cls_read_timeout)
+ local res, err = client:request_uri(uri, params)
+ client:close()
+ return res, err
+end
+
+
+function _M.send_cls_request(self, pb_obj)
+ -- recovery of stored pb_store
+ pb.state(pb_state)
+
+ local ok, pb_data = pcall(pb.encode, "cls.LogGroupList", pb_obj)
Review Comment:
Need to recover the pb_state after used.
##########
t/plugin/tencent-cloud-cls.t:
##########
@@ -0,0 +1,304 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+use t::APISIX 'no_plan';
+
+log_level('debug');
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "tencent-cloud-cls body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "tencent-cloud-cls headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+ server {
+ listen 10421;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.exit(500)
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests;
+
+__DATA__
+
+=== TEST 1: schema check
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ secret_key = "secret_key",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+done
+
+
+
+=== TEST 2: cls config missing
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+property "secret_key" is required
+done
+
+
+
+=== TEST 3: add plugin for incorrect server
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "tencent-cloud-cls": {
+ "cls_host": "127.0.0.1:10421",
+ "cls_topic": "143b5d70-139b-4aec-b54e-bb97756916de",
+ "secret_id": "secret_id",
+ "secret_key": "secret_key",
+ "batch_max_size": 1,
+ "max_retry_count": 1,
+ "retry_delay": 2,
+ "buffer_duration": 2,
+ "inactive_timeout": 2
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 4: incorrect server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] failed to process entries [1/1]: got wrong status: 500
+--- wait: 0.5
+
+
+
+=== TEST 5: add plugin
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "tencent-cloud-cls": {
+ "cls_host": "127.0.0.1:10420",
+ "cls_topic": "143b5d70-139b-4aec-b54e-bb97756916de",
+ "secret_id": "secret_id",
+ "secret_key": "secret_key",
+ "batch_max_size": 1,
+ "max_retry_count": 1,
+ "retry_delay": 2,
+ "buffer_duration": 2,
+ "inactive_timeout": 2
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 6: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] successfully processed the entries
+--- wait: 0.5
+
+
+
+=== TEST 7: verify request
+--- extra_init_by_lua
+ local cls = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+ cls.send_to_cls = function(self, logs)
+ if (#logs ~= 1) then
+ ngx.log(ngx.ERR, "unexpected logs length: ", #logs)
+ return
+ end
+ return true
+ end
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] successfully processed the entries
+--- wait: 0.5
+
+
+
+=== TEST 8: verify cls api request
+--- extra_init_by_lua
+ local cls = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+ cls.send_cls_request = function(self, pb_obj)
+ if (#pb_obj.logGroupList ~= 1) then
+ ngx.log(ngx.ERR, "unexpected logGroupList length: ", #pb_obj.logGroupList)
+ return false
+ end
+ local log_group = pb_obj.logGroupList[1]
+ if #log_group.logs ~= 1 then
+ ngx.log(ngx.ERR, "unexpected logs length: ", #log_group.logs)
+ return false
+ end
+ local log = log_group.logs[1]
+ if #log.contents == 0 then
+ ngx.log(ngx.ERR, "unexpected contents length: ", #log.contents)
+ return false
+ end
+ return true
+ end
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- no_error_log
+[error]
+
+
+
+=== TEST 9: plugin metadata
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/plugin_metadata/tencent-cloud-cls',
+ ngx.HTTP_PUT,
+ [[{
+ "log_format": {
+ "host": "$host",
+ "@timestamp": "$time_iso8601",
+ "client_ip": "$remote_addr"
+ }
+ }]]
+ )
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- response_body
+passed
+--- no_error_log
+[error]
Review Comment:
Ditto.
And also the `--- request` can be saved.
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,315 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local protoc = require("protoc").new()
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+local setmetatable = setmetatable
+local pcall = pcall
+
+-- api doc https://www.tencentcloud.com/document/product/614/16873
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
+local auth_expire_time = 60
+local cls_conn_timeout = 1000
+local cls_read_timeout = 10000
+local cls_send_timeout = 10000
+
+local headers_cache = {}
+local params_cache = {
+ ssl_verify = false,
+ headers = headers_cache,
+}
+
+
+local function get_ip(hostname)
+ local _, resolved = socket.dns.toip(hostname)
+ local ip_list = {}
+ for _, v in ipairs(resolved.ip) do
+ insert_tab(ip_list, v)
+ end
+ return ip_list
+end
+
+local host_ip = tostring(unpack(get_ip(core_gethostname())))
+local log_group_list = {}
+local log_group_list_pb = {
+ logGroupList = log_group_list,
+}
+
+
+local function sha1(msg)
+ return str_util.to_hex(ngx_sha1_bin(msg))
+end
+
+
+local function sha1_hmac(key, msg)
+ return str_util.to_hex(ngx_hmac_sha1(key, msg))
+end
+
+
+-- sign algorithm https://cloud.tencent.com/document/product/614/12445
+local function sign(secret_id, secret_key)
+ local method = "post"
+ local format_params = ""
+ local format_headers = ""
+ local sign_algorithm = "sha1"
+ local http_request_info = fmt("%s\n%s\n%s\n%s\n",
+ method, cls_api_path, format_params, format_headers)
+ local cur_time = ngx_time()
+ local sign_time = fmt("%d;%d", cur_time, cur_time + auth_expire_time)
+ local string_to_sign = fmt("%s\n%s\n%s\n", sign_algorithm, sign_time, sha1(http_request_info))
+
+ local sign_key = sha1_hmac(secret_key, sign_time)
+ local signature = sha1_hmac(sign_key, string_to_sign)
+
+ local arr = {
+ "q-sign-algorithm=sha1",
+ "q-ak=" .. secret_id,
+ "q-sign-time=" .. sign_time,
+ "q-key-time=" .. sign_time,
+ "q-header-list=",
+ "q-url-param-list=",
+ "q-signature=" .. signature,
+ }
+
+ return concat_tab(arr, '&')
+end
+
+
+-- normalized log data for CLS API
+local function normalize_log(log)
+ local normalized_log = {}
+ local log_size = 4 -- empty obj alignment
+ for k, v in pairs(log) do
+ local v_type = type(v)
+ local field = { key = k, value = "" }
+ if v_type == "string" then
+ field["value"] = v
+ elseif v_type == "number" then
+ field["value"] = tostring(v)
+ elseif v_type == "table" then
+ field["value"] = json_encode(v)
+ else
+ field["value"] = tostring(v)
+ core.log.warn("unexpected type " .. v_type .. " for field " .. k)
+ end
+ if #field.value > MAX_SINGLE_VALUE_SIZE then
+ core.log.warn(field.key, " value size over ", MAX_SINGLE_VALUE_SIZE, " , truncated")
+ field.value = field.value:sub(1, MAX_SINGLE_VALUE_SIZE)
+ end
+ insert_tab(normalized_log, field)
+ log_size = log_size + #field.key + #field.value
+ end
+ return normalized_log, log_size
+end
+
+
+local _M = { version = 0.1 }
+local mt = { __index = _M }
+
+local pb_state
+local function init_pb_state()
+ local old_pb_state = pb.state(nil)
+ protoc.reload()
+ local cls_sdk_protoc = protoc.new()
+ -- proto file in https://www.tencentcloud.com/document/product/614/42787
+ local ok, err = pcall(cls_sdk_protoc.load, cls_sdk_protoc, [[
+package cls;
+
+message Log
+{
+ message Content
+ {
+ required string key = 1; // Key of each field group
+ required string value = 2; // Value of each field group
+ }
+ required int64 time = 1; // Unix timestamp
+ repeated Content contents = 2; // Multiple key-value pairs in one log
+}
+
+message LogTag
+{
+ required string key = 1;
+ required string value = 2;
+}
+
+message LogGroup
+{
+ repeated Log logs = 1; // Log array consisting of multiple logs
+ optional string contextFlow = 2; // This parameter does not take effect currently
+ optional string filename = 3; // Log filename
+ optional string source = 4; // Log source, which is generally the machine IP
+ repeated LogTag logTags = 5;
+}
+
+message LogGroupList
+{
+ repeated LogGroup logGroupList = 1; // Log group list
+}
+ ]], "tencent-cloud-cls/cls.proto")
+ if not ok then
+ cls_sdk_protoc:reset()
+ pb.state(old_pb_state)
+ return "failed to load cls.proto: ".. err
+ end
+ pb_state = pb.state(old_pb_state)
+end
+
+
+function _M.new(host, topic, secret_id, secret_key)
+ if not pb_state then
+ local err = init_pb_state()
+ if err then
+ return nil, err
+ end
+ end
+ local self = {
+ host = host,
+ topic = topic,
+ secret_id = secret_id,
+ secret_key = secret_key,
+ }
+ return setmetatable(self, mt)
+end
+
+
+local function do_request_uri(uri, params)
+ local client = http:new()
+ client:set_timeouts(cls_conn_timeout, cls_send_timeout, cls_read_timeout)
+ local res, err = client:request_uri(uri, params)
+ client:close()
+ return res, err
+end
+
+
+function _M.send_cls_request(self, pb_obj)
+ -- recovery of stored pb_store
+ pb.state(pb_state)
+
+ local ok, pb_data = pcall(pb.encode, "cls.LogGroupList", pb_obj)
+ if not ok or not pb_data then
+ core.log.error("failed to encode LogGroupList, err: ", pb_data)
+ return false, pb_data
+ end
+
+ local client = http:new()
+ client:set_timeouts(cls_conn_timeout, cls_send_timeout, cls_read_timeout)
Review Comment:
The client is unused now?
##########
t/plugin/tencent-cloud-cls.t:
##########
@@ -0,0 +1,304 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+use t::APISIX 'no_plan';
+
+log_level('debug');
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "tencent-cloud-cls body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "tencent-cloud-cls headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+ server {
+ listen 10421;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.exit(500)
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests;
+
+__DATA__
+
+=== TEST 1: schema check
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ secret_key = "secret_key",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+done
+
+
+
+=== TEST 2: cls config missing
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+property "secret_key" is required
+done
+
+
+
+=== TEST 3: add plugin for incorrect server
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "tencent-cloud-cls": {
+ "cls_host": "127.0.0.1:10421",
+ "cls_topic": "143b5d70-139b-4aec-b54e-bb97756916de",
+ "secret_id": "secret_id",
+ "secret_key": "secret_key",
+ "batch_max_size": 1,
+ "max_retry_count": 1,
+ "retry_delay": 2,
+ "buffer_duration": 2,
+ "inactive_timeout": 2
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 4: incorrect server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] failed to process entries [1/1]: got wrong status: 500
+--- wait: 0.5
+
+
+
+=== TEST 5: add plugin
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "tencent-cloud-cls": {
+ "cls_host": "127.0.0.1:10420",
+ "cls_topic": "143b5d70-139b-4aec-b54e-bb97756916de",
+ "secret_id": "secret_id",
+ "secret_key": "secret_key",
+ "batch_max_size": 1,
+ "max_retry_count": 1,
+ "retry_delay": 2,
+ "buffer_duration": 2,
+ "inactive_timeout": 2
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 6: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] successfully processed the entries
+--- wait: 0.5
+
+
+
+=== TEST 7: verify request
+--- extra_init_by_lua
+ local cls = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+ cls.send_to_cls = function(self, logs)
+ if (#logs ~= 1) then
+ ngx.log(ngx.ERR, "unexpected logs length: ", #logs)
+ return
+ end
+ return true
+ end
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] successfully processed the entries
+--- wait: 0.5
+
+
+
+=== TEST 8: verify cls api request
+--- extra_init_by_lua
+ local cls = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+ cls.send_cls_request = function(self, pb_obj)
+ if (#pb_obj.logGroupList ~= 1) then
+ ngx.log(ngx.ERR, "unexpected logGroupList length: ", #pb_obj.logGroupList)
+ return false
+ end
+ local log_group = pb_obj.logGroupList[1]
+ if #log_group.logs ~= 1 then
+ ngx.log(ngx.ERR, "unexpected logs length: ", #log_group.logs)
+ return false
+ end
+ local log = log_group.logs[1]
+ if #log.contents == 0 then
+ ngx.log(ngx.ERR, "unexpected contents length: ", #log.contents)
+ return false
+ end
+ return true
+ end
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- no_error_log
+[error]
+
+
+
+=== TEST 9: plugin metadata
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/plugin_metadata/tencent-cloud-cls',
+ ngx.HTTP_PUT,
+ [[{
+ "log_format": {
+ "host": "$host",
+ "@timestamp": "$time_iso8601",
+ "client_ip": "$remote_addr"
+ }
+ }]]
+ )
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 10: log use log_format
Review Comment:
Could we also check the log content like `=== TEST 8: verify cls api request`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r950974029
##########
apisix/plugins/tencent-cloud-cls.lua:
##########
@@ -0,0 +1,143 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local log_util = require("apisix.utils.log-util")
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local cls_sdk = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+local plugin = require("apisix.plugin")
+local math = math
+local ngx = ngx
+local pairs = pairs
+
+
+local plugin_name = "tencent-cloud-cls"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+local schema = {
+ type = "object",
+ properties = {
+ cls_host = { type = "string" },
+ cls_topic = { type = "string" },
+ -- https://console.cloud.tencent.com/capi
+ secret_id = { type = "string" },
+ secret_key = { type = "string" },
+ sample_ratio = {
+ type = "number",
+ minimum = 0.00001,
+ maximum = 1,
+ default = 1
+ },
+ include_req_body = { type = "boolean", default = false },
+ include_resp_body = { type = "boolean", default = false },
+ global_tag = { type = "object" },
+ },
+ required = { "cls_host", "cls_topic", "secret_id", "secret_key" }
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 397,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+
+ local ok, err = core.schema.check(schema, conf)
+ if not ok then
+ return nil, err
+ end
+ return log_util.check_log_schema(conf)
+end
+
+
+function _M.access(conf, ctx)
+ -- sample if set
+ ctx.cls_sample = false
+ if conf.sample_ratio == 1 or math.random() < conf.sample_ratio then
+ core.log.debug("cls sampled")
+ ctx.cls_sample = true
+ return
+ end
+ core.log.debug("cls not sampled")
Review Comment:
ping @ychensha you need to respond to the review comments, not just close them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r948722875
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,308 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local protoc = require("protoc").new()
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+local setmetatable = setmetatable
+local pcall = pcall
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
+local auth_expire_time = 60
+local cls_conn_timeout = 1000
+local cls_read_timeout = 10000
+local cls_send_timeout = 10000
+
+local headers_cache = {}
+local params_cache = {
+ ssl_verify = false,
+ headers = headers_cache,
+}
+
+
+local function get_ip(hostname)
+ local _, resolved = socket.dns.toip(hostname)
+ local ip_list = {}
+ for _, v in ipairs(resolved.ip) do
+ insert_tab(ip_list, v)
+ end
+ return ip_list
+end
+
+local host_ip = tostring(unpack(get_ip(core_gethostname())))
+local log_group_list = {}
+local log_group_list_pb = {
+ logGroupList = log_group_list,
+}
+
+
+local function sha1(msg)
+ return str_util.to_hex(ngx_sha1_bin(msg))
+end
+
+
+local function sha1_hmac(key, msg)
+ return str_util.to_hex(ngx_hmac_sha1(key, msg))
+end
+
+
+-- sign algorithm https://cloud.tencent.com/document/product/614/12445
+local function sign(secret_id, secret_key)
+ local method = "post"
+ local format_params = ""
+ local format_headers = ""
+ local sign_algorithm = "sha1"
+ local http_request_info = fmt("%s\n%s\n%s\n%s\n",
+ method, cls_api_path, format_params, format_headers)
+ local cur_time = ngx_time()
+ local sign_time = fmt("%d;%d", cur_time, cur_time + auth_expire_time)
+ local string_to_sign = fmt("%s\n%s\n%s\n", sign_algorithm, sign_time, sha1(http_request_info))
+
+ local sign_key = sha1_hmac(secret_key, sign_time)
+ local signature = sha1_hmac(sign_key, string_to_sign)
+
+ local arr = {
+ "q-sign-algorithm=sha1",
+ "q-ak=" .. secret_id,
+ "q-sign-time=" .. sign_time,
+ "q-key-time=" .. sign_time,
+ "q-header-list=",
+ "q-url-param-list=",
+ "q-signature=" .. signature,
+ }
+
+ return concat_tab(arr, '&')
+end
+
+
+local function send_cls_request(host, topic, secret_id, secret_key, pb_obj)
+ local ok, pb_data = pcall(pb.encode, "cls.LogGroupList", pb_obj)
+ if not ok or not pb_data then
+ core.log.error("failed to encode LogGroupList, err: ", pb_data)
+ return false, pb_data
+ end
+
+ local client = http:new()
+ client:set_timeouts(cls_conn_timeout, cls_send_timeout, cls_read_timeout)
+
+ clear_tab(headers_cache)
+ headers_cache["Host"] = host
+ headers_cache["Content-Type"] = "application/x-protobuf"
+ headers_cache["Authorization"] = sign(secret_id, secret_key, cls_api_path)
+
+ -- TODO: support lz4/zstd compress
+ params_cache.method = "POST"
+ params_cache.body = pb_data
+
+ local cls_url = "http://" .. host .. cls_api_path .. "?topic_id=" .. topic
+ core.log.debug("CLS request URL: ", cls_url)
+
+ local res, err = client:request_uri(cls_url, params_cache)
+ if not res then
+ return false, err
+ end
+
+ if res.status ~= 200 then
+ err = fmt("got wrong status: %s, headers: %s, body, %s",
+ res.status, json.encode(res.headers), res.body)
+ -- 413, 404, 401, 403 are not retryable
+ if res.status == 413 or res.status == 404 or res.status == 401 or res.status == 403 then
+ core.log.error(err, ", not retryable")
+ return true
+ end
+
+ return false, err
+ end
+
+ core.log.debug("CLS report success")
+ return true
+end
+
+
+-- normalized log data for CLS API
+local function normalize_log(log)
+ local normalized_log = {}
+ local log_size = 4 -- empty obj alignment
+ for k, v in pairs(log) do
+ local v_type = type(v)
+ local field = { key = k, value = "" }
+ if v_type == "string" then
+ field["value"] = v
+ elseif v_type == "number" then
+ field["value"] = tostring(v)
+ elseif v_type == "table" then
+ field["value"] = json_encode(v)
+ else
+ field["value"] = tostring(v)
+ core.log.warn("unexpected type " .. v_type .. " for field " .. k)
+ end
+ if #field.value > MAX_SINGLE_VALUE_SIZE then
+ core.log.warn(field.key, " value size over ", MAX_SINGLE_VALUE_SIZE, " , truncated")
+ field.value = field.value:sub(1, MAX_SINGLE_VALUE_SIZE)
+ end
+ insert_tab(normalized_log, field)
+ log_size = log_size + #field.key + #field.value
+ end
+ return normalized_log, log_size
+end
+
+
+local _M = { version = 0.1 }
+local mt = { __index = _M }
+
+local pb_state
+local function init_pb_state()
+ pb.state(nil)
+ protoc.reload()
+ local cls_sdk_protoc = protoc.new()
+ if not cls_sdk_protoc.loaded["tencent-cloud-cls/cls.proto"] then
+ -- https://www.tencentcloud.com/document/product/614/42787
+ local ok, err = pcall(cls_sdk_protoc.load, cls_sdk_protoc, [[
+package cls;
+
+message Log
+{
+ message Content
+ {
+ required string key = 1; // Key of each field group
+ required string value = 2; // Value of each field group
+ }
+ required int64 time = 1; // Unix timestamp
+ repeated Content contents = 2; // Multiple key-value pairs in one log
+}
+
+message LogTag
+{
+ required string key = 1;
+ required string value = 2;
+}
+
+message LogGroup
+{
+ repeated Log logs = 1; // Log array consisting of multiple logs
+ optional string contextFlow = 2; // This parameter does not take effect currently
+ optional string filename = 3; // Log filename
+ optional string source = 4; // Log source, which is generally the machine IP
+ repeated LogTag logTags = 5;
+}
+
+message LogGroupList
+{
+ repeated LogGroup logGroupList = 1; // Log group list
+}
+ ]], "tencent-cloud-cls/cls.proto")
+ if not ok then
+ cls_sdk_protoc:reset()
+ return "failed to load cls.proto: ".. err
+ end
+ end
+ pb_state = pb.state(nil)
+end
+
+
+function _M.new(host, topic, secret_id, secret_key)
+ if not pb_state then
+ local err = init_pb_state()
+ if err then
+ return nil, err
+ end
+ end
+ local self = {
+ host = host,
+ topic = topic,
+ secret_id = secret_id,
+ secret_key = secret_key,
+ }
+ return setmetatable(self, mt)
+end
+
+
+function _M.send_to_cls(self, logs)
+ clear_tab(log_group_list)
+ local now = ngx_now() * 1000
+
+ -- recovery of stored pb_store
+ pb.state(pb_state)
Review Comment:
Better to store / restore state in the function we used it (send_cls_request in this case)
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,308 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local protoc = require("protoc").new()
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+local setmetatable = setmetatable
+local pcall = pcall
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
+local auth_expire_time = 60
+local cls_conn_timeout = 1000
+local cls_read_timeout = 10000
+local cls_send_timeout = 10000
+
+local headers_cache = {}
+local params_cache = {
+ ssl_verify = false,
+ headers = headers_cache,
+}
+
+
+local function get_ip(hostname)
+ local _, resolved = socket.dns.toip(hostname)
+ local ip_list = {}
+ for _, v in ipairs(resolved.ip) do
+ insert_tab(ip_list, v)
+ end
+ return ip_list
+end
+
+local host_ip = tostring(unpack(get_ip(core_gethostname())))
+local log_group_list = {}
+local log_group_list_pb = {
+ logGroupList = log_group_list,
+}
+
+
+local function sha1(msg)
+ return str_util.to_hex(ngx_sha1_bin(msg))
+end
+
+
+local function sha1_hmac(key, msg)
+ return str_util.to_hex(ngx_hmac_sha1(key, msg))
+end
+
+
+-- sign algorithm https://cloud.tencent.com/document/product/614/12445
+local function sign(secret_id, secret_key)
+ local method = "post"
+ local format_params = ""
+ local format_headers = ""
+ local sign_algorithm = "sha1"
+ local http_request_info = fmt("%s\n%s\n%s\n%s\n",
+ method, cls_api_path, format_params, format_headers)
+ local cur_time = ngx_time()
+ local sign_time = fmt("%d;%d", cur_time, cur_time + auth_expire_time)
+ local string_to_sign = fmt("%s\n%s\n%s\n", sign_algorithm, sign_time, sha1(http_request_info))
+
+ local sign_key = sha1_hmac(secret_key, sign_time)
+ local signature = sha1_hmac(sign_key, string_to_sign)
+
+ local arr = {
+ "q-sign-algorithm=sha1",
+ "q-ak=" .. secret_id,
+ "q-sign-time=" .. sign_time,
+ "q-key-time=" .. sign_time,
+ "q-header-list=",
+ "q-url-param-list=",
+ "q-signature=" .. signature,
+ }
+
+ return concat_tab(arr, '&')
+end
+
+
+local function send_cls_request(host, topic, secret_id, secret_key, pb_obj)
+ local ok, pb_data = pcall(pb.encode, "cls.LogGroupList", pb_obj)
+ if not ok or not pb_data then
+ core.log.error("failed to encode LogGroupList, err: ", pb_data)
+ return false, pb_data
+ end
+
+ local client = http:new()
+ client:set_timeouts(cls_conn_timeout, cls_send_timeout, cls_read_timeout)
+
+ clear_tab(headers_cache)
+ headers_cache["Host"] = host
+ headers_cache["Content-Type"] = "application/x-protobuf"
+ headers_cache["Authorization"] = sign(secret_id, secret_key, cls_api_path)
+
+ -- TODO: support lz4/zstd compress
+ params_cache.method = "POST"
+ params_cache.body = pb_data
+
+ local cls_url = "http://" .. host .. cls_api_path .. "?topic_id=" .. topic
+ core.log.debug("CLS request URL: ", cls_url)
+
+ local res, err = client:request_uri(cls_url, params_cache)
+ if not res then
+ return false, err
+ end
+
+ if res.status ~= 200 then
+ err = fmt("got wrong status: %s, headers: %s, body, %s",
+ res.status, json.encode(res.headers), res.body)
+ -- 413, 404, 401, 403 are not retryable
+ if res.status == 413 or res.status == 404 or res.status == 401 or res.status == 403 then
+ core.log.error(err, ", not retryable")
+ return true
+ end
+
+ return false, err
+ end
+
+ core.log.debug("CLS report success")
+ return true
+end
+
+
+-- normalized log data for CLS API
+local function normalize_log(log)
+ local normalized_log = {}
+ local log_size = 4 -- empty obj alignment
+ for k, v in pairs(log) do
+ local v_type = type(v)
+ local field = { key = k, value = "" }
+ if v_type == "string" then
+ field["value"] = v
+ elseif v_type == "number" then
+ field["value"] = tostring(v)
+ elseif v_type == "table" then
+ field["value"] = json_encode(v)
+ else
+ field["value"] = tostring(v)
+ core.log.warn("unexpected type " .. v_type .. " for field " .. k)
+ end
+ if #field.value > MAX_SINGLE_VALUE_SIZE then
+ core.log.warn(field.key, " value size over ", MAX_SINGLE_VALUE_SIZE, " , truncated")
+ field.value = field.value:sub(1, MAX_SINGLE_VALUE_SIZE)
+ end
+ insert_tab(normalized_log, field)
+ log_size = log_size + #field.key + #field.value
+ end
+ return normalized_log, log_size
+end
+
+
+local _M = { version = 0.1 }
+local mt = { __index = _M }
+
+local pb_state
+local function init_pb_state()
+ pb.state(nil)
Review Comment:
Let's improve this part like https://github.com/apache/apisix/pull/7728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ychensha commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
ychensha commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1210239476
@spacewander @tzssangglass after code related conversations clsoed, I will do the function test again for the related code modification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ychensha commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
ychensha commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r941197391
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,218 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local assert = assert
+assert(pb.loadfile("apisix/plugins/tencent-cloud-cls/cls.pb"))
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
+local auth_expire_time = 60
+local cls_conn_timeout = 1000
+local cls_read_timeout = 10000
+local cls_send_timeout = 10000
+
+local headers_cache = {}
+local params_cache = {
+ ssl_verify = false,
+ headers = headers_cache,
+}
+
+local function get_ip(hostname)
+ local _, resolved = socket.dns.toip(hostname)
+ local ListTab = {}
+ for _, v in ipairs(resolved.ip) do
+ insert_tab(ListTab, v)
+ end
+ return ListTab
+end
+
+local host_ip = tostring(unpack(get_ip(core_gethostname())))
+local log_group_list = {}
+local log_group_list_pb = {
+ logGroupList = log_group_list,
Review Comment:
It is pb field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r950972272
##########
apisix/plugins/tencent-cloud-cls.lua:
##########
@@ -0,0 +1,144 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local log_util = require("apisix.utils.log-util")
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local cls_sdk = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+local plugin = require("apisix.plugin")
+local math = math
+local ngx = ngx
+local pairs = pairs
+
+
+local plugin_name = "tencent-cloud-cls"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+local schema = {
+ type = "object",
+ properties = {
+ cls_host = { type = "string" },
+ cls_topic = { type = "string" },
+ -- https://console.cloud.tencent.com/capi
Review Comment:
this API needs to do auth and have nothing to show directly, remove it or change another link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
soulbird commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1203443066
> >
>
> any doc for testing? I have no exp for env setup & PERL testing framework...
See: https://github.com/apache/apisix/blob/master/docs/en/latest/internal/testing-framework.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r941960914
##########
apisix/plugins/tencent-cloud-cls.lua:
##########
@@ -0,0 +1,143 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local log_util = require("apisix.utils.log-util")
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local cls_sdk = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+local plugin = require("apisix.plugin")
+local math = math
+local ngx = ngx
+local pairs = pairs
+
+
+local plugin_name = "tencent-cloud-cls"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+local schema = {
+ type = "object",
+ properties = {
+ cls_host = { type = "string" },
+ cls_topic = { type = "string" },
+ -- https://console.cloud.tencent.com/capi
Review Comment:
what is this link for?
##########
apisix/plugins/tencent-cloud-cls.lua:
##########
@@ -0,0 +1,143 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local log_util = require("apisix.utils.log-util")
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local cls_sdk = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+local plugin = require("apisix.plugin")
+local math = math
+local ngx = ngx
+local pairs = pairs
+
+
+local plugin_name = "tencent-cloud-cls"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+local schema = {
+ type = "object",
+ properties = {
+ cls_host = { type = "string" },
+ cls_topic = { type = "string" },
+ -- https://console.cloud.tencent.com/capi
+ secret_id = { type = "string" },
+ secret_key = { type = "string" },
+ sample_ratio = {
+ type = "number",
+ minimum = 0.00001,
+ maximum = 1,
+ default = 1
+ },
+ include_req_body = { type = "boolean", default = false },
+ include_resp_body = { type = "boolean", default = false },
+ global_tag = { type = "object" },
+ },
+ required = { "cls_host", "cls_topic", "secret_id", "secret_key" }
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 397,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+
+ local ok, err = core.schema.check(schema, conf)
+ if not ok then
+ return nil, err
+ end
+ return log_util.check_log_schema(conf)
+end
+
+
+function _M.access(conf, ctx)
+ -- sample if set
+ ctx.cls_sample = false
+ if conf.sample_ratio == 1 or math.random() < conf.sample_ratio then
+ core.log.debug("cls sampled")
+ ctx.cls_sample = true
+ return
+ end
+ core.log.debug("cls not sampled")
Review Comment:
No meaningful log?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1210101101
I have a question: Is it that we can't do the test case to verify that this plugin is working correctly?
Because it requires a Tencent account and to make sure that this account is always available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r950975499
##########
apisix/plugins/tencent-cloud-cls.lua:
##########
@@ -0,0 +1,144 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local log_util = require("apisix.utils.log-util")
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local cls_sdk = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+local plugin = require("apisix.plugin")
+local math = math
+local ngx = ngx
+local pairs = pairs
+
+
+local plugin_name = "tencent-cloud-cls"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+local schema = {
+ type = "object",
+ properties = {
+ cls_host = { type = "string" },
+ cls_topic = { type = "string" },
+ -- https://console.cloud.tencent.com/capi
+ secret_id = { type = "string" },
+ secret_key = { type = "string" },
+ sample_ratio = {
+ type = "number",
+ minimum = 0.00001,
+ maximum = 1,
+ default = 1
+ },
+ include_req_body = { type = "boolean", default = false },
+ include_resp_body = { type = "boolean", default = false },
+ global_tag = { type = "object" },
+ },
+ required = { "cls_host", "cls_topic", "secret_id", "secret_key" }
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 397,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+
+ local ok, err = core.schema.check(schema, conf)
+ if not ok then
+ return nil, err
+ end
+ return log_util.check_log_schema(conf)
+end
+
+
+function _M.access(conf, ctx)
Review Comment:
we can do this in the `rewrite` phase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r949857650
##########
t/plugin/tencent-cloud-cls.t:
##########
@@ -0,0 +1,311 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+use t::APISIX 'no_plan';
+
+log_level('debug');
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "tencent-cloud-cls body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "tencent-cloud-cls headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+ server {
+ listen 10421;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "tencent-cloud-cls body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "tencent-cloud-cls headers: " .. k .. ":" .. v)
+ end
Review Comment:
We can simply use `ngx.exit(500)` as we don't care about the input.
##########
t/plugin/tencent-cloud-cls.t:
##########
@@ -0,0 +1,311 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+use t::APISIX 'no_plan';
+
+log_level('debug');
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "tencent-cloud-cls body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "tencent-cloud-cls headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+ server {
+ listen 10421;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "tencent-cloud-cls body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "tencent-cloud-cls headers: " .. k .. ":" .. v)
+ end
+ ngx.exit(500)
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests;
+
+__DATA__
+
+=== TEST 1: schema check
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ secret_key = "secret_key",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+done
+
+
+
+=== TEST 2: cls config missing
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+property "secret_key" is required
+done
+
+
+
+=== TEST 3: add plugin for incorrect server
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "tencent-cloud-cls": {
+ "cls_host": "127.0.0.1:10421",
+ "cls_topic": "143b5d70-139b-4aec-b54e-bb97756916de",
+ "secret_id": "secret_id",
+ "secret_key": "secret_key",
+ "batch_max_size": 1,
+ "max_retry_count": 1,
+ "retry_delay": 2,
+ "buffer_duration": 2,
+ "inactive_timeout": 2
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 4: incorrect server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] failed to process entries [1/1]: got wrong status: 500
+--- wait: 0.5
+
+
+
+=== TEST 5: add plugin
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "tencent-cloud-cls": {
+ "cls_host": "127.0.0.1:10420",
+ "cls_topic": "143b5d70-139b-4aec-b54e-bb97756916de",
+ "secret_id": "secret_id",
+ "secret_key": "secret_key",
+ "batch_max_size": 1,
+ "max_retry_count": 1,
+ "retry_delay": 2,
+ "buffer_duration": 2,
+ "inactive_timeout": 2
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 6: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] successfully processed the entries
+--- wait: 0.5
+
+
+
+=== TEST 7: verify request
+--- extra_init_by_lua
+ local cls = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+ cls.send_to_cls = function(self, logs)
+ if (#logs ~= 1) then
+ ngx.log(ngx.ERR, "unexpected logs length: ", #logs)
+ return
+ end
+ return true
+ end
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] successfully processed the entries
+--- wait: 0.5
+
+
+
+=== TEST 8: verify cls api request
+--- extra_init_by_lua
+ local cls = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+ cls.send_cls_request = function(self, pb_obj)
+ if (#pb_obj.logGroupList ~= 1) then
+ ngx.log(ngx.ERR, "unexpected logGroupList length: ", #pb_obj.logGroupList)
+ return false
+ end
+ local log_group = pb_obj.logGroupList[1]
+ if #log_group.logs ~= 1 then
+ ngx.log(ngx.ERR, "unexpected logs length: ", #log_group.logs)
+ return false
+ end
+ local log = log_group.logs[1]
+ for k, v in ipairs(log.contents) do
+ ngx.log(ngx.ERR, "key:", v.key, ",value:", v.value)
+ end
+ return true
+ end
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] successfully processed the entries
+--- wait: 0.5
+
+
+
+=== TEST 9: plugin metadata
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/plugin_metadata/tencent-cloud-cls',
+ ngx.HTTP_PUT,
+ [[{
+ "log_format": {
+ "host": "$host",
+ "@timestamp": "$time_iso8601",
+ "client_ip": "$remote_addr"
+ }
+ }]]
+ )
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 10: log use log_format
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+"@timestamp":[true,"time_iso8601"]
Review Comment:
Where is this error_log from? Maybe we can use a longer log message so people can know its position.
##########
t/plugin/tencent-cloud-cls.t:
##########
@@ -0,0 +1,311 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+use t::APISIX 'no_plan';
+
+log_level('debug');
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "tencent-cloud-cls body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "tencent-cloud-cls headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+ server {
+ listen 10421;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "tencent-cloud-cls body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "tencent-cloud-cls headers: " .. k .. ":" .. v)
+ end
+ ngx.exit(500)
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests;
+
+__DATA__
+
+=== TEST 1: schema check
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ secret_key = "secret_key",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+done
+
+
+
+=== TEST 2: cls config missing
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+property "secret_key" is required
+done
+
+
+
+=== TEST 3: add plugin for incorrect server
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "tencent-cloud-cls": {
+ "cls_host": "127.0.0.1:10421",
+ "cls_topic": "143b5d70-139b-4aec-b54e-bb97756916de",
+ "secret_id": "secret_id",
+ "secret_key": "secret_key",
+ "batch_max_size": 1,
+ "max_retry_count": 1,
+ "retry_delay": 2,
+ "buffer_duration": 2,
+ "inactive_timeout": 2
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 4: incorrect server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] failed to process entries [1/1]: got wrong status: 500
+--- wait: 0.5
+
+
+
+=== TEST 5: add plugin
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "tencent-cloud-cls": {
+ "cls_host": "127.0.0.1:10420",
+ "cls_topic": "143b5d70-139b-4aec-b54e-bb97756916de",
+ "secret_id": "secret_id",
+ "secret_key": "secret_key",
+ "batch_max_size": 1,
+ "max_retry_count": 1,
+ "retry_delay": 2,
+ "buffer_duration": 2,
+ "inactive_timeout": 2
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 6: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] successfully processed the entries
+--- wait: 0.5
+
+
+
+=== TEST 7: verify request
+--- extra_init_by_lua
+ local cls = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+ cls.send_to_cls = function(self, logs)
+ if (#logs ~= 1) then
+ ngx.log(ngx.ERR, "unexpected logs length: ", #logs)
+ return
+ end
+ return true
+ end
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] successfully processed the entries
+--- wait: 0.5
+
+
+
+=== TEST 8: verify cls api request
+--- extra_init_by_lua
+ local cls = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+ cls.send_cls_request = function(self, pb_obj)
+ if (#pb_obj.logGroupList ~= 1) then
+ ngx.log(ngx.ERR, "unexpected logGroupList length: ", #pb_obj.logGroupList)
+ return false
+ end
+ local log_group = pb_obj.logGroupList[1]
+ if #log_group.logs ~= 1 then
+ ngx.log(ngx.ERR, "unexpected logs length: ", #log_group.logs)
+ return false
+ end
+ local log = log_group.logs[1]
+ for k, v in ipairs(log.contents) do
+ ngx.log(ngx.ERR, "key:", v.key, ",value:", v.value)
Review Comment:
It seems we doesn't check it in the error log?
##########
t/plugin/tencent-cloud-cls.t:
##########
@@ -0,0 +1,311 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+use t::APISIX 'no_plan';
+
+log_level('debug');
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "tencent-cloud-cls body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "tencent-cloud-cls headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+ server {
+ listen 10421;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "tencent-cloud-cls body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "tencent-cloud-cls headers: " .. k .. ":" .. v)
+ end
+ ngx.exit(500)
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests;
+
+__DATA__
+
+=== TEST 1: schema check
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ secret_key = "secret_key",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+done
+
+
+
+=== TEST 2: cls config missing
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+property "secret_key" is required
+done
+
+
+
+=== TEST 3: add plugin for incorrect server
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "tencent-cloud-cls": {
+ "cls_host": "127.0.0.1:10421",
+ "cls_topic": "143b5d70-139b-4aec-b54e-bb97756916de",
+ "secret_id": "secret_id",
+ "secret_key": "secret_key",
+ "batch_max_size": 1,
+ "max_retry_count": 1,
+ "retry_delay": 2,
+ "buffer_duration": 2,
+ "inactive_timeout": 2
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 4: incorrect server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] failed to process entries [1/1]: got wrong status: 500
+--- wait: 0.5
+
+
+
+=== TEST 5: add plugin
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "tencent-cloud-cls": {
+ "cls_host": "127.0.0.1:10420",
+ "cls_topic": "143b5d70-139b-4aec-b54e-bb97756916de",
+ "secret_id": "secret_id",
+ "secret_key": "secret_key",
+ "batch_max_size": 1,
+ "max_retry_count": 1,
+ "retry_delay": 2,
+ "buffer_duration": 2,
+ "inactive_timeout": 2
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 6: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] successfully processed the entries
+--- wait: 0.5
+
+
+
+=== TEST 7: verify request
+--- extra_init_by_lua
+ local cls = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+ cls.send_to_cls = function(self, logs)
+ if (#logs ~= 1) then
+ ngx.log(ngx.ERR, "unexpected logs length: ", #logs)
+ return
+ end
+ return true
+ end
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] successfully processed the entries
+--- wait: 0.5
+
+
+
+=== TEST 8: verify cls api request
+--- extra_init_by_lua
+ local cls = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+ cls.send_cls_request = function(self, pb_obj)
+ if (#pb_obj.logGroupList ~= 1) then
+ ngx.log(ngx.ERR, "unexpected logGroupList length: ", #pb_obj.logGroupList)
+ return false
+ end
+ local log_group = pb_obj.logGroupList[1]
+ if #log_group.logs ~= 1 then
+ ngx.log(ngx.ERR, "unexpected logs length: ", #log_group.logs)
Review Comment:
Let's use no_error_log to ensure the error log doesn't exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r947498338
##########
t/plugin/tencent-cloud-cls.t:
##########
@@ -0,0 +1,155 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+use t::APISIX 'no_plan';
+
+log_level('debug');
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "tencent-cloud-cls body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "tencent-cloud-cls headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests;
+
+__DATA__
+
+=== TEST 1: schema check
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ secret_key = "secret_key",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+done
+
+
+
+=== TEST 2: cls config missing
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+property "secret_key" is required
+done
+
+
+
+=== TEST 3: add plugin
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "tencent-cloud-cls": {
+ "cls_host": "127.0.0.1:10420",
+ "cls_topic": "143b5d70-139b-4aec-b54e-bb97756916de",
+ "secret_id": "secret_id",
+ "secret_key": "secret_key",
+ "batch_max_size": 1,
+ "max_retry_count": 1,
+ "retry_delay": 2,
+ "buffer_duration": 2,
+ "inactive_timeout": 2
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 4: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] successfully processed the entries
Review Comment:
We need to also verify the log data received by the stub server or via injected check.
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,308 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local protoc = require("protoc").new()
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+local setmetatable = setmetatable
+local pcall = pcall
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
+local auth_expire_time = 60
+local cls_conn_timeout = 1000
+local cls_read_timeout = 10000
+local cls_send_timeout = 10000
+
+local headers_cache = {}
+local params_cache = {
+ ssl_verify = false,
+ headers = headers_cache,
+}
+
+
+local function get_ip(hostname)
+ local _, resolved = socket.dns.toip(hostname)
+ local ip_list = {}
+ for _, v in ipairs(resolved.ip) do
+ insert_tab(ip_list, v)
+ end
+ return ip_list
+end
+
+local host_ip = tostring(unpack(get_ip(core_gethostname())))
+local log_group_list = {}
+local log_group_list_pb = {
+ logGroupList = log_group_list,
+}
+
+
+local function sha1(msg)
+ return str_util.to_hex(ngx_sha1_bin(msg))
+end
+
+
+local function sha1_hmac(key, msg)
+ return str_util.to_hex(ngx_hmac_sha1(key, msg))
+end
+
+
+-- sign algorithm https://cloud.tencent.com/document/product/614/12445
+local function sign(secret_id, secret_key)
+ local method = "post"
+ local format_params = ""
+ local format_headers = ""
+ local sign_algorithm = "sha1"
+ local http_request_info = fmt("%s\n%s\n%s\n%s\n",
+ method, cls_api_path, format_params, format_headers)
+ local cur_time = ngx_time()
+ local sign_time = fmt("%d;%d", cur_time, cur_time + auth_expire_time)
+ local string_to_sign = fmt("%s\n%s\n%s\n", sign_algorithm, sign_time, sha1(http_request_info))
+
+ local sign_key = sha1_hmac(secret_key, sign_time)
+ local signature = sha1_hmac(sign_key, string_to_sign)
+
+ local arr = {
+ "q-sign-algorithm=sha1",
+ "q-ak=" .. secret_id,
+ "q-sign-time=" .. sign_time,
+ "q-key-time=" .. sign_time,
+ "q-header-list=",
+ "q-url-param-list=",
+ "q-signature=" .. signature,
+ }
+
+ return concat_tab(arr, '&')
+end
+
+
+local function send_cls_request(host, topic, secret_id, secret_key, pb_obj)
+ local ok, pb_data = pcall(pb.encode, "cls.LogGroupList", pb_obj)
+ if not ok or not pb_data then
+ core.log.error("failed to encode LogGroupList, err: ", pb_data)
+ return false, pb_data
+ end
+
+ local client = http:new()
+ client:set_timeouts(cls_conn_timeout, cls_send_timeout, cls_read_timeout)
+
+ clear_tab(headers_cache)
+ headers_cache["Host"] = host
+ headers_cache["Content-Type"] = "application/x-protobuf"
+ headers_cache["Authorization"] = sign(secret_id, secret_key, cls_api_path)
+
+ -- TODO: support lz4/zstd compress
+ params_cache.method = "POST"
+ params_cache.body = pb_data
+
+ local cls_url = "http://" .. host .. cls_api_path .. "?topic_id=" .. topic
+ core.log.debug("CLS request URL: ", cls_url)
+
+ local res, err = client:request_uri(cls_url, params_cache)
+ if not res then
+ return false, err
+ end
+
+ if res.status ~= 200 then
+ err = fmt("got wrong status: %s, headers: %s, body, %s",
+ res.status, json.encode(res.headers), res.body)
+ -- 413, 404, 401, 403 are not retryable
+ if res.status == 413 or res.status == 404 or res.status == 401 or res.status == 403 then
+ core.log.error(err, ", not retryable")
+ return true
+ end
+
+ return false, err
+ end
+
+ core.log.debug("CLS report success")
+ return true
+end
+
+
+-- normalized log data for CLS API
+local function normalize_log(log)
+ local normalized_log = {}
+ local log_size = 4 -- empty obj alignment
+ for k, v in pairs(log) do
+ local v_type = type(v)
+ local field = { key = k, value = "" }
+ if v_type == "string" then
+ field["value"] = v
+ elseif v_type == "number" then
+ field["value"] = tostring(v)
+ elseif v_type == "table" then
+ field["value"] = json_encode(v)
+ else
+ field["value"] = tostring(v)
+ core.log.warn("unexpected type " .. v_type .. " for field " .. k)
+ end
+ if #field.value > MAX_SINGLE_VALUE_SIZE then
+ core.log.warn(field.key, " value size over ", MAX_SINGLE_VALUE_SIZE, " , truncated")
+ field.value = field.value:sub(1, MAX_SINGLE_VALUE_SIZE)
+ end
+ insert_tab(normalized_log, field)
+ log_size = log_size + #field.key + #field.value
+ end
+ return normalized_log, log_size
+end
+
+
+local _M = { version = 0.1 }
+local mt = { __index = _M }
+
+local pb_state
+local function init_pb_state()
+ pb.state(nil)
+ protoc.reload()
+ local cls_sdk_protoc = protoc.new()
+ if not cls_sdk_protoc.loaded["tencent-cloud-cls/cls.proto"] then
Review Comment:
we can skip this check like https://github.com/apache/apisix/pull/7686
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,308 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local protoc = require("protoc").new()
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+local setmetatable = setmetatable
+local pcall = pcall
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
Review Comment:
Could you give us the source of these two magic number? Thanks!
##########
t/plugin/tencent-cloud-cls.t:
##########
@@ -0,0 +1,155 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+use t::APISIX 'no_plan';
+
+log_level('debug');
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "tencent-cloud-cls body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "tencent-cloud-cls headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests;
+
+__DATA__
+
+=== TEST 1: schema check
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ secret_key = "secret_key",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+done
+
+
+
+=== TEST 2: cls config missing
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+property "secret_key" is required
+done
+
+
+
+=== TEST 3: add plugin
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "tencent-cloud-cls": {
+ "cls_host": "127.0.0.1:10420",
+ "cls_topic": "143b5d70-139b-4aec-b54e-bb97756916de",
+ "secret_id": "secret_id",
+ "secret_key": "secret_key",
+ "batch_max_size": 1,
+ "max_retry_count": 1,
+ "retry_delay": 2,
+ "buffer_duration": 2,
+ "inactive_timeout": 2
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 4: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[tencent-cloud-cls] successfully processed the entries
+--- wait: 0.5
Review Comment:
Let's add a test that we check the log data with the custom log format from plugin metadata.
##########
Makefile:
##########
@@ -345,6 +345,9 @@ install: runtime
$(ENV_INSTALL) -d $(ENV_INST_LUADIR)/apisix/plugins/zipkin
$(ENV_INSTALL) apisix/plugins/zipkin/*.lua $(ENV_INST_LUADIR)/apisix/plugins/zipkin/
+ $(ENV_INSTALL) -d $(ENV_INST_LUADIR)/apisix/plugins/tencent-cloud-cls
Review Comment:
Let's put it in alphabetical order https://github.com/apache/apisix/pull/7698
##########
t/plugin/tencent-cloud-cls.t:
##########
@@ -0,0 +1,155 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+use t::APISIX 'no_plan';
+
+log_level('debug');
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /structuredlog {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "tencent-cloud-cls body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "tencent-cloud-cls headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests;
+
+__DATA__
+
+=== TEST 1: schema check
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ secret_key = "secret_key",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+done
+
+
+
+=== TEST 2: cls config missing
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.tencent-cloud-cls")
+ local ok, err = plugin.check_schema({
+ cls_host = "ap-guangzhou.cls.tencentyun.com",
+ cls_topic = "143b5d70-139b-4aec-b54e-bb97756916de",
+ secret_id = "secret_id",
+ })
+ if not ok then
+ ngx.say(err)
+ end
+
+ ngx.say("done")
+ }
+ }
+--- response_body
+property "secret_key" is required
+done
+
+
+
+=== TEST 3: add plugin
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "tencent-cloud-cls": {
+ "cls_host": "127.0.0.1:10420",
Review Comment:
Let's add a test that the log data is sent to incorrect address
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1225121813
As this PR is big enough, let's open another PR for the doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r942051610
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,218 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local assert = assert
+assert(pb.loadfile("apisix/plugins/tencent-cloud-cls/cls.pb"))
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
+local auth_expire_time = 60
+local cls_conn_timeout = 1000
+local cls_read_timeout = 10000
+local cls_send_timeout = 10000
+
+local headers_cache = {}
+local params_cache = {
+ ssl_verify = false,
+ headers = headers_cache,
+}
+
+local function get_ip(hostname)
+ local _, resolved = socket.dns.toip(hostname)
+ local ListTab = {}
+ for _, v in ipairs(resolved.ip) do
+ insert_tab(ListTab, v)
+ end
+ return ListTab
+end
+
+local host_ip = tostring(unpack(get_ip(core_gethostname())))
+local log_group_list = {}
+local log_group_list_pb = {
+ logGroupList = log_group_list,
+}
+
+local function sha1(msg)
+ return str_util.to_hex(ngx_sha1_bin(msg))
+end
+
+local function sha1_hmac(key, msg)
+ return str_util.to_hex(ngx_hmac_sha1(key, msg))
+end
+
+-- sign algorithm https://cloud.tencent.com/document/product/614/12445
+local function sign(secret_id, secret_key)
+ local method = "post"
+ local format_params = ""
+ local format_headers = ""
+ local sign_algorithm = "sha1"
+ local http_request_info = fmt("%s\n%s\n%s\n%s\n",
+ method, cls_api_path, format_params, format_headers)
+ local cur_time = ngx_time()
+ local sign_time = fmt("%d;%d", cur_time, cur_time + auth_expire_time)
+ local string_to_sign = fmt("%s\n%s\n%s\n", sign_algorithm, sign_time, sha1(http_request_info))
+
+ local sign_key = sha1_hmac(secret_key, sign_time)
+ local signature = sha1_hmac(sign_key, string_to_sign)
+
+ local arr = {
+ "q-sign-algorithm=sha1",
+ "q-ak=" .. secret_id,
+ "q-sign-time=" .. sign_time,
+ "q-key-time=" .. sign_time,
+ "q-header-list=",
+ "q-url-param-list=",
+ "q-signature=" .. signature,
+ }
+
+ return concat_tab(arr, '&')
+end
+
+local function send_cls_request(host, topic, secret_id, secret_key, pb_data)
+ local http_new = http:new()
+ http_new:set_timeouts(cls_conn_timeout, cls_send_timeout, cls_read_timeout)
+
+ clear_tab(headers_cache)
+ headers_cache["Host"] = host
+ headers_cache["Content-Type"] = "application/x-protobuf"
+ headers_cache["Authorization"] = sign(secret_id, secret_key, cls_api_path)
+
+ -- TODO: support lz4/zstd compress
+ params_cache.method = "POST"
+ params_cache.body = pb_data
+
+ local cls_url = "http://" .. host .. cls_api_path .. "?topic_id=" .. topic
+ core.log.debug("CLS request URL: ", cls_url)
+
+ local res, err = http_new:request_uri(cls_url, params_cache)
+ if not res then
+ return false, err
+ end
+
+ if res.status ~= 200 then
+ err = fmt("got wrong status: %s, headers: %s, body, %s",
+ res.status, json.encode(res.headers), res.body)
+ -- 413, 404, 401, 403 are not retryable
+ if res.status == 413 or res.status == 404 or res.status == 401 or res.status == 403 then
+ core.log.error(err, ", not retryable")
+ return true
+ end
+
+ return false, err
+ end
+
+ core.log.debug("CLS report success")
+ return true
+end
+
+-- normalized log data for CLS API
+local function normalize_log(log)
+ local normalized_log = {}
+ local log_size = 4 -- empty obj alignment
+ for k, v in pairs(log) do
+ local v_type = type(v)
+ local field = { key = k, value = "" }
+ if v_type == "string" then
+ field["value"] = v
+ elseif v_type == "number" then
+ field["value"] = tostring(v)
+ elseif v_type == "table" then
+ field["value"] = json_encode(v)
+ else
+ field["value"] = tostring(v)
+ core.log.warn("unexpected type " .. v_type .. " for field " .. k)
+ end
+ if #field.value > MAX_SINGLE_VALUE_SIZE then
+ core.log.warn(field.key, " value size over ", MAX_SINGLE_VALUE_SIZE, " , truncated")
+ field.value = field.value:sub(1, MAX_SINGLE_VALUE_SIZE)
+ end
+ insert_tab(normalized_log, field)
+ log_size = log_size + #field.key + #field.value
+ end
+ return normalized_log, log_size
+end
+
+local function send_to_cls(secret_id, secret_key, host, topic_id, logs)
+ clear_tab(log_group_list)
+ local now = ngx_now() * 1000
+
+ local total_size = 0
+ local format_logs = new_tab(#logs, 0)
+ -- sums of all value in a LogGroup should be no more than 5MB
+ for i = 1, #logs, 1 do
+ local contents, log_size = normalize_log(logs[i])
+ if log_size > MAX_LOG_GROUP_VALUE_SIZE then
+ core.log.error("size of log is over 5MB, dropped")
+ goto continue
+ end
+ total_size = total_size + log_size
+ if total_size > MAX_LOG_GROUP_VALUE_SIZE then
+ insert_tab(log_group_list, {
+ logs = format_logs,
+ source = host_ip,
+ })
+ format_logs = new_tab(#logs - i, 0)
+ total_size = 0
+ local data = assert(pb.encode("cls.LogGroupList", log_group_list_pb))
+ send_cls_request(host, topic_id, secret_id, secret_key, data)
Review Comment:
We can simply log down the err?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ychensha commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
ychensha commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1203421112
>
any doc for testing? I have no exp for env setup & PERL testing framework...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r951000785
##########
apisix/plugins/tencent-cloud-cls.lua:
##########
@@ -0,0 +1,144 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local log_util = require("apisix.utils.log-util")
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local cls_sdk = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+local plugin = require("apisix.plugin")
+local math = math
+local ngx = ngx
+local pairs = pairs
+
+
+local plugin_name = "tencent-cloud-cls"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+local schema = {
+ type = "object",
+ properties = {
+ cls_host = { type = "string" },
+ cls_topic = { type = "string" },
+ -- https://console.cloud.tencent.com/capi
+ secret_id = { type = "string" },
+ secret_key = { type = "string" },
+ sample_ratio = {
+ type = "number",
+ minimum = 0.00001,
+ maximum = 1,
+ default = 1
+ },
+ include_req_body = { type = "boolean", default = false },
+ include_resp_body = { type = "boolean", default = false },
+ global_tag = { type = "object" },
+ },
+ required = { "cls_host", "cls_topic", "secret_id", "secret_key" }
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 397,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+
+ local ok, err = core.schema.check(schema, conf)
+ if not ok then
+ return nil, err
+ end
+ return log_util.check_log_schema(conf)
+end
+
+
+function _M.access(conf, ctx)
Review Comment:
my mistake, keep `access` phase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ychensha commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
ychensha commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1203847988
> > >
> >
> >
> > any doc for testing? I have no exp for env setup & PERL testing framework...
>
> See: https://github.com/apache/apisix/blob/master/docs/en/latest/internal/testing-framework.md
tried.. I work on Mac, and now trapped in etcd configuration for more tesing case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1211592119
> >
>
> If hook is ok for now, I will add test code like otel plugin.
Don't be afraid. Feel free to contact me once you have trouble with the hook.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1223617707
Let's merge master to make CI pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r937426192
##########
apisix/plugins/tencent-cloud-cls.lua:
##########
@@ -0,0 +1,94 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local log_util = require("apisix.utils.log-util")
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local cls_sdk = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+local random = math.random
+math.randomseed(ngx.time() + ngx.worker.pid())
Review Comment:
We don't need to randomseed in the plugin level. We already do it when APISIX starts.
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,218 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local assert = assert
+assert(pb.loadfile("apisix/plugins/tencent-cloud-cls/cls.pb"))
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
+local auth_expire_time = 60
+local cls_conn_timeout = 1000
+local cls_read_timeout = 10000
+local cls_send_timeout = 10000
+
+local headers_cache = {}
+local params_cache = {
+ ssl_verify = false,
+ headers = headers_cache,
+}
+
+local function get_ip(hostname)
+ local _, resolved = socket.dns.toip(hostname)
+ local ListTab = {}
Review Comment:
Let's use underscore style
##########
apisix/plugins/tencent-cloud-cls.lua:
##########
@@ -0,0 +1,94 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local log_util = require("apisix.utils.log-util")
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local cls_sdk = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+local random = math.random
+math.randomseed(ngx.time() + ngx.worker.pid())
+local ngx = ngx
+local pairs = pairs
+
+local plugin_name = "tencent-cloud-cls"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+local schema = {
+ type = "object",
+ properties = {
+ cls_host = { type = "string" },
+ cls_topic = { type = "string" },
+ -- https://console.cloud.tencent.com/capi
+ secret_id = { type = "string" },
+ secret_key = { type = "string" },
+ sample_rate = { type = "integer", minimum = 1, maximum = 100, default = 100 },
+ include_req_body = { type = "boolean", default = false },
+ include_resp_body = { type = "boolean", default = false },
+ global_tag = { type = "object" },
+ },
+ required = { "cls_host", "cls_topic", "secret_id", "secret_key" }
+}
+
+local _M = {
+ version = 0.1,
+ priority = 397,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+}
+
+function _M.check_schema(conf)
+ return core.schema.check(schema, conf)
+end
+
+function _M.body_filter(conf, ctx)
+ -- sample if set
+ if conf.sample_rate < 100 and random(1, 100) > conf.sample_rate then
+ core.log.debug("not sampled")
+ return
+ end
+ log_util.collect_body(conf, ctx)
+ ctx.cls_sample = true
+end
+
+function _M.log(conf, ctx)
+ -- sample if set
+ if ctx.cls_sample == nil then
+ core.log.debug("not sampled")
+ return
+ end
+ local entry = log_util.get_full_log(ngx, conf)
+ if not entry.route_id then
+ entry.route_id = "no-matched"
Review Comment:
Some plugins have this field because of their historical legacy. Is this field required in tencent cloud log service?
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,218 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local assert = assert
+assert(pb.loadfile("apisix/plugins/tencent-cloud-cls/cls.pb"))
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
+local auth_expire_time = 60
+local cls_conn_timeout = 1000
+local cls_read_timeout = 10000
+local cls_send_timeout = 10000
+
+local headers_cache = {}
+local params_cache = {
+ ssl_verify = false,
+ headers = headers_cache,
+}
+
+local function get_ip(hostname)
+ local _, resolved = socket.dns.toip(hostname)
+ local ListTab = {}
+ for _, v in ipairs(resolved.ip) do
+ insert_tab(ListTab, v)
+ end
+ return ListTab
+end
+
+local host_ip = tostring(unpack(get_ip(core_gethostname())))
+local log_group_list = {}
+local log_group_list_pb = {
+ logGroupList = log_group_list,
Review Comment:
Let's use underscore style
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,218 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local assert = assert
+assert(pb.loadfile("apisix/plugins/tencent-cloud-cls/cls.pb"))
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
+local auth_expire_time = 60
+local cls_conn_timeout = 1000
+local cls_read_timeout = 10000
+local cls_send_timeout = 10000
+
+local headers_cache = {}
+local params_cache = {
+ ssl_verify = false,
+ headers = headers_cache,
+}
+
+local function get_ip(hostname)
+ local _, resolved = socket.dns.toip(hostname)
+ local ListTab = {}
+ for _, v in ipairs(resolved.ip) do
+ insert_tab(ListTab, v)
+ end
+ return ListTab
+end
+
+local host_ip = tostring(unpack(get_ip(core_gethostname())))
+local log_group_list = {}
+local log_group_list_pb = {
+ logGroupList = log_group_list,
+}
+
+local function sha1(msg)
+ return str_util.to_hex(ngx_sha1_bin(msg))
+end
+
+local function sha1_hmac(key, msg)
+ return str_util.to_hex(ngx_hmac_sha1(key, msg))
+end
+
+-- sign algorithm https://cloud.tencent.com/document/product/614/12445
+local function sign(secret_id, secret_key)
+ local method = "post"
+ local format_params = ""
+ local format_headers = ""
+ local sign_algorithm = "sha1"
+ local http_request_info = fmt("%s\n%s\n%s\n%s\n",
+ method, cls_api_path, format_params, format_headers)
+ local cur_time = ngx_time()
+ local sign_time = fmt("%d;%d", cur_time, cur_time + auth_expire_time)
+ local string_to_sign = fmt("%s\n%s\n%s\n", sign_algorithm, sign_time, sha1(http_request_info))
+
+ local sign_key = sha1_hmac(secret_key, sign_time)
+ local signature = sha1_hmac(sign_key, string_to_sign)
+
+ local arr = {
+ "q-sign-algorithm=sha1",
+ "q-ak=" .. secret_id,
+ "q-sign-time=" .. sign_time,
+ "q-key-time=" .. sign_time,
+ "q-header-list=",
+ "q-url-param-list=",
+ "q-signature=" .. signature,
+ }
+
+ return concat_tab(arr, '&')
+end
+
+local function send_cls_request(host, topic, secret_id, secret_key, pb_data)
+ local http_new = http:new()
Review Comment:
Call it `httpc` or `client` would be better?
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,218 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local assert = assert
+assert(pb.loadfile("apisix/plugins/tencent-cloud-cls/cls.pb"))
Review Comment:
https://github.com/apache/apisix/blob/18a6cacf1681c80d7821b0791f6b704fe6a9a4d8/apisix/core/pubsub.lua#L50
loadfile will throw an error when failed
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,218 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local assert = assert
+assert(pb.loadfile("apisix/plugins/tencent-cloud-cls/cls.pb"))
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
+local auth_expire_time = 60
+local cls_conn_timeout = 1000
+local cls_read_timeout = 10000
+local cls_send_timeout = 10000
+
+local headers_cache = {}
+local params_cache = {
+ ssl_verify = false,
+ headers = headers_cache,
+}
+
+local function get_ip(hostname)
+ local _, resolved = socket.dns.toip(hostname)
+ local ListTab = {}
+ for _, v in ipairs(resolved.ip) do
+ insert_tab(ListTab, v)
+ end
+ return ListTab
+end
+
+local host_ip = tostring(unpack(get_ip(core_gethostname())))
+local log_group_list = {}
+local log_group_list_pb = {
+ logGroupList = log_group_list,
+}
+
+local function sha1(msg)
+ return str_util.to_hex(ngx_sha1_bin(msg))
+end
+
+local function sha1_hmac(key, msg)
+ return str_util.to_hex(ngx_hmac_sha1(key, msg))
+end
+
+-- sign algorithm https://cloud.tencent.com/document/product/614/12445
+local function sign(secret_id, secret_key)
+ local method = "post"
+ local format_params = ""
+ local format_headers = ""
+ local sign_algorithm = "sha1"
+ local http_request_info = fmt("%s\n%s\n%s\n%s\n",
+ method, cls_api_path, format_params, format_headers)
+ local cur_time = ngx_time()
+ local sign_time = fmt("%d;%d", cur_time, cur_time + auth_expire_time)
+ local string_to_sign = fmt("%s\n%s\n%s\n", sign_algorithm, sign_time, sha1(http_request_info))
+
+ local sign_key = sha1_hmac(secret_key, sign_time)
+ local signature = sha1_hmac(sign_key, string_to_sign)
+
+ local arr = {
+ "q-sign-algorithm=sha1",
+ "q-ak=" .. secret_id,
+ "q-sign-time=" .. sign_time,
+ "q-key-time=" .. sign_time,
+ "q-header-list=",
+ "q-url-param-list=",
+ "q-signature=" .. signature,
+ }
+
+ return concat_tab(arr, '&')
+end
+
+local function send_cls_request(host, topic, secret_id, secret_key, pb_data)
+ local http_new = http:new()
+ http_new:set_timeouts(cls_conn_timeout, cls_send_timeout, cls_read_timeout)
+
+ clear_tab(headers_cache)
+ headers_cache["Host"] = host
+ headers_cache["Content-Type"] = "application/x-protobuf"
+ headers_cache["Authorization"] = sign(secret_id, secret_key, cls_api_path)
+
+ -- TODO: support lz4/zstd compress
+ params_cache.method = "POST"
+ params_cache.body = pb_data
+
+ local cls_url = "http://" .. host .. cls_api_path .. "?topic_id=" .. topic
+ core.log.debug("CLS request URL: ", cls_url)
+
+ local res, err = http_new:request_uri(cls_url, params_cache)
+ if not res then
+ return false, err
+ end
+
+ if res.status ~= 200 then
+ err = fmt("got wrong status: %s, headers: %s, body, %s",
+ res.status, json.encode(res.headers), res.body)
+ -- 413, 404, 401, 403 are not retryable
+ if res.status == 413 or res.status == 404 or res.status == 401 or res.status == 403 then
+ core.log.error(err, ", not retryable")
+ return true
+ end
+
+ return false, err
+ end
+
+ core.log.debug("CLS report success")
+ return true
+end
+
+-- normalized log data for CLS API
+local function normalize_log(log)
+ local normalized_log = {}
+ local log_size = 4 -- empty obj alignment
+ for k, v in pairs(log) do
+ local v_type = type(v)
+ local field = { key = k, value = "" }
+ if v_type == "string" then
+ field["value"] = v
+ elseif v_type == "number" then
+ field["value"] = tostring(v)
+ elseif v_type == "table" then
+ field["value"] = json_encode(v)
+ else
+ field["value"] = tostring(v)
+ core.log.warn("unexpected type " .. v_type .. " for field " .. k)
+ end
+ if #field.value > MAX_SINGLE_VALUE_SIZE then
+ core.log.warn(field.key, " value size over ", MAX_SINGLE_VALUE_SIZE, " , truncated")
+ field.value = field.value:sub(1, MAX_SINGLE_VALUE_SIZE)
+ end
+ insert_tab(normalized_log, field)
+ log_size = log_size + #field.key + #field.value
+ end
+ return normalized_log, log_size
+end
+
+local function send_to_cls(secret_id, secret_key, host, topic_id, logs)
+ clear_tab(log_group_list)
+ local now = ngx_now() * 1000
+
+ local total_size = 0
+ local format_logs = new_tab(#logs, 0)
+ -- sums of all value in a LogGroup should be no more than 5MB
+ for i = 1, #logs, 1 do
+ local contents, log_size = normalize_log(logs[i])
+ if log_size > MAX_LOG_GROUP_VALUE_SIZE then
+ core.log.error("size of log is over 5MB, dropped")
+ goto continue
+ end
+ total_size = total_size + log_size
+ if total_size > MAX_LOG_GROUP_VALUE_SIZE then
+ insert_tab(log_group_list, {
+ logs = format_logs,
+ source = host_ip,
+ })
+ format_logs = new_tab(#logs - i, 0)
+ total_size = 0
+ local data = assert(pb.encode("cls.LogGroupList", log_group_list_pb))
+ send_cls_request(host, topic_id, secret_id, secret_key, data)
+ clear_tab(log_group_list)
+ end
+ insert_tab(format_logs, {
+ time = now,
+ contents = contents,
+ })
+ :: continue ::
+ end
+
+ insert_tab(log_group_list, {
+ logs = format_logs,
+ source = host_ip,
+ })
+ local data = assert(pb.encode("cls.LogGroupList", log_group_list_pb))
Review Comment:
https://github.com/apache/apisix/blob/18a6cacf1681c80d7821b0791f6b704fe6a9a4d8/apisix/core/pubsub.lua#L76
pb.encode will throw an error when failed
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,218 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local assert = assert
+assert(pb.loadfile("apisix/plugins/tencent-cloud-cls/cls.pb"))
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
+local auth_expire_time = 60
+local cls_conn_timeout = 1000
+local cls_read_timeout = 10000
+local cls_send_timeout = 10000
+
+local headers_cache = {}
+local params_cache = {
+ ssl_verify = false,
+ headers = headers_cache,
+}
+
+local function get_ip(hostname)
+ local _, resolved = socket.dns.toip(hostname)
+ local ListTab = {}
+ for _, v in ipairs(resolved.ip) do
+ insert_tab(ListTab, v)
+ end
+ return ListTab
+end
+
+local host_ip = tostring(unpack(get_ip(core_gethostname())))
+local log_group_list = {}
+local log_group_list_pb = {
+ logGroupList = log_group_list,
+}
+
+local function sha1(msg)
+ return str_util.to_hex(ngx_sha1_bin(msg))
+end
+
+local function sha1_hmac(key, msg)
+ return str_util.to_hex(ngx_hmac_sha1(key, msg))
+end
+
+-- sign algorithm https://cloud.tencent.com/document/product/614/12445
+local function sign(secret_id, secret_key)
+ local method = "post"
+ local format_params = ""
+ local format_headers = ""
+ local sign_algorithm = "sha1"
+ local http_request_info = fmt("%s\n%s\n%s\n%s\n",
+ method, cls_api_path, format_params, format_headers)
+ local cur_time = ngx_time()
+ local sign_time = fmt("%d;%d", cur_time, cur_time + auth_expire_time)
+ local string_to_sign = fmt("%s\n%s\n%s\n", sign_algorithm, sign_time, sha1(http_request_info))
+
+ local sign_key = sha1_hmac(secret_key, sign_time)
+ local signature = sha1_hmac(sign_key, string_to_sign)
+
+ local arr = {
+ "q-sign-algorithm=sha1",
+ "q-ak=" .. secret_id,
+ "q-sign-time=" .. sign_time,
+ "q-key-time=" .. sign_time,
+ "q-header-list=",
+ "q-url-param-list=",
+ "q-signature=" .. signature,
+ }
+
+ return concat_tab(arr, '&')
+end
+
+local function send_cls_request(host, topic, secret_id, secret_key, pb_data)
+ local http_new = http:new()
+ http_new:set_timeouts(cls_conn_timeout, cls_send_timeout, cls_read_timeout)
+
+ clear_tab(headers_cache)
+ headers_cache["Host"] = host
+ headers_cache["Content-Type"] = "application/x-protobuf"
+ headers_cache["Authorization"] = sign(secret_id, secret_key, cls_api_path)
+
+ -- TODO: support lz4/zstd compress
+ params_cache.method = "POST"
+ params_cache.body = pb_data
+
+ local cls_url = "http://" .. host .. cls_api_path .. "?topic_id=" .. topic
+ core.log.debug("CLS request URL: ", cls_url)
+
+ local res, err = http_new:request_uri(cls_url, params_cache)
+ if not res then
+ return false, err
+ end
+
+ if res.status ~= 200 then
+ err = fmt("got wrong status: %s, headers: %s, body, %s",
+ res.status, json.encode(res.headers), res.body)
+ -- 413, 404, 401, 403 are not retryable
+ if res.status == 413 or res.status == 404 or res.status == 401 or res.status == 403 then
+ core.log.error(err, ", not retryable")
+ return true
+ end
+
+ return false, err
+ end
+
+ core.log.debug("CLS report success")
+ return true
+end
+
+-- normalized log data for CLS API
+local function normalize_log(log)
+ local normalized_log = {}
+ local log_size = 4 -- empty obj alignment
+ for k, v in pairs(log) do
+ local v_type = type(v)
+ local field = { key = k, value = "" }
+ if v_type == "string" then
+ field["value"] = v
+ elseif v_type == "number" then
+ field["value"] = tostring(v)
+ elseif v_type == "table" then
+ field["value"] = json_encode(v)
+ else
+ field["value"] = tostring(v)
+ core.log.warn("unexpected type " .. v_type .. " for field " .. k)
+ end
+ if #field.value > MAX_SINGLE_VALUE_SIZE then
+ core.log.warn(field.key, " value size over ", MAX_SINGLE_VALUE_SIZE, " , truncated")
+ field.value = field.value:sub(1, MAX_SINGLE_VALUE_SIZE)
+ end
+ insert_tab(normalized_log, field)
+ log_size = log_size + #field.key + #field.value
+ end
+ return normalized_log, log_size
+end
+
+local function send_to_cls(secret_id, secret_key, host, topic_id, logs)
+ clear_tab(log_group_list)
+ local now = ngx_now() * 1000
+
+ local total_size = 0
+ local format_logs = new_tab(#logs, 0)
+ -- sums of all value in a LogGroup should be no more than 5MB
+ for i = 1, #logs, 1 do
+ local contents, log_size = normalize_log(logs[i])
+ if log_size > MAX_LOG_GROUP_VALUE_SIZE then
+ core.log.error("size of log is over 5MB, dropped")
+ goto continue
+ end
+ total_size = total_size + log_size
+ if total_size > MAX_LOG_GROUP_VALUE_SIZE then
+ insert_tab(log_group_list, {
+ logs = format_logs,
+ source = host_ip,
+ })
+ format_logs = new_tab(#logs - i, 0)
+ total_size = 0
+ local data = assert(pb.encode("cls.LogGroupList", log_group_list_pb))
+ send_cls_request(host, topic_id, secret_id, secret_key, data)
Review Comment:
Not error handling?
##########
apisix/plugins/tencent-cloud-cls.lua:
##########
@@ -0,0 +1,94 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local log_util = require("apisix.utils.log-util")
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local cls_sdk = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+local random = math.random
+math.randomseed(ngx.time() + ngx.worker.pid())
+local ngx = ngx
+local pairs = pairs
+
+local plugin_name = "tencent-cloud-cls"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+local schema = {
+ type = "object",
+ properties = {
+ cls_host = { type = "string" },
+ cls_topic = { type = "string" },
+ -- https://console.cloud.tencent.com/capi
+ secret_id = { type = "string" },
+ secret_key = { type = "string" },
+ sample_rate = { type = "integer", minimum = 1, maximum = 100, default = 100 },
+ include_req_body = { type = "boolean", default = false },
+ include_resp_body = { type = "boolean", default = false },
+ global_tag = { type = "object" },
+ },
+ required = { "cls_host", "cls_topic", "secret_id", "secret_key" }
+}
+
+local _M = {
+ version = 0.1,
+ priority = 397,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+}
+
+function _M.check_schema(conf)
+ return core.schema.check(schema, conf)
+end
+
+function _M.body_filter(conf, ctx)
+ -- sample if set
Review Comment:
Not every request has a body. We should sample early like https://github.com/apache/apisix/blob/18a6cacf1681c80d7821b0791f6b704fe6a9a4d8/apisix/plugins/skywalking.lua#L81.
And for a consistent style, would you like to use `sample_ratio` (max = 1) like skywalking/zipkin/proxy-mirror plugins?
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,218 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local assert = assert
+assert(pb.loadfile("apisix/plugins/tencent-cloud-cls/cls.pb"))
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
+local auth_expire_time = 60
+local cls_conn_timeout = 1000
+local cls_read_timeout = 10000
+local cls_send_timeout = 10000
+
+local headers_cache = {}
+local params_cache = {
+ ssl_verify = false,
+ headers = headers_cache,
+}
+
+local function get_ip(hostname)
+ local _, resolved = socket.dns.toip(hostname)
+ local ListTab = {}
+ for _, v in ipairs(resolved.ip) do
+ insert_tab(ListTab, v)
+ end
+ return ListTab
+end
+
+local host_ip = tostring(unpack(get_ip(core_gethostname())))
+local log_group_list = {}
+local log_group_list_pb = {
+ logGroupList = log_group_list,
+}
+
+local function sha1(msg)
+ return str_util.to_hex(ngx_sha1_bin(msg))
+end
+
+local function sha1_hmac(key, msg)
+ return str_util.to_hex(ngx_hmac_sha1(key, msg))
+end
+
+-- sign algorithm https://cloud.tencent.com/document/product/614/12445
+local function sign(secret_id, secret_key)
+ local method = "post"
+ local format_params = ""
+ local format_headers = ""
+ local sign_algorithm = "sha1"
+ local http_request_info = fmt("%s\n%s\n%s\n%s\n",
+ method, cls_api_path, format_params, format_headers)
+ local cur_time = ngx_time()
+ local sign_time = fmt("%d;%d", cur_time, cur_time + auth_expire_time)
+ local string_to_sign = fmt("%s\n%s\n%s\n", sign_algorithm, sign_time, sha1(http_request_info))
+
+ local sign_key = sha1_hmac(secret_key, sign_time)
+ local signature = sha1_hmac(sign_key, string_to_sign)
+
+ local arr = {
+ "q-sign-algorithm=sha1",
+ "q-ak=" .. secret_id,
+ "q-sign-time=" .. sign_time,
+ "q-key-time=" .. sign_time,
+ "q-header-list=",
+ "q-url-param-list=",
+ "q-signature=" .. signature,
+ }
+
+ return concat_tab(arr, '&')
+end
+
+local function send_cls_request(host, topic, secret_id, secret_key, pb_data)
+ local http_new = http:new()
+ http_new:set_timeouts(cls_conn_timeout, cls_send_timeout, cls_read_timeout)
+
+ clear_tab(headers_cache)
+ headers_cache["Host"] = host
+ headers_cache["Content-Type"] = "application/x-protobuf"
+ headers_cache["Authorization"] = sign(secret_id, secret_key, cls_api_path)
+
+ -- TODO: support lz4/zstd compress
+ params_cache.method = "POST"
+ params_cache.body = pb_data
+
+ local cls_url = "http://" .. host .. cls_api_path .. "?topic_id=" .. topic
+ core.log.debug("CLS request URL: ", cls_url)
+
+ local res, err = http_new:request_uri(cls_url, params_cache)
+ if not res then
+ return false, err
Review Comment:
Would be better to close the http client. What about introducing a new function?
```
client = xxx
local ok, err = do_something()
client:close()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1219499035
@ychensha pls don't use force-push to update code, or reviews' comments may be ignore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ychensha commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
ychensha commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r941130262
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,218 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local assert = assert
+assert(pb.loadfile("apisix/plugins/tencent-cloud-cls/cls.pb"))
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
+local auth_expire_time = 60
+local cls_conn_timeout = 1000
+local cls_read_timeout = 10000
+local cls_send_timeout = 10000
+
+local headers_cache = {}
+local params_cache = {
+ ssl_verify = false,
+ headers = headers_cache,
+}
+
+local function get_ip(hostname)
+ local _, resolved = socket.dns.toip(hostname)
+ local ListTab = {}
+ for _, v in ipairs(resolved.ip) do
+ insert_tab(ListTab, v)
+ end
+ return ListTab
+end
+
+local host_ip = tostring(unpack(get_ip(core_gethostname())))
+local log_group_list = {}
+local log_group_list_pb = {
+ logGroupList = log_group_list,
+}
+
+local function sha1(msg)
+ return str_util.to_hex(ngx_sha1_bin(msg))
+end
+
+local function sha1_hmac(key, msg)
+ return str_util.to_hex(ngx_hmac_sha1(key, msg))
+end
+
+-- sign algorithm https://cloud.tencent.com/document/product/614/12445
+local function sign(secret_id, secret_key)
+ local method = "post"
+ local format_params = ""
+ local format_headers = ""
+ local sign_algorithm = "sha1"
+ local http_request_info = fmt("%s\n%s\n%s\n%s\n",
+ method, cls_api_path, format_params, format_headers)
+ local cur_time = ngx_time()
+ local sign_time = fmt("%d;%d", cur_time, cur_time + auth_expire_time)
+ local string_to_sign = fmt("%s\n%s\n%s\n", sign_algorithm, sign_time, sha1(http_request_info))
+
+ local sign_key = sha1_hmac(secret_key, sign_time)
+ local signature = sha1_hmac(sign_key, string_to_sign)
+
+ local arr = {
+ "q-sign-algorithm=sha1",
+ "q-ak=" .. secret_id,
+ "q-sign-time=" .. sign_time,
+ "q-key-time=" .. sign_time,
+ "q-header-list=",
+ "q-url-param-list=",
+ "q-signature=" .. signature,
+ }
+
+ return concat_tab(arr, '&')
+end
+
+local function send_cls_request(host, topic, secret_id, secret_key, pb_data)
+ local http_new = http:new()
+ http_new:set_timeouts(cls_conn_timeout, cls_send_timeout, cls_read_timeout)
+
+ clear_tab(headers_cache)
+ headers_cache["Host"] = host
+ headers_cache["Content-Type"] = "application/x-protobuf"
+ headers_cache["Authorization"] = sign(secret_id, secret_key, cls_api_path)
+
+ -- TODO: support lz4/zstd compress
+ params_cache.method = "POST"
+ params_cache.body = pb_data
+
+ local cls_url = "http://" .. host .. cls_api_path .. "?topic_id=" .. topic
+ core.log.debug("CLS request URL: ", cls_url)
+
+ local res, err = http_new:request_uri(cls_url, params_cache)
+ if not res then
+ return false, err
+ end
+
+ if res.status ~= 200 then
+ err = fmt("got wrong status: %s, headers: %s, body, %s",
+ res.status, json.encode(res.headers), res.body)
+ -- 413, 404, 401, 403 are not retryable
+ if res.status == 413 or res.status == 404 or res.status == 401 or res.status == 403 then
+ core.log.error(err, ", not retryable")
+ return true
+ end
+
+ return false, err
+ end
+
+ core.log.debug("CLS report success")
+ return true
+end
+
+-- normalized log data for CLS API
+local function normalize_log(log)
+ local normalized_log = {}
+ local log_size = 4 -- empty obj alignment
+ for k, v in pairs(log) do
+ local v_type = type(v)
+ local field = { key = k, value = "" }
+ if v_type == "string" then
+ field["value"] = v
+ elseif v_type == "number" then
+ field["value"] = tostring(v)
+ elseif v_type == "table" then
+ field["value"] = json_encode(v)
+ else
+ field["value"] = tostring(v)
+ core.log.warn("unexpected type " .. v_type .. " for field " .. k)
+ end
+ if #field.value > MAX_SINGLE_VALUE_SIZE then
+ core.log.warn(field.key, " value size over ", MAX_SINGLE_VALUE_SIZE, " , truncated")
+ field.value = field.value:sub(1, MAX_SINGLE_VALUE_SIZE)
+ end
+ insert_tab(normalized_log, field)
+ log_size = log_size + #field.key + #field.value
+ end
+ return normalized_log, log_size
+end
+
+local function send_to_cls(secret_id, secret_key, host, topic_id, logs)
+ clear_tab(log_group_list)
+ local now = ngx_now() * 1000
+
+ local total_size = 0
+ local format_logs = new_tab(#logs, 0)
+ -- sums of all value in a LogGroup should be no more than 5MB
+ for i = 1, #logs, 1 do
+ local contents, log_size = normalize_log(logs[i])
+ if log_size > MAX_LOG_GROUP_VALUE_SIZE then
+ core.log.error("size of log is over 5MB, dropped")
+ goto continue
+ end
+ total_size = total_size + log_size
+ if total_size > MAX_LOG_GROUP_VALUE_SIZE then
+ insert_tab(log_group_list, {
+ logs = format_logs,
+ source = host_ip,
+ })
+ format_logs = new_tab(#logs - i, 0)
+ total_size = 0
+ local data = assert(pb.encode("cls.LogGroupList", log_group_list_pb))
+ send_cls_request(host, topic_id, secret_id, secret_key, data)
Review Comment:
Any suggestion? Retry in batch processor is hard to understand here...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ychensha closed pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
ychensha closed pull request #7593: feat: support Tencent Cloud Log Service
URL: https://github.com/apache/apisix/pull/7593
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ychensha commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
ychensha commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r941291946
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,218 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local assert = assert
+assert(pb.loadfile("apisix/plugins/tencent-cloud-cls/cls.pb"))
Review Comment:
I will drop .proto file too... `protoc.load` like otel lib
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ychensha commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
ychensha commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1210227438
>
I have same question too. But same question can be applie to other plugins that have env matters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1210208755
> I have a question: Is it that we can't do the test case to verify that this plugin is working correctly? Because it requires a Tencent account and to make sure that this account is always available.
We can inject a hook or mock into the code? Like:
https://github.com/apache/apisix/blob/b3d4b464045b9bdfed3170cbd7a343ed5c40b24a/t/plugin/zipkin2.t#L35
https://github.com/apache/apisix/blob/b3d4b464045b9bdfed3170cbd7a343ed5c40b24a/t/plugin/opentelemetry.t#L553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander merged pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander merged PR #7593:
URL: https://github.com/apache/apisix/pull/7593
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ychensha commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
ychensha commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r938378483
##########
apisix/plugins/tencent-cloud-cls.lua:
##########
@@ -0,0 +1,94 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local log_util = require("apisix.utils.log-util")
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local cls_sdk = require("apisix.plugins.tencent-cloud-cls.cls-sdk")
+local random = math.random
+math.randomseed(ngx.time() + ngx.worker.pid())
+local ngx = ngx
+local pairs = pairs
+
+local plugin_name = "tencent-cloud-cls"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+local schema = {
+ type = "object",
+ properties = {
+ cls_host = { type = "string" },
+ cls_topic = { type = "string" },
+ -- https://console.cloud.tencent.com/capi
+ secret_id = { type = "string" },
+ secret_key = { type = "string" },
+ sample_rate = { type = "integer", minimum = 1, maximum = 100, default = 100 },
+ include_req_body = { type = "boolean", default = false },
+ include_resp_body = { type = "boolean", default = false },
+ global_tag = { type = "object" },
+ },
+ required = { "cls_host", "cls_topic", "secret_id", "secret_key" }
+}
+
+local _M = {
+ version = 0.1,
+ priority = 397,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+}
+
+function _M.check_schema(conf)
+ return core.schema.check(schema, conf)
+end
+
+function _M.body_filter(conf, ctx)
+ -- sample if set
+ if conf.sample_rate < 100 and random(1, 100) > conf.sample_rate then
+ core.log.debug("not sampled")
+ return
+ end
+ log_util.collect_body(conf, ctx)
+ ctx.cls_sample = true
+end
+
+function _M.log(conf, ctx)
+ -- sample if set
+ if ctx.cls_sample == nil then
+ core.log.debug("not sampled")
+ return
+ end
+ local entry = log_util.get_full_log(ngx, conf)
+ if not entry.route_id then
+ entry.route_id = "no-matched"
Review Comment:
No need. APISIX log is easy for debugging, if route_id field is missing, we known what happened
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r937505431
##########
apisix/plugins/tencent-cloud-cls/cls.proto:
##########
@@ -0,0 +1,50 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+// https://cloud.tencent.com/document/product/614/59470
+package cls;
+
+message Log
+{
+ message Content
+ {
+ required string key = 1; // 每组字段的 key
Review Comment:
Please use English
##########
apisix/plugins/tencent-cloud-cls/cls.proto:
##########
@@ -0,0 +1,50 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+// https://cloud.tencent.com/document/product/614/59470
+package cls;
Review Comment:
Let's update https://github.com/apache/apisix/blob/18a6cacf1681c80d7821b0791f6b704fe6a9a4d8/Makefile#L251 to install the proto file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r937504811
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,218 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local assert = assert
+assert(pb.loadfile("apisix/plugins/tencent-cloud-cls/cls.pb"))
Review Comment:
And we can load the proto file directly? So we don't need to store another .pb file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1211031910
> We can inject a hook or mock into the code? Like:
@ychensha Let's do that for now, there's no better way to do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r947496825
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,218 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local assert = assert
+assert(pb.loadfile("apisix/plugins/tencent-cloud-cls/cls.pb"))
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
+
+local cls_api_path = "/structuredlog"
+local auth_expire_time = 60
+local cls_conn_timeout = 1000
+local cls_read_timeout = 10000
+local cls_send_timeout = 10000
+
+local headers_cache = {}
+local params_cache = {
+ ssl_verify = false,
+ headers = headers_cache,
+}
+
+local function get_ip(hostname)
+ local _, resolved = socket.dns.toip(hostname)
+ local ListTab = {}
+ for _, v in ipairs(resolved.ip) do
+ insert_tab(ListTab, v)
+ end
+ return ListTab
+end
+
+local host_ip = tostring(unpack(get_ip(core_gethostname())))
+local log_group_list = {}
+local log_group_list_pb = {
+ logGroupList = log_group_list,
+}
+
+local function sha1(msg)
+ return str_util.to_hex(ngx_sha1_bin(msg))
+end
+
+local function sha1_hmac(key, msg)
+ return str_util.to_hex(ngx_hmac_sha1(key, msg))
+end
+
+-- sign algorithm https://cloud.tencent.com/document/product/614/12445
+local function sign(secret_id, secret_key)
+ local method = "post"
+ local format_params = ""
+ local format_headers = ""
+ local sign_algorithm = "sha1"
+ local http_request_info = fmt("%s\n%s\n%s\n%s\n",
+ method, cls_api_path, format_params, format_headers)
+ local cur_time = ngx_time()
+ local sign_time = fmt("%d;%d", cur_time, cur_time + auth_expire_time)
+ local string_to_sign = fmt("%s\n%s\n%s\n", sign_algorithm, sign_time, sha1(http_request_info))
+
+ local sign_key = sha1_hmac(secret_key, sign_time)
+ local signature = sha1_hmac(sign_key, string_to_sign)
+
+ local arr = {
+ "q-sign-algorithm=sha1",
+ "q-ak=" .. secret_id,
+ "q-sign-time=" .. sign_time,
+ "q-key-time=" .. sign_time,
+ "q-header-list=",
+ "q-url-param-list=",
+ "q-signature=" .. signature,
+ }
+
+ return concat_tab(arr, '&')
+end
+
+local function send_cls_request(host, topic, secret_id, secret_key, pb_data)
+ local http_new = http:new()
+ http_new:set_timeouts(cls_conn_timeout, cls_send_timeout, cls_read_timeout)
+
+ clear_tab(headers_cache)
+ headers_cache["Host"] = host
+ headers_cache["Content-Type"] = "application/x-protobuf"
+ headers_cache["Authorization"] = sign(secret_id, secret_key, cls_api_path)
+
+ -- TODO: support lz4/zstd compress
+ params_cache.method = "POST"
+ params_cache.body = pb_data
+
+ local cls_url = "http://" .. host .. cls_api_path .. "?topic_id=" .. topic
+ core.log.debug("CLS request URL: ", cls_url)
+
+ local res, err = http_new:request_uri(cls_url, params_cache)
+ if not res then
+ return false, err
Review Comment:
@ychensha
Please address this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a diff in pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on code in PR #7593:
URL: https://github.com/apache/apisix/pull/7593#discussion_r948718631
##########
apisix/plugins/tencent-cloud-cls/cls-sdk.lua:
##########
@@ -0,0 +1,308 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local pb = require "pb"
+local protoc = require("protoc").new()
+local http = require("resty.http")
+local socket = require("socket")
+local str_util = require("resty.string")
+local core = require("apisix.core")
+local core_gethostname = require("apisix.core.utils").gethostname
+local json = core.json
+local json_encode = json.encode
+local ngx = ngx
+local ngx_time = ngx.time
+local ngx_now = ngx.now
+local ngx_sha1_bin = ngx.sha1_bin
+local ngx_hmac_sha1 = ngx.hmac_sha1
+local fmt = string.format
+local table = table
+local concat_tab = table.concat
+local clear_tab = table.clear
+local new_tab = table.new
+local insert_tab = table.insert
+local ipairs = ipairs
+local pairs = pairs
+local type = type
+local tostring = tostring
+local setmetatable = setmetatable
+local pcall = pcall
+
+local MAX_SINGLE_VALUE_SIZE = 1 * 1024 * 1024
+local MAX_LOG_GROUP_VALUE_SIZE = 5 * 1024 * 1024 -- 5MB
Review Comment:
@ychensha
Let's add a comment for the source
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ychensha commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
ychensha commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1210237304
> >
>
> I have same question too. But same question can be applie to other plugins that have env matters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on pull request #7593: feat: support Tencent Cloud Log Service
Posted by GitBox <gi...@apache.org>.
spacewander commented on PR #7593:
URL: https://github.com/apache/apisix/pull/7593#issuecomment-1216225852
To fix https://github.com/apache/apisix/runs/7851928549?check_suite_focus=true
Let's add tencent-cloud-cls to https://github.com/apache/apisix/blob/b0934f9ed45ec352c3de09f59e6912a7ba71936e/Makefile#L344
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org