You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/01/27 06:25:15 UTC
[GitHub] [apisix] zhendongcmss opened a new pull request #6215: feat: clickhouse logger
zhendongcmss opened a new pull request #6215:
URL: https://github.com/apache/apisix/pull/6215
### What this PR does / why we need it:
<!--- Why is this change required? What problem does it solve? -->
<!--- If it fixes an open issue, please link to the issue here. -->
### Pre-submission checklist:
<!--
Please follow the PR manners:
1. Use Draft if the PR is not ready to be reviewed
2. Test is required for the feat/fix PR, unless you have a good reason
3. Doc is required for the feat PR
4. Use a new commit to resolve review instead of `push -f`
5. If you need to resolve merge conflicts after the PR is reviewed, please merge master but do not rebase
6. Use "request review" to notify the reviewer once you have resolved the review
7. Only reviewer can click "Resolve conversation" to mark the reviewer's review resolved
-->
* [ ] Did you explain what problem does this PR solve? Or what new features have been added?
* [ ] Have you added corresponding test cases?
* [ ] Have you modified the corresponding document?
* [ ] Is this PR backward compatible? **If it is not backward compatible, please discuss on the [mailing list](https://github.com/apache/apisix/tree/master#community) first**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r805543150
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+apisix:
+ node_listen: 1984
+ admin_key: null
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:10420/clickhouse-logger/test",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":5
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 5: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- grep_error_log_out
+"clickhouse body: INSERT INTO t FORMAT JSONEachRow"
+"clickhouse headers: X-ClickHouse-Key:a"
+"clickhouse headers: X-ClickHouse-User:default"
+"clickhouse headers: X-ClickHouse-Database:default"
Review comment:
I have reconsidered the test.
What about using error_log like https://github.com/apache/apisix/blob/537a8da24d723caa5c85561d36b3b931ce1f1410/t/plugin/loggly.t#L572-L575?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r803322743
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,227 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "loggly body: ", data)
+ ngx.log(ngx.ERR, "loggly tags: " .. require("toolkit.json").encode(headers["X-LOGGLY-TAG"]))
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r803333776
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
Review comment:
Clickhouse providers HTTP restful style to access DB, prevent SQL injection need DBA reasonably assigns permissions for apisix. Other than that, I don't know what protection means there are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r794164504
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,231 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local batch_processor = require("apisix.utils.batch-processor")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local timer_at = ngx.timer.at
+
+local plugin_name = "clickhouse-logger"
+local stale_timer_running = false
+local buffers = {}
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 399,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
+ ok, err = httpc:ssl_handshake(true, host, false)
+ if not ok then
+ return false, "failed to perform SSL with host[" .. host .. "] "
+ .. "port[" .. tostring(port) .. "] " .. err
+ end
+ end
+ url_decoded.query['database'] = conf.database
+
+ local httpc_res, httpc_err = httpc:request({
+ method = "POST",
+ path = url_decoded.path,
+ query = url_decoded.query,
+ body = "INSERT INTO " .. conf.logtable .." FORMAT JSONEachRow " .. log_message,
+ headers = {
+ ["Host"] = url_decoded.host,
+ ["Content-Type"] = "application/json",
+ ["X-ClickHouse-User"] = conf.user,
+ ["X-ClickHouse-Key"] = conf.password,
+ }
+ })
+
+ if not httpc_res then
+ return false, "error while sending data to [" .. host .. "] port["
+ .. tostring(port) .. "] " .. httpc_err
+ end
+
+ -- some error occurred in the server
+ if httpc_res.status >= 400 then
+ res = false
+ err_msg = "server returned status code[" .. httpc_res.status .. "] host["
+ .. host .. "] port[" .. tostring(port) .. "] "
+ .. "body[" .. httpc_res:read_body() .. "]"
+ end
+
+ return res, err_msg
+end
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, conf: ",
+ core.json.delay_encode(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+function _M.log(conf, ctx)
+ local metadata = plugin.plugin_metadata("http-logger")
+ core.log.info("metadata: ", core.json.delay_encode(metadata))
+ local entry
+
+ if metadata and metadata.value.log_format
+ and core.table.nkeys(metadata.value.log_format) > 0
+ then
+ entry = log_util.get_custom_format_log(ctx, metadata.value.log_format)
+ else
+ entry = log_util.get_full_log(ngx, conf)
+ end
+
+ if not entry.route_id then
Review comment:
Remove L171 - 173 ? Assign `entry.route_id` a default value on `batch-processor-manager` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r794220090
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 398,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
Review comment:
I think it is necessary to add it. For example `Python` request module add it.
```
import requests
response = requests.get(url='https://.githu.com', verify=False)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r794159753
##########
File path: docs/en/latest/plugins/clickhouse-logger.md
##########
@@ -0,0 +1,157 @@
+---
+title: clickhouse-logger
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Metadata**](#metadata)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`clickhouse-logger` is a plugin which push Log data requests to clickhouse.
+
+## Attributes
+
+| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 |
+| ---------------- | ------- | ------ | ------------- | ------- | ------------------------------------------------ |
+| endpoint_addr | string | required | | | The `clickhouse` endpoint. |
+| database | string | required | | | The DB name to store log. |
+| logtable | string | required | | | The table name. |
+| user | string | required | | |clickhouse user. |
+| password | string | required | | |clickhouse password. |
+| timeout | integer | optional | 3 | [1,...] | Time to keep the connection alive after sending a request. |
+| name | string | optional | "clickhouse logger" | | A unique identifier to identity the logger. |
+| batch_max_size | integer | optional | 100 | [1,...] | Set the maximum number of logs sent in each batch. When the number of logs reaches the set maximum, all logs will be automatically pushed to the clickhouse. |
+| max_retry_count | integer | optional | 0 | [0,...] | Maximum number of retries before removing from the processing pipe line. |
+| retry_delay | integer | optional | 1 | [0,...] | Number of seconds the process execution should be delayed if the execution fails. |
+
+## How To Enable
+
+The following is an example of how to enable the `click-logger` for a specific route.
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r793415236
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,231 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local batch_processor = require("apisix.utils.batch-processor")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local timer_at = ngx.timer.at
+
+local plugin_name = "clickhouse-logger"
+local stale_timer_running = false
+local buffers = {}
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 399,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
+ ok, err = httpc:ssl_handshake(true, host, false)
+ if not ok then
+ return false, "failed to perform SSL with host[" .. host .. "] "
+ .. "port[" .. tostring(port) .. "] " .. err
+ end
+ end
+ url_decoded.query['database'] = conf.database
+
+ local httpc_res, httpc_err = httpc:request({
+ method = "POST",
+ path = url_decoded.path,
+ query = url_decoded.query,
+ body = "INSERT INTO " .. conf.logtable .." FORMAT JSONEachRow " .. log_message,
+ headers = {
+ ["Host"] = url_decoded.host,
+ ["Content-Type"] = "application/json",
+ ["X-ClickHouse-User"] = conf.user,
+ ["X-ClickHouse-Key"] = conf.password,
+ }
+ })
+
+ if not httpc_res then
+ return false, "error while sending data to [" .. host .. "] port["
+ .. tostring(port) .. "] " .. httpc_err
+ end
+
+ -- some error occurred in the server
+ if httpc_res.status >= 400 then
+ res = false
+ err_msg = "server returned status code[" .. httpc_res.status .. "] host["
+ .. host .. "] port[" .. tostring(port) .. "] "
+ .. "body[" .. httpc_res:read_body() .. "]"
+ end
+
+ return res, err_msg
+end
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, conf: ",
+ core.json.delay_encode(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+function _M.log(conf, ctx)
+ local metadata = plugin.plugin_metadata("http-logger")
Review comment:
It's better to create and use the metadata of `clickhouse-logger` to decouple dependencies from other plugins.
##########
File path: conf/config-default.yaml
##########
@@ -381,6 +381,7 @@ plugins: # plugin list (sorted by priority)
- rocketmq-logger # priority: 402
- syslog # priority: 401
- udp-logger # priority: 400
+ #- clickhouse-logger # priority: 399
Review comment:
ditto
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,231 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local batch_processor = require("apisix.utils.batch-processor")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local timer_at = ngx.timer.at
+
+local plugin_name = "clickhouse-logger"
+local stale_timer_running = false
+local buffers = {}
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 399,
Review comment:
It should have its own priority and cannot conflict with other plugins.
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,231 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local batch_processor = require("apisix.utils.batch-processor")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local timer_at = ngx.timer.at
+
+local plugin_name = "clickhouse-logger"
+local stale_timer_running = false
+local buffers = {}
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 399,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
+ ok, err = httpc:ssl_handshake(true, host, false)
+ if not ok then
+ return false, "failed to perform SSL with host[" .. host .. "] "
+ .. "port[" .. tostring(port) .. "] " .. err
+ end
+ end
+ url_decoded.query['database'] = conf.database
+
+ local httpc_res, httpc_err = httpc:request({
+ method = "POST",
+ path = url_decoded.path,
+ query = url_decoded.query,
+ body = "INSERT INTO " .. conf.logtable .." FORMAT JSONEachRow " .. log_message,
+ headers = {
+ ["Host"] = url_decoded.host,
+ ["Content-Type"] = "application/json",
+ ["X-ClickHouse-User"] = conf.user,
+ ["X-ClickHouse-Key"] = conf.password,
+ }
+ })
+
+ if not httpc_res then
+ return false, "error while sending data to [" .. host .. "] port["
+ .. tostring(port) .. "] " .. httpc_err
+ end
+
+ -- some error occurred in the server
+ if httpc_res.status >= 400 then
+ res = false
+ err_msg = "server returned status code[" .. httpc_res.status .. "] host["
+ .. host .. "] port[" .. tostring(port) .. "] "
+ .. "body[" .. httpc_res:read_body() .. "]"
+ end
+
+ return res, err_msg
+end
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, conf: ",
+ core.json.delay_encode(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
Review comment:
You can use `apisix.utils.batch-processor-manager` to reduce redundancy of similar code in the `logger` classification plugin.
refer to: https://github.com/apache/apisix/blob/c38e94f20144b0ba01018970fa954c935a325ac1/apisix/plugins/splunk-hec-logging.lua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r794158591
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,231 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local batch_processor = require("apisix.utils.batch-processor")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local timer_at = ngx.timer.at
+
+local plugin_name = "clickhouse-logger"
+local stale_timer_running = false
+local buffers = {}
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 399,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
+ ok, err = httpc:ssl_handshake(true, host, false)
+ if not ok then
+ return false, "failed to perform SSL with host[" .. host .. "] "
+ .. "port[" .. tostring(port) .. "] " .. err
+ end
+ end
+ url_decoded.query['database'] = conf.database
+
+ local httpc_res, httpc_err = httpc:request({
+ method = "POST",
+ path = url_decoded.path,
+ query = url_decoded.query,
+ body = "INSERT INTO " .. conf.logtable .." FORMAT JSONEachRow " .. log_message,
+ headers = {
+ ["Host"] = url_decoded.host,
+ ["Content-Type"] = "application/json",
+ ["X-ClickHouse-User"] = conf.user,
+ ["X-ClickHouse-Key"] = conf.password,
+ }
+ })
+
+ if not httpc_res then
+ return false, "error while sending data to [" .. host .. "] port["
+ .. tostring(port) .. "] " .. httpc_err
+ end
+
+ -- some error occurred in the server
+ if httpc_res.status >= 400 then
+ res = false
+ err_msg = "server returned status code[" .. httpc_res.status .. "] host["
+ .. host .. "] port[" .. tostring(port) .. "] "
+ .. "body[" .. httpc_res:read_body() .. "]"
+ end
+
+ return res, err_msg
+end
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, conf: ",
+ core.json.delay_encode(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
Review comment:
Try to refine it. This plugin was developed on v2.10.1 on our internal repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r800274036
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,230 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "loggly body: ", data)
+ ngx.log(ngx.ERR, "loggly tags: " .. require("toolkit.json").encode(headers["X-LOGGLY-TAG"]))
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({log_id = "syslog",
+ max_retry_count = 0,
+ retry_delay = 1,
+ buffer_duration = 60,
+ inactive_timeout = 10,
+ batch_max_size = 100,
+ })
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:8123"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:8123",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":5
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
Review comment:
Already add on Line 27
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r803322204
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,227 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "loggly body: ", data)
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] starsz commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
starsz commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r804285442
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ ssl_verify = {type = "boolean", default = true},
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 398,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if not port then
+ if url_decoded.scheme == "https" then
+ port = 443
+ else
+ port = 80
+ end
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" and conf.ssl_verify then
+ ok, err = httpc:ssl_handshake(true, host, true)
Review comment:
Hi @zhendongcmss
I think the logic should be like this:
```
if url_decoded.scheme == "https" then
ok, err = httpc:ssl_handshake(true, host, conf.ssl_verify)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r801216324
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,223 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "loggly body: ", data)
+ ngx.log(ngx.ERR, "loggly tags: " .. require("toolkit.json").encode(headers["X-LOGGLY-TAG"]))
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:8123"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:8123",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":5
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
Review comment:
```suggestion
}
--- response_body
passed
```
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,223 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "loggly body: ", data)
+ ngx.log(ngx.ERR, "loggly tags: " .. require("toolkit.json").encode(headers["X-LOGGLY-TAG"]))
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- yaml_config
+plugins:
+ - clickhouse-logger
Review comment:
```suggestion
apisix:
node_listen: 1984
admin_key: null
plugins:
- clickhouse-logger
```
This will solve the 401 problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r794183325
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 398,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
Review comment:
```suggestion
if not port then
if url_decoded.scheme == "https" then
port = 443
else
port = 80
end
end
```
is better?
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,231 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local batch_processor = require("apisix.utils.batch-processor")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local timer_at = ngx.timer.at
+
+local plugin_name = "clickhouse-logger"
+local stale_timer_running = false
+local buffers = {}
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 399,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
+ ok, err = httpc:ssl_handshake(true, host, false)
+ if not ok then
+ return false, "failed to perform SSL with host[" .. host .. "] "
+ .. "port[" .. tostring(port) .. "] " .. err
+ end
+ end
+ url_decoded.query['database'] = conf.database
+
+ local httpc_res, httpc_err = httpc:request({
+ method = "POST",
+ path = url_decoded.path,
+ query = url_decoded.query,
+ body = "INSERT INTO " .. conf.logtable .." FORMAT JSONEachRow " .. log_message,
+ headers = {
+ ["Host"] = url_decoded.host,
+ ["Content-Type"] = "application/json",
+ ["X-ClickHouse-User"] = conf.user,
+ ["X-ClickHouse-Key"] = conf.password,
+ }
+ })
+
+ if not httpc_res then
+ return false, "error while sending data to [" .. host .. "] port["
+ .. tostring(port) .. "] " .. httpc_err
+ end
+
+ -- some error occurred in the server
+ if httpc_res.status >= 400 then
+ res = false
+ err_msg = "server returned status code[" .. httpc_res.status .. "] host["
+ .. host .. "] port[" .. tostring(port) .. "] "
+ .. "body[" .. httpc_res:read_body() .. "]"
+ end
+
+ return res, err_msg
+end
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, conf: ",
+ core.json.delay_encode(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+function _M.log(conf, ctx)
+ local metadata = plugin.plugin_metadata("http-logger")
Review comment:
I think `clickhouse-logger` is better, it just gets the clickhouse-logger's own `log_format` from the plugin_metadata
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,231 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local batch_processor = require("apisix.utils.batch-processor")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local timer_at = ngx.timer.at
+
+local plugin_name = "clickhouse-logger"
+local stale_timer_running = false
+local buffers = {}
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 399,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
+ ok, err = httpc:ssl_handshake(true, host, false)
+ if not ok then
+ return false, "failed to perform SSL with host[" .. host .. "] "
+ .. "port[" .. tostring(port) .. "] " .. err
+ end
+ end
+ url_decoded.query['database'] = conf.database
+
+ local httpc_res, httpc_err = httpc:request({
+ method = "POST",
+ path = url_decoded.path,
+ query = url_decoded.query,
+ body = "INSERT INTO " .. conf.logtable .." FORMAT JSONEachRow " .. log_message,
+ headers = {
+ ["Host"] = url_decoded.host,
+ ["Content-Type"] = "application/json",
+ ["X-ClickHouse-User"] = conf.user,
+ ["X-ClickHouse-Key"] = conf.password,
+ }
+ })
+
+ if not httpc_res then
+ return false, "error while sending data to [" .. host .. "] port["
+ .. tostring(port) .. "] " .. httpc_err
+ end
+
+ -- some error occurred in the server
+ if httpc_res.status >= 400 then
+ res = false
+ err_msg = "server returned status code[" .. httpc_res.status .. "] host["
+ .. host .. "] port[" .. tostring(port) .. "] "
+ .. "body[" .. httpc_res:read_body() .. "]"
+ end
+
+ return res, err_msg
+end
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, conf: ",
+ core.json.delay_encode(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+function _M.log(conf, ctx)
+ local metadata = plugin.plugin_metadata("http-logger")
Review comment:
I know, you follow `http-logger`
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 398,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
Review comment:
```suggestion
local ssl_verify = url_decoded.scheme == "https"
if ssl_verify then
```
Do we need to add `ssl_verify` to the attribute and let the user choose?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r794220090
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 398,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
Review comment:
I think it is necessary to keep it. For example `Python` request module keep it.
```
import requests
response = requests.get(url='https://.githu.com', verify=False)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#issuecomment-1024851343
Hi, @spacewander
How fix this test case ?
https://github.com/apache/apisix/runs/4989817705?check_suite_focus=true#step:7:22
```
Installing collected packages: zhon
Successfully installed zhon-1.1.5
find broken newline in file: docs/zh/latest/plugins/clickhouse-logger.md
cat: /tmp/error.log: No such file or directory
Error: Process completed with exit code 1.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r794230745
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 398,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
Review comment:
```suggestion
if url_decoded.scheme == "https" and conf.ssl_verify then
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#issuecomment-1025060200
> Hi, @spacewander How fix this test case ? [apache/apisix/runs/4989817705?check_suite_focus=true#step:7:22](https://github.com/apache/apisix/runs/4989817705?check_suite_focus=true#step:7:22)
>
> ```
> Installing collected packages: zhon
> Successfully installed zhon-1.1.5
> find broken newline in file: docs/zh/latest/plugins/clickhouse-logger.md
> cat: /tmp/error.log: No such file or directory
> Error: Process completed with exit code 1.
> ```
Sorry, it is the bug in our lint script. Fixed in https://github.com/apache/apisix/pull/6239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r795132834
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,182 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
Review comment:
Soga
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r806382732
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+apisix:
+ node_listen: 1984
+ admin_key: null
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:10420/clickhouse-logger/test",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":5
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 5: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- grep_error_log_out
+"clickhouse body: INSERT INTO t FORMAT JSONEachRow"
+"clickhouse headers: X-ClickHouse-Key:a"
+"clickhouse headers: X-ClickHouse-User:default"
+"clickhouse headers: X-ClickHouse-Database:default"
Review comment:
It doesn't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r806566122
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,219 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
Review comment:
I thought there was a timer that called this callback function periodically and also if the batch_max_size value was reached. But I added a wait of `--- wait: 5` below, it seem doesn'twork.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r807658625
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,221 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test",
+ "batch_max_size":1,
+ "inactive_timeout":1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:10420/clickhouse-logger/test",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":1
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 5: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+clickhouse body: INSERT INTO t FORMAT JSONEachRow
+clickhouse headers: x-clickHouse-key:a
Review comment:
```suggestion
clickhouse headers: x-clickhouse-key:a
```
Let's update similar places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r800271982
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,230 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "loggly body: ", data)
+ ngx.log(ngx.ERR, "loggly tags: " .. require("toolkit.json").encode(headers["X-LOGGLY-TAG"]))
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({log_id = "syslog",
+ max_retry_count = 0,
+ retry_delay = 1,
+ buffer_duration = 60,
+ inactive_timeout = 10,
+ batch_max_size = 100,
+ })
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
Review comment:
Sorry, I don't understand. On the test case, there is no clickhouse can be used. The password doesn't be verfied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r805543150
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+apisix:
+ node_listen: 1984
+ admin_key: null
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:10420/clickhouse-logger/test",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":5
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 5: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- grep_error_log_out
+"clickhouse body: INSERT INTO t FORMAT JSONEachRow"
+"clickhouse headers: X-ClickHouse-Key:a"
+"clickhouse headers: X-ClickHouse-User:default"
+"clickhouse headers: X-ClickHouse-Database:default"
Review comment:
I have reconsidered the test.
What about using error_log like https://github.com/apache/apisix/blob/537a8da24d723caa5c85561d36b3b931ce1f1410/t/plugin/loggly.t#L572-L577?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r794162338
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,231 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local batch_processor = require("apisix.utils.batch-processor")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local timer_at = ngx.timer.at
+
+local plugin_name = "clickhouse-logger"
+local stale_timer_running = false
+local buffers = {}
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 399,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
+ ok, err = httpc:ssl_handshake(true, host, false)
+ if not ok then
+ return false, "failed to perform SSL with host[" .. host .. "] "
+ .. "port[" .. tostring(port) .. "] " .. err
+ end
+ end
+ url_decoded.query['database'] = conf.database
+
+ local httpc_res, httpc_err = httpc:request({
+ method = "POST",
+ path = url_decoded.path,
+ query = url_decoded.query,
+ body = "INSERT INTO " .. conf.logtable .." FORMAT JSONEachRow " .. log_message,
+ headers = {
+ ["Host"] = url_decoded.host,
+ ["Content-Type"] = "application/json",
+ ["X-ClickHouse-User"] = conf.user,
+ ["X-ClickHouse-Key"] = conf.password,
+ }
+ })
+
+ if not httpc_res then
+ return false, "error while sending data to [" .. host .. "] port["
+ .. tostring(port) .. "] " .. httpc_err
+ end
+
+ -- some error occurred in the server
+ if httpc_res.status >= 400 then
+ res = false
+ err_msg = "server returned status code[" .. httpc_res.status .. "] host["
+ .. host .. "] port[" .. tostring(port) .. "] "
+ .. "body[" .. httpc_res:read_body() .. "]"
+ end
+
+ return res, err_msg
+end
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, conf: ",
+ core.json.delay_encode(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+function _M.log(conf, ctx)
+ local metadata = plugin.plugin_metadata("http-logger")
Review comment:
I think there is only one access log format. Thinking configurated http-logger and clickhouse-logger, if the log format is different, front page not esay to process it and show.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r800269554
##########
File path: conf/config-default.yaml
##########
@@ -382,6 +382,7 @@ plugins: # plugin list (sorted by priority)
- syslog # priority: 401
- udp-logger # priority: 400
- file-logger # priority: 399
+ #- clickhouse-logger # priority: 398
Review comment:
It is safe. The principal of plugins is enabled by default ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] starsz commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
starsz commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r801632446
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ ssl_verify = {type = "boolean", default = true},
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 398,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if not port then
+ if url_decoded.scheme == "https" then
+ port = 443
+ else
+ port = 80
+ end
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" and conf.ssl_verify then
+ ok, err = httpc:ssl_handshake(true, host, false)
Review comment:
conf.ssl_verify is true.But send false in the ssl_handshake function? 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#issuecomment-1032605770
hi @zhendongcmss, you can do like:https://github.com/apache/apisix/pull/5831/files#diff-0143af7fae95d5cd58bb1e460c89bbe37e8c35826b5ce783d9e2c3a13ba91775R114 to make `t/admin/plugins.t` pass.
maybe you put `clickhouse-logger` in the wrong place.
```
# loggly
# http-logger
# -clickhouse-logger
# splunk-hec-logging
# skywalking-logger
# google-cloud-logging
# @@ -56,6 +55,7 @@
# syslog
# udp-logger
# file-logger
# +clickhouse-logger
# example-plugin
# aws-lambda
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] starsz commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
starsz commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r802645783
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ ssl_verify = {type = "boolean", default = true},
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 398,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if not port then
+ if url_decoded.scheme == "https" then
+ port = 443
+ else
+ port = 80
+ end
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" and conf.ssl_verify then
+ ok, err = httpc:ssl_handshake(true, host, false)
Review comment:
You can check the third arg of this function.
https://github.com/ledgetech/lua-resty-http#ssl_handshake
Maybe you mistake the `ssl verify` with `use SSL` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r807686549
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,221 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test",
+ "batch_max_size":1,
+ "inactive_timeout":1
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:10420/clickhouse-logger/test",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":1
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 5: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+clickhouse body: INSERT INTO t FORMAT JSONEachRow
+clickhouse headers: x-clickHouse-key:a
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander merged pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander merged pull request #6215:
URL: https://github.com/apache/apisix/pull/6215
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r806384162
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+apisix:
+ node_listen: 1984
+ admin_key: null
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:10420/clickhouse-logger/test",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":5
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 5: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- grep_error_log_out
+"clickhouse body: INSERT INTO t FORMAT JSONEachRow"
+"clickhouse headers: X-ClickHouse-Key:a"
+"clickhouse headers: X-ClickHouse-User:default"
+"clickhouse headers: X-ClickHouse-Database:default"
Review comment:
It doesn't work. I don't know why. I think `--- grep_error_log_out` is easier to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
membphis commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r811594010
##########
File path: docs/en/latest/plugins/clickhouse-logger.md
##########
@@ -0,0 +1,148 @@
+---
+title: clickhouse-logger
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Metadata**](#metadata)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`clickhouse-logger` is a plugin which push Log data requests to clickhouse.
+
+## Attributes
+
+| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 |
Review comment:
I created an issue, let's fix it: https://github.com/apache/apisix/issues/6414
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] starsz commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
starsz commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r805454048
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ ssl_verify = {type = "boolean", default = true},
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 398,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if not port then
+ if url_decoded.scheme == "https" then
+ port = 443
+ else
+ port = 80
+ end
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" and conf.ssl_verify then
Review comment:
We don't need to check the conf.ssl_verify here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] starsz commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
starsz commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r805454048
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ ssl_verify = {type = "boolean", default = true},
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 398,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if not port then
+ if url_decoded.scheme == "https" then
+ port = 443
+ else
+ port = 80
+ end
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" and conf.ssl_verify then
Review comment:
We don't need to check the conf.ssl_verify here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r803271948
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,227 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "loggly body: ", data)
Review comment:
Ditto
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,227 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "loggly body: ", data)
+ ngx.log(ngx.ERR, "loggly tags: " .. require("toolkit.json").encode(headers["X-LOGGLY-TAG"]))
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- yaml_config
+apisix:
+ node_listen: 1984
+ admin_key: null
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:8123"
Review comment:
Let's configure to use the fake upstream and check the upload log data via grep_error_log_out
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
Review comment:
Should we use pattern to validate logtable to prevent SQL injection?
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,227 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "loggly body: ", data)
+ ngx.log(ngx.ERR, "loggly tags: " .. require("toolkit.json").encode(headers["X-LOGGLY-TAG"]))
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
Review comment:
Since clickhouse-logger is now enabled by default, we don't need to configure it in yaml_config anymore.
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,227 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
Review comment:
As this is test for clickhouse, better to use a new path name relative to clickhouse, even it's just a fake upstream.
##########
File path: docs/en/latest/plugins/clickhouse-logger.md
##########
@@ -0,0 +1,148 @@
+---
+title: clickhouse-logger
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Metadata**](#metadata)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`clickhouse-logger` is a plugin which push Log data requests to clickhouse.
+
+## Attributes
+
+| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 |
+| ---------------- | ------- | ------ | ------------- | ------- | ------------------------------------------------ |
+| endpoint_addr | string | required | | | The `clickhouse` endpoint. |
+| database | string | required | | | The DB name to store log. |
+| logtable | string | required | | | The table name. |
+| user | string | required | | | clickhouse user. |
+| password | string | required | | | clickhouse password. |
+| timeout | integer | optional | 3 | [1,...] | Time to keep the connection alive after sending a request. |
+| name | string | optional | "clickhouse logger" | | A unique identifier to identity the logger. |
+| batch_max_size | integer | optional | 100 | [1,...] | Set the maximum number of logs sent in each batch. When the number of logs reaches the set maximum, all logs will be automatically pushed to the clickhouse. |
+| max_retry_count | integer | optional | 0 | [0,...] | Maximum number of retries before removing from the processing pipe line. |
+| retry_delay | integer | optional | 1 | [0,...] | Number of seconds the process execution should be delayed if the execution fails. |
+| ssl_verify | boolean | optional | true | [true,false] | verify ssl. |
+
+## How To Enable
+
+The following is an example of how to enable the `clickhouse-logger` for a specific route.
+
+```shell
+curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
+{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "test",
+ "endpoint_addr": "http://127.0.0.1:8123"
+ }
+ },
+ "upstream": {
+ "type": "roundrobin",
+ "nodes": {
+ "127.0.0.1:1980": 1
+ }
+ },
+ "uri": "/hello"
+}'
+```
+
+## Test Plugin
+
+> success:
+
+```shell
+$ curl -i http://127.0.0.1:9080/hello
+HTTP/1.1 200 OK
+...
+hello, world
+```
+
+## Metadata
+
+| Name | Type | Requirement | Default | Valid | Description |
+| ---------------- | ------- | ----------- | ------------- | ------- | ---------------------------------------------------------------------------------------- |
+| log_format | object | optional | {"host": "$host", "@timestamp": "$time_iso8601", "client_ip": "$remote_addr"} | | Log format declared as key value pair in JSON format. Only string is supported in the `value` part. If the value starts with `$`, it means to get `APISIX` variables or [Nginx variable](http://nginx.org/en/docs/varindex.html). |
Review comment:
Let's turn "`APISIX` variables" into a link like other logger's doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r803320498
##########
File path: docs/en/latest/plugins/clickhouse-logger.md
##########
@@ -0,0 +1,148 @@
+---
+title: clickhouse-logger
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Metadata**](#metadata)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`clickhouse-logger` is a plugin which push Log data requests to clickhouse.
+
+## Attributes
+
+| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 |
+| ---------------- | ------- | ------ | ------------- | ------- | ------------------------------------------------ |
+| endpoint_addr | string | required | | | The `clickhouse` endpoint. |
+| database | string | required | | | The DB name to store log. |
+| logtable | string | required | | | The table name. |
+| user | string | required | | | clickhouse user. |
+| password | string | required | | | clickhouse password. |
+| timeout | integer | optional | 3 | [1,...] | Time to keep the connection alive after sending a request. |
+| name | string | optional | "clickhouse logger" | | A unique identifier to identity the logger. |
+| batch_max_size | integer | optional | 100 | [1,...] | Set the maximum number of logs sent in each batch. When the number of logs reaches the set maximum, all logs will be automatically pushed to the clickhouse. |
+| max_retry_count | integer | optional | 0 | [0,...] | Maximum number of retries before removing from the processing pipe line. |
+| retry_delay | integer | optional | 1 | [0,...] | Number of seconds the process execution should be delayed if the execution fails. |
+| ssl_verify | boolean | optional | true | [true,false] | verify ssl. |
+
+## How To Enable
+
+The following is an example of how to enable the `clickhouse-logger` for a specific route.
+
+```shell
+curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
+{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "test",
+ "endpoint_addr": "http://127.0.0.1:8123"
+ }
+ },
+ "upstream": {
+ "type": "roundrobin",
+ "nodes": {
+ "127.0.0.1:1980": 1
+ }
+ },
+ "uri": "/hello"
+}'
+```
+
+## Test Plugin
+
+> success:
+
+```shell
+$ curl -i http://127.0.0.1:9080/hello
+HTTP/1.1 200 OK
+...
+hello, world
+```
+
+## Metadata
+
+| Name | Type | Requirement | Default | Valid | Description |
+| ---------------- | ------- | ----------- | ------------- | ------- | ---------------------------------------------------------------------------------------- |
+| log_format | object | optional | {"host": "$host", "@timestamp": "$time_iso8601", "client_ip": "$remote_addr"} | | Log format declared as key value pair in JSON format. Only string is supported in the `value` part. If the value starts with `$`, it means to get `APISIX` variables or [Nginx variable](http://nginx.org/en/docs/varindex.html). |
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r803321789
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,227 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r804384280
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,220 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "clickhouse body: ", data)
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- yaml_config
Review comment:
Let's remove empty `--- yaml_config`
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,220 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "clickhouse body: ", data)
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- yaml_config
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- yaml_config
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- yaml_config
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- yaml_config
+apisix:
+ node_listen: 1984
+ admin_key: null
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:10420/clickhouse-logger/test",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":5
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 5: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- grep_error_log_out
Review comment:
We can check the content of the body, like:
```suggestion
--- grep_error_log eval
qr/clickhouse body: [^,]+/
--- grep_error_log_out
clickhouse body: blahblah...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r794158591
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,231 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local batch_processor = require("apisix.utils.batch-processor")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local timer_at = ngx.timer.at
+
+local plugin_name = "clickhouse-logger"
+local stale_timer_running = false
+local buffers = {}
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 399,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
+ ok, err = httpc:ssl_handshake(true, host, false)
+ if not ok then
+ return false, "failed to perform SSL with host[" .. host .. "] "
+ .. "port[" .. tostring(port) .. "] " .. err
+ end
+ end
+ url_decoded.query['database'] = conf.database
+
+ local httpc_res, httpc_err = httpc:request({
+ method = "POST",
+ path = url_decoded.path,
+ query = url_decoded.query,
+ body = "INSERT INTO " .. conf.logtable .." FORMAT JSONEachRow " .. log_message,
+ headers = {
+ ["Host"] = url_decoded.host,
+ ["Content-Type"] = "application/json",
+ ["X-ClickHouse-User"] = conf.user,
+ ["X-ClickHouse-Key"] = conf.password,
+ }
+ })
+
+ if not httpc_res then
+ return false, "error while sending data to [" .. host .. "] port["
+ .. tostring(port) .. "] " .. httpc_err
+ end
+
+ -- some error occurred in the server
+ if httpc_res.status >= 400 then
+ res = false
+ err_msg = "server returned status code[" .. httpc_res.status .. "] host["
+ .. host .. "] port[" .. tostring(port) .. "] "
+ .. "body[" .. httpc_res:read_body() .. "]"
+ end
+
+ return res, err_msg
+end
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, conf: ",
+ core.json.delay_encode(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
Review comment:
Try to refine it. This plugin was developed on v2.10.1 on our internal repo. No `batch-processor-manager` on v2.10.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#issuecomment-1032148042
> > hi @zhendongcmss CI is broken
>
> @spacewander @tzssangglass sorry， I don't know how to fix. I need help, can you tell me how to fix it ?
>
> ```
> # Failed test 'TEST 4: add plugin on routes - status code ok'
> # at /apisix/test-nginx/lib/Test/Nginx/Socket.pm line 948.
> # got: '401'
> # expected: '200'
>
> # Failed test 'TEST 5: access local server - status code ok'
> # at /apisix/test-nginx/lib/Test/Nginx/Socket.pm line 948.
> # got: '404'
> # expected: '200'
>
> # Failed test 'TEST 5: access local server - response_body - response is expected (repeated req 0, req 0)'
> # at /apisix/test-nginx/lib/Test/Nginx/Socket.pm line 1589.
> # got: '{"error_msg":"404 Route Not Found"}
> # '
> # expected: 'opentracing
> ```
My comments above solved the 401 problem with test 4, you can continue to see the problem with test 5, which is related to the code and configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r800345618
##########
File path: conf/config-default.yaml
##########
@@ -382,6 +382,7 @@ plugins: # plugin list (sorted by priority)
- syslog # priority: 401
- udp-logger # priority: 400
- file-logger # priority: 399
+ #- clickhouse-logger # priority: 398
Review comment:
This plugin doesn't require any setup and doesn't affect other components, so I think it's safe enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r804426163
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,220 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "clickhouse body: ", data)
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- yaml_config
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r805325388
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,216 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "clickhouse body: ", data)
Review comment:
Let's use `ngx.WARN` to avoid `pattern "[error]" should not match any line in error.log but matches line` error.
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,216 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "clickhouse body: ", data)
+ ngx.say("ok")
Review comment:
Let's check X-ClickHouse-XXX headers too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r806382732
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+apisix:
+ node_listen: 1984
+ admin_key: null
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:10420/clickhouse-logger/test",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":5
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 5: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- grep_error_log_out
+"clickhouse body: INSERT INTO t FORMAT JSONEachRow"
+"clickhouse headers: X-ClickHouse-Key:a"
+"clickhouse headers: X-ClickHouse-User:default"
+"clickhouse headers: X-ClickHouse-Database:default"
Review comment:
It doesn't work.
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+apisix:
+ node_listen: 1984
+ admin_key: null
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:10420/clickhouse-logger/test",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":5
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 5: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- grep_error_log_out
+"clickhouse body: INSERT INTO t FORMAT JSONEachRow"
+"clickhouse headers: X-ClickHouse-Key:a"
+"clickhouse headers: X-ClickHouse-User:default"
+"clickhouse headers: X-ClickHouse-Database:default"
Review comment:
It doesn't work. I don't know why. I think `--- grep_error_log_out` is easier to use.
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,219 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
Review comment:
I thought there was a timer that called this callback function periodically and also if the batch_max_size value was reached. But I added a wait of `--- wait: 5` below, it seem doesn'twork.
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+apisix:
+ node_listen: 1984
+ admin_key: null
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:10420/clickhouse-logger/test",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":5
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 5: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- grep_error_log_out
+"clickhouse body: INSERT INTO t FORMAT JSONEachRow"
+"clickhouse headers: X-ClickHouse-Key:a"
+"clickhouse headers: X-ClickHouse-User:default"
+"clickhouse headers: X-ClickHouse-Database:default"
Review comment:
Use `--- error_log` fialed. https://github.com/apache/apisix/runs/5197064703?check_suite_focus=true#step:14:891
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r794168886
##########
File path: conf/config-default.yaml
##########
@@ -381,6 +381,7 @@ plugins: # plugin list (sorted by priority)
- rocketmq-logger # priority: 402
- syslog # priority: 401
- udp-logger # priority: 400
+ #- clickhouse-logger # priority: 399
Review comment:
sorry, I found file-logger priority is 399.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r795120935
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,182 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
Review comment:
These fields can be injected with
https://github.com/apache/apisix/blob/82d17abf9987528856cbd20ce5f17f2ae0a799ce/apisix/plugins/sls-logger.lua#L49
##########
File path: docs/en/latest/plugins/clickhouse-logger.md
##########
@@ -0,0 +1,158 @@
+---
+title: clickhouse-logger
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Metadata**](#metadata)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`clickhouse-logger` is a plugin which push Log data requests to clickhouse.
+
+## Attributes
+
+| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 |
+| ---------------- | ------- | ------ | ------------- | ------- | ------------------------------------------------ |
+| endpoint_addr | string | required | | | The `clickhouse` endpoint. |
+| database | string | required | | | The DB name to store log. |
+| logtable | string | required | | | The table name. |
+| user | string | required | | |clickhouse user. |
+| password | string | required | | |clickhouse password. |
+| timeout | integer | optional | 3 | [1,...] | Time to keep the connection alive after sending a request. |
+| name | string | optional | "clickhouse logger" | | A unique identifier to identity the logger. |
+| batch_max_size | integer | optional | 100 | [1,...] | Set the maximum number of logs sent in each batch. When the number of logs reaches the set maximum, all logs will be automatically pushed to the clickhouse. |
+| max_retry_count | integer | optional | 0 | [0,...] | Maximum number of retries before removing from the processing pipe line. |
+| retry_delay | integer | optional | 1 | [0,...] | Number of seconds the process execution should be delayed if the execution fails. |
+| ssl_verify | boolean | optional | true | [true,false] | verify ssl. |
+
+## How To Enable
+
+The following is an example of how to enable the `clickhouse-logger` for a specific route.
+
+```shell
+curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
+{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "test",
+ "endpoint_addr": "http://127.0.0.1:8123"
+ }
+ },
+ "upstream": {
+ "type": "roundrobin",
+ "nodes": {
+ "127.0.0.1:1980": 1
+ }
+ },
+ "uri": "/hello"
+}'
+```
+
+## Test Plugin
+
+> success:
+
+```shell
+$ curl -i http://127.0.0.1:9080/hello
+HTTP/1.1 200 OK
+...
+hello, world
+```
+
+## Metadata
+
+| Name | Type | Requirement | Default | Valid | Description |
+| ---------------- | ------- | ----------- | ------------- | ------- | ---------------------------------------------------------------------------------------- |
+| log_format | object | optional | {"host": "$host", "@timestamp": "$time_iso8601", "client_ip": "$remote_addr"} | | Log format declared as key value pair in JSON format. Only string is supported in the `value` part. If the value starts with `$`, it means to get `APISIX` variables or [Nginx variable](http://nginx.org/en/docs/varindex.html). |
Review comment:
Let's update the doc according to the latest http-logger's: https://github.com/apache/apisix/blob/82d17abf9987528856cbd20ce5f17f2ae0a799ce/docs/en/latest/plugins/http-logger.md?plain=1#L90
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r803492459
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,227 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "loggly body: ", data)
+ ngx.log(ngx.ERR, "loggly tags: " .. require("toolkit.json").encode(headers["X-LOGGLY-TAG"]))
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- yaml_config
+apisix:
+ node_listen: 1984
+ admin_key: null
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:8123"
Review comment:
wonderful, it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r803493821
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,227 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
Review comment:
It means that the test framework will setup a fake http server and listen 10420, right ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r802694478
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ ssl_verify = {type = "boolean", default = true},
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 398,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if not port then
+ if url_decoded.scheme == "https" then
+ port = 443
+ else
+ port = 80
+ end
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" and conf.ssl_verify then
+ ok, err = httpc:ssl_handshake(true, host, false)
Review comment:
sorry, my fault.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss edited a comment on pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss edited a comment on pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#issuecomment-1031231204
> hi @zhendongcmss CI is broken
@spacewander @tzssangglass
sorry， I don't know how to fix. I need help, can you tell me how to fix it ?
```
# Failed test 'TEST 4: add plugin on routes - status code ok'
# at /apisix/test-nginx/lib/Test/Nginx/Socket.pm line 948.
# got: '401'
# expected: '200'
# Failed test 'TEST 5: access local server - status code ok'
# at /apisix/test-nginx/lib/Test/Nginx/Socket.pm line 948.
# got: '404'
# expected: '200'
# Failed test 'TEST 5: access local server - response_body - response is expected (repeated req 0, req 0)'
# at /apisix/test-nginx/lib/Test/Nginx/Socket.pm line 1589.
# got: '{"error_msg":"404 Route Not Found"}
# '
# expected: 'opentracing
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r800173837
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,230 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "loggly body: ", data)
+ ngx.log(ngx.ERR, "loggly tags: " .. require("toolkit.json").encode(headers["X-LOGGLY-TAG"]))
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({log_id = "syslog",
+ max_retry_count = 0,
+ retry_delay = 1,
+ buffer_duration = 60,
+ inactive_timeout = 10,
+ batch_max_size = 100,
+ })
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
Review comment:
Let's use a correct password to make CI pass
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,230 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "loggly body: ", data)
+ ngx.log(ngx.ERR, "loggly tags: " .. require("toolkit.json").encode(headers["X-LOGGLY-TAG"]))
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({log_id = "syslog",
+ max_retry_count = 0,
+ retry_delay = 1,
+ buffer_duration = 60,
+ inactive_timeout = 10,
+ batch_max_size = 100,
+ })
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:8123"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:8123",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":5
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
Review comment:
We can remove those sections as it is set globally in:
```
if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
$block->set_value("no_error_log", "[error]");
}
if (!defined $block->request) {
$block->set_value("request", "GET /t");
}
```
##########
File path: conf/config-default.yaml
##########
@@ -382,6 +382,7 @@ plugins: # plugin list (sorted by priority)
- syslog # priority: 401
- udp-logger # priority: 400
- file-logger # priority: 399
+ #- clickhouse-logger # priority: 398
Review comment:
Looks like it's safe to turn on the logger by default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r794158245
##########
File path: conf/config-default.yaml
##########
@@ -381,6 +381,7 @@ plugins: # plugin list (sorted by priority)
- rocketmq-logger # priority: 402
- syslog # priority: 401
- udp-logger # priority: 400
+ #- clickhouse-logger # priority: 399
Review comment:
Priority 410 - 400 all for push log via network, clickhouse-logger is via http request to push log too. It is fewer people using than http-logger，so the priority is low than http-logger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r794146392
##########
File path: docs/en/latest/plugins/clickhouse-logger.md
##########
@@ -0,0 +1,157 @@
+---
+title: clickhouse-logger
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Metadata**](#metadata)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`clickhouse-logger` is a plugin which push Log data requests to clickhouse.
+
+## Attributes
+
+| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 |
+| ---------------- | ------- | ------ | ------------- | ------- | ------------------------------------------------ |
+| endpoint_addr | string | required | | | The `clickhouse` endpoint. |
+| database | string | required | | | The DB name to store log. |
+| logtable | string | required | | | The table name. |
+| user | string | required | | |clickhouse user. |
+| password | string | required | | |clickhouse password. |
+| timeout | integer | optional | 3 | [1,...] | Time to keep the connection alive after sending a request. |
+| name | string | optional | "clickhouse logger" | | A unique identifier to identity the logger. |
+| batch_max_size | integer | optional | 100 | [1,...] | Set the maximum number of logs sent in each batch. When the number of logs reaches the set maximum, all logs will be automatically pushed to the clickhouse. |
+| max_retry_count | integer | optional | 0 | [0,...] | Maximum number of retries before removing from the processing pipe line. |
+| retry_delay | integer | optional | 1 | [0,...] | Number of seconds the process execution should be delayed if the execution fails. |
+
+## How To Enable
+
+The following is an example of how to enable the `click-logger` for a specific route.
Review comment:
```suggestion
The following is an example of how to enable the `clickhoush-logger` for a specific route.
```
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,231 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local batch_processor = require("apisix.utils.batch-processor")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local timer_at = ngx.timer.at
+
+local plugin_name = "clickhouse-logger"
+local stale_timer_running = false
+local buffers = {}
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 399,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
+ ok, err = httpc:ssl_handshake(true, host, false)
+ if not ok then
+ return false, "failed to perform SSL with host[" .. host .. "] "
+ .. "port[" .. tostring(port) .. "] " .. err
+ end
+ end
+ url_decoded.query['database'] = conf.database
+
+ local httpc_res, httpc_err = httpc:request({
+ method = "POST",
+ path = url_decoded.path,
+ query = url_decoded.query,
+ body = "INSERT INTO " .. conf.logtable .." FORMAT JSONEachRow " .. log_message,
+ headers = {
+ ["Host"] = url_decoded.host,
+ ["Content-Type"] = "application/json",
+ ["X-ClickHouse-User"] = conf.user,
+ ["X-ClickHouse-Key"] = conf.password,
+ }
+ })
+
+ if not httpc_res then
+ return false, "error while sending data to [" .. host .. "] port["
+ .. tostring(port) .. "] " .. httpc_err
+ end
+
+ -- some error occurred in the server
+ if httpc_res.status >= 400 then
+ res = false
+ err_msg = "server returned status code[" .. httpc_res.status .. "] host["
+ .. host .. "] port[" .. tostring(port) .. "] "
+ .. "body[" .. httpc_res:read_body() .. "]"
+ end
+
+ return res, err_msg
+end
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, conf: ",
+ core.json.delay_encode(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+function _M.log(conf, ctx)
+ local metadata = plugin.plugin_metadata("http-logger")
+ core.log.info("metadata: ", core.json.delay_encode(metadata))
+ local entry
+
+ if metadata and metadata.value.log_format
+ and core.table.nkeys(metadata.value.log_format) > 0
+ then
+ entry = log_util.get_custom_format_log(ctx, metadata.value.log_format)
+ else
+ entry = log_util.get_full_log(ngx, conf)
+ end
+
+ if not entry.route_id then
Review comment:
Could we remove this special logic for route_id? Actually, we want to remove it from http-logger if we can break the compatibility.
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,194 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({
+ "timeout": 3,
+ "retry_delay": 1,
+ "batch_max_size": 500,
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:8123",
+ "max_retry_count": 1,
+ "name": "clickhouse logger"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:8123"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({
+ log_id = "syslog",
+ max_retry_count = 0,
+ retry_delay = 1,
+ buffer_duration = 60,
+ inactive_timeout = 10,
+ batch_max_size = 100,
+ })
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+value should match only one schema, but matches none
+
+
+
+=== TEST 4: add plugin on routes
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:8123"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "node": {
+ "value": {
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:8123"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ },
+ "key": "/apisix/routes/1"
+ },
+ "action": "set"
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- response_body
+passed
+--- no_error_log
+[error]
+
+
+
+=== TEST 5: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- error_log
+Batch Processor[http logger] successfully processed the entries
Review comment:
We need to use a mock server to check the upload data, like https://github.com/apache/apisix/blob/5f6d16041f64402eb2af2c8380d532e6238c8717/t/plugin/loggly.t#L42-L61
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r794220090
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 398,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
Review comment:
I think it is necessary to add it. For example `Python` request module add it.
```
import requests
response = requests.get(url='https://githu.com', verify=False)
```
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 398,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
Review comment:
I think it is necessary to add it. For example `Python` request module add it.
```
import requests
response = requests.get(url='https://github.com', verify=False)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r805325388
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,216 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "clickhouse body: ", data)
Review comment:
Let's use `ngx.WARN` to avoid `pattern "[error]" should not match any line in error.log but matches line` error.
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,216 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "clickhouse body: ", data)
+ ngx.say("ok")
Review comment:
Let's check X-ClickHouse-XXX headers too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r806421093
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,219 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
Review comment:
```suggestion
"batch_max_size":1,
```
So the report can be triggered in the CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r805471329
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ ssl_verify = {type = "boolean", default = true},
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 398,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if not port then
+ if url_decoded.scheme == "https" then
+ port = 443
+ else
+ port = 80
+ end
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" and conf.ssl_verify then
Review comment:
ok
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,216 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "clickhouse body: ", data)
+ ngx.say("ok")
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
membphis commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r811593762
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
Review comment:
I created an issue, let's fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r794218333
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,231 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local batch_processor = require("apisix.utils.batch-processor")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local timer_at = ngx.timer.at
+
+local plugin_name = "clickhouse-logger"
+local stale_timer_running = false
+local buffers = {}
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 399,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
+ ok, err = httpc:ssl_handshake(true, host, false)
+ if not ok then
+ return false, "failed to perform SSL with host[" .. host .. "] "
+ .. "port[" .. tostring(port) .. "] " .. err
+ end
+ end
+ url_decoded.query['database'] = conf.database
+
+ local httpc_res, httpc_err = httpc:request({
+ method = "POST",
+ path = url_decoded.path,
+ query = url_decoded.query,
+ body = "INSERT INTO " .. conf.logtable .." FORMAT JSONEachRow " .. log_message,
+ headers = {
+ ["Host"] = url_decoded.host,
+ ["Content-Type"] = "application/json",
+ ["X-ClickHouse-User"] = conf.user,
+ ["X-ClickHouse-Key"] = conf.password,
+ }
+ })
+
+ if not httpc_res then
+ return false, "error while sending data to [" .. host .. "] port["
+ .. tostring(port) .. "] " .. httpc_err
+ end
+
+ -- some error occurred in the server
+ if httpc_res.status >= 400 then
+ res = false
+ err_msg = "server returned status code[" .. httpc_res.status .. "] host["
+ .. host .. "] port[" .. tostring(port) .. "] "
+ .. "body[" .. httpc_res:read_body() .. "]"
+ end
+
+ return res, err_msg
+end
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, conf: ",
+ core.json.delay_encode(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+function _M.log(conf, ctx)
+ local metadata = plugin.plugin_metadata("http-logger")
Review comment:
2 vs 1，I will add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#issuecomment-1031192234
hi @zhendongcmss CI is broken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r803453401
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,227 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /loggly/bulk/tok/tag/bulk {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.ERR, "loggly body: ", data)
+ ngx.log(ngx.ERR, "loggly tags: " .. require("toolkit.json").encode(headers["X-LOGGLY-TAG"]))
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:8123"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- yaml_config
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- yaml_config
+apisix:
+ node_listen: 1984
+ admin_key: null
+plugins:
+ - clickhouse-logger
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:8123"
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r806645448
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+apisix:
+ node_listen: 1984
+ admin_key: null
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:10420/clickhouse-logger/test",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":5
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 5: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- grep_error_log_out
+"clickhouse body: INSERT INTO t FORMAT JSONEachRow"
+"clickhouse headers: X-ClickHouse-Key:a"
+"clickhouse headers: X-ClickHouse-User:default"
+"clickhouse headers: X-ClickHouse-Database:default"
Review comment:
Use `--- error_log` fialed. https://github.com/apache/apisix/runs/5197064703?check_suite_focus=true#step:14:891
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r795132939
##########
File path: docs/en/latest/plugins/clickhouse-logger.md
##########
@@ -0,0 +1,158 @@
+---
+title: clickhouse-logger
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Metadata**](#metadata)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`clickhouse-logger` is a plugin which push Log data requests to clickhouse.
+
+## Attributes
+
+| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 |
+| ---------------- | ------- | ------ | ------------- | ------- | ------------------------------------------------ |
+| endpoint_addr | string | required | | | The `clickhouse` endpoint. |
+| database | string | required | | | The DB name to store log. |
+| logtable | string | required | | | The table name. |
+| user | string | required | | |clickhouse user. |
+| password | string | required | | |clickhouse password. |
+| timeout | integer | optional | 3 | [1,...] | Time to keep the connection alive after sending a request. |
+| name | string | optional | "clickhouse logger" | | A unique identifier to identity the logger. |
+| batch_max_size | integer | optional | 100 | [1,...] | Set the maximum number of logs sent in each batch. When the number of logs reaches the set maximum, all logs will be automatically pushed to the clickhouse. |
+| max_retry_count | integer | optional | 0 | [0,...] | Maximum number of retries before removing from the processing pipe line. |
+| retry_delay | integer | optional | 1 | [0,...] | Number of seconds the process execution should be delayed if the execution fails. |
+| ssl_verify | boolean | optional | true | [true,false] | verify ssl. |
+
+## How To Enable
+
+The following is an example of how to enable the `clickhouse-logger` for a specific route.
+
+```shell
+curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
+{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "test",
+ "endpoint_addr": "http://127.0.0.1:8123"
+ }
+ },
+ "upstream": {
+ "type": "roundrobin",
+ "nodes": {
+ "127.0.0.1:1980": 1
+ }
+ },
+ "uri": "/hello"
+}'
+```
+
+## Test Plugin
+
+> success:
+
+```shell
+$ curl -i http://127.0.0.1:9080/hello
+HTTP/1.1 200 OK
+...
+hello, world
+```
+
+## Metadata
+
+| Name | Type | Requirement | Default | Valid | Description |
+| ---------------- | ------- | ----------- | ------------- | ------- | ---------------------------------------------------------------------------------------- |
+| log_format | object | optional | {"host": "$host", "@timestamp": "$time_iso8601", "client_ip": "$remote_addr"} | | Log format declared as key value pair in JSON format. Only string is supported in the `value` part. If the value starts with `$`, it means to get `APISIX` variables or [Nginx variable](http://nginx.org/en/docs/varindex.html). |
Review comment:
will add.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r794158591
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,231 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local batch_processor = require("apisix.utils.batch-processor")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local timer_at = ngx.timer.at
+
+local plugin_name = "clickhouse-logger"
+local stale_timer_running = false
+local buffers = {}
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ max_retry_count = {type = "integer", minimum = 0, default = 0},
+ retry_delay = {type = "integer", minimum = 0, default = 1},
+ batch_max_size = {type = "integer", minimum = 1, default = 100}
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 399,
+ name = plugin_name,
+ schema = schema,
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if ((not port) and url_decoded.scheme == "https") then
+ port = 443
+ elseif not port then
+ port = 80
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" then
+ ok, err = httpc:ssl_handshake(true, host, false)
+ if not ok then
+ return false, "failed to perform SSL with host[" .. host .. "] "
+ .. "port[" .. tostring(port) .. "] " .. err
+ end
+ end
+ url_decoded.query['database'] = conf.database
+
+ local httpc_res, httpc_err = httpc:request({
+ method = "POST",
+ path = url_decoded.path,
+ query = url_decoded.query,
+ body = "INSERT INTO " .. conf.logtable .." FORMAT JSONEachRow " .. log_message,
+ headers = {
+ ["Host"] = url_decoded.host,
+ ["Content-Type"] = "application/json",
+ ["X-ClickHouse-User"] = conf.user,
+ ["X-ClickHouse-Key"] = conf.password,
+ }
+ })
+
+ if not httpc_res then
+ return false, "error while sending data to [" .. host .. "] port["
+ .. tostring(port) .. "] " .. httpc_err
+ end
+
+ -- some error occurred in the server
+ if httpc_res.status >= 400 then
+ res = false
+ err_msg = "server returned status code[" .. httpc_res.status .. "] host["
+ .. host .. "] port[" .. tostring(port) .. "] "
+ .. "body[" .. httpc_res:read_body() .. "]"
+ end
+
+ return res, err_msg
+end
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, conf: ",
+ core.json.delay_encode(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
Review comment:
Try to refine it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#issuecomment-1041469519
@zhendongcmss
Let's add the logger in https://github.com/apache/apisix/blob/master/README.md!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] juzhiyuan commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
juzhiyuan commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r808727629
##########
File path: docs/en/latest/plugins/clickhouse-logger.md
##########
@@ -0,0 +1,148 @@
+---
+title: clickhouse-logger
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Metadata**](#metadata)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`clickhouse-logger` is a plugin which push Log data requests to clickhouse.
+
+## Attributes
+
+| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 |
Review comment:
Hi @zhendongcmss, here have some Chinese 😄 Would you like to translate them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#issuecomment-1031231204
> hi @zhendongcmss CI is broken
@spacewander @tzssangglass
There are many test cases raised those errors. I don't know how to fix. I need help, can you tell me how to fix it ?
```
# Failed test 'TEST 4: add plugin on routes - status code ok'
# at /apisix/test-nginx/lib/Test/Nginx/Socket.pm line 948.
# got: '401'
# expected: '200'
# Failed test 'TEST 5: access local server - status code ok'
# at /apisix/test-nginx/lib/Test/Nginx/Socket.pm line 948.
# got: '404'
# expected: '200'
# Failed test 'TEST 5: access local server - response_body - response is expected (repeated req 0, req 0)'
# at /apisix/test-nginx/lib/Test/Nginx/Socket.pm line 1589.
# got: '{"error_msg":"404 Route Not Found"}
# '
# expected: 'opentracing
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r801208708
##########
File path: docs/zh/latest/plugins/clickhouse-logger.md
##########
@@ -0,0 +1,146 @@
+---
+title: clickhouse-logger
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## 目录
+
+- [**定义**](#定义)
+- [**属性列表**](#属性列表)
+- [**如何开启**](#如何开启)
+- [**测试插件**](#测试插件)
+- [**插件元数据设置**](#插件元数据设置)
+- [**禁用插件**](#禁用插件)
+
+## 定义
+
+`clickhouse-logger` 是一个插件，可将Log数据请求推送到clickhouse服务器。
+
+## 属性列表
+
+| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 |
+| ---------------- | ------- | ------ | ------------- | ------- | ------------------------------------------------ |
+| endpoint_addr | string | 必须 | | | `clickhouse` 服务器的 endpoint。 |
+| database | string | 必须 | | | 使用的数据库。 |
+| logtable | string | 必须 | | | 写入的表名 。 |
+| user | string | 必须 | | |clickhouse的用户。 |
+| password | string | 必须 | | |clickhouse的密码 。 |
Review comment:
```suggestion
| user | string | 必须 | | | clickhouse的用户。 |
| password | string | 必须 | | | clickhouse的密码 。 |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r801643518
##########
File path: apisix/plugins/clickhouse-logger.lua
##########
@@ -0,0 +1,179 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local bp_manager_mod = require("apisix.utils.batch-processor-manager")
+local log_util = require("apisix.utils.log-util")
+local core = require("apisix.core")
+local http = require("resty.http")
+local url = require("net.url")
+local plugin = require("apisix.plugin")
+
+local ngx = ngx
+local tostring = tostring
+
+local plugin_name = "clickhouse-logger"
+local batch_processor_manager = bp_manager_mod.new(plugin_name)
+
+local schema = {
+ type = "object",
+ properties = {
+ endpoint_addr = core.schema.uri_def,
+ user = {type = "string", default = ""},
+ password = {type = "string", default = ""},
+ database = {type = "string", default = ""},
+ logtable = {type = "string", default = ""},
+ timeout = {type = "integer", minimum = 1, default = 3},
+ name = {type = "string", default = "clickhouse logger"},
+ ssl_verify = {type = "boolean", default = true},
+ },
+ required = {"endpoint_addr", "user", "password", "database", "logtable"}
+}
+
+
+local metadata_schema = {
+ type = "object",
+ properties = {
+ log_format = log_util.metadata_schema_log_format,
+ },
+}
+
+
+local _M = {
+ version = 0.1,
+ priority = 398,
+ name = plugin_name,
+ schema = batch_processor_manager:wrap_schema(schema),
+ metadata_schema = metadata_schema,
+}
+
+
+function _M.check_schema(conf, schema_type)
+ if schema_type == core.schema.TYPE_METADATA then
+ return core.schema.check(metadata_schema, conf)
+ end
+ return core.schema.check(schema, conf)
+end
+
+
+local function send_http_data(conf, log_message)
+ local err_msg
+ local res = true
+ local url_decoded = url.parse(conf.endpoint_addr)
+ local host = url_decoded.host
+ local port = url_decoded.port
+
+ core.log.info("sending a batch logs to ", conf.endpoint_addr)
+
+ if not port then
+ if url_decoded.scheme == "https" then
+ port = 443
+ else
+ port = 80
+ end
+ end
+
+ local httpc = http.new()
+ httpc:set_timeout(conf.timeout * 1000)
+ local ok, err = httpc:connect(host, port)
+
+ if not ok then
+ return false, "failed to connect to host[" .. host .. "] port["
+ .. tostring(port) .. "] " .. err
+ end
+
+ if url_decoded.scheme == "https" and conf.ssl_verify then
+ ok, err = httpc:ssl_handshake(true, host, false)
Review comment:
Call this function means ssl = true.
https://github.com/ledgetech/lua-resty-http/blob/master/lib/resty/http.lua#L1060
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] zhendongcmss commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
zhendongcmss commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r805671193
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+apisix:
+ node_listen: 1984
+ admin_key: null
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r805543150
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+apisix:
+ node_listen: 1984
+ admin_key: null
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:10420/clickhouse-logger/test",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":5
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 5: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- grep_error_log_out
+"clickhouse body: INSERT INTO t FORMAT JSONEachRow"
+"clickhouse headers: X-ClickHouse-Key:a"
+"clickhouse headers: X-ClickHouse-User:default"
+"clickhouse headers: X-ClickHouse-Database:default"
Review comment:
I have reconsidered the test.
What about using error_log like https://github.com/apache/apisix/blob/537a8da24d723caa5c85561d36b3b931ce1f1410/t/plugin/loggly.t#L572?
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+apisix:
+ node_listen: 1984
+ admin_key: null
Review comment:
We can remove this now as we already use the default `--- yaml_config` configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r805543150
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+apisix:
+ node_listen: 1984
+ admin_key: null
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
+ "max_retry_count":0,
+ "retry_delay":1,
+ "ssl_verify":true,
+ "endpoint_addr":"http://127.0.0.1:10420/clickhouse-logger/test",
+ "password":"a",
+ "buffer_duration":60,
+ "timeout":3,
+ "user":"default",
+ "name":"clickhouse-logger",
+ "database":"default",
+ "logtable":"t",
+ "inactive_timeout":5
+ }
+ },
+ "id":"1"
+ },
+ "key":"/apisix/routes/1"
+ }
+ }]]
+ )
+
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 5: access local server
+--- request
+GET /opentracing
+--- response_body
+opentracing
+--- grep_error_log_out
+"clickhouse body: INSERT INTO t FORMAT JSONEachRow"
+"clickhouse headers: X-ClickHouse-Key:a"
+"clickhouse headers: X-ClickHouse-User:default"
+"clickhouse headers: X-ClickHouse-Database:default"
Review comment:
I have reconsidered the test.
What about using error_log like https://github.com/apache/apisix/blob/537a8da24d723caa5c85561d36b3b931ce1f1410/t/plugin/loggly.t#L572-L574?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r806421093
##########
File path: t/plugin/clickhouse-logger.t
##########
@@ -0,0 +1,219 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+log_level("info");
+repeat_each(1);
+no_long_string();
+no_root_location();
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+
+ if (!defined $block->request) {
+ $block->set_value("request", "GET /t");
+ }
+
+ my $http_config = $block->http_config // <<_EOC_;
+ server {
+ listen 10420;
+ location /clickhouse-logger/test {
+ content_by_lua_block {
+ ngx.req.read_body()
+ local data = ngx.req.get_body_data()
+ local headers = ngx.req.get_headers()
+ ngx.log(ngx.WARN, "clickhouse body: ", data)
+ for k, v in pairs(headers) do
+ ngx.log(ngx.WARN, "clickhouse headers: " .. k .. ":" .. v)
+ end
+ ngx.say("ok")
+ }
+ }
+ }
+_EOC_
+
+ $block->set_value("http_config", $http_config);
+});
+
+run_tests();
+
+__DATA__
+
+=== TEST 1: Full configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({timeout = 3,
+ retry_delay = 1,
+ batch_max_size = 500,
+ user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test",
+ max_retry_count = 1,
+ name = "clickhouse logger",
+ ssl_verify = false
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 2: Basic configuration verification
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t",
+ endpoint_addr = "http://127.0.0.1:10420/clickhouse-logger/test"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+passed
+
+
+
+=== TEST 3: auth configure undefined
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.clickhouse-logger")
+ local ok, err = plugin.check_schema({user = "default",
+ password = "a",
+ database = "default",
+ logtable = "t"
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- response_body
+property "endpoint_addr" is required
+
+
+
+=== TEST 4: add plugin on routes
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "clickhouse-logger": {
+ "user": "default",
+ "password": "a",
+ "database": "default",
+ "logtable": "t",
+ "endpoint_addr": "http://127.0.0.1:10420/clickhouse-logger/test"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1982": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/opentracing"
+ }]],
+ [[{
+ "action":"set",
+ "node":{
+ "value":{
+ "uri":"/opentracing",
+ "upstream":{
+ "scheme":"http",
+ "nodes":{
+ "127.0.0.1:1982":1
+ }
+ },
+ "plugins":{
+ "clickhouse-logger":{
+ "batch_max_size":1000,
Review comment:
```suggestion
"batch_max_size":1,
```
So the report can be triggered in the CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] juzhiyuan commented on a change in pull request #6215: feat: clickhouse logger
Posted by GitBox <gi...@apache.org>.
juzhiyuan commented on a change in pull request #6215:
URL: https://github.com/apache/apisix/pull/6215#discussion_r811559195
##########
File path: docs/en/latest/plugins/clickhouse-logger.md
##########
@@ -0,0 +1,148 @@
+---
+title: clickhouse-logger
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Metadata**](#metadata)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`clickhouse-logger` is a plugin which push Log data requests to clickhouse.
+
+## Attributes
+
+| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 |
Review comment:
Hi, @yzeng25, do you have interests to do this? 🤗
##########
File path: docs/en/latest/plugins/clickhouse-logger.md
##########
@@ -0,0 +1,148 @@
+---
+title: clickhouse-logger
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Metadata**](#metadata)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`clickhouse-logger` is a plugin which push Log data requests to clickhouse.
+
+## Attributes
+
+| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 |
Review comment:
Hi, @yzeng25, do you have interest to do this? 🤗
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org