You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2020/03/21 16:42:44 UTC
[GitHub] [incubator-apisix] Akayeshmantha opened a new pull request #1312:
Kafka Logger.
Akayeshmantha opened a new pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312
Adding kafka plugin.
Issues resolved
Fix #1259
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] Akayeshmantha commented on issue #1312: Kafka
Logger.
Posted by GitBox <gi...@apache.org>.
Akayeshmantha commented on issue #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#issuecomment-602070573
@moonming @membphis can we review the PR ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] membphis commented on a change in pull request
#1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
membphis commented on a change in pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#discussion_r396169847
##########
File path: t/plugin/kafka-logger.t
##########
@@ -0,0 +1,240 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+use t::APISIX 'no_plan';
+
+repeat_each(1);
+no_long_string();
+no_root_location();
+run_tests;
+
+__DATA__
+=== TEST 1: sanity
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.kafka-logger")
+ local ok, err = plugin.check_schema({
+ kafka_topic = "test",
+ key = "key1",
+ broker_list = {
+ ["127.0.0.1"] = 3
+ }
+ })
+ if not ok then
+ ngx.say(err)
+ end
+ ngx.say("done")
+ }
+ }
+--- request
+GET /t
+--- response_body
+done
+--- no_error_log
+[error]
+
+=== TEST 2: missing broker list
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.kafka-logger")
+ local ok, err = plugin.check_schema({kafka_topic = "test", key= "key1"})
+ if not ok then
+ ngx.say(err)
+ end
+ ngx.say("done")
+ }
+ }
+--- request
+GET /t
+--- response_body
+property "broker_list" is required
+done
+--- no_error_log
+[error]
+
+=== TEST 3: wrong type of string
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.kafka-logger")
+ local ok, err = plugin.check_schema({
+ broker_list = {
+ ["127.0.0.1"] = 3000
+ },
+ timeout = "10",
+ kafka_topic ="test",
+ key= "key1"
+ })
+ if not ok then
+ ngx.say(err)
+ end
+ ngx.say("done")
+ }
+ }
+--- request
+GET /t
+--- response_body
+property "timeout" validation failed: wrong type: expected integer, got string
+done
+--- no_error_log
+[error]
+
+=== TEST 4: add plugin
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "kafka-logger": {
+ "broker_list" :
+ {
+ "127.0.0.1":9092
+ },
+ "kafka_topic" : "test2",
+ "key" : "key1"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]],
+ [[{
+ "node": {
+ "value": {
+ "plugins": {
+ "kafka-logger": {
+ "broker_list" :
+ {
+ "127.0.0.1":9092
+ },
+ "kafka_topic" : "test2",
+ "key" : "key1"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ },
+ "key": "/apisix/routes/1"
+ },
+ "action": "set"
+ }]]
+ )
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- response_body
+passed
+--- no_error_log
+[error]
+
+=== TEST 5: access
+--- request
+GET /hello
+--- response_body
+hello world
+--- no_error_log
+[error]
+--- wait: 0.2
+
+=== TEST 6: error log
Review comment:
use too `reindex` to check your test case style.
https://github.com/apache/incubator-apisix/blob/master/Contributing.md#check-code-style-and-test-case-style
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] moonming merged pull request #1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
moonming merged pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] Akayeshmantha commented on a change in pull
request #1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
Akayeshmantha commented on a change in pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#discussion_r396064153
##########
File path: .travis.yml
##########
@@ -56,6 +56,13 @@ before_cache:
- brew cleanup
before_install:
+ - docker pull bitnami/zookeeper:3.6.0
Review comment:
Sure.Since all tests passed which means we are currently skipping testcases run on osx?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] Akayeshmantha commented on a change in pull
request #1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
Akayeshmantha commented on a change in pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#discussion_r396122745
##########
File path: .travis.yml
##########
@@ -56,6 +56,13 @@ before_cache:
- brew cleanup
before_install:
+ - docker pull bitnami/zookeeper:3.6.0
Review comment:
@membphis
I hope in the future we will be not running unit tests on
OSNAME=linux_apisix_master_luarocks
OSNAME=linux_apisix_current_luarocks
Which I dont see currently we are running unit tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] Akayeshmantha commented on a change in pull
request #1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
Akayeshmantha commented on a change in pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#discussion_r396385631
##########
File path: t/plugin/kafka-logger.t
##########
@@ -0,0 +1,240 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+use t::APISIX 'no_plan';
+
+repeat_each(1);
+no_long_string();
+no_root_location();
+run_tests;
+
+__DATA__
+=== TEST 1: sanity
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.kafka-logger")
+ local ok, err = plugin.check_schema({
+ kafka_topic = "test",
+ key = "key1",
+ broker_list = {
+ ["127.0.0.1"] = 3
+ }
+ })
+ if not ok then
+ ngx.say(err)
+ end
+ ngx.say("done")
+ }
+ }
+--- request
+GET /t
+--- response_body
+done
+--- no_error_log
+[error]
+
+=== TEST 2: missing broker list
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.kafka-logger")
+ local ok, err = plugin.check_schema({kafka_topic = "test", key= "key1"})
+ if not ok then
+ ngx.say(err)
+ end
+ ngx.say("done")
+ }
+ }
+--- request
+GET /t
+--- response_body
+property "broker_list" is required
+done
+--- no_error_log
+[error]
+
+=== TEST 3: wrong type of string
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.kafka-logger")
+ local ok, err = plugin.check_schema({
+ broker_list = {
+ ["127.0.0.1"] = 3000
+ },
+ timeout = "10",
+ kafka_topic ="test",
+ key= "key1"
+ })
+ if not ok then
+ ngx.say(err)
+ end
+ ngx.say("done")
+ }
+ }
+--- request
+GET /t
+--- response_body
+property "timeout" validation failed: wrong type: expected integer, got string
+done
+--- no_error_log
+[error]
+
+=== TEST 4: add plugin
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code, body = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "plugins": {
+ "kafka-logger": {
+ "broker_list" :
+ {
+ "127.0.0.1":9092
+ },
+ "kafka_topic" : "test2",
+ "key" : "key1"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ }]],
+ [[{
+ "node": {
+ "value": {
+ "plugins": {
+ "kafka-logger": {
+ "broker_list" :
+ {
+ "127.0.0.1":9092
+ },
+ "kafka_topic" : "test2",
+ "key" : "key1"
+ }
+ },
+ "upstream": {
+ "nodes": {
+ "127.0.0.1:1980": 1
+ },
+ "type": "roundrobin"
+ },
+ "uri": "/hello"
+ },
+ "key": "/apisix/routes/1"
+ },
+ "action": "set"
+ }]]
+ )
+ if code >= 300 then
+ ngx.status = code
+ end
+ ngx.say(body)
+ }
+ }
+--- request
+GET /t
+--- response_body
+passed
+--- no_error_log
+[error]
+
+=== TEST 5: access
+--- request
+GET /hello
+--- response_body
+hello world
+--- no_error_log
+[error]
+--- wait: 0.2
+
+=== TEST 6: error log
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] Akayeshmantha commented on a change in pull
request #1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
Akayeshmantha commented on a change in pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#discussion_r396144585
##########
File path: .travis.yml
##########
@@ -56,6 +56,13 @@ before_cache:
- brew cleanup
before_install:
+ - docker pull bitnami/zookeeper:3.6.0
Review comment:
@membphis updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] Akayeshmantha commented on issue #1312: Kafka
Logger.
Posted by GitBox <gi...@apache.org>.
Akayeshmantha commented on issue #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#issuecomment-603233395
@membphis done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] Akayeshmantha commented on a change in pull
request #1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
Akayeshmantha commented on a change in pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#discussion_r396385561
##########
File path: lua/apisix/plugins/kafka-logger.lua
##########
@@ -0,0 +1,106 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+local core = require("apisix.core")
+local log_util = require("apisix.utils.log-util")
+local producer = require ("resty.kafka.producer")
+local pairs = pairs
+local type = type
+local table = table
+
+local plugin_name = "kafka-logger"
+local ngx = ngx
+
+local timer_at = ngx.timer.at
+
+local schema = {
+ type = "object",
+ properties = {
+ broker_list = {
+ type = "object"
+ },
+ timeout = { -- timeout in milliseconds
+ type = "integer", minimum = 1, default= 2000
+ },
+ kafka_topic = {type = "string"},
+ async = {type = "boolean", default = false},
+ key = {type = "string"},
+ max_retry = {type = "integer", minimum = 0 , default = 3},
+ },
+ required = {"broker_list", "kafka_topic", "key"}
+}
+
+local _M = {
+ version = 0.1,
+ priority = 403,
+ name = plugin_name,
+ schema = schema,
+}
+
+function _M.check_schema(conf)
+ return core.schema.check(schema, conf)
+end
+
+local function log(premature, conf, log_message)
+ if premature then
+ return
+ end
+
+ if core.table.nkeys(conf.broker_list) == 0 then
+ core.log.error("failed to identify the broker specified")
+ end
+
+ local broker_list = {}
+ local broker_config = {}
+
+ for host, port in pairs(conf.broker_list) do
+ if type(host) == 'string'
+ and type(port) == 'number' then
+
+ local broker = {
+ host = host, port = port
+ }
+ table.insert(broker_list,broker)
+ end
+ end
+
+ broker_config["request_timeout"] = conf.timeout
+ broker_config["max_retry"] = conf.max_retry
+
+ --Async producers will queue logs and push them when the buffer exceeds.
+ if conf.async then
+ broker_config["producer_type"] = "async"
+ end
+
+ local prod, err = producer:new(broker_list,broker_config)
+ if err then
+ core.log.error("failed to identify the broker specified", err)
+
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] membphis commented on a change in pull request
#1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
membphis commented on a change in pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#discussion_r396170025
##########
File path: lua/apisix/plugins/kafka-logger.lua
##########
@@ -0,0 +1,106 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+local core = require("apisix.core")
+local log_util = require("apisix.utils.log-util")
+local producer = require ("resty.kafka.producer")
+local pairs = pairs
+local type = type
+local table = table
+
+local plugin_name = "kafka-logger"
+local ngx = ngx
+
+local timer_at = ngx.timer.at
+
+local schema = {
+ type = "object",
+ properties = {
+ broker_list = {
+ type = "object"
+ },
+ timeout = { -- timeout in milliseconds
+ type = "integer", minimum = 1, default= 2000
+ },
+ kafka_topic = {type = "string"},
+ async = {type = "boolean", default = false},
+ key = {type = "string"},
+ max_retry = {type = "integer", minimum = 0 , default = 3},
+ },
+ required = {"broker_list", "kafka_topic", "key"}
+}
+
+local _M = {
+ version = 0.1,
+ priority = 403,
+ name = plugin_name,
+ schema = schema,
+}
+
+function _M.check_schema(conf)
+ return core.schema.check(schema, conf)
+end
+
+local function log(premature, conf, log_message)
+ if premature then
+ return
+ end
+
+ if core.table.nkeys(conf.broker_list) == 0 then
+ core.log.error("failed to identify the broker specified")
+ end
+
+ local broker_list = {}
+ local broker_config = {}
+
+ for host, port in pairs(conf.broker_list) do
+ if type(host) == 'string'
+ and type(port) == 'number' then
+
+ local broker = {
+ host = host, port = port
+ }
+ table.insert(broker_list,broker)
+ end
+ end
+
+ broker_config["request_timeout"] = conf.timeout
+ broker_config["max_retry"] = conf.max_retry
+
+ --Async producers will queue logs and push them when the buffer exceeds.
+ if conf.async then
+ broker_config["producer_type"] = "async"
+ end
+
+ local prod, err = producer:new(broker_list,broker_config)
+ if err then
+ core.log.error("failed to identify the broker specified", err)
+
+ return
+ end
+
+ local ok, err = prod:send(conf.kafka_topic, conf.key, log_message)
+ if not ok then
+ core.log.error("failed to send data to Kafka topic", err)
+ end
+
Review comment:
remove this blank line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] membphis commented on issue #1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#issuecomment-603604329
@Akayeshmantha many thx
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] Akayeshmantha commented on a change in pull
request #1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
Akayeshmantha commented on a change in pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#discussion_r396364390
##########
File path: .travis/osx_openresty_runner.sh
##########
@@ -33,6 +33,26 @@ before_install() {
sudo cpanm --notest Test::Nginx >build.log 2>&1 || (cat build.log && exit 1)
export_or_prefix
luarocks install --lua-dir=${OPENRESTY_PREFIX}/luajit luacov-coveralls --local --tree=deps
+
+ # spin up kafka cluster for tests (1 zookeper and 1 kafka instance)
Review comment:
oh okays. I didnt know that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] membphis commented on a change in pull request
#1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
membphis commented on a change in pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#discussion_r396169991
##########
File path: lua/apisix/plugins/kafka-logger.lua
##########
@@ -0,0 +1,106 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+local core = require("apisix.core")
+local log_util = require("apisix.utils.log-util")
+local producer = require ("resty.kafka.producer")
+local pairs = pairs
+local type = type
+local table = table
+
+local plugin_name = "kafka-logger"
+local ngx = ngx
+
+local timer_at = ngx.timer.at
+
+local schema = {
+ type = "object",
+ properties = {
+ broker_list = {
+ type = "object"
+ },
+ timeout = { -- timeout in milliseconds
+ type = "integer", minimum = 1, default= 2000
+ },
+ kafka_topic = {type = "string"},
+ async = {type = "boolean", default = false},
+ key = {type = "string"},
+ max_retry = {type = "integer", minimum = 0 , default = 3},
+ },
+ required = {"broker_list", "kafka_topic", "key"}
+}
+
+local _M = {
+ version = 0.1,
+ priority = 403,
+ name = plugin_name,
+ schema = schema,
+}
+
+function _M.check_schema(conf)
+ return core.schema.check(schema, conf)
+end
+
+local function log(premature, conf, log_message)
+ if premature then
+ return
+ end
+
+ if core.table.nkeys(conf.broker_list) == 0 then
+ core.log.error("failed to identify the broker specified")
+ end
+
+ local broker_list = {}
+ local broker_config = {}
+
+ for host, port in pairs(conf.broker_list) do
+ if type(host) == 'string'
+ and type(port) == 'number' then
+
+ local broker = {
+ host = host, port = port
+ }
+ table.insert(broker_list,broker)
+ end
+ end
+
+ broker_config["request_timeout"] = conf.timeout
+ broker_config["max_retry"] = conf.max_retry
+
+ --Async producers will queue logs and push them when the buffer exceeds.
+ if conf.async then
+ broker_config["producer_type"] = "async"
+ end
+
+ local prod, err = producer:new(broker_list,broker_config)
+ if err then
+ core.log.error("failed to identify the broker specified", err)
+
Review comment:
need to remove this blank line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] Akayeshmantha commented on issue #1312: Kafka
Logger.
Posted by GitBox <gi...@apache.org>.
Akayeshmantha commented on issue #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#issuecomment-603466683
Added @membphis. And rebased with the master :smiley:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] Akayeshmantha commented on a change in pull
request #1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
Akayeshmantha commented on a change in pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#discussion_r396385434
##########
File path: .travis/osx_openresty_runner.sh
##########
@@ -33,6 +33,26 @@ before_install() {
sudo cpanm --notest Test::Nginx >build.log 2>&1 || (cat build.log && exit 1)
export_or_prefix
luarocks install --lua-dir=${OPENRESTY_PREFIX}/luajit luacov-coveralls --local --tree=deps
+
+ # spin up kafka cluster for tests (1 zookeper and 1 kafka instance)
Review comment:
@membphis done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] Akayeshmantha commented on a change in pull
request #1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
Akayeshmantha commented on a change in pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#discussion_r396144585
##########
File path: .travis.yml
##########
@@ -56,6 +56,13 @@ before_cache:
- brew cleanup
before_install:
+ - docker pull bitnami/zookeeper:3.6.0
Review comment:
@membphis updated
sorry I had to push few times since I don't have a mac with me
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] membphis commented on a change in pull request
#1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
membphis commented on a change in pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#discussion_r396168881
##########
File path: .travis/osx_openresty_runner.sh
##########
@@ -33,6 +33,26 @@ before_install() {
sudo cpanm --notest Test::Nginx >build.log 2>&1 || (cat build.log && exit 1)
export_or_prefix
luarocks install --lua-dir=${OPENRESTY_PREFIX}/luajit luacov-coveralls --local --tree=deps
+
+ # spin up kafka cluster for tests (1 zookeper and 1 kafka instance)
Review comment:
we do not need to install the `zookeeper`, `kafka`.
only run the mini test cases(t/admin/*.t) for OSX system.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] membphis commented on a change in pull request
#1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
membphis commented on a change in pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#discussion_r396084353
##########
File path: .travis.yml
##########
@@ -56,6 +56,13 @@ before_cache:
- brew cleanup
before_install:
+ - docker pull bitnami/zookeeper:3.6.0
Review comment:
After this PR was merged, we started running test cases for the OSX system in each PR.
https://github.com/apache/incubator-apisix/pull/1314/files
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] Akayeshmantha commented on a change in pull
request #1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
Akayeshmantha commented on a change in pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#discussion_r396385703
##########
File path: lua/apisix/plugins/kafka-logger.lua
##########
@@ -0,0 +1,106 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+local core = require("apisix.core")
+local log_util = require("apisix.utils.log-util")
+local producer = require ("resty.kafka.producer")
+local pairs = pairs
+local type = type
+local table = table
+
+local plugin_name = "kafka-logger"
+local ngx = ngx
+
+local timer_at = ngx.timer.at
+
+local schema = {
+ type = "object",
+ properties = {
+ broker_list = {
+ type = "object"
+ },
+ timeout = { -- timeout in milliseconds
+ type = "integer", minimum = 1, default= 2000
+ },
+ kafka_topic = {type = "string"},
+ async = {type = "boolean", default = false},
+ key = {type = "string"},
+ max_retry = {type = "integer", minimum = 0 , default = 3},
+ },
+ required = {"broker_list", "kafka_topic", "key"}
+}
+
+local _M = {
+ version = 0.1,
+ priority = 403,
+ name = plugin_name,
+ schema = schema,
+}
+
+function _M.check_schema(conf)
+ return core.schema.check(schema, conf)
+end
+
+local function log(premature, conf, log_message)
+ if premature then
+ return
+ end
+
+ if core.table.nkeys(conf.broker_list) == 0 then
+ core.log.error("failed to identify the broker specified")
+ end
+
+ local broker_list = {}
+ local broker_config = {}
+
+ for host, port in pairs(conf.broker_list) do
+ if type(host) == 'string'
+ and type(port) == 'number' then
+
+ local broker = {
+ host = host, port = port
+ }
+ table.insert(broker_list,broker)
+ end
+ end
+
+ broker_config["request_timeout"] = conf.timeout
+ broker_config["max_retry"] = conf.max_retry
+
+ --Async producers will queue logs and push them when the buffer exceeds.
+ if conf.async then
+ broker_config["producer_type"] = "async"
+ end
+
+ local prod, err = producer:new(broker_list,broker_config)
+ if err then
+ core.log.error("failed to identify the broker specified", err)
+
+ return
+ end
+
+ local ok, err = prod:send(conf.kafka_topic, conf.key, log_message)
+ if not ok then
+ core.log.error("failed to send data to Kafka topic", err)
+ end
+
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] membphis commented on issue #1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#issuecomment-602309054
@Akayeshmantha please rebase your branch with `master` branch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] membphis commented on a change in pull request
#1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
membphis commented on a change in pull request #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#discussion_r396039110
##########
File path: .travis.yml
##########
@@ -56,6 +56,13 @@ before_cache:
- brew cleanup
before_install:
+ - docker pull bitnami/zookeeper:3.6.0
Review comment:
Not all CI environments support docker, such as OSX.
So we should complete the installation in a specific system script.
Here is an example: https://github.com/apache/incubator-apisix/blob/master/.travis/linux_openresty_runner.sh#L35
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-apisix] membphis commented on issue #1312: Kafka Logger.
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #1312: Kafka Logger.
URL: https://github.com/apache/incubator-apisix/pull/1312#issuecomment-603241569
@Akayeshmantha It missing doc now.
The source code is fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services