You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/01/26 17:12:13 UTC

[GitHub] [apisix] tzssangglass opened a new pull request #3429: feat: kafka-logger implemented reuse kafka producer

tzssangglass opened a new pull request #3429:
URL: https://github.com/apache/apisix/pull/3429


   ### What this PR does / why we need it:
   <!--- Why is this change required? What problem does it solve? -->
   When a topic has multiple partitions, the kafka-logger plugin creates a new producer instance to connect to kafka every time it pushes a message, which causes kafka to reallocate the partition each time, and all messages are placed in the same partition.
   
   This PR reuse producer via lrucache to solve above problem.
   <!--- If it fixes an open issue, please link to the issue here. -->
   fix #3384
   ### Pre-submission checklist:
   
   * [x] Did you explain what problem does this PR solve? Or what new features have been added?
   * [ ] Have you added corresponding test cases?
   * [ ] Have you modified the corresponding document?
   * [x] Is this PR backward compatible? **If it is not backward compatible, please discuss on the [mailing list](https://github.com/apache/apisix/tree/master#community) first**
   
   How do I build a kafka cluster with multiple brokers? In https://github.com/apache/apisix/blob/3e19b06293463f52c8d26f0958542fc475ff0fcd/.github/workflows/centos7-ci.yml#L69? I need to use it when I write test cases


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] spacewander commented on a change in pull request #3429: feat: kafka-logger implemented reuse kafka producer

Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #3429:
URL: https://github.com/apache/apisix/pull/3429#discussion_r566539621



##########
File path: apisix/plugins/kafka-logger.lua
##########
@@ -66,33 +70,28 @@ function _M.check_schema(conf)
 end
 
 
-local function send_kafka_data(conf, log_message)
-    if core.table.nkeys(conf.broker_list) == 0 then
-        core.log.error("failed to identify the broker specified")
+local function partition_id(sendbuffer, topic, log_message)
+    if not sendbuffer.topics[topic] then
+        core.log.info("current topic in sendbuffer has no message")
+        return nil
     end
-
-    local broker_list = {}
-    local broker_config = {}
-
-    for host, port  in pairs(conf.broker_list) do
-        if type(host) == 'string'
-            and type(port) == 'number' then
-
-            local broker = {
-                host = host, port = port
-            }
-            table.insert(broker_list,broker)
+    for i, message in pairs(sendbuffer.topics[topic]) do
+        if log_message == message.queue[2] then
+            return i
         end
     end
+end
 
-    broker_config["request_timeout"] = conf.timeout * 1000
 
-    local prod, err = producer:new(broker_list,broker_config)
-    if err then
-        return nil, "failed to identify the broker specified: " .. err
+local function send_kafka_data(conf, log_message, prod)
+    if core.table.nkeys(conf.broker_list) == 0 then

Review comment:
       Should we move this before creating the producer?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] spacewander commented on pull request #3429: feat: kafka-logger implemented reuse kafka producer

Posted by GitBox <gi...@apache.org>.
spacewander commented on pull request #3429:
URL: https://github.com/apache/apisix/pull/3429#issuecomment-768844006


   The CI is up now. You might need to submit a new commit to trigger it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] tokers commented on a change in pull request #3429: feat: kafka-logger implemented reuse kafka producer

Posted by GitBox <gi...@apache.org>.
tokers commented on a change in pull request #3429:
URL: https://github.com/apache/apisix/pull/3429#discussion_r565758151



##########
File path: apisix/plugins/kafka-logger.lua
##########
@@ -72,6 +71,10 @@ end
 
 
 local function partition_id(sendbuffer, topic, offset)
+    if not sendbuffer.topics[topic] then
+        core.log.info("current topic in sendbuffer has no messages")

Review comment:
       `has no messages` => `has no message`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] spacewander merged pull request #3429: feat: kafka-logger implemented reuse kafka producer

Posted by GitBox <gi...@apache.org>.
spacewander merged pull request #3429:
URL: https://github.com/apache/apisix/pull/3429


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] Firstsawyou commented on a change in pull request #3429: feat: kafka-logger implemented reuse kafka producer

Posted by GitBox <gi...@apache.org>.
Firstsawyou commented on a change in pull request #3429:
URL: https://github.com/apache/apisix/pull/3429#discussion_r565204972



##########
File path: apisix/plugins/kafka-logger.lua
##########
@@ -141,6 +143,29 @@ function _M.log(conf, ctx)
         return
     end
 
+    -- reuse producer via lrucache to avoid unbalanced partitions of messages in kafka
+    local broker_list = {}
+    local broker_config = {}
+
+    for host, port in pairs(conf.broker_list) do
+        if type(host) == 'string'
+                and type(port) == 'number' then
+
+            local broker = {
+                host = host, port = port
+            }

Review comment:
       I think this style would be better:
   
   ```
    local broker = {
           host = host, 
           port = port
    }
   ```

##########
File path: apisix/plugins/kafka-logger.lua
##########
@@ -141,6 +143,29 @@ function _M.log(conf, ctx)
         return
     end
 
+    -- reuse producer via lrucache to avoid unbalanced partitions of messages in kafka
+    local broker_list = {}
+    local broker_config = {}
+
+    for host, port in pairs(conf.broker_list) do
+        if type(host) == 'string'
+                and type(port) == 'number' then

Review comment:
       Bad style, I think this would be better:
   
   ```
   if type(host) == 'string'
      and type(port) == 'number' then
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] tzssangglass commented on pull request #3429: feat: kafka-logger implemented reuse kafka producer

Posted by GitBox <gi...@apache.org>.
tzssangglass commented on pull request #3429:
URL: https://github.com/apache/apisix/pull/3429#issuecomment-768844399


   got


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] tokers commented on a change in pull request #3429: feat: kafka-logger implemented reuse kafka producer

Posted by GitBox <gi...@apache.org>.
tokers commented on a change in pull request #3429:
URL: https://github.com/apache/apisix/pull/3429#discussion_r566520492



##########
File path: apisix/plugins/kafka-logger.lua
##########
@@ -141,6 +146,29 @@ function _M.log(conf, ctx)
         return
     end
 
+    -- reuse producer via lrucache to avoid unbalanced partitions of messages in kafka
+    local broker_list = core.table.new(0, core.table.nkeys(conf.broker_list))

Review comment:
       The `broker_list` is an array-like table, you should initialize its array part, not the hash table part.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] tzssangglass commented on pull request #3429: feat: kafka-logger implemented reuse kafka producer

Posted by GitBox <gi...@apache.org>.
tzssangglass commented on pull request #3429:
URL: https://github.com/apache/apisix/pull/3429#issuecomment-767694406


   add test cases later


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] spacewander commented on a change in pull request #3429: feat: kafka-logger implemented reuse kafka producer

Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #3429:
URL: https://github.com/apache/apisix/pull/3429#discussion_r564944011



##########
File path: apisix/plugins/kafka-logger.lua
##########
@@ -141,6 +153,10 @@ function _M.log(conf, ctx)
         return
     end
 
+    local api_ctx = {}
+    api_ctx.conf_type = ctx.conf_type
+    api_ctx.conf_id = ctx.conf_id
+    api_ctx.conf_version = ctx.conf_version

Review comment:
       Maybe we can create the producer here and pass it to the `func`. So there is no need to create a copy of ctx.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] tokers commented on a change in pull request #3429: feat: kafka-logger implemented reuse kafka producer

Posted by GitBox <gi...@apache.org>.
tokers commented on a change in pull request #3429:
URL: https://github.com/apache/apisix/pull/3429#discussion_r565193643



##########
File path: apisix/plugins/kafka-logger.lua
##########
@@ -153,10 +143,29 @@ function _M.log(conf, ctx)
         return
     end
 
-    local api_ctx = {}
-    api_ctx.conf_type = ctx.conf_type
-    api_ctx.conf_id = ctx.conf_id
-    api_ctx.conf_version = ctx.conf_version
+    -- reuse producer via lrucache to avoid unbalanced partitions of messages in kafka
+    local broker_list = {}
+    local broker_config = {}
+
+    for host, port in pairs(conf.broker_list) do
+        if type(host) == 'string'

Review comment:
       It seems the line is not too long, why break it?

##########
File path: apisix/plugins/kafka-logger.lua
##########
@@ -153,10 +143,29 @@ function _M.log(conf, ctx)
         return
     end
 
-    local api_ctx = {}
-    api_ctx.conf_type = ctx.conf_type
-    api_ctx.conf_id = ctx.conf_id
-    api_ctx.conf_version = ctx.conf_version
+    -- reuse producer via lrucache to avoid unbalanced partitions of messages in kafka
+    local broker_list = {}
+    local broker_config = {}
+
+    for host, port in pairs(conf.broker_list) do
+        if type(host) == 'string'
+                and type(port) == 'number' then
+
+            local broker = {
+                host = host, port = port
+            }
+            table.insert(broker_list,broker)

Review comment:
       Style: keep an empty space after `,`.

##########
File path: apisix/plugins/kafka-logger.lua
##########
@@ -130,6 +114,12 @@ local function remove_stale_objects(premature)
 end
 
 
+local function create_producer(broker_list, broker_config)
+    core.log.info("create new kafka producer instance")
+    return producer:new(broker_list,broker_config)

Review comment:
       Style: need an empty space after `,`.

##########
File path: apisix/plugins/kafka-logger.lua
##########
@@ -153,10 +143,29 @@ function _M.log(conf, ctx)
         return
     end
 
-    local api_ctx = {}
-    api_ctx.conf_type = ctx.conf_type
-    api_ctx.conf_id = ctx.conf_id
-    api_ctx.conf_version = ctx.conf_version
+    -- reuse producer via lrucache to avoid unbalanced partitions of messages in kafka
+    local broker_list = {}

Review comment:
       The `broker_list` can be allocated with `core.table.new`, to pre-allocate the table.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] tzssangglass commented on pull request #3429: feat: kafka-logger implemented reuse kafka producer

Posted by GitBox <gi...@apache.org>.
tzssangglass commented on pull request #3429:
URL: https://github.com/apache/apisix/pull/3429#issuecomment-768824378


   @spacewander CI is broken?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] tzssangglass commented on pull request #3429: feat: kafka-logger implemented reuse kafka producer

Posted by GitBox <gi...@apache.org>.
tzssangglass commented on pull request #3429:
URL: https://github.com/apache/apisix/pull/3429#issuecomment-769506773


   update, @spacewander @tokers @Firstsawyou pls take a look


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] tzssangglass edited a comment on pull request #3429: feat: kafka-logger implemented reuse kafka producer

Posted by GitBox <gi...@apache.org>.
tzssangglass edited a comment on pull request #3429:
URL: https://github.com/apache/apisix/pull/3429#issuecomment-768824378


   @spacewander @tokers @Firstsawyou  CI is broken?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [apisix] tzssangglass commented on a change in pull request #3429: feat: kafka-logger implemented reuse kafka producer

Posted by GitBox <gi...@apache.org>.
tzssangglass commented on a change in pull request #3429:
URL: https://github.com/apache/apisix/pull/3429#discussion_r564948817



##########
File path: apisix/plugins/kafka-logger.lua
##########
@@ -141,6 +153,10 @@ function _M.log(conf, ctx)
         return
     end
 
+    local api_ctx = {}
+    api_ctx.conf_type = ctx.conf_type
+    api_ctx.conf_id = ctx.conf_id
+    api_ctx.conf_version = ctx.conf_version

Review comment:
       got




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org