You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/01/06 10:06:37 UTC
[GitHub] [apisix] guxingke opened a new issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
guxingke opened a new issue #6033:
URL: https://github.com/apache/apisix/issues/6033
### Issue description
always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
### Environment
- apisix version (cmd: `apisix version`):
- OS (cmd: `uname -a`):
- OpenResty / Nginx version (cmd: `nginx -V` or `openresty -V`):
- etcd version, if have (cmd: run `curl http://127.0.0.1:9090/v1/server_info` to get the info from server-info API):
- apisix-dashboard version, if have:
- the plugin runner version, if the issue is about a plugin runner (cmd: depended on the kind of runner):
- luarocks version, if the issue is about installation (cmd: `luarocks --version`):
### Steps to reproduce
1. use kafka logger
2. make karka borker down
3. until apisix error log apprear like 'not found borker' logs.
4. make kafka borker up
5. apisix not auto recover
### Actual result
after kafka borker up , send logs.
### Error log
buffered messages send to kafka err: not found broker, retryable: true, ....
### Expected result
notings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1010603177
I don't know how to reproduce `not found broker` 😂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1007208185
I need a step by step replication of how you build kafka, how you configure route, the Kafka-logger plugin, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] guxingke edited a comment on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
guxingke edited a comment on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1007868955
> If I start kafka with docker, shut down the container after starting it and start it again, can I simulate the broker down scenario you mentioned?
yes, this can.
> maybe we need to add refresh_interval to skip this bug.
yeah, It's best if it can recover automatically
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1009961895
> maybe your kafka recover too quickly? I could not provider more information about this .😂
I got `connection refused`, I need to be able to reproduce this exception accurately in order to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] guxingke edited a comment on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
guxingke edited a comment on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1007227481
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] guxingke commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
guxingke commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1007093826
> provide reproduction steps and environment & version information.
description edited
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] guxingke commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
guxingke commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1007227481
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1009748824
> Maybe we didn't repeating it because our configuration was different.
I need to be able to simulate the kafka broker down de scenario you mentioned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1006578942
provide reproduction steps and environment & version information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1007210459
> 2. make karka borker down
how to do this?
> 4. make kafka borker up
Is kafka broken definitely available at this point?
In my experience, the kafka producer has a lrucache, so you can update the kafka-logger configuration with the `cluster_name` parameter to generate a new kafka producer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] guxingke commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
guxingke commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1007227481
the kafka brown in other case .
---
when i restart kafka borkers , and waiting the apisix send message auto recover. but not , about 5 minute later , relaod the apisix, it works well.
---
> In my experience, the kafka producer has a lrucache, so you can update the kafka-logger configuration with the cluster_name parameter to generate a new kafka producer?
Maybe in the next time, I will try this solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass edited a comment on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
tzssangglass edited a comment on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1007208185
~~I need a step by step replication of how you build kafka, how you configure route, the Kafka-logger plugin, etc.~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] guxingke edited a comment on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
guxingke edited a comment on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1007227481
the kafka broken in other case .
---
when i restart kafka borkers , and waiting the apisix send message auto recover. but not , about 5 minute later , relaod the apisix, it works well.
---
> In my experience, the kafka producer has a lrucache, so you can update the kafka-logger configuration with the cluster_name parameter to generate a new kafka producer?
Maybe in the next time, I will try this solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] guxingke commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
guxingke commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1007868955
> If I start kafka with docker, shut down the container after starting it and start it again, can I simulate the broker down scenario you mentioned?
yes, this can.
maybe we need to add refresh_interval to skip this bug.
> yeah, It's best if it can recover automatically
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] guxingke commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
guxingke commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1009667534
my kafka logger configuration
```js
{
"batch_max_size": 100,
"broker_list": {
"10.200.32.69": 9092,
"10.200.32.70": 9092,
"10.200.32.71": 9092
},
"disable": false,
"kafka_topic": "nginx_www"
}
```
and it as a global plugin.
and it use user-define log_format.
-------
Maybe we didn't repeating it because our configuration was different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass edited a comment on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
tzssangglass edited a comment on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1007208185
~~I need a step by step replication of how you build kafka, how you configure route, the Kafka-logger plugin, etc.~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] guxingke commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
guxingke commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1010576076
Retrieved the logs and found both errors. the logs like below.
```
2022/01/06 16:05:12 [error] 32298#32298: *518806498 [lua] producer.lua:272: buffered messages send to kafka err: connection refused, retryable: true, topic: nginx_www, partition_id: 1, lengt
h: 1, context: ngx.timer, client: y, server: x
2022/01/06 16:05:12 [error] 32296#32296: *518806551 [lua] producer.lua:272: buffered messages send to kafka err: connection refused, retryable: true, topic: nginx_www, partition_id: 1, length: 1, context: ngx.timer, client: y, server: x
2022/01/06 16:05:13 [error] 32299#32299: *518806730 [lua] producer.lua:272: buffered messages send to kafka err: not found broker, retryable: true, topic: nginx_www, partition_id: 1, length: 1, context: ngx.timer, client: y server: x
2022/01/06 16:05:13 [error] 32294#32294: *518806744 [lua] producer.lua:272: buffered messages send to kafka err: not found broker, retryable: true, topic: nginx_www, partition_id: 1, length: 1, context: ngx.timer, client: y, server:x
```
and the first time for `connection refused` at '16:04:44', the other at '16:05:13' .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1007208185
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] guxingke commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
guxingke commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1009842570
maybe your kafka recover too quickly? I could not provider more information about this .😂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1007634743
> when i restart kafka borkers , and waiting the apisix send message auto recover. but not , about 5 minute later , relaod the apisix, it works well.
refer to: https://github.com/doujiang24/lua-resty-kafka/issues/70
maybe we need to add `refresh_interval` to skip this bug.
@guxingke If I start kafka with docker, shut down the container after starting it and start it again, can I simulate the broker down scenario you mentioned?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6033: bug: kafka-logger, always log `not found broker` after kafka-logger kafka broker up. until reload apisix.
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6033:
URL: https://github.com/apache/apisix/issues/6033#issuecomment-1008558652
> If I start kafka with docker, shut down the container after starting it and start it again, can I simulate the broker down scenario you mentioned
@guxingke I was unable to reproduce the scenario with this on the master branch.
Here is my configuration of route:
```
{
"plugins": {
"kafka-logger": {
"broker_list": {
"127.0.0.1": 9092
},
"kafka_topic": "apisix_traffic_logs",
"key": "key1",
"timeout": 1,
"batch_max_size": 1
}
},
"upstream": {
"nodes": {
"127.0.0.1:1980": 1
},
"type": "roundrobin"
},
"uri": "/hello"
}
```
Here is my exception log:
```
2022/01/10 13:36:36 [error] 18553#25478878: *43867 [lua] client.lua:150: refresh(): all brokers failed in fetch topic metadata, context: ngx.timer, client: 127.0.0.1, server: 0.0.0.0:9080
2022/01/10 13:36:36 [error] 18553#25478878: *43867 [lua] client.lua:150: refresh(): all brokers failed in fetch topic metadata, context: ngx.timer, client: 127.0.0.1, server: 0.0.0.0:9080
2022/01/10 13:36:36 [error] 18553#25478878: *43867 [lua] client.lua:150: refresh(): all brokers failed in fetch topic metadata, context: ngx.timer, client: 127.0.0.1, server: 0.0.0.0:9080
2022/01/10 13:36:36 [error] 18553#25478878: *43867 [lua] producer.lua:272: buffered messages send to kafka err: connection refused, retryable: true, topic: apisix_traffic_logs, partition_id: 0, length: 1, context: ngx.timer, client: 127.0.0.1, server: 0.0.0.0:9080
```
When I restarted the kafka container, it continued to push messages to kafka normally again.
Note: I did not make the kafka volume, and the restarted kafka is a new cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org