You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "gaoxingliang (via GitHub)" <gi...@apache.org> on 2023/10/19 02:40:33 UTC
[I] Kafka logger not working [apisix]
gaoxingliang opened a new issue, #10359:
URL: https://github.com/apache/apisix/issues/10359
### Description
I try to use kafka logger to record all the requests and response and replace the clickhouse-logger(it's working)
after I add it in the apisix dashboard, it shows nothing messages about kafka.
and when I triggered a request and found the data is never sent to kafka.
at same time, I created another plugin response-rewrite and found it's working.
The apisix conf shows it's enabled:
```
plugins: # plugin list
- api-breaker
- authz-keycloak
- basic-auth
- batch-requests
- consumer-restriction
- cors
- echo
- fault-injection
- file-logger
- grpc-transcode
- grpc-web
- hmac-auth
- http-logger
- ip-restriction
- ua-restriction
- jwt-auth
- kafka-logger
- key-auth
- limit-conn
- limit-count
- limit-req
- node-status
- openid-connect
- authz-casbin
- prometheus
- proxy-cache
- proxy-mirror
- proxy-rewrite
- redirect
- referer-restriction
- request-id
- request-validation
- response-rewrite
- serverless-post-function
- serverless-pre-function
- sls-logger
- syslog
- tcp-logger
- udp-logger
- uri-blocker
- wolf-rbac
- zipkin
- traffic-split
- gzip
- real-ip
- ext-plugin-pre-req
- ext-plugin-post-req
- ext-plugin-post-resp
- server-info
- elasticsearch-logger
- clickhouse-logger
- forward-auth
stream_plugins:
- mqtt-proxy
- ip-restriction
- limit-conn
```
It seems this plugin is DISAPPEARED.
### Environment
- APISIX version (run `apisix version`): 3.3.0
- APISIX dashboard: 3.0.0
- Operating system (run `uname -a`):
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): 3.5.0
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka logger not working [apisix]
Posted by "gaoxingliang (via GitHub)" <gi...@apache.org>.
gaoxingliang commented on issue #10359:
URL: https://github.com/apache/apisix/issues/10359#issuecomment-1770115685
@shreemaan-abhishek I don;t see any error in the apisix error log. I created a new version cluster with 3.6.0. and it worked. and I also found the apisix occasionally got error of topic not found (even it's there):
```
2023/10/19 05:56:00 [error] 51#51: *1062684 [lua] batch-processor.lua:95: Batch Processor[apisix-kafkalogger] failed to process entries: failed to send data to Kafka topic: not found topic, brokers: null, context: ngx.timer, client: 10.233.113.0, server: 0.0.0.0:9080
2023/10/19 05:56:00 [error] 51#51: *1062684 [lua] batch-processor.lua:104: Batch Processor[apisix-kafkalogger] exceeded the max_retry_count[1] dropping the entries, context: ngx.timer, client: 10.233.113.0, server: 0.0.0.0:9080
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka logger not working [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on issue #10359:
URL: https://github.com/apache/apisix/issues/10359#issuecomment-1769850031
Are there any errors in error.log?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka logger not working [apisix]
Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10359:
URL: https://github.com/apache/apisix/issues/10359#issuecomment-1770147887
@gaoxingliang This might be an issue with lua-resty-kafka when FQDN is used instead of IP.
Relevant issues:
https://github.com/doujiang24/lua-resty-kafka/issues/6
https://github.com/doujiang24/lua-resty-kafka/issues/5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka logger not working [apisix]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10359:
URL: https://github.com/apache/apisix/issues/10359#issuecomment-3139319237
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka logger not working [apisix]
Posted by "kayx23 (via GitHub)" <gi...@apache.org>.
kayx23 commented on issue #10359:
URL: https://github.com/apache/apisix/issues/10359#issuecomment-2259771605
Please try the examples in this doc to see if anything helps with your case: https://docs.api7.ai/hub/kafka-logger
They were tested to be working and the doc is more current.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka logger not working [apisix]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10359:
URL: https://github.com/apache/apisix/issues/10359#issuecomment-3077905076
This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka logger not working [apisix]
Posted by "gaoxingliang (via GitHub)" <gi...@apache.org>.
gaoxingliang commented on issue #10359:
URL: https://github.com/apache/apisix/issues/10359#issuecomment-1972709789
@monkeyDluffy6017 no. It's unstable. I have to do the log manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka logger not working [apisix]
Posted by "monkeyDluffy6017 (via GitHub)" <gi...@apache.org>.
monkeyDluffy6017 commented on issue #10359:
URL: https://github.com/apache/apisix/issues/10359#issuecomment-1851497802
@gaoxingliang have you solved your problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka logger not working [apisix]
Posted by "gaoxingliang (via GitHub)" <gi...@apache.org>.
gaoxingliang commented on issue #10359:
URL: https://github.com/apache/apisix/issues/10359#issuecomment-1820504886
This issue persist.
The apisix version is 3.6.0.
and the conf:
```
{
"_meta": {
"disable": false
},
"brokers": [
{
"host": "192.168.102.227",
"port": 39091
}
],
"include_req_body": true,
"kafka_topic": "apisix-log-test",
"name": "apisix-kafkalogger"
}
```
and the error log:
```
2023/11/21 09:02:25 [warn] 55#55: *1345942 [lua] plugins.lua:54: query parameter "all" will be deprecated soon., client: 10.233.113.84, server: , request: "GET /apisix/admin/plugins?all=true HTTP/1.1", host: "apisix-admin.apisix.svc.cluster.local:9180"
2023/11/21 09:02:28 [error] 57#57: *1350184 [lua] batch-processor.lua:95: Batch Processor[apisix-kafkalogger] failed to process entries: failed to send data to Kafka topic: not found broker, brokers: null, context: ngx.timer, client: 10.233.107.0, server: 0.0.0.0:9080
2023/11/21 09:02:28 [error] 57#57: *1350184 [lua] batch-processor.lua:104: Batch Processor[apisix-kafkalogger] exceeded the max_retry_count[1] dropping the entries, context: ngx.timer, client: 10.233.107.0, server: 0.0.0.0:9080
```
I don't know why it said the not found broker. I telnet the kafka and it's connected.
I found the kafka logger plugin code error msg use the deprecated `broker_list`. so I also try with the conf:
```
"broker_list": {
"192.168.102.227": 39091
},
```
It also says the broker list is null. I think the problem is: the conf get from etcd or somewhere is NULL.
the source code is here:
https://github.com/apache/apisix/blob/master/apisix/plugins/kafka-logger.lua#L203
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka logger not working [apisix]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #10359: Kafka logger not working
URL: https://github.com/apache/apisix/issues/10359
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org