You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "karthi5634 (via GitHub)" <gi...@apache.org> on 2025/05/09 14:18:32 UTC
[I] help request: I want to send JSON data from apisix to kafka topic but not able to do it, can anyone assist. [apisix]
karthi5634 opened a new issue, #12201:
URL: https://github.com/apache/apisix/issues/12201
### Description
data is being logged in consumer as
{"route_id":"kafkaroute","message":"{\r\n \"event_id\": \"event10\",\r\n \"event_type\": \"apisix\",\r\n \"user_id\": \"user10\",\r\n \"amount\": 900.22,\r\n \"timestamp\": 1712832480000\r\n}"}
but i need actual JSON data so that i can ingest the same in Pinot table
Kindly help.
### Environment
- APISIX version (run `apisix version`):
- Operating system (run `uname -a`):
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: I want to send JSON data from apisix to kafka topic but not able to do it, can anyone assist. [apisix]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12201:
URL: https://github.com/apache/apisix/issues/12201#issuecomment-2965900588
You can refer to https://docs.api7.ai/apisix/how-to-guide/custom-plugins/create-plugin-in-lua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: I want to send JSON data from apisix to kafka topic but not able to do it, can anyone assist. [apisix]
Posted by "karthi5634 (via GitHub)" <gi...@apache.org>.
karthi5634 commented on issue #12201:
URL: https://github.com/apache/apisix/issues/12201#issuecomment-2876421027
Below is my Apisix route:
{
"uri": "/send-json",
"methods": ["POST"],
"name": "kafkaroute",
"plugins": {
"kafka-logger": {
"batch_max_size": 1,
"broker_list": {
"172.18.0.7:19092": 1,
"172.18.0.8:19092": 1,
"172.18.0.9:19092": 1
},
"disable": false,
"kafka_topic": "apisix",
"key": "apisix-key",
"name": "kafka-logger",
"timeout": 60,
"retry_count": 5,
"producer_type": "async",
"include_resp_body": true,
"include_req_body": true,
"log_format": {
"event_id": "event10",
"event_type": "apisix",
"user_id": "user10",
"amount": "$amount"
}
},
"upstream_id": "kafkaupstream"
}
Below is the JSON data sending from API:
{
"event_id": "event10",
"event_type": "apisix",
"user_id": "user10",
"amount": 101.22,
"timestamp": 1712832480000
}
Could you please let me know the log_format syntax for above data. Currenly I am providing the static values directly but it has pick the data which i send from API.
Please help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: I want to send JSON data from apisix to kafka topic but not able to do it, can anyone assist. [apisix]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop closed issue #12201: help request: I want to send JSON data from apisix to kafka topic but not able to do it, can anyone assist.
URL: https://github.com/apache/apisix/issues/12201
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: I want to send JSON data from apisix to kafka topic but not able to do it, can anyone assist. [apisix]
Posted by "karthi5634 (via GitHub)" <gi...@apache.org>.
karthi5634 commented on issue #12201:
URL: https://github.com/apache/apisix/issues/12201#issuecomment-2916653031
hi @Baoyuantop and Team
Any update on my above request?
Kindly let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: I want to send JSON data from apisix to kafka topic but not able to do it, can anyone assist. [apisix]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12201:
URL: https://github.com/apache/apisix/issues/12201#issuecomment-2965621714
log_format currently only supports string values, you can choose to handle this format on the consumer side or develop a customized plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: I want to send JSON data from apisix to kafka topic but not able to do it, can anyone assist. [apisix]
Posted by "karthi5634 (via GitHub)" <gi...@apache.org>.
karthi5634 commented on issue #12201:
URL: https://github.com/apache/apisix/issues/12201#issuecomment-2890738626
Can anyone please help by referring to above details
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: I want to send JSON data from apisix to kafka topic but not able to do it, can anyone assist. [apisix]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12201:
URL: https://github.com/apache/apisix/issues/12201#issuecomment-2969177432
If there is still a problem, please open it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: I want to send JSON data from apisix to kafka topic but not able to do it, can anyone assist. [apisix]
Posted by "karthi5634 (via GitHub)" <gi...@apache.org>.
karthi5634 commented on issue #12201:
URL: https://github.com/apache/apisix/issues/12201#issuecomment-2898504032
hi @Baoyuantop
Any idea or thoughts on the above request?
Kindly help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: I want to send JSON data from apisix to kafka topic but not able to do it, can anyone assist. [apisix]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12201:
URL: https://github.com/apache/apisix/issues/12201#issuecomment-2875510174
The kafka-logger plugin will send log entries to Kafka as JSON objects by default, but if the original data is already in JSON format, it will be escaped and embedded as a string. Using a custom `log_format` gives you more control over the exact format of the data sent to Kafka.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: I want to send JSON data from apisix to kafka topic but not able to do it, can anyone assist. [apisix]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12201:
URL: https://github.com/apache/apisix/issues/12201#issuecomment-2892655250
Is it ok to include only request_body?
```
"log_format": {
"request_body": "$request_body"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: I want to send JSON data from apisix to kafka topic but not able to do it, can anyone assist. [apisix]
Posted by "karthi5634 (via GitHub)" <gi...@apache.org>.
karthi5634 commented on issue #12201:
URL: https://github.com/apache/apisix/issues/12201#issuecomment-2892658581
let me give a try and update you shotly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: I want to send JSON data from apisix to kafka topic but not able to do it, can anyone assist. [apisix]
Posted by "karthi5634 (via GitHub)" <gi...@apache.org>.
karthi5634 commented on issue #12201:
URL: https://github.com/apache/apisix/issues/12201#issuecomment-2892663733
no it is working, Kafka consumer is ready the like below

Please check and let me know
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: I want to send JSON data from apisix to kafka topic but not able to do it, can anyone assist. [apisix]
Posted by "karthi5634 (via GitHub)" <gi...@apache.org>.
karthi5634 commented on issue #12201:
URL: https://github.com/apache/apisix/issues/12201#issuecomment-2965627424
Thanks for your suggestion.
Can you let me know how to develop a customized plugin?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org