You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/07/07 07:08:22 UTC
[GitHub] [apisix] tianqiuming opened a new issue #4550: bug: 启用Kafka-logger，批量上报到后台的日志格式为JSONArray，而非JSONObject格式，跟JAVA客户端的批量上报实现有区别。
tianqiuming opened a new issue #4550:
URL: https://github.com/apache/apisix/issues/4550
### Issue description
添加Kafka-logger后，批量上报日志到kafka后，变成JSONArray格式
### Environment
Bug report without environment information will be ignored or closed.
* apisix version (cmd: `apisix version`):2.7
* OS (cmd: `uname -a`):centos7.2
* OpenResty / Nginx version (cmd: `nginx -V` or `openresty -V`):1.19.3.1
* etcd version, if have (cmd: run `curl http://127.0.0.1:9090/v1/server_info` to get the info from server-info API):3.5.0
* apisix-dashboard version, if have:2.7
* luarocks version, if the issue is about installation (cmd: `luarocks --version`):
### Minimal test code / Steps to reproduce the issue
Bug report without steps to reproduce will be ignored or closed.
1.正确配置上游、路由和服务。
2.启用kafka-logger插件，正确配置broker_list和kafka_topic参数，其他参数用默认值。
3.通过网关连续多次上游服务。
### What's the actual result? (including assertion message & call stack if applicable)
上报到kafka后台的数据变成了JSONArray格式。
### What's the expected result?
单条的JSONObject格式数据。
注:当batch_max_size配置为1时，上报到kafka的数据为JSONObject格式数据，如果用户想要JSONObject，只能不启用批量上报日志功能。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jamesislebron commented on issue #4550: bug: When Kafka-logger is enabled, the log format for batch reporting to the backend is JSONArray instead of JSONObject format, which is different from the batch reporting implementation of the JAVA client.
Posted by GitBox <gi...@apache.org>.
jamesislebron commented on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-892623989
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tianqiuming edited a comment on issue #4550: bug: When Kafka-logger is enabled, the log format for batch reporting to the backend is JSONArray instead of JSONObject format, which is different from the batch reporting implementation of the JAVA client.
Posted by GitBox <gi...@apache.org>.
tianqiuming edited a comment on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-893077775
是的，Kafka有批量上报的特性，但是我们没有用这个特性，而是变相的通过jsonArray包装成单条数据实现批量上报的功能。这会导致接入方需要判断每条数据是jsonObject还是jsonArray数据，非常不方便。
Yes, Kafka does have a batch reporting feature, but we don't use this feature. Instead, we use jsonArray to wrap a single piece of data into a batch reporting function. This will cause the access party to determine whether each piece of data is jsonObject or jsonArray data, which is very inconvenient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tianqiuming commented on issue #4550: bug: When Kafka-logger is enabled, the log format for batch reporting to the backend is JSONArray instead of JSONObject format, which is different from the batch reporting implementation of the JAVA client.
Posted by GitBox <gi...@apache.org>.
tianqiuming commented on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-893077775
是的，Kafka有批量上报的特性，但是我们没有用这个特性，而是变相的通过jsonArray数组通过单条上报来实际批量上报的功能。这会导致接入方需要判断每条数据是jsonObject还是jsonArray数据，非常不方便。
Yes, Kafka has a batch report feature, but we don't use this feature. Instead, we use the jsonArray array to actually report in batches. This will cause the access party to determine whether each piece of data is jsonObject or jsonArray data, which is very inconvenient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dmsolr commented on issue #4550: bug: When Kafka-logger is enabled, the log format for batch reporting to the backend is JSONArray instead of JSONObject format, which is different from the batch reporting implementation of the JAVA client.
Posted by GitBox <gi...@apache.org>.
dmsolr commented on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-891937275
I think @tianqiuming is asking why it is JsonObject format when the batch size was 1 and it is JsonArrary when there are multi messages in the batch. That is confusing for you, right?
In personal, I prefer to use JsonArrary in any case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jamesislebron commented on issue #4550: bug: When Kafka-logger is enabled, the log format for batch reporting to the backend is JSONArray instead of JSONObject format, which is different from the batch reporting implementation of the JAVA client.
Posted by GitBox <gi...@apache.org>.
jamesislebron commented on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-892656371
是的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dmsolr commented on issue #4550: bug: When Kafka-logger is enabled, the log format for batch reporting to the backend is JSONArray instead of JSONObject format, which is different from the batch reporting implementation of the JAVA client.
Posted by GitBox <gi...@apache.org>.
dmsolr commented on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-892633024
I see what you mean. every piece message is a separate Kafka message, right? and then send them through Kafka's batch feature. If so, the PR is welcome.
----------------
我知道你的意思了。每条消息都是一条独立的 Kafka 消息对吧，然后通过 Kafka 的 Batch 特征。如果是的话，欢迎提交 PR。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tianqiuming edited a comment on issue #4550: bug: When Kafka-logger is enabled, the log format for batch reporting to the backend is JSONArray instead of JSONObject format, which is different from the batch reporting implementation of the JAVA client.
Posted by GitBox <gi...@apache.org>.
tianqiuming edited a comment on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-893077775
是的，Kafka有批量上报的特性，但是我们没有用这个特性，而是变相的通过jsonArray包装成单条数据实现批量上报的功能。这会导致接入方需要判断每条数据是jsonObject还是jsonArray数据，非常不方便。
Yes, Kafka does have a batch reporting feature, but we don't use this feature. Instead, we use jsonArray to wrap a single piece of data into a batch reporting function. This will cause the access party to determine whether each piece of data is jsonObject or jsonArray data, which is very inconvenient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tianqiuming commented on issue #4550: bug: When Kafka-logger is enabled, the log format for batch reporting to the backend is JSONArray instead of JSONObject format, which is different from the batch reporting implementation of the JAVA client.
Posted by GitBox <gi...@apache.org>.
tianqiuming commented on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-893077775
是的，Kafka有批量上报的特性，但是我们没有用这个特性，而是变相的通过jsonArray数组通过单条上报来实际批量上报的功能。这会导致接入方需要判断每条数据是jsonObject还是jsonArray数据，非常不方便。
Yes, Kafka has a batch report feature, but we don't use this feature. Instead, we use the jsonArray array to actually report in batches. This will cause the access party to determine whether each piece of data is jsonObject or jsonArray data, which is very inconvenient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jamesislebron commented on issue #4550: bug: When Kafka-logger is enabled, the log format for batch reporting to the backend is JSONArray instead of JSONObject format, which is different from the batch reporting implementation of the JAVA client.
Posted by GitBox <gi...@apache.org>.
jamesislebron commented on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-891634851
> By saying `JSONArray`, you mean the format like:
>
> ```json
> [
> {...},
> {...},
> {...}
> ]
> ```
>
> Or just like the following:
>
> ```json
> {...}
> {...}
> {...}
> ```
I have the same problem, `JSONArray` means format like
```
[
{...},
{...},
{...}
]
```
What I want is just json log one by one:
```
{...}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #4550: bug: When Kafka-logger is enabled, the log format for batch reporting to the backend is JSONArray instead of JSONObject format, which is different from the batch reporting implementation of the JAVA client.
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-891828820
The problem is that when there is only one log in a batch, the data format is JsonObject, when there are multiple logs, the data format is JsonArray, and I prefer to use JsonArray consistently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dmsolr edited a comment on issue #4550: bug: When Kafka-logger is enabled, the log format for batch reporting to the backend is JSONArray instead of JSONObject format, which is different from the batch reporting implementation of the JAVA client.
Posted by GitBox <gi...@apache.org>.
dmsolr edited a comment on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-892633024
I see what you mean. every piece message is a separate Kafka message, right? and then send them through Kafka's batch feature. If so, the PR is welcome.
----------------
我知道你的意思了。每条消息都是一条独立的 Kafka 消息对吧，然后通过 Kafka 的 Batch 特征。如果是的话，欢迎提交 PR。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dmsolr edited a comment on issue #4550: bug: When Kafka-logger is enabled, the log format for batch reporting to the backend is JSONArray instead of JSONObject format, which is different from the batch reporting implementation of the JAVA client.
Posted by GitBox <gi...@apache.org>.
dmsolr edited a comment on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-892633024
I see what you mean. every piece message is a separate Kafka message, right? and then send them through Kafka's batch feature. If so, the PR is welcome.
----------------
我知道你的意思了。每条消息都是一条独立的 Kafka 消息对吧，然后通过 Kafka 的 Batch 特征。如果是的话，欢迎提交 PR。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jamesislebron commented on issue #4550: bug: When Kafka-logger is enabled, the log format for batch reporting to the backend is JSONArray instead of JSONObject format, which is different from the batch reporting implementation of the JAVA client.
Posted by GitBox <gi...@apache.org>.
jamesislebron commented on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-892623989
抱歉，英文太差，我就中文直接回了：
当日志收集到日志平台之后，这种`JsonArrary`的格式，在做一些日志检索或者排错的时候，就不好做了；
而且kafka有批量发送消息的特性，直接通过这个特性批量上报，但是每个消息都是JsonObject格式的，这样性能上应该也不会差；
所以，是否考虑支持可以配置为JsonArray或者JsonObject呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #4550: bug: 启用Kafka-logger，批量上报到后台的日志格式为JSONArray，而非JSONObject格式，跟JAVA客户端的批量上报实现有区别。
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-875550442
By saying `JSONArray`, you mean the format like:
```json
[
{...},
{...},
{...}
]
```
Or just like the following:
```json
{...}
{...}
{...}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #4550: bug: When Kafka-logger is enabled, the log format for batch reporting to the backend is JSONArray instead of JSONObject format, which is different from the batch reporting implementation of the JAVA client.
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-891745562
Maybe the format can be a configurable item?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] dmsolr commented on issue #4550: bug: When Kafka-logger is enabled, the log format for batch reporting to the backend is JSONArray instead of JSONObject format, which is different from the batch reporting implementation of the JAVA client.
Posted by GitBox <gi...@apache.org>.
dmsolr commented on issue #4550:
URL: https://github.com/apache/apisix/issues/4550#issuecomment-892633024
I see what you mean. every piece message is a separate Kafka message, right? and then send them through Kafka's batch feature. If so, the PR is welcome.
----------------
我知道你的意思了。每条消息都是一条独立的 Kafka 消息对吧，然后通过 Kafka 的 Batch 特征。如果是的话，欢迎提交 PR。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org