You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/07/13 10:46:39 UTC
[GitHub] [apisix] caozhi6655 opened a new issue, #7454: clickhouse-logger can't support datetime or date field in clickhouse
caozhi6655 opened a new issue, #7454:
URL: https://github.com/apache/apisix/issues/7454
### Description
I use clickhouse to instore access logs of apisix，and i need to delete history data example three month before。I want use TTL feature of clickhouse to delete data auto，and the TTL field must be datetime or date，and the datetime or date format is 'yyyy-MM-dd hh:mm:ss'。In apisix there is no varible this format，so the data can't be writen。
I think the clickhouse-logger plugin log-format can't support more，could you give a way to solve this？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] caozhi6655 commented on issue #7454: clickhouse-logger can't support datetime or date field in clickhouse
Posted by GitBox <gi...@apache.org>.
caozhi6655 commented on issue #7454:
URL: https://github.com/apache/apisix/issues/7454#issuecomment-1186736211
> > I didn't catch your answer，can you please give me a more information or operation steps？
>
> We can add the following code to `clickhouse-logger` to register a custom variable:
>
> ```lua
> core.ctx.register_var("fmt_localtime", function(ctx)
> return ngx.localtime()
> end)
> ```
>
> Then use this variable in the metadata:
>
> ```
> curl http://127.0.0.1:9080/apisix/admin/plugin_metadata/clickhouse-logger -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
> {
> "log_format": {
> "host": "$host",
> "@timestamp": "$fmt_localtime",
> "client_ip": "$remote_addr"
> }
> }'
> ```
So，this needs to update the clickhouse-logger plugins's code，right？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] caozhi6655 commented on issue #7454: clickhouse-logger can't support datetime or date field in clickhouse
Posted by GitBox <gi...@apache.org>.
caozhi6655 commented on issue #7454:
URL: https://github.com/apache/apisix/issues/7454#issuecomment-1183932047
> the default `timestamp` used in clickhouse-logger is `time_iso8601`, which looks like `2022-01-17T10:03:10+08:00`, you can update the metadata of clickhouse-logger to specify the format of the `timestamp`, see: https://github.com/apache/apisix/blob/master/docs/en/latest/plugins/clickhouse-logger.md#metadata
>
> such as changing `$time_iso8601` to `$time_local`, which looks like `2022/07/11 18:52:41`.
I tried $time_local，it didn't work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] caozhi6655 commented on issue #7454: clickhouse-logger can't support datetime or date field in clickhouse
Posted by GitBox <gi...@apache.org>.
caozhi6655 commented on issue #7454:
URL: https://github.com/apache/apisix/issues/7454#issuecomment-1185086293
>
I didn't catch your answer，can you please give me a more information or operation steps？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7454: clickhouse-logger can't support datetime or date field in clickhouse
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7454:
URL: https://github.com/apache/apisix/issues/7454#issuecomment-1190975129
hi @zhendongcmss, what's your option?
clickhouse use the format like `2000-01-01 12:00:00` as the default time format? If so, would it be more convenient for the clickhouse plugin to push the logs in this time format?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7454: clickhouse-logger can't support datetime or date field in clickhouse
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7454:
URL: https://github.com/apache/apisix/issues/7454#issuecomment-1185210621
> I didn't catch your answer，can you please give me a more information or operation steps？
We can add the following code to `clickhouse-logger` to register a custom variable:
```lua
core.ctx.register_var("fmt_localtime", function(ctx)
return ngx.localtime()
end)
```
Then use this variable in the metadata:
```
curl http://127.0.0.1:9080/apisix/admin/plugin_metadata/clickhouse-logger -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"log_format": {
"host": "$host",
"@timestamp": "$fmt_localtime",
"client_ip": "$remote_addr"
}
}'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] github-actions[bot] commented on issue #7454: clickhouse-logger can't support datetime or date field in clickhouse
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7454:
URL: https://github.com/apache/apisix/issues/7454#issuecomment-1623395853
This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] github-actions[bot] closed issue #7454: clickhouse-logger can't support datetime or date field in clickhouse
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7454: clickhouse-logger can't support datetime or date field in clickhouse
URL: https://github.com/apache/apisix/issues/7454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] caozhi6655 commented on issue #7454: clickhouse-logger can't support datetime or date field in clickhouse
Posted by GitBox <gi...@apache.org>.
caozhi6655 commented on issue #7454:
URL: https://github.com/apache/apisix/issues/7454#issuecomment-1183932783
> the default `timestamp` used in clickhouse-logger is `time_iso8601`, which looks like `2022-01-17T10:03:10+08:00`, you can update the metadata of clickhouse-logger to specify the format of the `timestamp`, see: https://github.com/apache/apisix/blob/master/docs/en/latest/plugins/clickhouse-logger.md#metadata
>
> such as changing `$time_iso8601` to `$time_local`, which looks like `2022/07/11 18:52:41`.
I tried $time_local，it didn't work because the time format is not match；- can't be /。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7454: clickhouse-logger can't support datetime or date field in clickhouse
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7454:
URL: https://github.com/apache/apisix/issues/7454#issuecomment-1183929786
the default `timestamp` used in clickhouse-logger is `time_iso8601`, which looks like `2022-01-17T10:03:10+08:00`, you can update the metadata of clickhouse-logger to specify the format of the `timestamp`, see: https://github.com/apache/apisix/blob/master/docs/en/latest/plugins/clickhouse-logger.md#metadata
such as changing `$time_iso8601` to `$time_local`, which looks like `2022/07/11 18:52:41`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7454: clickhouse-logger can't support datetime or date field in clickhouse
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7454:
URL: https://github.com/apache/apisix/issues/7454#issuecomment-1184534058
> It must match strictly i think，and there is no other varible's format is like `2000-01-01 12:00:00`
Then you can modify the format of the regular extraction. AFAIK, the common time format used by Nginx is `$time_iso8601`or `$time_local`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7454: clickhouse-logger can't support datetime or date field in clickhouse
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7454:
URL: https://github.com/apache/apisix/issues/7454#issuecomment-1187188290
> So，this needs to update the clickhouse-logger plugins's code，right？
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] caozhi6655 commented on issue #7454: clickhouse-logger can't support datetime or date field in clickhouse
Posted by GitBox <gi...@apache.org>.
caozhi6655 commented on issue #7454:
URL: https://github.com/apache/apisix/issues/7454#issuecomment-1190016598
> > So，this needs to update the clickhouse-logger plugins's code，right？
>
> yes
hope to support it quickly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] github-actions[bot] commented on issue #7454: clickhouse-logger can't support datetime or date field in clickhouse
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7454:
URL: https://github.com/apache/apisix/issues/7454#issuecomment-1645333726
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org