You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/08/09 15:44:53 UTC

[GitHub] [apisix] ccxhwmy opened a new issue, #7636: feat: As a user, I want to report APISIX's logs to Elasticsearch, so that i can check the logs with Kibana easily

ccxhwmy opened a new issue, #7636:
URL: https://github.com/apache/apisix/issues/7636

### Description

As a user, I want to report APISIX's logs to Elasticsearch, so that i can check the logs with Kibana easily.

Elasticsearch is the distributed search and analytics engine that provides
a unified data store for solutions built on the Elastic Stack. According to the
latest database search engine ranking [1], Elasticsearch has been ranked
second Kibana provides access to the Elastic Observability, Security, and
Enterprise Search solutions, enables you to interactively explore,
visualize, and share insights into your data. APISIX report logs with
Elasticsearch's Bulk API [2].

The following are the design and technical details:

1. Name

elasticsearch-logging

2. Configuration

```json
{
"endpoint": {
"uri": "https://127.0.0.1:9200/<target>",
"index": "service",
"type": "collector",
"username": "elastic",
"password": "elastic",
"ssl": true
},
"inactive_timeout": 10,
"max_retry_count": 0,
"buffer_duration": 60,
"retry_delay": 1,
"batch_max_size": 1
}
```

- `endpoint` Elasticsearch HTTP/HTTPS endpoint
- `endpoint.uri` Elasticsearch HTTP/HTTPS request uri,
`elasticsearch-logging` will add "_bulk" at the end of it [3].
- `endpoint.index` Elasticsearch HTTP/HTTPS "_index" field [4]
- `endpoint.type` Elasticsearch HTTP/HTTPS "_type" field [5]
- `endpoint.username` Elasticsearch HTTP/HTTPS

- `max_retry_count` maximum number of retries before removing the
entry from the processing pipeline when an error occurs.
- `retry_delay` number of seconds the process execution should
be delayed if the execution fails.
- `buffer_duration` max age in seconds of the oldest entry in a batch
before the batch must be processed
- `inactive_timeout` max age in seconds when the buffer will be flushed
if inactive
- `batch_max_size` max size of each batch

3. Details

3.1 Configuration process
1. Get the username and password of Elasticsearch if needed.
2. Set the request URI, index, type(not necessary, Elasticsearch will config
it with default config if it is null) to the plugin configuration.

3.2 HTTP Request process
1. Obtain and assemble request information data in the APISIX Log stage,
data format refer to [6].
2. Add the assemble request data to the batch queue.
3. When the threshold of the batch queue is triggered, the requested data
is subbmitted in batches to ElasticSearch.

[1] [https://db-engines.com/en/ranking/search+engine](https://github.com/apache/apisix/issues/[1]%20https://db-engines.com/en/ranking/search+engine)
[2] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk
[3] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk-api-request
[4] [https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-index-field.html#mapping-index-field](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/mapping-type-field.html#mapping-type-field)
[5] https://www.elastic.co/guide/en/elasticsearch/reference/7.17/mapping-type-field.html#mapping-type-field
[6] [https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk)

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] spacewander closed issue #7636: feat: As a user, I want to report APISIX's logs to Elasticsearch, so that i can check the logs with Kibana easily

Posted by GitBox <gi...@apache.org>.

spacewander closed issue #7636: feat: As a user, I want to report APISIX's logs to Elasticsearch, so that i can check the logs with Kibana easily
URL: https://github.com/apache/apisix/issues/7636


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] tzssangglass commented on issue #7636: feat: As a user, I want to report APISIX's logs to Elasticsearch, so that i can check the logs with Kibana easily

Posted by GitBox <gi...@apache.org>.

tzssangglass commented on issue #7636:
URL: https://github.com/apache/apisix/issues/7636#issuecomment-1210088794

   LGTM, would you plan to work on this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] ccxhwmy commented on issue #7636: feat: As a user, I want to report APISIX's logs to Elasticsearch, so that i can check the logs with Kibana easily

Posted by GitBox <gi...@apache.org>.

ccxhwmy commented on issue #7636:
URL: https://github.com/apache/apisix/issues/7636#issuecomment-1210096241

   > LGTM, would you plan to work on this?
   
   Yes, i'd like to.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org