You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apisix.apache.org by "ccxhwmy@gmail.com" <cc...@gmail.com> on 2022/08/09 15:49:10 UTC
[Proposal] support Elasticsearch logging plugin
Hi folks,
Elasticsearch is the distributed search and analytics engine that provides
a unified data store for solutions built on the Elastic Stack. According to the
latest database search engine ranking [1], Elasticsearch has been ranked
second Kibana provides access to the Elastic Observability, Security, and
Enterprise Search solutions, enables you to interactively explore,
visualize, and share insights into your data. APISIX report logs with
Elasticsearch's Bulk API [2].
The following are the design and technical details:
1. Name
elasticsearch-logging
1. Configuration
{
"endpoint": {
"uri": "https://127.0.0.1:9200/<target>",
"index": "service",
"type": "collector",
"username": "elastic",
"password": "elastic",
"ssl": true
},
"inactive_timeout": 10,
"max_retry_count": 0,
"buffer_duration": 60,
"retry_delay": 1,
"batch_max_size": 1
}
- `endpoint` ElasticSearch HTTP/HTTPS endpoint
- `endpoint.uri` ElasticSearch HTTP/HTTPS request uri,
`elasticsearch-logging` will add "_bulk" after it [3].
- `endpoint.index` ElasticSearch HTTP/HTTPS "_index" field [4]
- `endpoint.type` ElasticSearch HTTP/HTTPS "_type" field [5]
- `endpoint.username` ElasticSearch HTTP/HTTPS
- `max_retry_count` maximum number of retries before removing the
entry from the processing pipeline when an error occurs.
- `retry_delay` number of seconds the process execution should
be delayed if the execution fails.
- `buffer_duration` max age in seconds of the oldest entry in a batch
before the batch must be processed
- `inactive_timeout` max age in seconds when the buffer will be flushed
if inactive
- `batch_max_size` max size of each batch
3. Details
3.1 Configuration process
1. Get the username and password of Elasticsearch if needed.
2. Set the request URI, index, type(not necessary, Elasticsearch will config
it with default config if it is null) to the plugin configuration.
3.2 HTTP Request process
1. Obtain and assemble request information data in the APISIX Log stage,
data format refer to [6].
2. Add the assemble request data to the batch queue.
3. When the threshold of the batch queue is triggered, the requested data
is subbmitted in batches to ElasticSearch.
[1] https://db-engines.com/en/ranking/search+engine
[2] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk
[3] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk-api-request
[4] https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-index-field.html#mapping-index-field
[5] https://www.elastic.co/guide/en/elasticsearch/reference/7.17/mapping-type-field.html#mapping-type-field
[6] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk
By the way, I have create a issue about it
Thanks,
ccxhwmy@gmail.com
Re: [Proposal] support Elasticsearch logging plugin
Posted by Ming Wen <we...@apache.org>.
Cool, looks good to me:)
Thanks,
Ming Wen, Apache APISIX PMC Chair
Twitter: _WenMing
ccxhwmy@gmail.com <cc...@gmail.com> 于2022年8月9日周二 23:49写道：
> Hi folks,
>
> Elasticsearch is the distributed search and analytics engine that provides
> a unified data store for solutions built on the Elastic Stack. According
> to the
> latest database search engine ranking [1], Elasticsearch has been ranked
> second Kibana provides access to the Elastic Observability, Security, and
> Enterprise Search solutions, enables you to interactively explore,
> visualize, and share insights into your data. APISIX report logs with
> Elasticsearch's Bulk API [2].
>
> The following are the design and technical details:
>
> 1. Name
>
> elasticsearch-logging
>
> 1. Configuration
>
> {
> "endpoint": {
> "uri": "https://127.0.0.1:9200/<target>",
> "index": "service",
> "type": "collector",
> "username": "elastic",
> "password": "elastic",
> "ssl": true
> },
> "inactive_timeout": 10,
> "max_retry_count": 0,
> "buffer_duration": 60,
> "retry_delay": 1,
> "batch_max_size": 1
> }
>
> - `endpoint` ElasticSearch HTTP/HTTPS endpoint
> - `endpoint.uri` ElasticSearch HTTP/HTTPS request uri,
> `elasticsearch-logging` will add "_bulk" after it [3].
> - `endpoint.index` ElasticSearch HTTP/HTTPS "_index" field [4]
> - `endpoint.type` ElasticSearch HTTP/HTTPS "_type" field [5]
> - `endpoint.username` ElasticSearch HTTP/HTTPS
> - `max_retry_count` maximum number of retries before removing the
> entry from the processing pipeline when an error occurs.
> - `retry_delay` number of seconds the process execution should
> be delayed if the execution fails.
> - `buffer_duration` max age in seconds of the oldest entry in a batch
> before the batch must be processed
> - `inactive_timeout` max age in seconds when the buffer will be flushed
> if inactive
> - `batch_max_size` max size of each batch
>
> 3. Details
>
> 3.1 Configuration process
> 1. Get the username and password of Elasticsearch if needed.
> 2. Set the request URI, index, type(not necessary, Elasticsearch will
> config
> it with default config if it is null) to the plugin configuration.
>
> 3.2 HTTP Request process
> 1. Obtain and assemble request information data in the APISIX Log stage,
> data format refer to [6].
> 2. Add the assemble request data to the batch queue.
> 3. When the threshold of the batch queue is triggered, the requested data
> is subbmitted in batches to ElasticSearch.
>
> [1] https://db-engines.com/en/ranking/search+engine
> [2]
> https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk
> [3]
> https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk-api-request
> [4]
> https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-index-field.html#mapping-index-field
> [5]
> https://www.elastic.co/guide/en/elasticsearch/reference/7.17/mapping-type-field.html#mapping-type-field
> [6]
> https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk
>
> By the way, I have create a issue about it
>
>
>
> Thanks,
> ccxhwmy@gmail.com
>
Re: Re: [Proposal] support Elasticsearch logging plugin
Posted by Zexuan Luo <sp...@apache.org>.
> It seems to be ambiguous, is it more suitable to use `"ssl_verify": true`?
Agree.
ccxhwmy@gmail.com <cc...@gmail.com> 于2022年8月10日周三 14:38写道：
>
> That is used to determine whether to use `ssl_verify = true` in the lua-resy-http
> refer to option `ssl_verify` in:
> https://github.com/ledgetech/lua-resty-http/blob/master/README.md#connect
>
> It seems to be ambiguous, is it more suitable to use `"ssl_verify": true`?
>
>
>
> ccxhwmy@gmail.com
>
> From: Zexuan Luo
> Date: 2022-08-10 11:46
> To: dev
> Subject: Re: [Proposal] support Elasticsearch logging plugin
> ```
> "endpoint": {
> "uri": "https://127.0.0.1:9200/<targe
> t>",
> "index": "service",
> "type": "collector",
> "username": "elastic",
> "password": "elastic",
> "ssl": true
> },
> ```
> I am curious why we need the `ssl` field when the scheme is specified
> in the URL?
>
> ccxhwmy@gmail.com <cc...@gmail.com> 于2022年8月9日周二 23:49写道：
> >
> > Hi folks,
> >
> > Elasticsearch is the distributed search and analytics engine that provides
> > a unified data store for solutions built on the Elastic Stack. According to the
> > latest database search engine ranking [1], Elasticsearch has been ranked
> > second Kibana provides access to the Elastic Observability, Security, and
> > Enterprise Search solutions, enables you to interactively explore,
> > visualize, and share insights into your data. APISIX report logs with
> > Elasticsearch's Bulk API [2].
> >
> > The following are the design and technical details:
> >
> > 1. Name
> >
> > elasticsearch-logging
> >
> > 1. Configuration
> >
> > {
> > "endpoint": {
> > "uri": "https://127.0.0.1:9200/<target>",
> > "index": "service",
> > "type": "collector",
> > "username": "elastic",
> > "password": "elastic",
> > "ssl": true
> > },
> > "inactive_timeout": 10,
> > "max_retry_count": 0,
> > "buffer_duration": 60,
> > "retry_delay": 1,
> > "batch_max_size": 1
> > }
> >
> > - `endpoint` ElasticSearch HTTP/HTTPS endpoint
> > - `endpoint.uri` ElasticSearch HTTP/HTTPS request uri,
> > `elasticsearch-logging` will add "_bulk" after it [3].
> > - `endpoint.index` ElasticSearch HTTP/HTTPS "_index" field [4]
> > - `endpoint.type` ElasticSearch HTTP/HTTPS "_type" field [5]
> > - `endpoint.username` ElasticSearch HTTP/HTTPS
> > - `max_retry_count` maximum number of retries before removing the
> > entry from the processing pipeline when an error occurs.
> > - `retry_delay` number of seconds the process execution should
> > be delayed if the execution fails.
> > - `buffer_duration` max age in seconds of the oldest entry in a batch
> > before the batch must be processed
> > - `inactive_timeout` max age in seconds when the buffer will be flushed
> > if inactive
> > - `batch_max_size` max size of each batch
> >
> > 3. Details
> >
> > 3.1 Configuration process
> > 1. Get the username and password of Elasticsearch if needed.
> > 2. Set the request URI, index, type(not necessary, Elasticsearch will config
> > it with default config if it is null) to the plugin configuration.
> >
> > 3.2 HTTP Request process
> > 1. Obtain and assemble request information data in the APISIX Log stage,
> > data format refer to [6].
> > 2. Add the assemble request data to the batch queue.
> > 3. When the threshold of the batch queue is triggered, the requested data
> > is subbmitted in batches to ElasticSearch.
> >
> > [1] https://db-engines.com/en/ranking/search+engine
> > [2] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk
> > [3] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk-api-request
> > [4] https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-index-field.html#mapping-index-field
> > [5] https://www.elastic.co/guide/en/elasticsearch/reference/7.17/mapping-type-field.html#mapping-type-field
> > [6] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk
> >
> > By the way, I have create a issue about it
> >
> >
> >
> > Thanks,
> > ccxhwmy@gmail.com
Re: Re: [Proposal] support Elasticsearch logging plugin
Posted by "ccxhwmy@gmail.com" <cc...@gmail.com>.
That is used to determine whether to use `ssl_verify = true` in the lua-resy-http
refer to option `ssl_verify` in:
https://github.com/ledgetech/lua-resty-http/blob/master/README.md#connect
It seems to be ambiguous, is it more suitable to use `"ssl_verify": true`?
ccxhwmy@gmail.com
From: Zexuan Luo
Date: 2022-08-10 11:46
To: dev
Subject: Re: [Proposal] support Elasticsearch logging plugin
```
"endpoint": {
"uri": "https://127.0.0.1:9200/<targe
t>",
"index": "service",
"type": "collector",
"username": "elastic",
"password": "elastic",
"ssl": true
},
```
I am curious why we need the `ssl` field when the scheme is specified
in the URL?
ccxhwmy@gmail.com <cc...@gmail.com> 于2022年8月9日周二 23:49写道：
>
> Hi folks,
>
> Elasticsearch is the distributed search and analytics engine that provides
> a unified data store for solutions built on the Elastic Stack. According to the
> latest database search engine ranking [1], Elasticsearch has been ranked
> second Kibana provides access to the Elastic Observability, Security, and
> Enterprise Search solutions, enables you to interactively explore,
> visualize, and share insights into your data. APISIX report logs with
> Elasticsearch's Bulk API [2].
>
> The following are the design and technical details:
>
> 1. Name
>
> elasticsearch-logging
>
> 1. Configuration
>
> {
> "endpoint": {
> "uri": "https://127.0.0.1:9200/<target>",
> "index": "service",
> "type": "collector",
> "username": "elastic",
> "password": "elastic",
> "ssl": true
> },
> "inactive_timeout": 10,
> "max_retry_count": 0,
> "buffer_duration": 60,
> "retry_delay": 1,
> "batch_max_size": 1
> }
>
> - `endpoint` ElasticSearch HTTP/HTTPS endpoint
> - `endpoint.uri` ElasticSearch HTTP/HTTPS request uri,
> `elasticsearch-logging` will add "_bulk" after it [3].
> - `endpoint.index` ElasticSearch HTTP/HTTPS "_index" field [4]
> - `endpoint.type` ElasticSearch HTTP/HTTPS "_type" field [5]
> - `endpoint.username` ElasticSearch HTTP/HTTPS
> - `max_retry_count` maximum number of retries before removing the
> entry from the processing pipeline when an error occurs.
> - `retry_delay` number of seconds the process execution should
> be delayed if the execution fails.
> - `buffer_duration` max age in seconds of the oldest entry in a batch
> before the batch must be processed
> - `inactive_timeout` max age in seconds when the buffer will be flushed
> if inactive
> - `batch_max_size` max size of each batch
>
> 3. Details
>
> 3.1 Configuration process
> 1. Get the username and password of Elasticsearch if needed.
> 2. Set the request URI, index, type(not necessary, Elasticsearch will config
> it with default config if it is null) to the plugin configuration.
>
> 3.2 HTTP Request process
> 1. Obtain and assemble request information data in the APISIX Log stage,
> data format refer to [6].
> 2. Add the assemble request data to the batch queue.
> 3. When the threshold of the batch queue is triggered, the requested data
> is subbmitted in batches to ElasticSearch.
>
> [1] https://db-engines.com/en/ranking/search+engine
> [2] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk
> [3] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk-api-request
> [4] https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-index-field.html#mapping-index-field
> [5] https://www.elastic.co/guide/en/elasticsearch/reference/7.17/mapping-type-field.html#mapping-type-field
> [6] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk
>
> By the way, I have create a issue about it
>
>
>
> Thanks,
> ccxhwmy@gmail.com
Re: [Proposal] support Elasticsearch logging plugin
Posted by Zexuan Luo <sp...@apache.org>.
```
"endpoint": {
"uri": "https://127.0.0.1:9200/<targe
t>",
"index": "service",
"type": "collector",
"username": "elastic",
"password": "elastic",
"ssl": true
},
```
I am curious why we need the `ssl` field when the scheme is specified
in the URL?
ccxhwmy@gmail.com <cc...@gmail.com> 于2022年8月9日周二 23:49写道：
>
> Hi folks,
>
> Elasticsearch is the distributed search and analytics engine that provides
> a unified data store for solutions built on the Elastic Stack. According to the
> latest database search engine ranking [1], Elasticsearch has been ranked
> second Kibana provides access to the Elastic Observability, Security, and
> Enterprise Search solutions, enables you to interactively explore,
> visualize, and share insights into your data. APISIX report logs with
> Elasticsearch's Bulk API [2].
>
> The following are the design and technical details:
>
> 1. Name
>
> elasticsearch-logging
>
> 1. Configuration
>
> {
> "endpoint": {
> "uri": "https://127.0.0.1:9200/<target>",
> "index": "service",
> "type": "collector",
> "username": "elastic",
> "password": "elastic",
> "ssl": true
> },
> "inactive_timeout": 10,
> "max_retry_count": 0,
> "buffer_duration": 60,
> "retry_delay": 1,
> "batch_max_size": 1
> }
>
> - `endpoint` ElasticSearch HTTP/HTTPS endpoint
> - `endpoint.uri` ElasticSearch HTTP/HTTPS request uri,
> `elasticsearch-logging` will add "_bulk" after it [3].
> - `endpoint.index` ElasticSearch HTTP/HTTPS "_index" field [4]
> - `endpoint.type` ElasticSearch HTTP/HTTPS "_type" field [5]
> - `endpoint.username` ElasticSearch HTTP/HTTPS
> - `max_retry_count` maximum number of retries before removing the
> entry from the processing pipeline when an error occurs.
> - `retry_delay` number of seconds the process execution should
> be delayed if the execution fails.
> - `buffer_duration` max age in seconds of the oldest entry in a batch
> before the batch must be processed
> - `inactive_timeout` max age in seconds when the buffer will be flushed
> if inactive
> - `batch_max_size` max size of each batch
>
> 3. Details
>
> 3.1 Configuration process
> 1. Get the username and password of Elasticsearch if needed.
> 2. Set the request URI, index, type(not necessary, Elasticsearch will config
> it with default config if it is null) to the plugin configuration.
>
> 3.2 HTTP Request process
> 1. Obtain and assemble request information data in the APISIX Log stage,
> data format refer to [6].
> 2. Add the assemble request data to the batch queue.
> 3. When the threshold of the batch queue is triggered, the requested data
> is subbmitted in batches to ElasticSearch.
>
> [1] https://db-engines.com/en/ranking/search+engine
> [2] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk
> [3] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk-api-request
> [4] https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-index-field.html#mapping-index-field
> [5] https://www.elastic.co/guide/en/elasticsearch/reference/7.17/mapping-type-field.html#mapping-type-field
> [6] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk
>
> By the way, I have create a issue about it
>
>
>
> Thanks,
> ccxhwmy@gmail.com