Flume elasticsearch sink version upgrade

[徐文平 <we...@163.com>]

Hi, I want to upgrade the version of flume-elasticsearch-sink elasticsearch.




At present, flume uses elasticsearch as a category of sink, but the currently supported version of elasticsearch is too low, 0.90.1, and the latest version has reached 8.x.x




At the same time, elasticsearch removed the type syntax between 6.x.x and 7.x.x, which is a bit too weak for the existing flume-elasticsearch-sink component.




Based on this, I plan to upgrade the elasticsearch version here. The overall process generally includes the following processes:




1. List the existing flume-elasticsearch-sink configuration and supported functions, and write the schema characteristics of elasticsearch.

This is because the current stage uses tcp and http to write data, but in the higher version of elasticsearch, it may not be supported.




2. Use the Java Low Level REST Client provided by elasticsearch to realize the sink function.

The low level rest client provided by elasticsearch complies with the apache protocol, as shown in:

https://github.com/elastic/elasticsearch/tree/main/licenses

https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/current/java-rest-low-usage-maven.html

According to the official website, The low-level REST client is compatible with all Elasticsearch versions.

I have successfully written data to elasticsearch 0.90.1 through the 6.8.23 version Http protocol

<dependency>

     <groupId>org.elasticsearch.client</groupId>

     <artifactId>elasticsearch-rest-client</artifactId>

     <version>6.8.23</version>

</dependency>




3. Version compatibility test.

Considering the existing version, it is necessary to conduct a compatibility test for each major version of elasticsearch in the past. Here, I understand that the main focus is on writing the results of the following versions

|
0.90.13
|
|
1.7.6
|
|
2.4.6
|
|
5.6.16
|
|
6.8.23
|
|
7.17.9
|
| 8.6.2 |




Finally, I hope you can give some suggestions and opinions, and hope that I can successfully complete the upgrade of elasticsearch, thank you.