You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Samir Desai (JIRA)" <ji...@apache.org> on 2019/02/26 17:08:00 UTC
[jira] [Comment Edited] (FLINK-11747) Elasticsearch 6 connector -
Expose RestHighLevelClient to allow for custom sniffing
[ https://issues.apache.org/jira/browse/FLINK-11747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778196#comment-16778196 ]
Samir Desai edited comment on FLINK-11747 at 2/26/19 5:07 PM:
--------------------------------------------------------------
[~Salatich], That is correct. The issue is more that it's very difficult to get access to the high level rest client using the ES flink connector out-of-box. The [base sink|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkBase.java#L299] calls {{bridge.createClient()}}, and the [Elasticsearch 6 sink|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-elasticsearch6/src/main/java/org/apache/flink/streaming/connectors/elasticsearch6/ElasticsearchSink.java#L72] internally instantiates the call bridge, so adding sniffing requires inheriting or reimplementing these classes.
When attempting to add sniffing, I realized that many of the classes (including the base sink and the Elasticsearch6 sink) cannot be extended easily, because of private member variables and constructors. Additionally, they're often marked with {{@PublicEvolving}} or {{@Internal}}, and you have reimplement the interface with your own implementation, but this requires copying a significant amount of code.
Additionally, the Elasticsearch6 sink relies on a package private [BulkProcessorIndexer|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-elasticsearch6/src/main/java/org/apache/flink/streaming/connectors/elasticsearch6/Elasticsearch6BulkProcessorIndexer.java], so if you reimplement the Elasticsearch6 sink in your project, you have to copy this file into your own project.
was (Author: yosami):
That is correct. The issue is more that it's very difficult to get access to the high level rest client using the ES flink connector out-of-box. The [base sink|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkBase.java#L299] calls {{bridge.createClient()}}, and the [Elasticsearch 6 sink|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-elasticsearch6/src/main/java/org/apache/flink/streaming/connectors/elasticsearch6/ElasticsearchSink.java#L72] internally instantiates the call bridge, so adding sniffing requires inheriting or reimplementing these classes.
When attempting to add sniffing, I realized that many of the classes (including the base sink and the Elasticsearch6 sink) cannot be extended easily, because of private member variables and constructors. Additionally, they're often marked with {{@PublicEvolving}} or {{@Internal}}, and you have reimplement the interface with your own implementation, but this requires copying a significant amount of code.
Additionally, the Elasticsearch6 sink relies on a package private [BulkProcessorIndexer|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-elasticsearch6/src/main/java/org/apache/flink/streaming/connectors/elasticsearch6/Elasticsearch6BulkProcessorIndexer.java], so if you reimplement the Elasticsearch6 sink in your project, you have to copy this file into your own project.
> Elasticsearch 6 connector - Expose RestHighLevelClient to allow for custom sniffing
> -----------------------------------------------------------------------------------
>
> Key: FLINK-11747
> URL: https://issues.apache.org/jira/browse/FLINK-11747
> Project: Flink
> Issue Type: Improvement
> Components: ElasticSearch Connector
> Reporter: Samir Desai
> Priority: Major
>
> In the Elasticsearch6 connector, the [RestClientFactory|https://github.com/apache/flink/blob/release-1.6/flink-connectors/flink-connector-elasticsearch6/src/main/java/org/apache/flink/streaming/connectors/elasticsearch6/RestClientFactory.java#L31] allows users to customize the {{RestClientBuilder}}. However, certain customizations like adding [Sniffing|https://www.elastic.co/guide/en/elasticsearch/client/java-rest/master/_usage.html]
> require access to the low-level rest client, which can be obtained through the high level rest client. The {{RestHighLevelClient}} is [instantiated|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-elasticsearch6/src/main/java/org/apache/flink/streaming/connectors/elasticsearch6/Elasticsearch6ApiCallBridge.java#L75] in the api call bridge, and is never exposed to the user.
> To my knowledge, the current Elasticsearch6 connector does not utilize sniffing or provide a way to add it in. The Elasticsearch6 connector should expose some type of access to the RestHighLevelClient to allow for custom sniffing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)