Secure TLS/SSL ElasticSearch connector for current and future connector

[Fritz Budiyanto <fb...@icloud.com>]

Hi All,

Anyone know if Flink has TLS/SSL support for the current ES connector ?
If yes, any sample configuration/code ?
If not, would TLS/SSL be support in the upcoming ES connector using Java High Level client ?

Thanks,
Fritz

Re: Secure TLS/SSL ElasticSearch connector for current and future connector

[Fritz Budiyanto <fb...@icloud.com>]

Hi Christophe,

Thanks so much for the pointers. That helps.

Looking at the latest update on https://issues.apache.org/jira/browse/FLINK-8101 <https://issues.apache.org/jira/browse/FLINK-8101>, there was an issue related to HLR retry handling. If I read this correctly, there is a bug in ES/HLR and some tests were failed because of that and hence this PR cant be merged. 

Also have you tried HLR based connector at scale, is it stable?

--
Fritz

> On Mar 26, 2018, at 3:18 PM, Christophe Jolif <cj...@gmail.com> wrote:
> 
> Hi Fritz,
> 
> I think the High Level Rest Client implementation in this PR: https://github.com/apache/flink/pull/5374 <https://github.com/apache/flink/pull/5374> should work. If you don't get the certificate properly available in your Java certs, you might want to redefine the createClient method to do something along those lines to get the context aware of it:
> 
> https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_encrypted_communication.html <https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_encrypted_communication.html>
> 
> We might want to amend the code to make that even easier (and also manage basic auth: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_basic_authentication.html <https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_basic_authentication.html>)
> 
> That said I'm unsure how the community wants to pursue on this next generation ES connector, because despite the obvious need from the number of requests, it does not seem there is a lot of traction to get something actually merged.
> 
> My current thinking would be to build a brand new Java REST High Level client-only based client, possibly breaking some compatibility with old APIs (that said my PR above it trying to keep the compatibility to the price of a few cast). This would leave undone the 5.2-6.0 support. And start back working on 6.1+. But at least there would be something "correct" for the future.
> 
> --
> Christophe
> 
> On Mon, Mar 26, 2018 at 11:38 PM, Fritz Budiyanto <fbudiyan@icloud.com <ma...@icloud.com>> wrote:
> Hi All,
> 
> Anyone know if Flink has TLS/SSL support for the current ES connector ?
> If yes, any sample configuration/code ?
> If not, would TLS/SSL be support in the upcoming ES connector using Java High Level client ?
> 
> Thanks,
> Fritz
> 
> 
> 

Re: Secure TLS/SSL ElasticSearch connector for current and future connector

[Christophe Jolif <cj...@gmail.com>]

Hi Fritz,

I think the High Level Rest Client implementation in this PR:
https://github.com/apache/flink/pull/5374 should work. If you don't get the
certificate properly available in your Java certs, you might want to
redefine the createClient method to do something along those lines to get
the context aware of it:

https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_encrypted_communication.html

We might want to amend the code to make that even easier (and also manage
basic auth:
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_basic_authentication.html
)

That said I'm unsure how the community wants to pursue on this next
generation ES connector, because despite the obvious need from the number
of requests, it does not seem there is a lot of traction to get something
actually merged.

My current thinking would be to build a brand new Java REST High Level
client-only based client, possibly breaking some compatibility with old
APIs (that said my PR above it trying to keep the compatibility to the
price of a few cast). This would leave undone the 5.2-6.0 support. And
start back working on 6.1+. But at least there would be something "correct"
for the future.

--
Christophe

On Mon, Mar 26, 2018 at 11:38 PM, Fritz Budiyanto <fb...@icloud.com>
wrote:

> Hi All,
>
> Anyone know if Flink has TLS/SSL support for the current ES connector ?
> If yes, any sample configuration/code ?
> If not, would TLS/SSL be support in the upcoming ES connector using Java
> High Level client ?
>
> Thanks,
> Fritz