You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Jonathan Greenleaf <jo...@gmail.com> on 2017/01/28 14:44:51 UTC

SSL with Elasticsearch / Shield

I have been able to start the server on 8443, but have not been able to make a connection to my Elasticsearch data node over SSL.  I verified I can curl (curl --insecure -v -u user:pwd https:xx.xx.xx.xx:9200/...) the ES box with the Shield credentials.

Within the interpreter I have
name -> value
shield.user -> user:pwd
shield.transport.ssl -> true

we use port 9200 and force the gets/searches to require https.

and I don't know if this is even used but I included this dependency:
/zeppelin/interpreter/elasticsearch/shield-2.4.4.jar
based on what I read here:  https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/elasticsearch.html

/zeppelin/logs/zeppelin-root-ip-10-2-3-144.log
 INFO [2017-01-27 20:38:44,556] ({main} AbstractConnector.java[doStart]:266) - Started ServerConnector@30aba78f{SSL-HTTP/1.1}{0.0.0.0:8443}

elasticsearch log complains:
[2017-01-27 21:23:18,161][WARN ][shield.transport.netty   ] [esdata3] received plaintext http traffic on a https channel, closing connection [id: 0xf43a9b2f, /xx.xx.xx.xx:36188 => /xx.xx.xx.xx:9200]

I built from source - 0.8.0-SNAPSHOT.  I also added this to /zeppelin/elasticsearch/pom.xml

      <!-- add the shield jar as a dependency -->
      <dependency>
         <groupId>org.elasticsearch.plugin</groupId>
         <artifactId>shield</artifactId>
         <version>2.4.4</version>
      </dependency>

I'm confused what I need to do with Shield on the Zeppelin server.  Do I need to copy a cert from my existing Shield setup on my data node?

Any pointers appreciated.  
Thanks - Jonathan

Re: SSL with Elasticsearch / Shield

Posted by Bruno Bonnin <bb...@gmail.com>.
Hello,

I have created a ticket in JIRA to address this (
https://issues.apache.org/jira/browse/ZEPPELIN-2031), because it is not
supported with this version of Shield.

Bruno

2017-01-30 17:45 GMT+01:00 Jonathan Greenleaf <jo...@gmail.com>:

> Thank you Bruno.  I have switched to trying to use just Shield on 9300.  I
> have copied the latest Shield jar here:   /zeppelin/interpreter/elasticsearch/shield-2.4.4.jar
> and edited the /zeppelin/elasticsearch/pom.xml dependency per:
> https://www.elastic.co/guide/en/shield/current/_using_
> elasticsearch_java_clients_with_shield.html
>
> I have restarted my ES interpreter with the correct shield settings
> (shield.ssl.user, shield.ssl.keystore.path) - the same that my main
> application uses successfully.
>
> My issue it seems is that the Shield plugin is not being loaded.  Per
> zeppelin-interpreter-elasticsearch-root-ip-xx-xx-xx-xx.log
>
>  INFO [2017-01-30 16:06:29,471] ({pool-2-thread-2}
> ElasticsearchInterpreter.java[open]:132)- prop={elasticsearch.result.
> size=10...
>  INFO [2017-01-30 16:06:29,572] ({pool-2-thread-2}
> PluginsService.java[<init>]:180) - [Vashti] modules [], plugins [], sites
> []
>  INFO [2017-01-30 16:06:31,248] ({pool-2-thread-2}
> TransportClientNodesService.java[doSample]:420) - [Vashti] failed to get
> node info for {#transport#-1}{xx.xx.xx.xx}{xx.xx.xx.xx:9300},
> disconnecting...
>
> I would expect to see something in the plugins[] ^.  Any additional
> pointers appreciated.
> Thanks, Jonathan
>
> On 2017-01-28 15:18 (-0500), Bruno Bonnin <bb...@gmail.com> wrote:
> > Hello,
> >
> > For the moment, in Zeppelin, the HTTP client for elasticsearch does not
> > support SSL.
> > If you want to use the elasticsearch transport client, maybe you should
> try
> > to use the port 9300 and for SSL, you have to add other parameters, such
> as
> > "shield.ssl.keystore.path" and "shield.ssl.keystore.password" (there are
> a
> > description of some of them here:
> > https://www.elastic.co/guide/en/shield/current/_using_
> elasticsearch_java_clients_with_shield.html
> > )
> >
> > Bruno
> >
> >
> > 2017-01-28 15:44 GMT+01:00 Jonathan Greenleaf <
> jonathangreenleaf@gmail.com>:
> >
> > > I have been able to start the server on 8443, but have not been able to
> > > make a connection to my Elasticsearch data node over SSL.  I verified
> I can
> > > curl (curl --insecure -v -u user:pwd https:xx.xx.xx.xx:9200/...) the
> ES box
> > > with the Shield credentials.
> > >
> > > Within the interpreter I have
> > > name -> value
> > > shield.user -> user:pwd
> > > shield.transport.ssl -> true
> > >
> > > we use port 9200 and force the gets/searches to require https.
> > >
> > > and I don't know if this is even used but I included this dependency:
> > > /zeppelin/interpreter/elasticsearch/shield-2.4.4.jar
> > > based on what I read here:  https://zeppelin.apache.org/
> > > docs/0.7.0-SNAPSHOT/interpreter/elasticsearch.html
> > >
> > > /zeppelin/logs/zeppelin-root-ip-10-2-3-144.log
> > >  INFO [2017-01-27 20:38:44,556] ({main} AbstractConnector.java[
> doStart]:266)
> > > - Started ServerConnector@30aba78f{SSL-HTTP/1.1}{0.0.0.0:8443}
> > >
> > > elasticsearch log complains:
> > > [2017-01-27 21:23:18,161][WARN ][shield.transport.netty   ] [esdata3]
> > > received plaintext http traffic on a https channel, closing connection
> [id:
> > > 0xf43a9b2f, /xx.xx.xx.xx:36188 => /xx.xx.xx.xx:9200]
> > >
> > > I built from source - 0.8.0-SNAPSHOT.  I also added this to
> > > /zeppelin/elasticsearch/pom.xml
> > >
> > >       <!-- add the shield jar as a dependency -->
> > >       <dependency>
> > >          <groupId>org.elasticsearch.plugin</groupId>
> > >          <artifactId>shield</artifactId>
> > >          <version>2.4.4</version>
> > >       </dependency>
> > >
> > > I'm confused what I need to do with Shield on the Zeppelin server.  Do
> I
> > > need to copy a cert from my existing Shield setup on my data node?
> > >
> > > Any pointers appreciated.
> > > Thanks - Jonathan
> > >
> >
>

Re: SSL with Elasticsearch / Shield

Posted by Jonathan Greenleaf <jo...@gmail.com>.
Thank you Bruno.  I have switched to trying to use just Shield on 9300.  I have copied the latest Shield jar here:   /zeppelin/interpreter/elasticsearch/shield-2.4.4.jar and edited the /zeppelin/elasticsearch/pom.xml dependency per:
https://www.elastic.co/guide/en/shield/current/_using_elasticsearch_java_clients_with_shield.html

I have restarted my ES interpreter with the correct shield settings (shield.ssl.user, shield.ssl.keystore.path) - the same that my main application uses successfully. 

My issue it seems is that the Shield plugin is not being loaded.  Per zeppelin-interpreter-elasticsearch-root-ip-xx-xx-xx-xx.log

 INFO [2017-01-30 16:06:29,471] ({pool-2-thread-2} ElasticsearchInterpreter.java[open]:132)- prop={elasticsearch.result.size=10...
 INFO [2017-01-30 16:06:29,572] ({pool-2-thread-2} PluginsService.java[<init>]:180) - [Vashti] modules [], plugins [], sites []
 INFO [2017-01-30 16:06:31,248] ({pool-2-thread-2} TransportClientNodesService.java[doSample]:420) - [Vashti] failed to get node info for {#transport#-1}{xx.xx.xx.xx}{xx.xx.xx.xx:9300}, disconnecting...

I would expect to see something in the plugins[] ^.  Any additional pointers appreciated.  
Thanks, Jonathan

On 2017-01-28 15:18 (-0500), Bruno Bonnin <bb...@gmail.com> wrote: 
> Hello,
> 
> For the moment, in Zeppelin, the HTTP client for elasticsearch does not
> support SSL.
> If you want to use the elasticsearch transport client, maybe you should try
> to use the port 9300 and for SSL, you have to add other parameters, such as
> "shield.ssl.keystore.path" and "shield.ssl.keystore.password" (there are a
> description of some of them here:
> https://www.elastic.co/guide/en/shield/current/_using_elasticsearch_java_clients_with_shield.html
> )
> 
> Bruno
> 
> 
> 2017-01-28 15:44 GMT+01:00 Jonathan Greenleaf <jo...@gmail.com>:
> 
> > I have been able to start the server on 8443, but have not been able to
> > make a connection to my Elasticsearch data node over SSL.  I verified I can
> > curl (curl --insecure -v -u user:pwd https:xx.xx.xx.xx:9200/...) the ES box
> > with the Shield credentials.
> >
> > Within the interpreter I have
> > name -> value
> > shield.user -> user:pwd
> > shield.transport.ssl -> true
> >
> > we use port 9200 and force the gets/searches to require https.
> >
> > and I don't know if this is even used but I included this dependency:
> > /zeppelin/interpreter/elasticsearch/shield-2.4.4.jar
> > based on what I read here:  https://zeppelin.apache.org/
> > docs/0.7.0-SNAPSHOT/interpreter/elasticsearch.html
> >
> > /zeppelin/logs/zeppelin-root-ip-10-2-3-144.log
> >  INFO [2017-01-27 20:38:44,556] ({main} AbstractConnector.java[doStart]:266)
> > - Started ServerConnector@30aba78f{SSL-HTTP/1.1}{0.0.0.0:8443}
> >
> > elasticsearch log complains:
> > [2017-01-27 21:23:18,161][WARN ][shield.transport.netty   ] [esdata3]
> > received plaintext http traffic on a https channel, closing connection [id:
> > 0xf43a9b2f, /xx.xx.xx.xx:36188 => /xx.xx.xx.xx:9200]
> >
> > I built from source - 0.8.0-SNAPSHOT.  I also added this to
> > /zeppelin/elasticsearch/pom.xml
> >
> >       <!-- add the shield jar as a dependency -->
> >       <dependency>
> >          <groupId>org.elasticsearch.plugin</groupId>
> >          <artifactId>shield</artifactId>
> >          <version>2.4.4</version>
> >       </dependency>
> >
> > I'm confused what I need to do with Shield on the Zeppelin server.  Do I
> > need to copy a cert from my existing Shield setup on my data node?
> >
> > Any pointers appreciated.
> > Thanks - Jonathan
> >
>

Re: SSL with Elasticsearch / Shield

Posted by Bruno Bonnin <bb...@gmail.com>.
Hello,

For the moment, in Zeppelin, the HTTP client for elasticsearch does not
support SSL.
If you want to use the elasticsearch transport client, maybe you should try
to use the port 9300 and for SSL, you have to add other parameters, such as
"shield.ssl.keystore.path" and "shield.ssl.keystore.password" (there are a
description of some of them here:
https://www.elastic.co/guide/en/shield/current/_using_elasticsearch_java_clients_with_shield.html
)

Bruno


2017-01-28 15:44 GMT+01:00 Jonathan Greenleaf <jo...@gmail.com>:

> I have been able to start the server on 8443, but have not been able to
> make a connection to my Elasticsearch data node over SSL.  I verified I can
> curl (curl --insecure -v -u user:pwd https:xx.xx.xx.xx:9200/...) the ES box
> with the Shield credentials.
>
> Within the interpreter I have
> name -> value
> shield.user -> user:pwd
> shield.transport.ssl -> true
>
> we use port 9200 and force the gets/searches to require https.
>
> and I don't know if this is even used but I included this dependency:
> /zeppelin/interpreter/elasticsearch/shield-2.4.4.jar
> based on what I read here:  https://zeppelin.apache.org/
> docs/0.7.0-SNAPSHOT/interpreter/elasticsearch.html
>
> /zeppelin/logs/zeppelin-root-ip-10-2-3-144.log
>  INFO [2017-01-27 20:38:44,556] ({main} AbstractConnector.java[doStart]:266)
> - Started ServerConnector@30aba78f{SSL-HTTP/1.1}{0.0.0.0:8443}
>
> elasticsearch log complains:
> [2017-01-27 21:23:18,161][WARN ][shield.transport.netty   ] [esdata3]
> received plaintext http traffic on a https channel, closing connection [id:
> 0xf43a9b2f, /xx.xx.xx.xx:36188 => /xx.xx.xx.xx:9200]
>
> I built from source - 0.8.0-SNAPSHOT.  I also added this to
> /zeppelin/elasticsearch/pom.xml
>
>       <!-- add the shield jar as a dependency -->
>       <dependency>
>          <groupId>org.elasticsearch.plugin</groupId>
>          <artifactId>shield</artifactId>
>          <version>2.4.4</version>
>       </dependency>
>
> I'm confused what I need to do with Shield on the Zeppelin server.  Do I
> need to copy a cert from my existing Shield setup on my data node?
>
> Any pointers appreciated.
> Thanks - Jonathan
>