[jira] [Commented] (METRON-1465) X-pack support for Elasticsearch

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/METRON-1465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16422724#comment-16422724 ] 

ASF GitHub Bot commented on METRON-1465:
----------------------------------------

Github user mmiklavc commented on a diff in the pull request:

    https://github.com/apache/metron/pull/946#discussion_r178584154
  
    --- Diff: metron-deployment/Kerberos-manual-setup.md ---
    @@ -533,3 +534,211 @@ In order to correct this, you should:
     ### References
     
     * [https://github.com/apache/storm/blob/master/SECURITY.md](https://github.com/apache/storm/blob/master/SECURITY.md)
    +
    +X-Pack
    +------
    +
    +First, stop the random_access_indexing topology through the Storm UI or from the CLI, e.g.
    +
    +```
    +storm kill random_access_indexing
    +```
    +
    +Here are instructions for enabling X-Pack with Elasticsearch and Kibana: https://www.elastic.co/guide/en/x-pack/5.6/installing-xpack.html
    +
    +You need to be sure to add the appropriate username and password for Elasticsearch and Kibana to enable external connections from Metron components. e.g. the following will create a user "transport_client_user" with password "changeme" and "superuser" credentials.
    +
    +```
    +sudo /usr/share/elasticsearch/bin/x-pack/users useradd transport_client_user -p changeme -r superuser
    +```
    +
    +Once you've picked a password to connect to ES, you need to upload a 1-line file to HDFS with that password in it. Metron will use this file to securely read the password in order to connect to ES securely.
    +
    +Here is an example using "changeme" as the password
    +
    +```
    +echo changeme > /tmp/xpack-password
    +sudo -u hdfs hdfs dfs -mkdir /apps/metron/elasticsearch/
    +sudo -u hdfs hdfs dfs -put /tmp/xpack-password /apps/metron/elasticsearch/
    +sudo -u hdfs hdfs dfs -chown metron:metron /apps/metron/elasticsearch/xpack-password
    +```
    +
    +New settings have been added to configure the Elasticsearch client. By default the client will run as the normal ES prebuilt transport client. If you enable X-Pack you should set the es.client.class as shown below.
    +
    +Add the es settings to global.json
    +
    +```
    +/usr/metron/0.4.3/config/zookeeper/global.json ->
    +
    +  "es.client.settings" : {
    +      "es.client.class" : "org.elasticsearch.xpack.client.PreBuiltXPackTransportClient",
    +      "es.xpack.username" : "transport_client_user",
    +      "es.xpack.password.file" : "/apps/metron/elasticsearch/xpack-password"
    +  }
    +```
    +
    +Submit the update to Zookeeper
    +
    +```
    +$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i METRON_HOME/config/zookeeper/ -z $ZOOKEEPER
    +```
    +
    +The last step before restarting the topology is to create a custom X-Pack shaded and relocated jar. This is up to you because of licensing restrictions, but here is a sample Maven pom file that should help.
    +
    +```
    +<?xml version="1.0" encoding="UTF-8"?>
    +<!--
    +  Licensed to the Apache Software
    +	Foundation (ASF) under one or more contributor license agreements. See the
    +	NOTICE file distributed with this work for additional information regarding
    +	copyright ownership. The ASF licenses this file to You under the Apache License,
    +	Version 2.0 (the "License"); you may not use this file except in compliance
    +	with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
    +	Unless required by applicable law or agreed to in writing, software distributed
    +	under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
    +	OR CONDITIONS OF ANY KIND, either express or implied. See the License for
    +  the specific language governing permissions and limitations under the License.
    +  -->
    +<project xmlns="http://maven.apache.org/POM/4.0.0"
    +         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    +         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    +    <modelVersion>4.0.0</modelVersion>
    +    <groupId>org.elasticsearch</groupId>
    +    <artifactId>elasticsearch-xpack-shaded</artifactId>
    +    <name>elasticsearch-xpack-shaded</name>
    +    <packaging>jar</packaging>
    +    <version>5.6.2</version>
    +    <repositories>
    +        <repository>
    +            <id>elasticsearch-releases</id>
    +            <url>https://artifacts.elastic.co/maven</url>
    +            <releases>
    +                <enabled>true</enabled>
    +            </releases>
    +            <snapshots>
    +                <enabled>false</enabled>
    +            </snapshots>
    +        </repository>
    +    </repositories>
    +    <dependencies>
    +        <dependency>
    +            <groupId>org.elasticsearch.client</groupId>
    +            <artifactId>x-pack-transport</artifactId>
    +            <version>5.6.2</version>
    +            <exclusions>
    +              <exclusion>
    --- End diff --
    
    @nickwallen It is necessary, Otherwise the x-pack client will have conflicts. We can't package it due to licensing, and we shouldn't leave users completely on their own to figure out what needs to be excluded, shaded and relocated.


> X-pack support for Elasticsearch
> --------------------------------
>
>                 Key: METRON-1465
>                 URL: https://issues.apache.org/jira/browse/METRON-1465
>             Project: Metron
>          Issue Type: Bug
>    Affects Versions: 0.4.2
>            Reporter: Ward Bekker
>            Priority: Major
>             Fix For: 0.4.3
>
>
> Provide support for X-pack secured Elasticsearch clusters for the Elasticsearch writer and the DAO used by the rest service.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)