You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by justinleet <gi...@git.apache.org> on 2017/04/07 12:45:59 UTC

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

GitHub user justinleet opened a pull request:

    https://github.com/apache/incubator-metron/pull/518

    METRON-799: The MPack should function in a kerberized cluster

    ## Contributor Comments
    Allows the Ambari Kerberos wizard to handle Metron setup.
    
    Changes include:
    - Creation of Keytabs
    - Running everything as the Metron user, including Storm topology workers (on a Kerberized cluster).
    - Setup for Metron user to actually be able to run (client_jaas setup, home Storm dir setup, etc.)
    - Adjusting perms to 0755.  The exception is the HDFS output folder on a non-kerb cluster is left as 0775 because we don't have Storm running workers as metron user on.  When Kerberizing, the permissions will be restricted down to 0755.
    - Kafka ACLs
    - HBase ACLs
    - Refactored Topic creation to use a common function so I didn't have to edit the same thing 3 times.
    - Automated updating of Storm configs (the AutoTGT and running workers as user)
    
    There's still more testing I want to do, but this is definitely far enough along to submit a PR.
    
    I've spun this up on full dev with the now modified Kerberos setup instructions, with the caveat that Ambari's Storm service check fails (it's harmless, as far as I can tell).  See below for more details.  As this does not touch the sensors, data will need to be pushed manually (same as the old instructions).  I've been able to push data from Kafka to Elasticsearch/HDFS.
    
    ### The Bad News
    I would love insight on a problem, if anybody has some.  I haven't edited the docs to reflect this yet, in the hopes it'll be resolved.
    
    Storm's service check will fail during (and after) Kerberization.  Metron can immediately be started perfectly fine.  Nothing is legit wrong, but this setup means that the storm user is unable to submit to the cluster (it doesn't have it's home directory setup with some configs).  Unfortunately, Ambari runs the service check as the storm user.
    
    This can be worked around by creating ~storm/.storm/storm.yaml
    ```
    nimbus.seeds : ['node1']
    java.security.auth.login.config : '/usr/hdp/current/storm-supervisor/conf/storm_jaas.conf'
    storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin
    ```
    `java.security.auth.login.conf` can also be `/etc/storm/conf/storm_jaas.conf`, but the value above leads me to my next point.  All of these values already exist in storm.yaml.  The fact that they need to be specified again in the user's home is really strange. And it'll give an error that the TGT found is not renewable, not something you'd expected.
    
    I'm unsure if there are restrictions on where Ambari chooses to run service check, so it's possible this would have to be setup on every node Storm lives on the cluster. I'm also unsure if we can actually have Ambari automate this if it turns out to be necessary, since we aren't the Storm service.
    
    ## Pull Request Checklist
    
    Thank you for submitting a contribution to Apache Metron (Incubating).  
    Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.  
    Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.  
    
    
    In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
    
    ### For all changes:
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). 
    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
    
    
    ### For code changes:
    - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
    - [x] Have you included steps or a guide to how the change may be verified and tested manually?
    - [ ] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
      ```
      mvn -q clean integration-test install && build_utils/verify_licenses.sh 
      ```
    
    - ~Have you written or updated unit tests and or integration tests to verify your changes?~
    - ~If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?~ 
    - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
    
    ### For documentation related changes:
    - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
    
      ```
      cd site-book
      bin/generate-md.sh
      mvn site:site
      ```
    
    #### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
    It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/justinleet/incubator-metron mpack-kerberos

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/518.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #518
    
----
commit 7934bec0c8f302bba3f68b6afaaf89271b0efce5
Author: justinleet <ju...@gmail.com>
Date:   2017-03-15T18:55:06Z

    hbase kinit for ambari
    
    * hbase kinit
    
    * Adding kinits around kafka topic creation
    
    * adding in hbase acl
    
    * All status checks, etc. seem to work now
    
    * Default to PLAINTEXT

commit fe065f2ea39c3146a60fe23504655089309f5481
Author: justinleet <ju...@gmail.com>
Date:   2017-03-15T19:42:51Z

    Splitting out ACLs from Kafka and HBase setup (#3)

commit ee8d35dc5dcdaee507ff906aa83e83033d0606ea
Author: justinleet <ju...@gmail.com>
Date:   2017-03-15T20:41:56Z

    Kerb testing (#4)
    
    * Splitting out ACLs from Kafka and HBase setup
    
    * Moving user outside

commit 7c0c5588e1eb4100fe455c199b0729088f47546f
Author: justinjleet <ju...@gmail.com>
Date:   2017-04-06T02:40:59Z

    mostly working

commit f75b4c37e6f4c7e72ca5cdd22a4f8b830db3ecc3
Author: justinjleet <ju...@gmail.com>
Date:   2017-04-06T12:49:14Z

    working now again after some refactoring.

commit 91a89429201d3d45ece33dc3c36ee9c6a34ebbb6
Author: justinjleet <ju...@gmail.com>
Date:   2017-04-06T18:59:21Z

    fixes, refactoring, and improvements

commit fd5238f632a9d4fc62a3e59ca8d3bb24e2e3c55b
Author: justinjleet <ju...@gmail.com>
Date:   2017-04-06T19:23:31Z

    Cleanup

commit 569be68b17d0e84ece51015140e6eade755f3cb5
Author: justinjleet <ju...@gmail.com>
Date:   2017-04-07T12:09:28Z

    error topics plus fixes plus documentation

commit 5418a2eb914faac0450f7749b6c76f75a2da4e19
Author: justinjleet <ju...@gmail.com>
Date:   2017-04-07T12:22:35Z

    Removing leftover TODO

commit 049cf524f919f608dbce2b6c5dacbd4552ce3efb
Author: justinjleet <ju...@gmail.com>
Date:   2017-04-07T12:37:12Z

    Caught a testing change in the last git add

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110383817
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py ---
    @@ -0,0 +1,74 @@
    +"""
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +    http://www.apache.org/licenses/LICENSE-2.0
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +"""
    +
    +import os.path
    +from resource_management.core.source import Template
    +from resource_management.core.resources.system import Directory, File
    +from resource_management.core import global_lock
    +from resource_management.core.logger import Logger
    +from resource_management.core.resources.system import Execute
    +from resource_management.libraries.functions import format as ambari_format
    +
    +
    +# Convenience function for ensuring home dirs are setup consistently.
    +def storm_security_setup(params):
    +    if params.security_enabled:
    +        # I don't think there's an Ambari way to get a user's local home dir , so have Python perform tilde expansion.
    +        # Ambari's Directory doesn't do tilde expansion.
    +        metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
    +        metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
    +        Directory(metron_storm_dir,
    +                  mode=0755,
    +                  owner=params.metron_user,
    +                  group=params.metron_group
    +                  )
    +
    +        File(ambari_format('{client_jaas_path}'),
    +             content=Template('client_jaas.conf.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    --- End diff --
    
    Does this file end up on all the supervisor nodes too? Does it need to?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #518: METRON-799: The MPack should function in a kerb...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:

    https://github.com/apache/incubator-metron/pull/518
  
    Yeah, reading is FUNdamental. I didn't see that it was the Storm service check that was failing. I kind of made up in my head that it was the Metron service check, because that would make sense. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #518: METRON-799: The MPack should function in a kerb...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/518
  
    Updated to use builtin check for security in stack advisor.  Old versions caused problems because of typing in Python and my lack of catching what it actually does.  Params files should still be fine, because the config objects underneath the covers in Ambari explicitly convert strings to booleans if appropriate.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110385480
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py ---
    @@ -0,0 +1,74 @@
    +"""
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +    http://www.apache.org/licenses/LICENSE-2.0
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +"""
    +
    +import os.path
    +from resource_management.core.source import Template
    +from resource_management.core.resources.system import Directory, File
    +from resource_management.core import global_lock
    +from resource_management.core.logger import Logger
    +from resource_management.core.resources.system import Execute
    +from resource_management.libraries.functions import format as ambari_format
    +
    +
    +# Convenience function for ensuring home dirs are setup consistently.
    +def storm_security_setup(params):
    +    if params.security_enabled:
    +        # I don't think there's an Ambari way to get a user's local home dir , so have Python perform tilde expansion.
    +        # Ambari's Directory doesn't do tilde expansion.
    +        metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
    +        metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
    +        Directory(metron_storm_dir,
    +                  mode=0755,
    +                  owner=params.metron_user,
    +                  group=params.metron_group
    +                  )
    +
    +        File(ambari_format('{client_jaas_path}'),
    +             content=Template('client_jaas.conf.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    --- End diff --
    
    All the stuff in (or referenced in the case of client_jaas) ~metron/.storm should just need to be on the Metron node, because everything gets kicked off there.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110383752
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py ---
    @@ -0,0 +1,74 @@
    +"""
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +    http://www.apache.org/licenses/LICENSE-2.0
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +"""
    +
    +import os.path
    +from resource_management.core.source import Template
    +from resource_management.core.resources.system import Directory, File
    +from resource_management.core import global_lock
    +from resource_management.core.logger import Logger
    +from resource_management.core.resources.system import Execute
    +from resource_management.libraries.functions import format as ambari_format
    +
    +
    +# Convenience function for ensuring home dirs are setup consistently.
    +def storm_security_setup(params):
    +    if params.security_enabled:
    +        # I don't think there's an Ambari way to get a user's local home dir , so have Python perform tilde expansion.
    +        # Ambari's Directory doesn't do tilde expansion.
    +        metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
    +        metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
    +        Directory(metron_storm_dir,
    +                  mode=0755,
    +                  owner=params.metron_user,
    +                  group=params.metron_group
    +                  )
    +
    +        File(ambari_format('{client_jaas_path}'),
    +             content=Template('client_jaas.conf.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    +        File(metron_storm_dir + '/storm.yaml',
    +             content=Template('storm.yaml.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    --- End diff --
    
    Does this override the storm.yaml that is at /etc/conf and managed by Ambari?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110384328
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py ---
    @@ -131,47 +153,35 @@ def init_geo(self):
             self.set_geo_configured()
     
         def init_kafka_topics(self):
    -        Logger.info('Creating Kafka topics')
    -        command_template = """{0}/kafka-topics.sh \
    -                                --zookeeper {1} \
    -                                --create \
    -                                --topic {2} \
    -                                --partitions {3} \
    -                                --replication-factor {4} \
    -                                --config retention.bytes={5}"""
    -        num_partitions = 1
    -        replication_factor = 1
    -        retention_gigabytes = int(self.__params.metron_topic_retention)
    -        retention_bytes = retention_gigabytes * 1024 * 1024 * 1024
    -
    -        Logger.info("Creating topics for enrichment")
    -        topics = [self.__enrichment_topic]
    -        for topic in topics:
    -            Logger.info("Creating topic'{0}'".format(topic))
    -            Execute(command_template.format(self.__params.kafka_bin_dir,
    -                                            self.__params.zookeeper_quorum,
    -                                            topic,
    -                                            num_partitions,
    -                                            replication_factor,
    -                                            retention_bytes))
    -
    -        Logger.info("Done creating Kafka topics")
    +        Logger.info('Creating Kafka topics for enrichment')
    +        # All errors go to indexing topics, so create it here if it's not already
    +        metron_service.init_kafka_topics(self.__params, [self.__enrichment_topic, self.__params.metron_error_topic])
             self.set_kafka_configured()
     
    +    def init_kafka_acls(self):
    +        Logger.info('Creating Kafka topics')
    --- End diff --
    
    Did you mean something like "Setting ACL" or even to omit this line?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110409879
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py ---
    @@ -0,0 +1,74 @@
    +"""
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +    http://www.apache.org/licenses/LICENSE-2.0
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +"""
    +
    +import os.path
    +from resource_management.core.source import Template
    +from resource_management.core.resources.system import Directory, File
    +from resource_management.core import global_lock
    +from resource_management.core.logger import Logger
    +from resource_management.core.resources.system import Execute
    +from resource_management.libraries.functions import format as ambari_format
    +
    +
    +# Convenience function for ensuring home dirs are setup consistently.
    +def storm_security_setup(params):
    +    if params.security_enabled:
    +        # I don't think there's an Ambari way to get a user's local home dir , so have Python perform tilde expansion.
    +        # Ambari's Directory doesn't do tilde expansion.
    +        metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
    +        metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
    +        Directory(metron_storm_dir,
    +                  mode=0755,
    +                  owner=params.metron_user,
    +                  group=params.metron_group
    +                  )
    +
    +        File(ambari_format('{client_jaas_path}'),
    +             content=Template('client_jaas.conf.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    +        File(metron_storm_dir + '/storm.yaml',
    +             content=Template('storm.yaml.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    --- End diff --
    
    Just to confirm, the shadowed nimbus.seeds is changed when I change the value for Storm via Ambari?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110385183
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py ---
    @@ -0,0 +1,74 @@
    +"""
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +    http://www.apache.org/licenses/LICENSE-2.0
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +"""
    +
    +import os.path
    +from resource_management.core.source import Template
    +from resource_management.core.resources.system import Directory, File
    +from resource_management.core import global_lock
    +from resource_management.core.logger import Logger
    +from resource_management.core.resources.system import Execute
    +from resource_management.libraries.functions import format as ambari_format
    +
    +
    +# Convenience function for ensuring home dirs are setup consistently.
    +def storm_security_setup(params):
    +    if params.security_enabled:
    +        # I don't think there's an Ambari way to get a user's local home dir , so have Python perform tilde expansion.
    +        # Ambari's Directory doesn't do tilde expansion.
    +        metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
    +        metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
    +        Directory(metron_storm_dir,
    +                  mode=0755,
    +                  owner=params.metron_user,
    +                  group=params.metron_group
    +                  )
    +
    +        File(ambari_format('{client_jaas_path}'),
    +             content=Template('client_jaas.conf.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    +        File(metron_storm_dir + '/storm.yaml',
    +             content=Template('storm.yaml.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    --- End diff --
    
    Just the properties in here.  Couple thoughts as you bring this up.
    
    We probably want to have a ticket to make sure turning off Kerberos works correctly in the future.  The properties in the file (except nimbus.seeds) are essentially set properties.  We need our own client_jaas, and the storm.thrift.transport has to be there for some reason and that's pretty much constant on a secure cluster.  That should also be made a property that flows down from Storm.
    
    nimbus.seeds is wrong and I need to carry that over from the actual Storm property. And I even made sure it was in params_linux and forgot to use it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #518: METRON-799: The MPack should function in a kerb...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/518
  
    So coming back to the AutoTGT discussion and making it easier to see.  There's basically two approaches we can takes that (should) work.
    
    1. The current use of AutoTGT.  This requires setting up .storm/storm.yaml for the Storm nodes, and users will have to do the same.
    2. The AutoHDFS / AutoHBase solution mentioned by @dlyle65535.  This requires symlinking some jars and configs to make them available to Storm.  Should only be necessary for Nimbuses, but it does mean that HDFS/HBase upgrades can make the symlinks stale.
    
    Either solution requires Ambari acting on a different node (potentially) than the one running the scripts.  I don't know if Ambari has any resources for handling that sort of things.  It could potentially be a command over ssh to another node, but presumably that requires passwordless ssh setup or someone to manually create the symlinks (which may be acceptable for this pass).
    
    I'm fairly strongly inclined towards the second one, primarily because it requires less effort on the users part.  Ambari work is fairly similar either way.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #518: METRON-799: The MPack should function in a kerb...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:

    https://github.com/apache/incubator-metron/pull/518
  
    I've had 2 successful runs:
    1) Full dev + kerberos.
    2) EC2 will full stack installed prior to enabling kerberos.
    
    Both ran without error.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110413918
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py ---
    @@ -0,0 +1,74 @@
    +"""
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +    http://www.apache.org/licenses/LICENSE-2.0
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +"""
    +
    +import os.path
    +from resource_management.core.source import Template
    +from resource_management.core.resources.system import Directory, File
    +from resource_management.core import global_lock
    +from resource_management.core.logger import Logger
    +from resource_management.core.resources.system import Execute
    +from resource_management.libraries.functions import format as ambari_format
    +
    +
    +# Convenience function for ensuring home dirs are setup consistently.
    +def storm_security_setup(params):
    +    if params.security_enabled:
    +        # I don't think there's an Ambari way to get a user's local home dir , so have Python perform tilde expansion.
    +        # Ambari's Directory doesn't do tilde expansion.
    +        metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
    +        metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
    +        Directory(metron_storm_dir,
    +                  mode=0755,
    +                  owner=params.metron_user,
    +                  group=params.metron_group
    +                  )
    +
    +        File(ambari_format('{client_jaas_path}'),
    +             content=Template('client_jaas.conf.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    +        File(metron_storm_dir + '/storm.yaml',
    +             content=Template('storm.yaml.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    --- End diff --
    
    Yeah, you're right, you should be able to change it via HA.  I just meant I can't go and change it to something in full dev.
    
    Re: the AutoTGT stuff
    I'm unsure if HA Storm would be an issue, but assuming Storm handles spinning up secure nimbuses correctly it shouldn't be.  New nimbuses are required to make sure they get all code of active topologies.  I would assume this includes getting any TGTs (although I haven't verified this) to ensure that active topologies can continue to run.  We'd need to test on an actual HA cluster.
    
    I disagree with AutoHDFS/AutoHBase being less complex, because if I recall correctly (and I don't, please correct me), that solution required setting up symlinks on each Storm node.  I don't even know that we easily have that capability in Ambari as the Metron service. Even if we do work around that, I don't see AutoTGT as complicated, and the Storm docs mention "On a kerberos secure cluster they should be set by default to point to org.apache.storm.security.auth.kerberos.AutoTGT."  It seems reasonable to go with Storm's recommendation unless we have a compelling reason not to.
    
    The custom storm.yaml only contains 3 configs that we need to run
    1. Nimbus seeds (gathered from Storm configs themselves)
    2. Jaas file
    3. Thrift protocol (which admittedly doesn't grab correctly from Storm, but is essentially a constant for the purpose of Kerberos).
    
    Of these, only the jaas file really matters for management, and all it does it define named stanzas and it's not anything particularly complicated.
    
    Having said that, the Storm service check could make AutoTGT a lot less attractive if it's not easily workable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110384006
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py ---
    @@ -72,55 +93,46 @@ def remote_repo():
                 raise ValueError("Unsupported repo type '{0}'".format(repo_type))
     
         def init_kafka_topics(self):
    -        Logger.info('Creating Kafka topics')
    -        command_template = """{0}/kafka-topics.sh \
    -                                --zookeeper {1} \
    -                                --create \
    -                                --topic {2} \
    -                                --partitions {3} \
    -                                --replication-factor {4} \
    -                                --config retention.bytes={5}"""
    -        num_partitions = 1
    -        replication_factor = 1
    -        retention_gigabytes = int(self.__params.metron_topic_retention)
    -        retention_bytes = retention_gigabytes * 1024 * 1024 * 1024
    -        Logger.info("Creating topics for indexing")
    -
    -        Logger.info("Creating topic'{0}'".format(self.__indexing))
    -        Execute(command_template.format(self.__params.kafka_bin_dir,
    -                                        self.__params.zookeeper_quorum,
    -                                        self.__indexing,
    -                                        num_partitions,
    -                                        replication_factor,
    -                                        retention_bytes))
    -        Logger.info("Done creating Kafka topics")
    +        Logger.info('Creating Kafka topics for indexing')
    +        metron_service.init_kafka_topics(self.__params, [self.__indexing])
    +
    +    def init_kafka_acls(self):
    +        Logger.info('Creating Kafka ACLs')
    +        # Indexed topic names matches the group
    +        metron_service.init_kafka_acls(self.__params, [self.__indexing], [self.__indexing])
     
         def init_hdfs_dir(self):
    -        Logger.info('Creating HDFS indexing directory')
    +        Logger.info('Setting up HDFS indexing directory')
    +
    +        # Non Kerberized Metron runs under 'storm', requiring write under the 'hadoop' group.
    +        # Kerberized Metron runs under it's own user.
    +        ownership = 0755 if self.__params.security_enabled else 0775
    +        Logger.info('HDFS indexing directory ownership is: ' + str(ownership))
             self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
                                        type="directory",
                                        action="create_on_execute",
                                        owner=self.__params.metron_user,
                                        group=self.__params.hadoop_group,
    --- End diff --
    
    Did you intend to leave this as hadoop group ownership or change it to the metron group?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #518: METRON-799: The MPack should function in a kerb...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/518
  
    Full dev is able to successfully startup, be Kerberized via Ambari without errors (including Storm service check), and have new data flow through.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110387708
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py ---
    @@ -0,0 +1,74 @@
    +"""
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +    http://www.apache.org/licenses/LICENSE-2.0
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +"""
    +
    +import os.path
    +from resource_management.core.source import Template
    +from resource_management.core.resources.system import Directory, File
    +from resource_management.core import global_lock
    +from resource_management.core.logger import Logger
    +from resource_management.core.resources.system import Execute
    +from resource_management.libraries.functions import format as ambari_format
    +
    +
    +# Convenience function for ensuring home dirs are setup consistently.
    +def storm_security_setup(params):
    +    if params.security_enabled:
    +        # I don't think there's an Ambari way to get a user's local home dir , so have Python perform tilde expansion.
    +        # Ambari's Directory doesn't do tilde expansion.
    +        metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
    +        metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
    +        Directory(metron_storm_dir,
    +                  mode=0755,
    +                  owner=params.metron_user,
    +                  group=params.metron_group
    +                  )
    +
    +        File(ambari_format('{client_jaas_path}'),
    +             content=Template('client_jaas.conf.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    +        File(metron_storm_dir + '/storm.yaml',
    +             content=Template('storm.yaml.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    --- End diff --
    
    So, if we switch on HA Storm after Metron kerberization, there's an issue?
    
    I'm a bit uncomfortable with all the all the additional management required to use AutoTGT rather than Auto[HDFS|HBase]TGT. I didn't deliberately choose those two when I started (I just happened to land on those docs first), but that seems less complex than what we're currently doing. Are there benefits I'm missing? Because managing a jaas file and storm.yaml from outside of the Ambari Storm Service seems to add difficulty and risk.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110386021
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py ---
    @@ -72,55 +93,46 @@ def remote_repo():
                 raise ValueError("Unsupported repo type '{0}'".format(repo_type))
     
         def init_kafka_topics(self):
    -        Logger.info('Creating Kafka topics')
    -        command_template = """{0}/kafka-topics.sh \
    -                                --zookeeper {1} \
    -                                --create \
    -                                --topic {2} \
    -                                --partitions {3} \
    -                                --replication-factor {4} \
    -                                --config retention.bytes={5}"""
    -        num_partitions = 1
    -        replication_factor = 1
    -        retention_gigabytes = int(self.__params.metron_topic_retention)
    -        retention_bytes = retention_gigabytes * 1024 * 1024 * 1024
    -        Logger.info("Creating topics for indexing")
    -
    -        Logger.info("Creating topic'{0}'".format(self.__indexing))
    -        Execute(command_template.format(self.__params.kafka_bin_dir,
    -                                        self.__params.zookeeper_quorum,
    -                                        self.__indexing,
    -                                        num_partitions,
    -                                        replication_factor,
    -                                        retention_bytes))
    -        Logger.info("Done creating Kafka topics")
    +        Logger.info('Creating Kafka topics for indexing')
    +        metron_service.init_kafka_topics(self.__params, [self.__indexing])
    +
    +    def init_kafka_acls(self):
    +        Logger.info('Creating Kafka ACLs')
    +        # Indexed topic names matches the group
    +        metron_service.init_kafka_acls(self.__params, [self.__indexing], [self.__indexing])
     
         def init_hdfs_dir(self):
    -        Logger.info('Creating HDFS indexing directory')
    +        Logger.info('Setting up HDFS indexing directory')
    +
    +        # Non Kerberized Metron runs under 'storm', requiring write under the 'hadoop' group.
    +        # Kerberized Metron runs under it's own user.
    +        ownership = 0755 if self.__params.security_enabled else 0775
    +        Logger.info('HDFS indexing directory ownership is: ' + str(ownership))
             self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
                                        type="directory",
                                        action="create_on_execute",
                                        owner=self.__params.metron_user,
                                        group=self.__params.hadoop_group,
    --- End diff --
    
    Seems reasonable to me too, thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #518: METRON-799: The MPack should function in a kerb...

Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:

    https://github.com/apache/incubator-metron/pull/518
  
    +1 I tested it on AWS and was able to get it to work. Great job!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110411025
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py ---
    @@ -0,0 +1,74 @@
    +"""
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +    http://www.apache.org/licenses/LICENSE-2.0
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +"""
    +
    +import os.path
    +from resource_management.core.source import Template
    +from resource_management.core.resources.system import Directory, File
    +from resource_management.core import global_lock
    +from resource_management.core.logger import Logger
    +from resource_management.core.resources.system import Execute
    +from resource_management.libraries.functions import format as ambari_format
    +
    +
    +# Convenience function for ensuring home dirs are setup consistently.
    +def storm_security_setup(params):
    +    if params.security_enabled:
    +        # I don't think there's an Ambari way to get a user's local home dir , so have Python perform tilde expansion.
    +        # Ambari's Directory doesn't do tilde expansion.
    +        metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
    +        metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
    +        Directory(metron_storm_dir,
    +                  mode=0755,
    +                  owner=params.metron_user,
    +                  group=params.metron_group
    +                  )
    +
    +        File(ambari_format('{client_jaas_path}'),
    +             content=Template('client_jaas.conf.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    +        File(metron_storm_dir + '/storm.yaml',
    +             content=Template('storm.yaml.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    --- End diff --
    
    It's not possible to change it via Ambari, but it will populate the variable appropriately.  There's some translation magic happening somewhere, because Ambari lists the value of nimbus.seeds as 'node1', but it actually passes it down as a list (which is why the new commit doesn't surround it with square brackets). I'm guessing this magic is also why the thrift config doesn't get passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110385802
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py ---
    @@ -0,0 +1,74 @@
    +"""
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +    http://www.apache.org/licenses/LICENSE-2.0
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +"""
    +
    +import os.path
    +from resource_management.core.source import Template
    +from resource_management.core.resources.system import Directory, File
    +from resource_management.core import global_lock
    +from resource_management.core.logger import Logger
    +from resource_management.core.resources.system import Execute
    +from resource_management.libraries.functions import format as ambari_format
    +
    +
    +# Convenience function for ensuring home dirs are setup consistently.
    +def storm_security_setup(params):
    +    if params.security_enabled:
    +        # I don't think there's an Ambari way to get a user's local home dir , so have Python perform tilde expansion.
    +        # Ambari's Directory doesn't do tilde expansion.
    +        metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
    +        metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
    +        Directory(metron_storm_dir,
    +                  mode=0755,
    +                  owner=params.metron_user,
    +                  group=params.metron_group
    +                  )
    +
    +        File(ambari_format('{client_jaas_path}'),
    +             content=Template('client_jaas.conf.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    --- End diff --
    
    Won't the supervisors need to read/write to HBase/HDFS? How are they authenticated?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #518: METRON-799: The MPack should function in a kerb...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:

    https://github.com/apache/incubator-metron/pull/518
  
    I validated this on full dev.  Ran a vagrant up, followed the instructions in Kerberos-setup.md, and verified Bro data being written to the index.
    
    +1.  Nice job.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #518: METRON-799: The MPack should function in a kerb...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/518
  
    Still need to do some additional testing (I'm spiin, but this is updated to pull the topology.auto-credentials down to the topology level.
    
    This should let Storm actually run service check (because AutoTGT isn't initializing).  At the same time, Metron should be able to set these appropriately and run them.   This involves passing the config down to the various files (the .properties files that feed Flux) and updating integration tests slightly.
    
    Oddly enough, the use of the tickets in the client_jaas file caused issues, so it's now using the ticket cache and running a kinit before acting on topologies.  I strongly dislike having the kinits, but there's no obvious reason for the difference in behavior between ticket cache and keytab.
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #518: METRON-799: The MPack should function in a kerb...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/518
  
    I've run this up on an EC2 cluster, and been able to get data from the core topologies from Kafka to ES/HDFS.  There are a couple caveats spelled out in the metron-deployment README now in a short Kerberos section mentioning that the mpack supports Ambari's Kerberization process.
    
    The client process managed to ensure keytabs existed and that the client_jaas was created appropriately.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110385969
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py ---
    @@ -131,47 +153,35 @@ def init_geo(self):
             self.set_geo_configured()
     
         def init_kafka_topics(self):
    -        Logger.info('Creating Kafka topics')
    -        command_template = """{0}/kafka-topics.sh \
    -                                --zookeeper {1} \
    -                                --create \
    -                                --topic {2} \
    -                                --partitions {3} \
    -                                --replication-factor {4} \
    -                                --config retention.bytes={5}"""
    -        num_partitions = 1
    -        replication_factor = 1
    -        retention_gigabytes = int(self.__params.metron_topic_retention)
    -        retention_bytes = retention_gigabytes * 1024 * 1024 * 1024
    -
    -        Logger.info("Creating topics for enrichment")
    -        topics = [self.__enrichment_topic]
    -        for topic in topics:
    -            Logger.info("Creating topic'{0}'".format(topic))
    -            Execute(command_template.format(self.__params.kafka_bin_dir,
    -                                            self.__params.zookeeper_quorum,
    -                                            topic,
    -                                            num_partitions,
    -                                            replication_factor,
    -                                            retention_bytes))
    -
    -        Logger.info("Done creating Kafka topics")
    +        Logger.info('Creating Kafka topics for enrichment')
    +        # All errors go to indexing topics, so create it here if it's not already
    +        metron_service.init_kafka_topics(self.__params, [self.__enrichment_topic, self.__params.metron_error_topic])
             self.set_kafka_configured()
     
    +    def init_kafka_acls(self):
    +        Logger.info('Creating Kafka topics')
    --- End diff --
    
    Yes, yes I did.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #518: METRON-799: The MPack should function in a kerb...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/518
  
    Update on this:
    
    After a significant amount of pain involved in making this work, we need to have the client_jaas.conf and the metron keytab distributed to the various supervisor nodes so that they can actually be authenticated/authorized as the metron user.
    
    The best discovered solution to this is to have a client that essentially just sets them up (so we can actually create things on the various supervisor servers).  It's not ideal, but @dlyle65535 and I are testing that it works.
    
    To actually set this up, just install Metron clients on all Storm supervisor nodes prior to Kerberization.  Docs will be updated once we confirm that this approach works.
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110405590
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py ---
    @@ -0,0 +1,74 @@
    +"""
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +    http://www.apache.org/licenses/LICENSE-2.0
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +"""
    +
    +import os.path
    +from resource_management.core.source import Template
    +from resource_management.core.resources.system import Directory, File
    +from resource_management.core import global_lock
    +from resource_management.core.logger import Logger
    +from resource_management.core.resources.system import Execute
    +from resource_management.libraries.functions import format as ambari_format
    +
    +
    +# Convenience function for ensuring home dirs are setup consistently.
    +def storm_security_setup(params):
    +    if params.security_enabled:
    +        # I don't think there's an Ambari way to get a user's local home dir , so have Python perform tilde expansion.
    +        # Ambari's Directory doesn't do tilde expansion.
    +        metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
    +        metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
    +        Directory(metron_storm_dir,
    +                  mode=0755,
    +                  owner=params.metron_user,
    +                  group=params.metron_group
    +                  )
    +
    +        File(ambari_format('{client_jaas_path}'),
    +             content=Template('client_jaas.conf.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    +        File(metron_storm_dir + '/storm.yaml',
    +             content=Template('storm.yaml.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    --- End diff --
    
    nimbus.seeds is updated to grab the actual value.  Oddly, storm.thrift.transport doesn't end up flowing correctly to the template.  I double checked all the spelling and so on, and it always ended up just being the variable, not the value.  I'm inclined to let that one go, since it's only even created in a Kerberized environment anyway and it's a constant.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110388704
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py ---
    @@ -0,0 +1,74 @@
    +"""
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +    http://www.apache.org/licenses/LICENSE-2.0
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +"""
    +
    +import os.path
    +from resource_management.core.source import Template
    +from resource_management.core.resources.system import Directory, File
    +from resource_management.core import global_lock
    +from resource_management.core.logger import Logger
    +from resource_management.core.resources.system import Execute
    +from resource_management.libraries.functions import format as ambari_format
    +
    +
    +# Convenience function for ensuring home dirs are setup consistently.
    +def storm_security_setup(params):
    +    if params.security_enabled:
    +        # I don't think there's an Ambari way to get a user's local home dir , so have Python perform tilde expansion.
    +        # Ambari's Directory doesn't do tilde expansion.
    +        metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
    +        metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
    +        Directory(metron_storm_dir,
    +                  mode=0755,
    +                  owner=params.metron_user,
    +                  group=params.metron_group
    +                  )
    +
    +        File(ambari_format('{client_jaas_path}'),
    +             content=Template('client_jaas.conf.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    --- End diff --
    
    My understanding, and correct me if I'm wrong, is that nimbus distributed and renews the TGT from Kerberos.  I'm honestly not sure how we'd get everything to line up otherwise if we have to distribute keytabs and jaas files and everything else out.
    
    This does raise the question of what happens when the max renewal of a Kerberos ticket has passed. I don't know enough about the implementation to know what happens, and I'd be interested in thoughts.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #518: METRON-799: The MPack should function in a kerb...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:

    https://github.com/apache/incubator-metron/pull/518
  
    I have a (likely wrong) idea about the service check. A couple of things to check- can you verify that Ambari runs the service check as the storm user, because that's just weird, isn't it? If it does, make sure the user identity it's using is in nimbus.admins. If that doesn't work, maybe try adding the metron user to nimbus.admins.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-metron/pull/518


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #518: METRON-799: The MPack should function in a kerb...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/518
  
    @dlyle65535 I'll take a look at nimbus.admins.  I'm pretty confident it's running as Storm user though,
    Partial output from Ambari:
    ```
    Traceback (most recent call last):
     ...
        user=params.storm_user
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110385783
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py ---
    @@ -72,55 +93,46 @@ def remote_repo():
                 raise ValueError("Unsupported repo type '{0}'".format(repo_type))
     
         def init_kafka_topics(self):
    -        Logger.info('Creating Kafka topics')
    -        command_template = """{0}/kafka-topics.sh \
    -                                --zookeeper {1} \
    -                                --create \
    -                                --topic {2} \
    -                                --partitions {3} \
    -                                --replication-factor {4} \
    -                                --config retention.bytes={5}"""
    -        num_partitions = 1
    -        replication_factor = 1
    -        retention_gigabytes = int(self.__params.metron_topic_retention)
    -        retention_bytes = retention_gigabytes * 1024 * 1024 * 1024
    -        Logger.info("Creating topics for indexing")
    -
    -        Logger.info("Creating topic'{0}'".format(self.__indexing))
    -        Execute(command_template.format(self.__params.kafka_bin_dir,
    -                                        self.__params.zookeeper_quorum,
    -                                        self.__indexing,
    -                                        num_partitions,
    -                                        replication_factor,
    -                                        retention_bytes))
    -        Logger.info("Done creating Kafka topics")
    +        Logger.info('Creating Kafka topics for indexing')
    +        metron_service.init_kafka_topics(self.__params, [self.__indexing])
    +
    +    def init_kafka_acls(self):
    +        Logger.info('Creating Kafka ACLs')
    +        # Indexed topic names matches the group
    +        metron_service.init_kafka_acls(self.__params, [self.__indexing], [self.__indexing])
     
         def init_hdfs_dir(self):
    -        Logger.info('Creating HDFS indexing directory')
    +        Logger.info('Setting up HDFS indexing directory')
    +
    +        # Non Kerberized Metron runs under 'storm', requiring write under the 'hadoop' group.
    +        # Kerberized Metron runs under it's own user.
    +        ownership = 0755 if self.__params.security_enabled else 0775
    +        Logger.info('HDFS indexing directory ownership is: ' + str(ownership))
             self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
                                        type="directory",
                                        action="create_on_execute",
                                        owner=self.__params.metron_user,
                                        group=self.__params.hadoop_group,
    --- End diff --
    
    I decided not to mess with it.  If we have a preference on it not being owned by metron:hadoop, we can go ahead and do that, but I think we probably need a more thorough discussion of how we want all that owned and permissioned anyway.  Leaving it only readable seemed like a reasonable compromise for now. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...

Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/518#discussion_r110411427
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py ---
    @@ -0,0 +1,74 @@
    +"""
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +    http://www.apache.org/licenses/LICENSE-2.0
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +"""
    +
    +import os.path
    +from resource_management.core.source import Template
    +from resource_management.core.resources.system import Directory, File
    +from resource_management.core import global_lock
    +from resource_management.core.logger import Logger
    +from resource_management.core.resources.system import Execute
    +from resource_management.libraries.functions import format as ambari_format
    +
    +
    +# Convenience function for ensuring home dirs are setup consistently.
    +def storm_security_setup(params):
    +    if params.security_enabled:
    +        # I don't think there's an Ambari way to get a user's local home dir , so have Python perform tilde expansion.
    +        # Ambari's Directory doesn't do tilde expansion.
    +        metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
    +        metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
    +        Directory(metron_storm_dir,
    +                  mode=0755,
    +                  owner=params.metron_user,
    +                  group=params.metron_group
    +                  )
    +
    +        File(ambari_format('{client_jaas_path}'),
    +             content=Template('client_jaas.conf.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    +        File(metron_storm_dir + '/storm.yaml',
    +             content=Template('storm.yaml.j2'),
    +             owner=params.metron_user,
    +             group=params.metron_group,
    +             mode=0755
    +             )
    +
    --- End diff --
    
    I think I can change it by setting up Storm HA. I think that's allowable via Ambari. But good deal that it populates it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---