You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by nickwallen <gi...@git.apache.org> on 2018/09/11 19:09:53 UTC

[GitHub] metron pull request #1192: METRON-1774 Allow user to configure JAAS client i...

GitHub user nickwallen opened a pull request:

    https://github.com/apache/metron/pull/1192

    METRON-1774 Allow user to configure JAAS client in Ambari

    Right now, the user can only configure Metron's client JAAS in limited ways.  They can configure the principal name and key tab path, but few other things.  For example, I cannot easily turn on debugging today.  If I make a change to `$METRON_HOME/client_jaas.conf` to set `debug=true`, Ambari will overwrite my change when any of the topologies are restarted.
    
    With this PR the user can now edit a template in Ambari that is used to generate `$METRON_HOME/client_jaas.conf`.  This allows user to turn on debug logs, change the service name, or add additional options required by their environment.
    
    ![screen shot 2018-09-11 at 2 12 35 pm](https://user-images.githubusercontent.com/2475409/45381641-d9b45080-b5d3-11e8-9afe-afd79c61ad4c.png)
    
    ## Testing
    
    1. Stand-up a development environment.
    
    1. Ensure alerts are visible in the Alerts UI.
    
    1. Kerberize that development environment.
    
    1. Ensure alerts are visible in the Alerts UI.
    
    1. Ensure that Metron's `client_jaas.conf` was created correctly.
    
        ```
        [root@node1 0.6.0]# source /etc/default/metron
        [root@node1 0.6.0]# cat $METRON_HOME/client_jaas.conf
    
        StormClient {
           com.sun.security.auth.module.Krb5LoginModule required
           useKeyTab=false
           storeKey=false
           useTicketCache=true
           serviceName="nimbus"
           principal="metron@EXAMPLE.COM";
        };
        Client {
           com.sun.security.auth.module.Krb5LoginModule required
           useKeyTab=true
           keyTab="/etc/security/keytabs/metron.headless.keytab"
           storeKey=true
           useTicketCache=false
           serviceName="zookeeper"
           principal="metron@EXAMPLE.COM";
        };
        KafkaClient {
           com.sun.security.auth.module.Krb5LoginModule required
           useKeyTab=true
           keyTab="/etc/security/keytabs/metron.headless.keytab"
           storeKey=true
           useTicketCache=false
           serviceName="kafka"
           principal="metron@EXAMPLE.COM";
        ```
    
    1. In Ambari, go to Metron > Configs > Advanced > Advanced metron-client-jaas-conf and alter the template in some way.  For example, you could add `debug=true` to each stanza.
    
    1. Restart the Enrichment topology.
    
    1. Validate that Metron's `client_jaas.conf` was updated based on the changes you made in Ambari.
    
        ```
        [root@node1 0.6.0]# cat client_jaas.conf
        StormClient {
           com.sun.security.auth.module.Krb5LoginModule required
           debug=true
           useKeyTab=false
           storeKey=false
           useTicketCache=true
           serviceName="nimbus"
           principal="metron@EXAMPLE.COM";
        };
        Client {
           com.sun.security.auth.module.Krb5LoginModule required
           debug=true
           useKeyTab=true
           keyTab="/etc/security/keytabs/metron.headless.keytab"
           storeKey=true
           useTicketCache=false
           serviceName="zookeeper"
           principal="metron@EXAMPLE.COM";
        };
        KafkaClient {
           com.sun.security.auth.module.Krb5LoginModule required
           debug=true
           useKeyTab=true
           keyTab="/etc/security/keytabs/metron.headless.keytab"
           storeKey=true
           useTicketCache=false
           serviceName="kafka"
           principal="metron@EXAMPLE.COM";
        ```
    
    ## Pull Request Checklist
    
    - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
    - [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
    - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
    - [ ] Have you included steps or a guide to how the change may be verified and tested manually?
    - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
    - [ ] Have you written or updated unit tests and or integration tests to verify your changes?
    - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
    - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nickwallen/metron METRON-1774

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/1192.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1192
    
----
commit e3138acf62098b3e15ffe93d0699d0181e453c10
Author: Nick Allen <ni...@...>
Date:   2018-09-11T19:01:35Z

    METRON-1774 Allow user to configure JAAS client in Ambari

----


---

[GitHub] metron issue #1192: METRON-1774 Allow user to configure JAAS client in Ambar...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:

    https://github.com/apache/metron/pull/1192
  
    I restarted indexing and the jaas shows up. I think this may just be missing a couple extra service dependencies, but it's close.


---

[GitHub] metron pull request #1192: METRON-1774 Allow user to configure JAAS client i...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:

    https://github.com/apache/metron/pull/1192#discussion_r218413157
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py ---
    @@ -46,7 +47,7 @@ def storm_security_setup(params):
                       )
     
             File(ambari_format('{client_jaas_path}'),
    -             content=Template('client_jaas.conf.j2'),
    +             content=InlineTemplate(params.metron_client_jaas_conf_template),
    --- End diff --
    
    Instead of populating the template from the static .j2 file, the template now comes from the configuration. In MPack parlance, its now using an `InlineTemplate`.


---

[GitHub] metron issue #1192: METRON-1774 Allow user to configure JAAS client in Ambar...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/1192
  
    Sure.  See  https://issues.apache.org/jira/browse/METRON-1783


---

[GitHub] metron pull request #1192: METRON-1774 Allow user to configure JAAS client i...

Posted by nickwallen <gi...@git.apache.org>.
GitHub user nickwallen reopened a pull request:

    https://github.com/apache/metron/pull/1192

    METRON-1774 Allow user to configure JAAS client in Ambari

    Right now, the user can only configure Metron's client JAAS in limited ways.  They can configure the principal name and key tab path, but few other things.  For example, I cannot easily turn on debugging today.  If I make a change to `$METRON_HOME/client_jaas.conf` to set `debug=true`, Ambari will overwrite my change when any of the topologies are restarted.
    
    With this PR the user can now edit a template in Ambari that is used to generate `$METRON_HOME/client_jaas.conf`.  This allows user to turn on debug logs, change the service name, or add additional options required by their environment.
    
    ![screen shot 2018-09-11 at 2 12 35 pm](https://user-images.githubusercontent.com/2475409/45381641-d9b45080-b5d3-11e8-9afe-afd79c61ad4c.png)
    
    ## Testing
    
    1. Stand-up a development environment.
    
    1. Ensure alerts are visible in the Alerts UI.
    
    1. Kerberize that development environment.
    
    1. Ensure alerts are visible in the Alerts UI.
    
    1. Ensure that Metron's `client_jaas.conf` was created correctly.
    
        ```
        [root@node1 0.6.0]# source /etc/default/metron
        [root@node1 0.6.0]# cat $METRON_HOME/client_jaas.conf
    
        StormClient {
           com.sun.security.auth.module.Krb5LoginModule required
           useKeyTab=false
           storeKey=false
           useTicketCache=true
           serviceName="nimbus"
           principal="metron@EXAMPLE.COM";
        };
        Client {
           com.sun.security.auth.module.Krb5LoginModule required
           useKeyTab=true
           keyTab="/etc/security/keytabs/metron.headless.keytab"
           storeKey=true
           useTicketCache=false
           serviceName="zookeeper"
           principal="metron@EXAMPLE.COM";
        };
        KafkaClient {
           com.sun.security.auth.module.Krb5LoginModule required
           useKeyTab=true
           keyTab="/etc/security/keytabs/metron.headless.keytab"
           storeKey=true
           useTicketCache=false
           serviceName="kafka"
           principal="metron@EXAMPLE.COM";
        ```
    
    1. In Ambari, go to Metron > Configs > Advanced > Advanced metron-client-jaas-conf and alter the template in some way.  For example, you could add `debug=true` to each stanza.
    
    1. Restart the Enrichment topology.
    
    1. Validate that Metron's `client_jaas.conf` was updated based on the changes you made in Ambari.
    
        ```
        [root@node1 0.6.0]# cat client_jaas.conf
        StormClient {
           com.sun.security.auth.module.Krb5LoginModule required
           debug=true
           useKeyTab=false
           storeKey=false
           useTicketCache=true
           serviceName="nimbus"
           principal="metron@EXAMPLE.COM";
        };
        Client {
           com.sun.security.auth.module.Krb5LoginModule required
           debug=true
           useKeyTab=true
           keyTab="/etc/security/keytabs/metron.headless.keytab"
           storeKey=true
           useTicketCache=false
           serviceName="zookeeper"
           principal="metron@EXAMPLE.COM";
        };
        KafkaClient {
           com.sun.security.auth.module.Krb5LoginModule required
           debug=true
           useKeyTab=true
           keyTab="/etc/security/keytabs/metron.headless.keytab"
           storeKey=true
           useTicketCache=false
           serviceName="kafka"
           principal="metron@EXAMPLE.COM";
        ```
    
    ## Pull Request Checklist
    
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
    - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
    - [x] Have you included steps or a guide to how the change may be verified and tested manually?
    - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
    - [ ] Have you written or updated unit tests and or integration tests to verify your changes?
    - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
    - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nickwallen/metron METRON-1774

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/1192.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1192
    
----
commit e3138acf62098b3e15ffe93d0699d0181e453c10
Author: Nick Allen <ni...@...>
Date:   2018-09-11T19:01:35Z

    METRON-1774 Allow user to configure JAAS client in Ambari

----


---

[GitHub] metron issue #1192: METRON-1774 Allow user to configure JAAS client in Ambar...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/1192
  
    @mmiklavc  Nothing changed about how the JAAS template gets deployed.  The template gets deployed when any service calls `storm_security_setup`, which includes Parser, Enrichment, Profiler, Indexing, and the mysterious Metron Client.  The UIs and REST never triggered a deployment of the JAAS template.  I assume because they don't directly use it.


---

[GitHub] metron issue #1192: METRON-1774 Allow user to configure JAAS client in Ambar...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/1192
  
    [CI failed](https://travis-ci.org/apache/metron/builds/427353381?utm_source=github_status&utm_medium=notification) due to unrelated transient test failure...
    ```
    SensorEnrichmentConfigControllerIntegrationTest.test:231 Status expected:<404> but was:<200>
    ```


---

[GitHub] metron pull request #1192: METRON-1774 Allow user to configure JAAS client i...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/metron/pull/1192


---

[GitHub] metron issue #1192: METRON-1774 Allow user to configure JAAS client in Ambar...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:

    https://github.com/apache/metron/pull/1192
  
    @nickwallen - sounds great to me. Can you create a Jira to track that and link it to this PR? +1


---

[GitHub] metron pull request #1192: METRON-1774 Allow user to configure JAAS client i...

Posted by nickwallen <gi...@git.apache.org>.
GitHub user nickwallen reopened a pull request:

    https://github.com/apache/metron/pull/1192

    METRON-1774 Allow user to configure JAAS client in Ambari

    Right now, the user can only configure Metron's client JAAS in limited ways.  They can configure the principal name and key tab path, but few other things.  For example, I cannot easily turn on debugging today.  If I make a change to `$METRON_HOME/client_jaas.conf` to set `debug=true`, Ambari will overwrite my change when any of the topologies are restarted.
    
    With this PR the user can now edit a template in Ambari that is used to generate `$METRON_HOME/client_jaas.conf`.  This allows user to turn on debug logs, change the service name, or add additional options required by their environment.
    
    ![screen shot 2018-09-11 at 2 12 35 pm](https://user-images.githubusercontent.com/2475409/45381641-d9b45080-b5d3-11e8-9afe-afd79c61ad4c.png)
    
    ## Testing
    
    1. Stand-up a development environment.
    
    1. Ensure alerts are visible in the Alerts UI.
    
    1. Kerberize that development environment.
    
    1. Ensure alerts are visible in the Alerts UI.
    
    1. Ensure that Metron's `client_jaas.conf` was created correctly.
    
        ```
        [root@node1 0.6.0]# source /etc/default/metron
        [root@node1 0.6.0]# cat $METRON_HOME/client_jaas.conf
    
        StormClient {
           com.sun.security.auth.module.Krb5LoginModule required
           useKeyTab=false
           storeKey=false
           useTicketCache=true
           serviceName="nimbus"
           principal="metron@EXAMPLE.COM";
        };
        Client {
           com.sun.security.auth.module.Krb5LoginModule required
           useKeyTab=true
           keyTab="/etc/security/keytabs/metron.headless.keytab"
           storeKey=true
           useTicketCache=false
           serviceName="zookeeper"
           principal="metron@EXAMPLE.COM";
        };
        KafkaClient {
           com.sun.security.auth.module.Krb5LoginModule required
           useKeyTab=true
           keyTab="/etc/security/keytabs/metron.headless.keytab"
           storeKey=true
           useTicketCache=false
           serviceName="kafka"
           principal="metron@EXAMPLE.COM";
        ```
    
    1. In Ambari, go to Metron > Configs > Advanced > Advanced metron-client-jaas-conf and alter the template in some way.  For example, you could add `debug=true` to each stanza.
    
    1. Restart the Enrichment topology.
    
    1. Validate that Metron's `client_jaas.conf` was updated based on the changes you made in Ambari.
    
        ```
        [root@node1 0.6.0]# cat client_jaas.conf
        StormClient {
           com.sun.security.auth.module.Krb5LoginModule required
           debug=true
           useKeyTab=false
           storeKey=false
           useTicketCache=true
           serviceName="nimbus"
           principal="metron@EXAMPLE.COM";
        };
        Client {
           com.sun.security.auth.module.Krb5LoginModule required
           debug=true
           useKeyTab=true
           keyTab="/etc/security/keytabs/metron.headless.keytab"
           storeKey=true
           useTicketCache=false
           serviceName="zookeeper"
           principal="metron@EXAMPLE.COM";
        };
        KafkaClient {
           com.sun.security.auth.module.Krb5LoginModule required
           debug=true
           useKeyTab=true
           keyTab="/etc/security/keytabs/metron.headless.keytab"
           storeKey=true
           useTicketCache=false
           serviceName="kafka"
           principal="metron@EXAMPLE.COM";
        ```
    
    ## Pull Request Checklist
    
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
    - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
    - [x] Have you included steps or a guide to how the change may be verified and tested manually?
    - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
    - [ ] Have you written or updated unit tests and or integration tests to verify your changes?
    - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
    - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nickwallen/metron METRON-1774

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/1192.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1192
    
----
commit e3138acf62098b3e15ffe93d0699d0181e453c10
Author: Nick Allen <ni...@...>
Date:   2018-09-11T19:01:35Z

    METRON-1774 Allow user to configure JAAS client in Ambari

----


---

[GitHub] metron pull request #1192: METRON-1774 Allow user to configure JAAS client i...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen closed the pull request at:

    https://github.com/apache/metron/pull/1192


---

[GitHub] metron issue #1192: METRON-1774 Allow user to configure JAAS client in Ambar...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/1192
  
    > @mmiklavc: Just curious, was that necessary to make the jaas config available for user modification in a text field in Ambari or for other reasons?
    
    Yes, that is the only way I've seen it done.  Its similar in the other Mpacks, like Kafka, etc.
    



---

[GitHub] metron issue #1192: METRON-1774 Allow user to configure JAAS client in Ambar...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:

    https://github.com/apache/metron/pull/1192
  
    Hey @nickwallen, I see a number of exceptions in the metron-rest.log. It dies shortly thereafter.
    
    ```
    18/09/18 00:51:41 WARN context.AnnotationConfigServletWebServerApplicationContext: Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'kafkaConfig': Unsatisfied dependency expressed through field 'zkClient'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'zkClient' defined in class path resource [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' threw exception; nested exception is java.lang.IllegalArgumentException: File /usr/metron/0.6.0/client_jaas.confcannot be read.
    18/09/18 00:51:41 ERROR imps.CuratorFrameworkImpl: Background exception was not retry-able or retry gave up
    java.lang.IllegalStateException: instance must be started before calling this method
            at com.google.common.base.Preconditions.checkState(Preconditions.java:176)
            at org.apache.curator.framework.imps.CuratorFrameworkImpl.getData(CuratorFrameworkImpl.java:363)
            at org.apache.curator.framework.recipes.cache.TreeCache$TreeNode.doRefreshData(TreeCache.java:244)
            at org.apache.curator.framework.recipes.cache.TreeCache$TreeNode.refresh(TreeCache.java:209)
            at org.apache.curator.framework.recipes.cache.TreeCache$TreeNode.wasCreated(TreeCache.java:263)
            at org.apache.curator.framework.recipes.cache.TreeCache$TreeNode.processResult(TreeCache.java:397)
            at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:728)
            at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:505)
            at org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:166)
            at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:590)
            at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
    18/09/18 00:51:41 INFO zookeeper.ZooKeeper: Session: 0x165e9a7dc4a002a closed
    18/09/18 00:51:41 INFO jpa.LocalContainerEntityManagerFactoryBean: Closing JPA EntityManagerFactory for persistence unit 'default'
    18/09/18 00:51:41 INFO hikari.HikariDataSource: HikariPool-1 - Shutdown initiated...
    18/09/18 00:51:41 INFO zookeeper.ClientCnxn: EventThread shut down
    18/09/18 00:51:41 INFO hikari.HikariDataSource: HikariPool-1 - Shutdown completed.
    Sep 18, 2018 12:51:41 AM org.apache.catalina.core.StandardService stopInternal
    INFO: Stopping service [Tomcat]
    18/09/18 00:51:41 INFO logging.ConditionEvaluationReportLoggingListener:
    
    Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
    18/09/18 00:51:41 ERROR boot.SpringApplication: Application run failed
    org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'kafkaConfig': Unsatisfied dependency expressed through field 'zkClient'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'zkClient' defined in class path resource [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' threw exception; nested exception is java.lang.IllegalArgumentException: File /usr/metron/0.6.0/client_jaas.confcannot be read.
            at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:587)
            at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:91)
            at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:373)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1344)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:578)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501)
            at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
            at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
            at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
            at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
            at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:760)
            at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
            at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
            at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)
            at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)
            at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395)
            at org.springframework.boot.SpringApplication.run(SpringApplication.java:327)
            at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
            at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)
            at org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:36)
    Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'zkClient' defined in class path resource [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' threw exception; nested exception is java.lang.IllegalArgumentException: File /usr/metron/0.6.0/client_jaas.confcannot be read.
            at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:541)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501)
            at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
            at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
            at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
            at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
            at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
            at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
            at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1065)
            at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:584)
            ... 19 more
    Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' threw exception; nested exception is java.lang.IllegalArgumentException: File /usr/metron/0.6.0/client_jaas.confcannot be read.
            at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
            at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:579)
            ... 31 more
    Caused by: java.lang.IllegalArgumentException: File /usr/metron/0.6.0/client_jaas.confcannot be read.
            at org.I0Itec.zkclient.ZkClient.isZkSaslEnabled(ZkClient.java:904)
            at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
            at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
            at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:97)
            at org.apache.metron.rest.config.ZookeeperConfig.zkClient(ZookeeperConfig.java:63)
            at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpringCGLIB$$e6070ab6.CGLIB$zkClient$1(<generated>)
            at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpringCGLIB$$e6070ab6$$FastClassBySpringCGLIB$$8bbe133d.invoke(<generated>)
            at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
            at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361)
            at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpringCGLIB$$e6070ab6.zkClient(<generated>)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
            ... 32 more
    ```


---

[GitHub] metron issue #1192: METRON-1774 Allow user to configure JAAS client in Ambar...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/1192
  
    [CI failed](https://travis-ci.org/apache/metron/builds/427377294?utm_source=github_status&utm_medium=notification) due to unrelated transient test failure...
    
    ```
    Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 289.241 sec <<< FAILURE! - in org.apache.metron.profiler.integration.ProfilerIntegrationTest
    testProcessingTimeWithTimeToLiveFlush(org.apache.metron.profiler.integration.ProfilerIntegrationTest)  Time elapsed: 135.735 sec  <<< FAILURE!
    java.lang.AssertionError
    	at org.junit.Assert.fail(Assert.java:86)
    	at org.junit.Assert.assertTrue(Assert.java:41)
    	at org.junit.Assert.assertTrue(Assert.java:52)
    	at org.apache.metron.profiler.integration.ProfilerIntegrationTest.testProcessingTimeWithTimeToLiveFlush(ProfilerIntegrationTest.java:210)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
    	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
    	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
    	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
    	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
    	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
    	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
    ```
    
    I am going to take a look at this failure (outside of this PR)


---

[GitHub] metron issue #1192: METRON-1774 Allow user to configure JAAS client in Ambar...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:

    https://github.com/apache/metron/pull/1192
  
    I think I see the problem. Is this meant to be only tied to a specific topology(ies) restarting to get deployed? There were some errors during service restart after Kerberos deployed which required me to restart a number of services manually. Since I'm running in full dev I only restarted a few necessary services, including REST and the UIs. Neither seems to have forced the jaas file to deploy. 


---

[GitHub] metron pull request #1192: METRON-1774 Allow user to configure JAAS client i...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen closed the pull request at:

    https://github.com/apache/metron/pull/1192


---

[GitHub] metron issue #1192: METRON-1774 Allow user to configure JAAS client in Ambar...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:

    https://github.com/apache/metron/pull/1192
  
    It sounds like this may be an existing bug. That stacktrace makes it pretty clear that the REST api wants the jaas file for configuring the zkClient.


---

[GitHub] metron pull request #1192: METRON-1774 Allow user to configure JAAS client i...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:

    https://github.com/apache/metron/pull/1192#discussion_r218412871
  
    --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-client-jaas-conf.xml ---
    @@ -0,0 +1,61 @@
    +<?xml version="1.0"?>
    +<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    +<!--
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +-->
    +<configuration supports_final="false" supports_adding_forbidden="true">
    +  <property>
    +    <name>content</name>
    +    <display-name>metron_client_jaas template</display-name>
    +    <description>Metron client JAAS configuration</description>
    --- End diff --
    
    The template that previously lived in `client_jaas.conf.j2` is now here.  This allows the user to edit the template in Ambari.


---

[GitHub] metron issue #1192: METRON-1774 Allow user to configure JAAS client in Ambar...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/1192
  
    Right.  So probably only shows up because kerberization in Full Dev never comes up cleanly (mainly because of the Elasticsearch service check always fails; ES is 'yellow' not 'green').
    
    I could add the `storm_security_setup` to the REST service.  Can we tackle that as a separate PR?  I'd want to refactor the code just a bit to make it clear that it is not necessarily just a "Storm" security setup, if in fact REST relies on it.



---

[GitHub] metron issue #1192: METRON-1774 Allow user to configure JAAS client in Ambar...

Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:

    https://github.com/apache/metron/pull/1192
  
    Took a look over the source and looks good to me there. I do want to spin this up in full dev before I give it my +1. One question regarding the change from j2/Jinja templates. Just curious, was that necessary to make the jaas config available for user modification in a text field in Ambari or for other reasons?


---