You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/09/11 18:42:00 UTC

[jira] [Commented] (METRON-1177) Stale running topologies seen post-kerberization and cause exceptions

    [ https://issues.apache.org/jira/browse/METRON-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161758#comment-16161758 ] 

ASF GitHub Bot commented on METRON-1177:
----------------------------------------

GitHub user nickwallen opened a pull request:

    https://github.com/apache/metron/pull/748

    METRON-1177 Stale running topologies seen post-kerberization and cause exceptions

    [METRON-1177](https://issues.apache.org/jira/browse/METRON-1177)
    
    ### Problem
    
    After running the Ambari Kerberization process on a cluster where Metron was installed with the MPack, often times the Kerberization process would complete successfully, but the running Metron topologies were stale and had not been restarted properly after all Kerberization steps completed.  In other cases, the Metron service check would fail when Ambari began restarting all cluster services.
    
    One clue that this has occurred is that when querying Storm using the Thrift API to check on topology status after kerberization would result in the following error.
    ```
    AuthorizationException(msg:getTopologyInfo on topology snort is not authorized)
    ```
    
    ### Solution
    
    * All Metron services have to be started before performing a Metron service check.
    * All external dependencies like Storm, HBase, Kafka, etc must complete their service check before performing the service check of a Metron service having those dependencies.
    * Added Storm as a start dependency for the Metron Profiler.
    * Metron Profiler has to be stopped before Storm is stopped.
    
    ### Testing
    
    This was tested by launching the Full Dev environment, kerberizing the environment, and then monitoring the order in which each of the service start, stop and status check actions occurred.  I was not able to replicate the failure condition with this fix.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nickwallen/metron METRON-1177

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/748.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #748
    
----
commit a115f726eb5ccd3d215a7282871d290317cc1052
Author: Nick Allen <ni...@nickallen.org>
Date:   2017-09-08T15:23:49Z

    METRON-1177 Stale running topologies seen post-kerberization and cause exceptions

----


> Stale running topologies seen post-kerberization and cause exceptions
> ---------------------------------------------------------------------
>
>                 Key: METRON-1177
>                 URL: https://issues.apache.org/jira/browse/METRON-1177
>             Project: Metron
>          Issue Type: Bug
>            Reporter: Anand Subramanian
>            Assignee: Nick Allen
>
> *Steps to Reproduce*
> 1. Install Ambari on 12 node cluster
> 2. Install Metron using MPack
> 3. Metron topologies are now running as user 'storm'
> 4. Setup standalone KDC on Ambari node by following these [steps|https://github.com/apache/metron/blob/master/metron-deployment/Kerberos-manual-setup.md#setup-a-kdc]
> 5. Initiate "kerberize" cluster using Ambari
> 6. Ambari triggers "Stop all services"
> 7. In the event of a race condition, the REST service shuts down (which prevents from being able to shutdown metron topologies)
> 8. Metron topology shut down occurs, but Metron Mpack cannot stop topologies because REST already stopped. Metron ignores this condition and continues on assuming the topologies are down
> 9. Storm service shuts down
> 10. Cluster kerberization occurs
> 11. Ambari triggers "Start all services"
> 12. Storm service starts. 
> 13. Since the Metron topologies were running when Storm was shutdown, these same topologies are attempted to be started again by Storm, but are declared as "already running"
> 14. All other services start successfully
> 15. Now, if we attempt to run a test with the stale topologies, following exceptions are seen:
> {code}
> AuthorizationException(msg:getTopologyInfo on topology snort is not authorized)
>         at org.apache.storm.generated.Nimbus$getTopologyInfo_result$getTopologyInfo_resultStandardScheme.read(Nimbus.java:36971)
>         at org.apache.storm.generated.Nimbus$getTopologyInfo_result$getTopologyInfo_resultStandardScheme.read(Nimbus.java:36939)
>         at org.apache.storm.generated.Nimbus$getTopologyInfo_result.read(Nimbus.java:36870)
>         at org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
>         at org.apache.storm.generated.Nimbus$Client.recv_getTopologyInfo(Nimbus.java:1182)
>         at org.apache.storm.generated.Nimbus$Client.getTopologyInfo(Nimbus.java:1169)
>         at org.hw.metron.lib.storm.StormClient.isTopologyRunning(StormClient.java:150)
>         at org.hw.metron.lib.storm.StormClient.startTopology(StormClient.java:68)
>         at functional.SnortTopologyTest.snortElasticsearchIndexTest(SnortTopologyTest.java:36)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:86)
>         at org.testng.internal.Invoker.invokeMethod(Invoker.java:643)
>         at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:820)
>         at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1128)
>         at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
>         at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
>         at org.testng.TestRunner.privateRun(TestRunner.java:782)
>         at org.testng.TestRunner.run(TestRunner.java:632)
>         at org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
>         at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361)
>         at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319)
>         at org.testng.SuiteRunner.run(SuiteRunner.java:268)
>         at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
>         at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
>         at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244)
>         at org.testng.TestNG.runSuitesLocally(TestNG.java:1169)
>         at org.testng.TestNG.run(TestNG.java:1064)
>         at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:295)
>         at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:84)
>         at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:90)
>         at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
>         at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
>         at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)