You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@metron.apache.org by mmiklavc <gi...@git.apache.org> on 2018/02/23 22:34:02 UTC

[GitHub] metron pull request #943: METRON-1462: Separate ES and Kibana from Metron Mp...

GitHub user mmiklavc opened a pull request:

    https://github.com/apache/metron/pull/943

    METRON-1462: Separate ES and Kibana from Metron Mpack

    ## Contributor Comments
    
    This splits ES and Kibana into their own MPacks. Two comments for reviewers:
    - The original discuss thread mentioned putting this in metron-contrib. Are we dead set on this, or is my split in metron-deployment reasonable?
    - I kept the Kibana templates and associated documentation in their current location and migrated them with the Kibana service. This is consistent with our current deployment and means we don't need to add a new set of features to the Metron MPack to publish them. A point in favor of moving this to Metron's MPack would be that we think this is in the purview of Metron and would make the Ambari design more consistent when we introduce Solr.
    
    **Testing**
    Spin up full dev and verify the Kibana dashboard is displaying information.
    
    ## Pull Request Checklist
    
    In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
    
    ### For all changes:
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
    
    
    ### For code changes:
    - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
    - [x] Have you included steps or a guide to how the change may be verified and tested manually?
    - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
      ```
      mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh 
      ```
    
    - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
    
    ### For documentation related changes:
    - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
    
      ```
      cd site-book
      mvn site
      ```

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mmiklavc/metron es-mpack-split

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/943.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #943
    
----
commit f266600bf67dd11974a4c5faff7120c1b1044629
Author: Michael Miklavcic <mi...@...>
Date:   2018-02-21T16:58:42Z

    Ready to test ES/Kibana MPack split. TODO: README updates

commit 941e767287e403873d893a304b5847033e4c9b16
Author: Michael Miklavcic <mi...@...>
Date:   2018-02-21T16:59:55Z

    Merge branch 'master' into es-mpack-split

commit 96e4b6188b04a5518064a9a1cd59aeaf3059cdc1
Author: Michael Miklavcic <mi...@...>
Date:   2018-02-23T21:40:03Z

    Document the ES and Kibana MPack split

----


---

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

Posted by mmiklavc <gi...@git.apache.org>.

Github user mmiklavc commented on the issue:

    https://github.com/apache/metron/pull/943
  
    @ottobackwards @merrimanr @anandsubbu - I've moved the Kibana dashboard from the Elasticsearch MPack to the Metron one. Can you reaffirm your +1's or let me know if there's anything more I need to address?


---

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

Posted by ottobackwards <gi...@git.apache.org>.

Github user ottobackwards commented on the issue:

    https://github.com/apache/metron/pull/943
  
    What about upgrading.md?  


---

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

Posted by merrimanr <gi...@git.apache.org>.

Github user merrimanr commented on the issue:

    https://github.com/apache/metron/pull/943
  
    I spun this up in full dev and everything worked as expected.  The organization looks good to me and I can't find anything wrong with it.  +1


---

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

Posted by anandsubbu <gi...@git.apache.org>.

Github user anandsubbu commented on the issue:

    https://github.com/apache/metron/pull/943
  
    +1
    
    Works as advertised. I used both mpacks to deploy a 12-node CentOS 7 cluster (Ambari 2.6.0.0 and HDP 2.6.3.0). Was able to kerberize the cluster and get bro indices into ES.


---

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

Posted by mmiklavc <gi...@git.apache.org>.

Github user mmiklavc commented on the issue:

    https://github.com/apache/metron/pull/943
  
    I re-tested this via full dev spin-up from scratch. The dashboard comes up as expected.


---

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

Posted by nickwallen <gi...@git.apache.org>.

Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/943
  
    I was able to replicate this same problem in master running on the ubuntu14 dev environment.  This does not seem to be a problem caused by your PR.  I will open a separate defect for this.
    
    +1 That being said, nice work!  


---

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

Posted by mmiklavc <gi...@git.apache.org>.

Github user mmiklavc commented on the issue:

    https://github.com/apache/metron/pull/943
  
    @ottobackwards I believe the repercussions are the same as they are for the existing MPack since the services are distinct from the MPack itself. We should probably consider a separate test for this.


---

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

Posted by nickwallen <gi...@git.apache.org>.

Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/943
  
    When I spin this up in Ubuntu, the Indexing topology is unable to write to Elasticsearch.  This may very well be a resource constraint issues in my environment, but I thought I'd mention it before I spend too much more time digging into this.
    ```
    2018-04-04 15:51:05.707 o.a.s.d.executor Thread-6-indexingBolt-executor[3 3] [ERROR] 
    org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{rZcTXccfSPq4fH4IRLQ0zg}{node1}{127.0.1.1:9300}]
    	at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347) ~[stormjar.jar:?]
    	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245) ~[stormjar.jar:?]
    	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59) ~[stormjar.jar:?]
    	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363) ~[stormjar.jar:?]
    	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[stormjar.jar:?]
    	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80) ~[stormjar.jar:?]
    	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54) ~[stormjar.jar:?]
    	at org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:92) ~[stormjar.jar:?]
    	at org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239) [stormjar.jar:?]
    	at org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217) [stormjar.jar:?]
    	at org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:236) [stormjar.jar:?]
    	at org.apache.storm.daemon.executor$fn__7590$tuple_action_fn__7592.invoke(executor.clj:730) [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
    	at org.apache.storm.daemon.executor$mk_task_receiver$fn__7511.invoke(executor.clj:462) [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
    	at org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40) [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
    	at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
    	at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
    	at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
    	at org.apache.storm.daemon.executor$fn__7590$fn__7603$fn__7656.invoke(executor.clj:849) [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
    	at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
    	at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
    	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
    ```


---