You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@metron.apache.org by mattf-horton <gi...@git.apache.org> on 2017/01/03 07:54:22 UTC

[GitHub] incubator-metron pull request #408: METRON-608 Mpack to install a single-nod...

GitHub user mattf-horton opened a pull request:

    https://github.com/apache/incubator-metron/pull/408

    METRON-608 Mpack to install a single-node test cluster

    This "metron-mpack-singlenode" is very similar to "metron-mpack", and can be usefully compared with it for code review, altho it also includes many bug fixes proposed in METRON-634.  As it says in the Jira, this is a short-term fix by providing a completely separate Mpack just for the single-node scenario, until we can work the bugs out of an mpack that works on all sizes of cluster.
    
    This was tested on a Centos7 VM with 16GB of RAM, and all prerequisites installed, including Python 2.7.11.  The usual install process was followed, with Ambari 2.4.2 and HDP stack 2.5.3.  After successful startup, a few thousand 'bro' packets were injected, and were seen to correctly propagate through parser, enricher, and indexer topologies, and finally into elasticsearch.
    
    An outstanding issue is that the configuration validator in the mpack doesn't work.  It fails with "unknown error" and must be ignored.  The multi-node mpack already had this problem, and I have run out of time to try to fix it here.  Consequently, my enhancement of trying to automatically flag the user if less than five Storm Supervisor Slots are allocated (commit 9ad8706), is untested.  I would like to open a new Jira for this issue, since it was pre-existing before these additions.
    Thanks,
    --Matt

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mattf-horton/incubator-metron METRON-608

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/408.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #408
    
----
commit 05571b501222b91457bd2f1b2ccc096af015f4cb
Author: mattf-horton <mf...@hortonworks.com>
Date:   2016-12-08T05:31:40Z

    METRON-608 Mpack to install a single-node test cluster

commit f2245900cd94389bc1c004c60f1606a49ca6ea8c
Author: mattf-horton <mf...@hortonworks.com>
Date:   2016-12-10T01:01:41Z

    Add quicklinks to Elasticsearch Service page in Ambari, for health and indexes.

commit 542a6f01376a656b539040992e9fa727efd8a047
Author: mattf-horton <mf...@hortonworks.com>
Date:   2016-12-10T01:02:40Z

    METRON-608 fix several bugs, especially in METRON use of 'es_url'

commit fb646fdda345107b0debd8395743630cc99995e0
Author: mattf-horton <mf...@hortonworks.com>
Date:   2016-12-10T09:05:43Z

    update version to 0.3.0

commit 8e30344e0b50fdff0cf8a1a66e7f41868a384171
Author: mattf-horton <mf...@hortonworks.com>
Date:   2016-12-10T09:09:30Z

    Merge/rebase updates from 0.3.0

commit 9ad8706e5209ca9506710ca6498ac387e667ddf0
Author: mattf-horton <mf...@hortonworks.com>
Date:   2016-12-13T02:03:07Z

    METRON-608 add service_advisor validation for number of Storm slots, enhance README, and a couple bug fixes

commit d4c97e6de4838265b72139727a5fd1082388117b
Author: mattf-horton <mf...@hortonworks.com>
Date:   2016-12-13T05:48:22Z

    tweak pom.xml file

commit 95d83702ef844bc94c2ce54bccaebb7b7cb0513a
Author: mattf-horton <mf...@hortonworks.com>
Date:   2016-12-20T00:54:07Z

    METRON-608 multiple small bug fixes

commit b5b3a348527c03bc778a5fc5dd0d94ec3a00470b
Author: mattf-horton <mf...@hortonworks.com>
Date:   2016-12-20T01:39:37Z

    rebase to Dec 19, 2016

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #408: METRON-608 Mpack to install a single-node test ...

Posted by mattf-horton <gi...@git.apache.org>.

Github user mattf-horton commented on the issue:

    https://github.com/apache/incubator-metron/pull/408
  
    @JonZeolla , I removed the obsolete comments about MySQL installation in Mpack.  Thanks for the suggestion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #408: METRON-608 Mpack to install a single-node test ...

Posted by dlyle65535 <gi...@git.apache.org>.

Github user dlyle65535 commented on the issue:

    https://github.com/apache/incubator-metron/pull/408
  
    @mattf-horton - I need an MPack that works for 1-N nodes in order to complete [METRON-671](https://issues.apache.org/jira/browse/METRON-671). I suspect I'm working through stuff you've already fixed. Would it make sense to push out a PR with the unified branch and let me test it? Then I could use finish METRON-671 on top of your changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #408: METRON-608 Mpack to install a single-node test ...

Posted by mattf-horton <gi...@git.apache.org>.

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/408

Hi @dlyle65535 , from a design perspective, I totally agree that we should have a single MPack that works on any size cluster, and I hope to replace both the 4(+)-node and single-node versions in the future with a more general solution. I even have a prototype of such. My reasons for proceeding with the split-off version at this time were entirely pragmatic, due to time pressure:

- I really needed a single-node that works with Ambari, for testing other PRs.
- I spent several days trying to make Elasticsearch work on a single node installation, while retaining the generality of the multi-node case, but couldn't make it work. This chewed up a lot of time.
- I put together a unified Mpack that used two different template files for Elasticsearch, depending whether it was for a single-node or multi-node install. That worked just fine, on the surface, but was clearly going to require in-depth testing of 1, 2, 3, and 4 node cases. I do not currently have time to thoroughly test all the special cases.
- Doing it this way I only had to test the single-node case, which was fairly fast.
- Having made one that works, I'd like to make it available to others.
- Now, if it is not accepted into the code, the high rate of change in metron-deployment will soon obsolete it.

That's pretty much it. It's a throw-away, but not until we do the work of actually producing the generalized solution and testing it thoroughly. Which I evaluate as lower priority than a number of other more urgent needs. Thanks.

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---