You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2016/10/03 22:01:08 UTC
[GitHub] incubator-metron pull request #290: METRON-421 Make Stellar Profiler Client ...
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/290
METRON-421 Make Stellar Profiler Client API Accessible in Parser and Enrichment Topologies
[METRON-421](https://issues.apache.org/jira/browse/METRON-421)
### Changes
These changes were required to make the Stellar Profiler Client API accessible from the Parser and Enrichment topologies.
* Refactored the Profiler into three projects.
* This dramatically helps with the clashing dependencies between Storm and HBase when the Profiler Client API is added to Enrichment.
* `metron-profiler-common` - Contains all code shared between the Profiler topology and client
* `metron-profiler` - Contains the topology used to build Profiles.
* `metron-profiler-client` - The client API used to access Profile data.
* Added the `metron-profiler-client` as a dependency to `metron-parser` and `metron-enrichment`. This allows Profile data to be accessed from both the Parser and Enrichment topologies.
* Added a README for the `metron-profiler-client` to describe its usage.
* Pushing the `metron-profiler-client` archive to the Metron host to make the library accessible to the Stellar Shell.
* Fixed an issue with potentially unsafe String to Byte conversion that may cause issues on different platforms.
### Testing
The changes were tested by deploying Metron using the 'Quick Dev' environment.
* [ ] Deployed and validated Parser topology using Snort and Bro
* [ ] Deployed and validated the Enrichment topology
* [ ] Deployed and validated the Indexing topology
* [ ] Deployed and validated the Profiler topology
* [ ] Used the Profiler Client API to retrieve Profile data
* [ ] Validated full end-to-end automated deployment with 'Quick Dev'
These changes can be manually tested by following the Getting Started instructions included in both the metron-profiler/README.md and metron-profiler-client/README.md.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-421
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/290.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #290
----
commit d994c5b6541b4187f36a7434cd2bc724baf6a8dc
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-19T21:25:28Z
METRON-421 Make Stellar Profiler Client API Accessible in Parser and Enrichment Topologies
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #290: METRON-421 Make Stellar Profiler Client API Acc...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/290
I can build and install Metron with the RPMs. I can build the Mpack. But I am not able to easily test installation of the Mpack. (It might make life easier to have Vagrant/Ansible spin-up with Ambari 2.4 so I could easily test this.)
I did a diff of /usr/metron as installed via Vagrant/Ansible to what is installed with the RPMs. The only notable gap that I would not expect was that the RPMs did not install `config/zookeeper/global.json`.
The other differences that are likely expected (need confirmation) is that the following files were installed via Vagrant/Ansible, but not by the RPM.
* `lib/metron-api-0.2.1BETA.jar`
* `lib/metron-maas-service-0.2.1BETA-uber.jar`
* `lib/metron-profiler-0.2.1BETA-uber.jar` and `lib/metron-profiler-client-0.2.1BETA.jar` as the Profiler is not installed via the RPMs currently.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #290: METRON-421 Make Stellar Profiler Client ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/290#discussion_r81747294
--- Diff: metron-analytics/metron-profiler/src/main/assembly/assembly.xml ---
@@ -63,7 +63,7 @@
<fileSet>
<directory>${project.basedir}/target</directory>
<includes>
- <include>${project.artifactId}-${project.version}.jar</include>
+ <include>${project.artifactId}-${project.version}-uber.jar</include>
</includes>
--- End diff --
I'd be glad to check. If you can point me to a README or specifics on what to check that would be appreciated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #290: METRON-421 Make Stellar Profiler Client API Acc...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/290
+1 on inspection
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #290: METRON-421 Make Stellar Profiler Client ...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/290#discussion_r81655831
--- Diff: metron-analytics/metron-profiler/src/main/assembly/assembly.xml ---
@@ -63,7 +63,7 @@
<fileSet>
<directory>${project.basedir}/target</directory>
<includes>
- <include>${project.artifactId}-${project.version}.jar</include>
+ <include>${project.artifactId}-${project.version}-uber.jar</include>
</includes>
--- End diff --
Is it important to change the jar names? If so, could you check for ripple changes to the RPM build scripts and the Ambari scripts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #290: METRON-421 Make Stellar Profiler Client ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/290
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #290: METRON-421 Make Stellar Profiler Client API Acc...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/290
Would love a review on this. It's getting aged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #290: METRON-421 Make Stellar Profiler Client ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/290#discussion_r81758410
--- Diff: metron-analytics/metron-profiler/src/main/assembly/assembly.xml ---
@@ -63,7 +63,7 @@
<fileSet>
<directory>${project.basedir}/target</directory>
<includes>
- <include>${project.artifactId}-${project.version}.jar</include>
+ <include>${project.artifactId}-${project.version}-uber.jar</include>
</includes>
--- End diff --
Think I've got it. Added some additional tests to the list in the PR description. Will have another commit coming that updates the RPM spec file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #290: METRON-421 Make Stellar Profiler Client API Acc...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/290
@nickwallen Looking over the spec change, since it's just a slight rename of things in the /lib folder (not adding a bunch of new files, or anything.), making sure the RPMs build, deploy, and the changed jars have correct perms should be good given the other testing of things.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #290: METRON-421 Make Stellar Profiler Client ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/290#discussion_r81746146
--- Diff: metron-analytics/metron-profiler/src/main/assembly/assembly.xml ---
@@ -63,7 +63,7 @@
<fileSet>
<directory>${project.basedir}/target</directory>
<includes>
- <include>${project.artifactId}-${project.version}.jar</include>
+ <include>${project.artifactId}-${project.version}-uber.jar</include>
</includes>
--- End diff --
Some of the projects did not have `<shadedClassifierName>uber</shadedClassifierName>` set. When trying to troubleshoot dependency issues, the dependencies coming from those projects would not show-up when running `mvn dependency:tree`. This was more than a little frustrating.
Once I turned that on so that I could actually see the dependencies being pulled in, the non-uber jars would be deployed and so nothing would actually work. These changes ensure that when troubleshooting you can actually see the dependencies and the right jar actually gets deployed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---