You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by ottobackwards <gi...@git.apache.org> on 2017/06/29 15:42:15 UTC
[GitHub] metron pull request #626: METRON-947 [DO NOT MERGE UNTIL METRON-942] Stellar...
GitHub user ottobackwards opened a pull request:
https://github.com/apache/metron/pull/626
METRON-947 [DO NOT MERGE UNTIL METRON-942] Stellar Management Function for Metron Parser Extension Configurations
METRON-777 adds the Metron Extension capability, the default parsers as extensions, and the parser archetype
METRON-942 adds metron-rest support for installing and uninstalling parser extensions, as well as the configuration primitives for extensions in zookeeper
This PR adds a Stellar Management command to retrieve those configurations.
## Testing
### Build and install some parser extensions
Follow the test plan for [METRON-942](https://github.com/apache/metron/pull/580), to the point where you have built and installed one or more parser extensions.
### deploy the metron-management jar per readme
### Run Stellar
```bash
[vagrant@node1 ~]$ /usr/metron/0.4.0/bin/stellar -z node1:2181
```
Run the command:
```bash
CONFIG_GET('PARSER_EXTENSION', 'metron-parser-nice-assembly-0_4_0')
```
verify that the configuration return matches what you would see in the rest client
if you executed Parser Extension Controller : get /api/v1/ext/parsers/{name} -> name = metron-parser-nice-assembly-0_4_0
### For all changes:
- [ x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x ] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [na ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x ] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ na] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ottobackwards/metron METRON-947
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/626.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #626
----
commit 864d320d91c522dfc2eb63fc12341f316a3f8952
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-03-17T04:56:49Z
Metron Extension system
Based on Apache Nifi Nars
NAR changes
* new lib , rebrand to bundles from NAR
* port to VFS/FileObject from File based
* ability to set property values
* Rework FileUtils so that you can derive and override
* added initializers to set 'classes' that we care about instead of hard coding them, still needs defaults
* added components nec. for integration tests ( do not want dep. on metron-* )
* VFSClassloader for NarClassLoader
* Hdfs based integration test version of unpacknars tests
* HDFS ( filesystem ) based fileutilities to cover for writes to hdfs, since VFS is currently R/O HDFS
* modified plugin to support configuration of outputs
* use class index not service loader ( both subclass and annotated supported )
Archetype
* Parser Extension archetyp
* incudes all configuration
* creates tar.gz with bundle and configuration
* class index support ( automatic generation )
Extensions
* new extensions modules
* parser
* archetype built module for each parser type
* support for configuration only parsers with tests
Parsers
* moved all but json, csv, grok to extensions
* Bolt now loads from bundle properties
Deployment
* rpms for parsers
* create extension directories
* ambari initializes zookeeper per parser
* amabri creates hdfs directories
* ISSUE: Writing to hdfs
Rest-API
* only test against parsers in metron-parsers
* still needs integration
commit 850154cb086bd9f825daa6797b575cde95392a87
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-04-14T12:55:37Z
fix package names
commit e8caaf51e3f3d640e9ec1c3c548bd722d595687d
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-04-17T12:19:02Z
Merge remote-tracking branch 'apache/master' into METRON-777
commit c18cbe3a8847fbed7b5f6ba9182b58f01d142c2c
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-04-23T16:14:15Z
merge apache/master
commit 7760425b02142f01f37d58876aa09fb88b197c82
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-04-27T10:45:37Z
Merge remote-tracking branch 'apache/master' into METRON-777
commit bc7cc2d823d54be58fae13ef94f671457d4c10f0
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-04-29T04:05:41Z
use correct testing bundle
commit c25585d10529ca2048d2ed0df4d8dbd904d3db17
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-04-29T04:07:11Z
use the configured extension list, do not hardcode
commit 158f463f121cbca2cf2c4d3c1cd9cf7b707e1d23
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-04-29T12:58:41Z
Merge remote-tracking branch 'apache/master' into METRON-777
commit e21efb89a37d034e36cb82dab6c0b862cad3777b
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-04-30T13:16:41Z
make dependency explicit, I think parent version is not what we think it is when running from archetype, causing the extension versio to be used for this dependency
commit ac92d7b88404f8b8bc825c71a4aeee5aa015d757
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-04-30T13:44:58Z
instead of hard-coding in metron-parsers-extensions, overload in the archetype to use metronVersion
commit fd021e42a2cdb80676426b65ae227ac9f44fcd6a
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-03T14:34:31Z
Merge remote-tracking branch 'apache/master' into METRON-777
commit 9cafe970b40d551a846fc442ac121591ab1d0d6d
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-04T02:18:29Z
specify the plugin to fix dependency problem
commit de48845634ae1cecb258175326592a28f5b3f8fe
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-04T02:40:55Z
second attempt to fix plugin errors with jacoco
commit 995a4d746fa21400a55e9571957878ef2d5b48d8
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-04T13:46:58Z
Merge remote-tracking branch 'apache/master' into METRON-777
commit 5f74fc3a15fe5e56f9e3483fa4c25961ffb284dd
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-04T20:25:18Z
be sure to clear out before and after, we cannot assume order of maven build
commit a3e63555e4e76486be58ddd6f818006ff42cdd1f
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-05T03:42:54Z
Merge remote-tracking branch 'apache/master' into METRON-777
commit 1e0e30562b4b526d9512bb1dc06077674d6b3277
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-05T18:18:32Z
update archetype based on changes to loading and configuration
commit e3b71aed74271cf255be5fad072a99506f553d2e
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-06T16:08:58Z
use simple json and not the JSONUtil Instance, there are issues in storm loading
commit 6c0201b853ffd079f02589011c57cb6a960475e8
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-09T13:55:31Z
Merge remote-tracking branch 'apache/master' into METRON-777
commit ab0aad849319117dd3d964247861cf09bc3e7822
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-10T12:57:21Z
Merge remote-tracking branch 'apache/master' into METRON-777
commit 5565ad661f4143234b67230f7cfd420ea1bce91a
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-10T19:47:58Z
Merge remote-tracking branch 'apache/master' into METRON-777
commit 225fb4e38d6e5ec5eff877be2a84c8fd29af0c3e
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-12T18:12:03Z
Merge remote-tracking branch 'apache/master' into METRON-777
commit 97c1be132c114a0f3f729e5f9887adb5aa49f582
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-17T02:22:33Z
Merge remote-tracking branch 'apache/master' into METRON-777
commit b73f35fc25982556ad469de409e15c72b0305a8a
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-19T15:33:30Z
Merge remote-tracking branch 'apache/master' into METRON-777
commit eb2062be4c14d0549a6ce2504dbbe29ea5510141
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-22T11:58:50Z
Merge remote-tracking branch 'apache/master' into METRON-777
commit 382a29e18f9ab161f0a21d92c1ba3ce6f2601eeb
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-22T12:01:38Z
Merge remote-tracking branch 'apache/master' into METRON-777
commit 90a51deb14078b57966f3ad8e2f7e000cbb74ca3
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-05-25T21:52:55Z
Merge branch 'METRON-777' of https://github.com/ottobackwards/metron into METRON-777
commit 2e66ea397ffc82caf6a76d47ba3651754361dd51
Author: Jon Zeolla <ze...@gmail.com>
Date: 2017-05-25T22:07:23Z
Trivial documentation changes
commit 0dd8cf8c9ab89b43d543de8b5237010a948540a9
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-26T01:58:05Z
Merge remote-tracking branch 'apache/master' into METRON-777
commit a2634f31d2b16f9ece00dbae49a8b0265f091cfd
Author: Otto Fowler <ot...@gmail.com>
Date: 2017-05-26T02:00:07Z
Merge branch 'METRON-777' of https://github.com/JonZeolla/metron into jz-metron-777
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #626: METRON-947 [DO NOT MERGE UNTIL METRON-942] Stellar Manage...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/626
I am closing this PR, I'll re-submit when METRON-942 is landed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #626: METRON-947 [DO NOT MERGE UNTIL METRON-942] Stellar...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards closed the pull request at:
https://github.com/apache/metron/pull/626
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---