You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/03/21 20:49:00 UTC
[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao
[ https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408564#comment-16408564 ]
ASF GitHub Bot commented on METRON-1421:
----------------------------------------
GitHub user justinleet opened a pull request:
https://github.com/apache/metron/pull/970
METRON-1421: Create a SolrMetaAlertDao
## Contributor Comments
Initial pass of adding meta alert functionality to the Solr implementation. It's extremely similar in style to the ES implementation (metaalerts field on an individual alert that list the metaalerts it belongs to, alerts field on metaalerts that has a denormalized copy of the individual alerts).
I need to do whatever running up we're set up for on this branch at this point. I got the build running and all the tests working. Sidenote, if anyone can tell me what exactly should be run up at this point, I would appreciate it.
Main things that are in this PR:
- `SolrMetaAlertDao` and integration tests.
- Schema changes as appropriate to support meta alerts.
- Lot of refactoring to have the ES and Solr implementations share a lot of metaalert code, seen in `AbstractMetaAlertDao`
- Lot of refactoring to have the ES and Solr integration tests share a lot of common code, seen in `MetaAlertIntegrationTest`
- There are some minor behavior differences, e.g. a metaalert with no alerts might return either no field or an empty list depending on Solr vs. ES. These (should) be limited to things we should be handling robustly elsewhere anyway.
- Some minor bugfixes (e.g. some leftover hardcoding, added some retry based result lookups, and so on).
- The index config needs to be available to Solr, so that we can determine what collection a given sensor belongs to.
- The `SolrComponent` can now have collections added while running, and tear them down between tests.
Things to look out for during review
- The use of a lot of the constants can be tricky to track, care should be given to make sure the appropriate choices are made (e.g. for some of the typing and such).
- The correctness of Solr queries. They're finicky, so make sure they make sense.
- Any stray` System.out`s.
- Stuff tends to be deferred to the implementations in certain cases. If there are cleaner ways to handle some of that, let me know.
- Any refactorings that could happen to merge anything else between impls.
Miscellaneous Notes
- Child documents in Solr do not get a field name. This implementation chooses to treat any child as a the child alert, and pass it back up the stack as the alert field to match ES. Similarly, going down the stack, Documents will have their alert field translated to the Solr field.
- Deleting all children of a parent in Solr seems to not work. I've seen references that this operation is undefined online, but hadn't seen it in the docs (although adding/removing in general is. It's weird). Instead, if all children would be gone, they are manually deleted before the parent update is set. If anyone knows better, I'd be thrilled.
- There's a bit of minor cleanup through some of the files that got touched. The test schema files in particular got autoformatted, because it was a pain to deal with sometimes when things were at different indentations and such.
- We're still apparently using ':' for all of our field names (e.g. source:type). Is there a particular reason other than inertia? Seems like we could be using '.' and my life would be easier here.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justinleet/metron METRON-1421
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/970.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #970
----
commit 17748f59c8eb2c5fa6cd77571588b4f2d7527427
Author: justinjleet <ju...@...>
Date: 2018-02-06T12:50:55Z
refactoring tests wip
commit be1883480eb5eb4b38ced1f7619d62a70fe40fbc
Author: justinjleet <ju...@...>
Date: 2018-02-09T14:27:49Z
WIP meta alerts stuff
commit 6c98c49ed14a3d8f2a40aafc5a2033cd06d22b82
Author: justinjleet <ju...@...>
Date: 2018-02-21T14:12:02Z
fix get all meta alerts
commit 988d9a3eade55e12adbf783473d1e563ef90da5d
Author: justinjleet <ju...@...>
Date: 2018-02-27T20:13:31Z
Fix some things
commit 9f9ad4efc13bbb6a65de2d9a3492371376fc6413
Author: justinjleet <ju...@...>
Date: 2018-02-28T16:25:05Z
Hack hardcoded fix to some update problems
commit 549a783bd17d78ccae5b322bac9b98efe64f612e
Author: justinjleet <ju...@...>
Date: 2018-02-28T19:36:26Z
More correct, but not actually correct lookup of sensor output collection
commit d442d6707cd8c224049376e6aa4b1bc8529f7d73
Author: justinjleet <ju...@...>
Date: 2018-03-01T16:15:24Z
WIP on getting search and remove working
commit 94215284d3a706213e79e409f7beb5fdeeee65e9
Author: justinjleet <ju...@...>
Date: 2018-03-05T14:28:15Z
more wip stuff, query parser issues
commit 86f9892cc7b2cfc6578451fc7f743e23d9389133
Author: justinjleet <ju...@...>
Date: 2018-03-05T15:42:28Z
Merging in from parent feature branch
commit a17132326d0d20f1944a3b422eda5fd42a3e063b
Author: justinjleet <ju...@...>
Date: 2018-03-05T18:42:00Z
fixing incorrect cast
commit f16362506d13454aac4f173805aed1ddc8c7e611
Author: justinjleet <ju...@...>
Date: 2018-03-13T12:37:28Z
fix plus some cleanup
commit 683643a9b7d4b967ffbb32014c6474c86663ebfe
Author: justinjleet <ju...@...>
Date: 2018-03-18T23:57:58Z
Parent Child Query fixes
commit dded631d6ff956a3dc33c248a0fc7b1557ee74eb
Author: justinjleet <ju...@...>
Date: 2018-03-19T15:19:47Z
Fixing child clause
commit 3361e7c0007801849ea9e1ab5a5f690425d451e2
Author: justinjleet <ju...@...>
Date: 2018-03-20T13:38:10Z
Fixes to nested alerts
commit 24fe1aa1ce7115fdaf69eff133cb4b699519730b
Author: justinjleet <ju...@...>
Date: 2018-03-20T16:00:00Z
Grouping working
commit ee848ad63fa10a8864c2eb963fc5286d97acc200
Author: justinjleet <ju...@...>
Date: 2018-03-20T16:26:31Z
cleanup and fixing some of the ES ripples
commit 2781ec7994ce6798cf0608aa30deb50f07d86e4b
Author: justinjleet <ju...@...>
Date: 2018-03-20T18:02:23Z
Fixing more ES
commit 305ae38724b1fd18fe2e1a342ede4eb2ad1755e2
Author: justinjleet <ju...@...>
Date: 2018-03-20T20:12:33Z
Fixing some index name shenanigans now that we aren't doing wildcard indices
commit d3610ddff9c88755335aa484b9637c8b06068cf0
Author: justinjleet <ju...@...>
Date: 2018-03-20T20:41:36Z
WIP indexing test fix
commit 17c7b93fc5e5150bfbe3c8f95571ba34062b3c97
Author: justinjleet <ju...@...>
Date: 2018-03-20T20:57:16Z
Merge in feature branch and fix integration test
commit cd888977ca2535b1677ef7302f859ae2abbd5ca2
Author: justinjleet <ju...@...>
Date: 2018-03-20T20:58:46Z
deleting old comment
commit 8f607adfcde58ee36079969141995329e3d50519
Author: justinjleet <ju...@...>
Date: 2018-03-20T21:06:35Z
Bunch of cleanup
commit f0fdd71090b298fea7c661ca97fbf916ea979b2f
Author: justinjleet <ju...@...>
Date: 2018-03-21T01:45:30Z
fixing ES remove to match nonfinite Double values
commit 063edf5dc51811616fa8857571eec9fb249030ec
Author: justinjleet <ju...@...>
Date: 2018-03-21T02:43:55Z
hacky workaround to the delete remaining docs problem
commit 5b7bec42e02f03f40afd4c6e01fc00fee05970d8
Author: justinjleet <ju...@...>
Date: 2018-03-21T02:54:17Z
Fixing create test to use proper retries
commit bdb16ea70838f75f892b467c8bd0484b3c0df9a2
Author: justinjleet <ju...@...>
Date: 2018-03-21T12:17:28Z
workaround for classpath issues, along with patch function fix for solr
commit 82dc8099a8338e4f4832ac46fa94c724ad3299fb
Author: justinjleet <ju...@...>
Date: 2018-03-21T13:17:54Z
Ensuring SolrComponent cleans up after itself.
commit 41811e05e7d38ddf3b94cfe210a8b8fe0a0e3c02
Author: justinjleet <ju...@...>
Date: 2018-03-21T14:00:47Z
Added missing annotations
commit e92e9e54705ad49de2a3f0a49a38e07ff0ff641e
Author: justinjleet <ju...@...>
Date: 2018-03-21T16:22:26Z
Lot of cleanup and guava fix for json patch
commit 896d26f2353c8602fddf7a5d1c42443f3318838a
Author: justinjleet <ju...@...>
Date: 2018-03-21T16:46:39Z
Merge branch 'feature/METRON-1416-upgrade-solr' into METRON-1421
----
> Create a SolrMetaAlertDao
> -------------------------
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
> Issue Type: Sub-task
> Reporter: Justin Leet
> Assignee: Justin Leet
> Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve implementing the various MetaAlertDao methods using the SolrJ library and also providing a SolrMetaAlertIntegrationTest (similar to ElasticsearchMetaAlertIntegrationTest).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)