You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/06/05 18:53:00 UTC
[jira] [Commented] (METRON-1585) SolrRetrieveLatestDao does not use
the collection lookup
[ https://issues.apache.org/jira/browse/METRON-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502303#comment-16502303 ]
ASF GitHub Bot commented on METRON-1585:
----------------------------------------
GitHub user justinleet opened a pull request:
https://github.com/apache/metron/pull/1050
METRON-1585: SolrRetrieveLatestDao does not use the collection lookup
## Contributor Comments
- Fixed the naming of the parameters in `getLatest()`.
- Updated the `SolrRetrieveLatestDao` to use the collection lookup. Includes updating constructor to have access to the appropriate config.
- Moved the utility function for the collection lookup to `SolrUtilities`, and changed `SolrUpdateDao` to use it.
- Added integration tests for the `SolrRetrieveLatestDao` to make sure this works as expected.
I also ran it up in full dev and did
```
curl -u user:password -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
"guid": "b42981ad-8b8f-493d-84fb-2c792d9fa237",
"sensorType": "bro"
}' 'http://node1:8082/api/v1/search/findOne' | jq '.'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
```
The response (after passing through jq for formatting):
```
{
"adapter.threatinteladapter.end.ts": 1528224091155,
"bro_timestamp": "1528224089.932079",
"ip_dst_port": 8080,
"enrichmentsplitterbolt.splitter.end.ts": 1528224091144,
"enrichmentsplitterbolt.splitter.begin.ts": 1528224091144,
"adapter.hostfromjsonlistadapter.end.ts": 1528224091147,
"adapter.geoadapter.begin.ts": 1528224091146,
"uid": "CUrRne3iLIxXavQtci",
"trans_depth": 225,
"protocol": "http",
"original_string": "HTTP | id.orig_p:50451 method:GET request_body_len:0 id.resp_p:8080 uri:/api/v1/persist/wizard-data?_=1484169340974 tags:[] uid:CUrRne3iLIxXavQtci referrer:http://node1:8080/ trans_depth:225 host:node1 id.orig_h:192.168.66.1 response_body_len:0 user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36 ts:1528224089.932079 id.resp_h:192.168.66.121",
"ip_dst_addr": "192.168.66.121",
"threatinteljoinbolt.joiner.ts": 1528224091158,
"host": "node1",
"enrichmentjoinbolt.joiner.ts": 1528224091149,
"adapter.hostfromjsonlistadapter.begin.ts": 1528224091147,
"threatintelsplitterbolt.splitter.begin.ts": 1528224091153,
"ip_src_addr": "192.168.66.1",
"user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36",
"timestamp": 1528224089932,
"method": "GET",
"request_body_len": 0,
"uri": "/api/v1/persist/wizard-data?_=1484169340974",
"source.type": "bro",
"adapter.geoadapter.end.ts": 1528224091146,
"referrer": "http://node1:8080/",
"threatintelsplitterbolt.splitter.end.ts": 1528224091153,
"adapter.threatinteladapter.begin.ts": 1528224091155,
"ip_src_port": 50451,
"guid": "b42981ad-8b8f-493d-84fb-2c792d9fa237",
"response_body_len": 0
}
```
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justinleet/metron retrieveLatest
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1050.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1050
----
commit b58ca1f239e4756d00903b9c2091175417242bf0
Author: justinjleet <ju...@...>
Date: 2018-06-05T17:30:56Z
Updating solr retrieve latest dao to use lookup
commit 050abf99a69654782ec6b40546bbec4667bc956a
Author: justinjleet <ju...@...>
Date: 2018-06-05T18:23:33Z
fixing test teardown
----
> SolrRetrieveLatestDao does not use the collection lookup
> --------------------------------------------------------
>
> Key: METRON-1585
> URL: https://issues.apache.org/jira/browse/METRON-1585
> Project: Metron
> Issue Type: Sub-task
> Reporter: Justin Leet
> Assignee: Justin Leet
> Priority: Major
>
> `getLatest` interface has the second arg as "sensorType", but the Solr DAO makes the assumption that it's "collection" and renames the arg and uses it without retrieving the actual collection. https://github.com/apache/metron/blob/feature/METRON-1416-upgrade-solr/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrRetrieveLatestDao.java#L47
> `getAllLatest` at https://github.com/apache/metron/blob/feature/METRON-1416-upgrade-solr/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrRetrieveLatestDao.java#L47
> This can affect other DAOs that defer to this DAO (e.g. update and metaalert)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)