You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by merrimanr <gi...@git.apache.org> on 2018/02/28 23:08:04 UTC
[GitHub] metron pull request #945: METRON-1464: Convert schemas to be compatible with...
GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/945
METRON-1464: Convert schemas to be compatible with Solr 5.5.2
## Contributor Comments
This PR makes the Solr schema compatible with Solr 5.5.2. The client code worked as is (compiled with version 6.6.2), no changes necessary.
To test this in full dev:
1. Spin up full dev
2. Install HDP Search ("Chapter 3: Installing HDP Search Using Ambari" in https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_solr-search-installation/bk_solr-search-installation.pdf)
3. Create the bro and snort collections manually:
```
/opt/lucidworks-hdpsearch/solr/bin/solr create -c bro -d /usr/metron/0.4.3/config/schema/bro
/opt/lucidworks-hdpsearch/solr/bin/solr create -c snort -d /usr/metron/0.4.3/config/schema/snort
```
4. Create a file named `metron_solr_template_installed_flag_file` in $METRON_HOME/config
5. Navigate to the Metron service in Ambari and change the "Solr Zookeeper Urls" setting in the Index Settings tab to `node1:2181/solr`
6. Change the "Random Access Search Engine" setting in the Indexing tab to `Solr`
7. Restart the Indexing component in Ambari
Navigate to node1:8983 and inspect the bro and snort collections. They should contain data. There will likely be additional Ambari work should we decide to move to HDP Search. This PR still assumes Solr 6.6.2 will be installed manually.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh
```
- [] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron METRON-1464
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/945.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #945
----
commit ac1de170af8fa769683f72783f6258cf12663f94
Author: merrimanr <me...@...>
Date: 2018-02-28T22:49:37Z
initial commit
----
---
[GitHub] metron issue #945: METRON-1464: Convert schemas to be compatible with Solr 5...
Posted by simonellistonball <gi...@git.apache.org>.
Github user simonellistonball commented on the issue:
https://github.com/apache/metron/pull/945
Are we losing anything by moving the scheme from Range to Trie types?, repeating my comment on
https://github.com/apache/metron/pull/922:
Given that our use case is heavily dependant on sorting, I wonder why not the Trie based indices for numeric fields. I may be completely wrong on their advantages but would love to hear the logic behind the choice of Point indices.
If there is a good reason, maybe we should consider retaining those for 6.6 in addition to the 5.5 clusters. Either way it would be be good to understand the basis for the type decision.
---
[GitHub] metron issue #945: METRON-1464: Convert schemas to be compatible with Solr 5...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/945
Ok, I'm cool with it. +1 by inspection; great work.
---
[GitHub] metron issue #945: METRON-1464: Convert schemas to be compatible with Solr 5...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/945
When looking at the reference guide, it's not obvious to me what the differences are between Point and Trie fields as it relates to our use case. Point field types for the corresponding Trie field types don't exist in 5.5 anyways so I suspect it's a moot point if supporting 5.5 is desirable.
Yes this has been tested with the HDP Search MPack (Solr 5.5.2) and the manually installed Solr 6.6.2.
---
[GitHub] metron pull request #945: METRON-1464: Convert schemas to be compatible with...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr closed the pull request at:
https://github.com/apache/metron/pull/945
---
[GitHub] metron issue #945: METRON-1464: Convert schemas to be compatible with Solr 5...
Posted by simonellistonball <gi...@git.apache.org>.
Github user simonellistonball commented on the issue:
https://github.com/apache/metron/pull/945
To be fair, my question is probably just as appropriate on a discuss thread and a separate ticket out of said thread if it comes to it.
---
[GitHub] metron issue #945: METRON-1464: Convert schemas to be compatible with Solr 5...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/945
@merrimanr you ran this up in Solr 5.5 as well as 6.6, right? If so, then I'm content with the change and give a +1 pending (other than holding for an answer to Simon's question, which I was wondering as well).
---