You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by nickwallen <gi...@git.apache.org> on 2018/06/07 17:36:19 UTC
[GitHub] metron pull request #1056: METRON-1609 Elasticsearch settings in Ambari shou...
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1056
METRON-1609 Elasticsearch settings in Ambari should not be required if Solr is the Indexer
NOTE: This PR has been submitted against the Solr feature branch; not master.
## Description
When a user deploys Metron with Solr, the Mpack requires that the user populate the Index Settings > Elasticsearch Hosts field to continue, even when deploying a Metron cluster that indexes to Solr and does not use Elasticsearch.
* The Elasticsearch related properties under the Ambari 'Index settings' tab should only be required if Elasticsearch is chosen as the indexer.
* The Solr related properties under the 'Index Settings' tab should only be required if Solr is chosen as the indexer.
## Changes
* Removed the static validation checks for each of the parameters on the 'Index Settings' tab. Static validation no longer works because validation depends on which Indexer the user has chosen.
* Added a function `check_indexing_parameters` that validates the parameters based on the Indexer that has been chosen by the user on the "Indexing" tab.
* The parameters are checked before any of the following actions:
* Start indexing
* Stop indexing
* Status check indexing
* Service checking indexing
* Install Elasticsearch Template
* Delete Elasticsearch Template
* Install Kibana Dashboard
* Update global properties
* When updating the global properties, we only update either Solr or Elasticsearch, not both as before.
* It seems logical to move the 'Indexing Settings' tab parameters to the `Index Settings` tab, especially when it on this page where the user selects the Indexer; Elasticsearch or Solr. I am not sure if there are other implications in doing this, so I did not do that as part of this PR.
## Testing
1. Start the development environment.
1. Validate that the Alerts UI is receiving alerts and that the Ambari 'Service Check' passes.
**Missing Elasticsearch parameters should be caught**
1. Stop the Indexing topology in Ambari.
1. Using Ambari, delete (define a blank value for) the `metron-env/es_binary_port` parameter in Metron > Index Settings.
1. Attempt to start the Indexing topology in Ambari. This should fail with the following error message.
```
resource_management.core.exceptions.Fail: Missing required indexing parameters(s): indexer=Elasticsearch, missing=['metron-env/es_binary_port']
```
1. Attempt the Elasticsearch Template Install. This should fail with the same error message.
1. Attempt the Elasticsearch Template Delete. This should fail with the same error message.
1. Attempt the Kibana Dashboard Install. This should fail with the same error message.
**Missing Elasticsearch parameters should not impact unrelated functions**
1. Restart another unrelated topology like the Profiler or Enrichment. These should restart successfully as they are not impacted by the missing Indexing parameters.
**Missing Elasticsearch parameters should not impact Solr**
1. Go to Metron > Indexing and change "Random Access Search Engine" to "Solr".
1. Attempt to start the Indexing topology in Ambari. This should succeed even though we are missing Elasticsearch parameters.
**Missing Solr parameters should be caught**
1. Stop the Indexing topology in Ambari.
1. Using Ambari, delete (define a blank value for) the `metron-env/solr_zookeeper_url` parameter in Metron > Index Settings.
1. Attempt to start the Indexing topology in Ambari. This should fail with the following error message.
```
resource_management.core.exceptions.Fail: Missing required indexing parameters(s): indexer=Solr, missing=['metron-env/solr_zookeeper_url']
```
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1609
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1056.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1056
----
commit 11acc9fa0be95ddb6b42cafe8b0a39cadc7d496e
Author: Nick Allen <ni...@...>
Date: 2018-06-07T15:59:27Z
METRON-1609 Elasticsearch settings in Ambari should not be required if Solr is the indexer
----
---
[GitHub] metron issue #1056: METRON-1609 Elasticsearch settings in Ambari should not ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1056
This has been merged
---
[GitHub] metron pull request #1056: METRON-1609 Elasticsearch settings in Ambari shou...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen closed the pull request at:
https://github.com/apache/metron/pull/1056
---
[GitHub] metron issue #1056: METRON-1609 Elasticsearch settings in Ambari should not ...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/1056
I spun this up in full dev and the only issue I could find was that starting REST doesn't throw an error when a required parameter is missing. The action succeeds but the app won't start and Ambari will still report it as down. Other than that everything else worked.
---
[GitHub] metron issue #1056: METRON-1609 Elasticsearch settings in Ambari should not ...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1056
I'm +1 by inspection assuming @merrimanr is good.
---
[GitHub] metron issue #1056: METRON-1609 Elasticsearch settings in Ambari should not ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1056
Thanks for the review and all the testing guys! Will merge now.
---
[GitHub] metron issue #1056: METRON-1609 Elasticsearch settings in Ambari should not ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1056
I deconflicted and pushed some changes to catch any problems when the REST service is started. I am in the middle of testing those changes now.
---
[GitHub] metron issue #1056: METRON-1609 Elasticsearch settings in Ambari should not ...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/1056
I tested this in full dev and went through the testing instructions. REST now throws an error on restart. Thanks! +1
---
[GitHub] metron issue #1056: METRON-1609 Elasticsearch settings in Ambari should not ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1056
Thanks @merrimanr. Good catch. I will fix that.
---
[GitHub] metron issue #1056: METRON-1609 Elasticsearch settings in Ambari should not ...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1056
@nickwallen Can you conflict this?
---
[GitHub] metron issue #1056: METRON-1609 Elasticsearch settings in Ambari should not ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1056
This is ready to review again. Cheers
---