You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/10/19 20:54:00 UTC
[jira] [Commented] (METRON-1834) Migrate Elasticsearch from
TransportClient to new Java REST API
[ https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657425#comment-16657425 ]
ASF GitHub Bot commented on METRON-1834:
----------------------------------------
GitHub user mmiklavc opened a pull request:
https://github.com/apache/metron/pull/1242
METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API
## Contributor Comments
https://issues.apache.org/jira/browse/METRON-1834
This task has been a long time coming after having completed the ES upgrade in https://issues.apache.org/jira/browse/METRON-939. Motivation for completing this now is that Elasticsearch will be deprecating use of the TransportClient in v 7.x. This PR migrates the Elasticsearch client from TransportClient to the newer Java REST API.
1. https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.6/client.html
2. https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-overview.html
3. https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high-level-migration.html
This builds off and finishes work started by @cestella here - https://github.com/cestella/incubator-metron/tree/es_rest_client. I condensed his branch into 1 flattened commit and built on top of it in order to provide attribution.
I have a number of tasks I'm still working through, but I wanted to get the review process started. I've minimally validated X-Pack auth and will have some follow-up for SSL. Test plans and a breakdown of the changes will be soon to follow. For starters, full dev should continue to work as normal and you should see data flowing into indexes for bro, snort, and yaf. There are some additional changes to how this client will be configured, which I'll be documenting shortly. The new client does not take a Map of settings any longer now that it is leverage Apache HTTP Async Client https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-low-usage-dependencies.html under the hood. This meant choosing a set of properties to expose and doing a translation to the builder pattern under the hood. Again, I'll have a write-up of this in the migration guide and update the README's accordingly.
NOTE: This checks off 2 items from this follow-on list https://github.com/apache/metron/pull/840#issuecomment-347281776
1. Fix Log4j logging problem - classpath issues
2. Migrate to new ES REST client
Per discussion in the Metron Slack channel, I will be updating the Jira ticket with a series of tasks to be completed prior to acceptance, including performance regression testing compared with the old API.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mmiklavc/metron es-rest-client
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1242.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1242
----
commit a7c7dc287b4f9c99c6780b934a0b6f433a03aa04
Author: cstella <ce...@...>
Date: 2018-10-09T00:06:52Z
Casey Stella - elasticsearch rest client migration base work
commit 10410ea9718a2a1b1d287fb4f22a6c98efb1fdaa
Author: Michael Miklavcic <mi...@...>
Date: 2018-10-09T00:07:22Z
Update shade plugin version
commit a33a16872118175ed35729df6ddde2959e49ae2f
Author: Michael Miklavcic <mi...@...>
Date: 2018-10-09T15:56:08Z
Fix es update dao test
commit 52c3c96d7657205e65d3bd0c0e35923a851911da
Author: Michael Miklavcic <mi...@...>
Date: 2018-10-09T21:26:16Z
Merge with master. Fix es search integration tests
commit 4742832869f9512e11cab4a32109c15a2b17a92e
Author: Michael Miklavcic <mi...@...>
Date: 2018-10-11T18:45:06Z
Merge branch 'master' into es-rest-client
commit 43809968320e586fd70411776140f3aa13a60195
Author: Michael Miklavcic <mi...@...>
Date: 2018-10-11T23:59:25Z
Get shade plugin working with the new ES client and the ClassIndexTransformer Shade plugin transformer.
commit af03f6f036e96c742db39733bf8ebc2cbf229129
Author: Michael Miklavcic <mi...@...>
Date: 2018-10-19T02:56:52Z
Introduce config classes for managing ES client configuration. Translate properties for new client.
commit dfcf9df2c9d41ddd777d02678c90fbb1e7dce1a3
Author: Michael Miklavcic <mi...@...>
Date: 2018-10-19T19:21:23Z
Resolve merge conflicts with master
commit babe86451abe0f56a2cd5ed92e49cbed6bca31ae
Author: Michael Miklavcic <mi...@...>
Date: 2018-10-19T19:50:31Z
Merge branch 'master' into es-rest-client
commit df2a77d15251d7c1ea77f87d238a0fba932ae4dd
Author: Michael Miklavcic <mi...@...>
Date: 2018-10-19T20:19:50Z
Remove extra deps in elasticsearch pom
----
> Migrate Elasticsearch from TransportClient to new Java REST API
> ---------------------------------------------------------------
>
> Key: METRON-1834
> URL: https://issues.apache.org/jira/browse/METRON-1834
> Project: Metron
> Issue Type: Improvement
> Reporter: Michael Miklavcic
> Assignee: Michael Miklavcic
> Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)