You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by JonZeolla <gi...@git.apache.org> on 2016/10/26 03:12:14 UTC
[GitHub] incubator-metron pull request #326: Update the bro_index elasticsearch templ...
GitHub user JonZeolla opened a pull request:
https://github.com/apache/incubator-metron/pull/326
Update the bro_index elasticsearch template to index *_body_len properly
## Problem
[METRON-510](https://issues.apache.org/jira/browse/METRON-510)
The bro *_body_len fields in [HTTP::Info](https://www.bro.org/sphinx/scripts/base/protocols/http/main.bro.html#type-HTTP::Info) can exceed the range of an int, and so writing to ElasticSearch fails with the following exception:
```
MapperParsingException[failed to parse [response_body_len]]; nested: JsonParseException[Numeric value (9876543210) out of range of int
```
## Solution
I updated the bro_index elasticsearch template to use a datatype of `long` for {request,response}_body_len, as opposed to an `integer`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JonZeolla/incubator-metron master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/326.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #326
----
commit d8efbf7af37a0a03131c7baaed74f197abc4f1de
Author: Jon Zeolla <ze...@gmail.com>
Date: 2016-10-24T13:29:21Z
Update the bro_index elasticsearch template to use a datatype of long for {request,response}_body_len
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #326: METRON-510: Update elasticsearch bro tem...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/326
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #326: METRON-510: Update the bro_index elasticsearch ...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/326
@JonZeolla - rgr that. Good deal. I'll try to give it a spin in quick dev today. Thanks again!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #326: METRON-510: Update the bro_index elasticsearch ...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/incubator-metron/pull/326
Not the best, but from my install node, inside the docker container I used to install, I inserted the updated template to my ES cluster via:
`curl --data "@/root/incubator-metron/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template" -XPUT server5:9200/_template/bro_index`
Then I put a message on the indexing topic with a response_body_len of 4261543936 (a value I saw cause this issue in the past):
`echo ${jsonBlob} | /usr/hdp/2.4.0.0-169/kafka/bin/kafka-console-producer.sh --topic indexing --broker-list server1:6667,server2:6667,server3:6667,server4:6667`
And watched the indexing topic, monitoring for the value in the response_body_len field:
`/usr/hdp/2.4.0.0-169/kafka/bin/kafka-console-consumer.sh --zookeeper $zk --topic indexing | grep 4261543936`
Finally, I successfully searched for a response_body_len of 4261543936 in kibana, as well as a more general response_body_len:>2147483647, which provided valid responses.
I also pursued testing this in quick-dev but it knocked over my laptop, so I'll be setting up a VM to do that in the future...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #326: METRON-510: Update the bro_index elasticsearch ...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/326
+1 ran it up on quick dev, looks good!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #326: METRON-510: Update the bro_index elasticsearch ...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/326
Hi @JonZeolla - this looks like a good change to me, thanks! Can you tell me how you tested it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---