You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by JonZeolla <gi...@git.apache.org> on 2016/11/13 03:59:02 UTC
[GitHub] incubator-metron pull request #358: METRON-517: Update elasticsearch bro tem...
GitHub user JonZeolla opened a pull request:
https://github.com/apache/incubator-metron/pull/358
METRON-517: Update elasticsearch bro templates for uri
## Problem
[METRON-517](https://issues.apache.org/jira/browse/METRON-517)
The bro uri field in [HTTP::Info](https://www.bro.org/sphinx/scripts/base/protocols/http/main.bro.html#type-HTTP::Info) can grow to a size which fails to insert the message into Elasticsearch. The related error message is:
`IllegalArgumentException[Document contains at least one immense term in field=\"uri\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped...`
## Short-term Solution
Set the elasticsearch template to [truncate the URI field](https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-above.html) at 10922 (32776 / 3), as [RFC 3986](https://tools.ietf.org/html/rfc3986) allows UTF-8 characters to be used in URIs. 32766 is a size limitation built into Lucene ([MAX_TERM_LENGTH](https://lucene.apache.org/core/6_2_1/core/constant-values.html#org.apache.lucene.index.IndexWriter.MAX_TERM_LENGTH)), and each UTF-8 character can be at most 4 bytes, but are predominantly [3 bytes](https://en.wikipedia.org/wiki/CJK_characters).
## Long-term Solution
Currently being tracked via [METRON-542](https://issues.apache.org/jira/browse/METRON-542)
## Testing
Inserted this into my bare metal cluster via `curl --data "@/root/incubator-metron/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template" -XPUT server5:9200/_template/bro_index`
Did some kibana queries and cluster monitoring on my bare metal cluster.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JonZeolla/incubator-metron METRON-517
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/358.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #358
----
commit b63040f0fa3b735fc9636a5eb98b966f164fe1f3
Author: Jon Zeolla <ze...@gmail.com>
Date: 2016-11-13T03:22:07Z
Restrict Bro HTTP URI field to 10922 characters so it will not surpass 32766 bytes (Assumes a UTF-8 maxb 3 bytes per character)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #358: METRON-517: Update elasticsearch bro templates ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/358
We are planning on dropping the BETA for 0.3.0 for the release being cut today. I think I'm ok with defaulting to the safer number (8191) and letting the user adjust in the template if they choose to not want that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #358: METRON-517: Update elasticsearch bro templates ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/358
I am +1 here under a condition. I would like to point out that despite what the ES documentation states, UTF-8 characters can occupy 4 bytes, so I'd ask that we move the limit to `32766/4 = 8191` Looking at the lucene source, I do not think they're doing anything to limit the characters to 3 bytes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #358: METRON-517: Update elasticsearch bro tem...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/358
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #358: METRON-517: Update elasticsearch bro templates ...
Posted by JonZeolla <gi...@git.apache.org>.
Github user JonZeolla commented on the issue:
https://github.com/apache/incubator-metron/pull/358
Right, I did take a look to see how realistic it would be for 4 byte characters to be used in real world URIs and it didn't look very concerning, which is why I went with 10922 instead of 8191. When I evaluated using 8191 I looked through 1 week of my http logs and found numerous legitimate URI fields which exceeded that, but none that exceeded 10922 (not that the latter doesn't happen, it's just much more rare).
Specifically I considered that legitimate URIs will usually contain CJK characters along with the standard URI delimiters. Because CJK characters only use up to 3 bytes (afaict), non-CJK characters are only rarely used (i.e. if enough of them are used that this problem is encountered, it is probably nefarious, but definitely an extremely rare event), as well as the fact that 8191 seems to truncate some legitimate URI fields, I went with 10922.
Of course, if the project is looking to drop its BETA flag before [METRON-542](https://issues.apache.org/jira/browse/METRON-542) (which would truncate based on URI length upstream from indexing and have other mitigations in place) has been implemented, I would agree that this should be 8191 for now.
If you'd still prefer 8191 after considering my previous comments, I'd be more than happy to swap it out, but I wanted to make sure to note that it may prevent a higher percent of legitimate URI fields from being indexed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---