You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by dlyle65535 <gi...@git.apache.org> on 2016/05/25 20:45:01 UTC
[GitHub] incubator-metron pull request: METRON-152: Add support for ES 2.3....
GitHub user dlyle65535 opened a pull request:
https://github.com/apache/incubator-metron/pull/135
METRON-152: Add support for ES 2.3.x and Kibana 4.5.0
Use ES2.3 and Kibana 4.5.0.
Currently does not support PCAP queries from the UI. Will open up separate JIRA.
Tested on Vagrant (quick-dev)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dlyle65535/incubator-metron METRON-152
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/135.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #135
----
commit c5a5e2cbdc81ec43120201cb15d4f7ea6f848c92
Author: David Lyle <dl...@gmail.com>
Date: 2016-05-25T15:26:16Z
METRON-152: Add support for ES 2.3.x and Kibana 4.5.0
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-152: Add support for ES 2.3....
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/135#issuecomment-221865759
FYI - Spinning up an Amazon deployment on this now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-152: Add support for ES 2.3....
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/135#issuecomment-222159039
I am still seeing failures. Tried a redeploy of the environment I already had setup along with a fresh install. Ran into this problem both times.
```
Unexpected character (''' (code 39)): was expecting either valid name character (for unquoted name)
or double-quote (for quoted) to start field name\n at [Source: [B@79b57c87; line: 1, column: 3]",
"type": "json_parse_exception"}
```
I confirmed that I have your latest commit.
```
commit f6eafc93256e26447806ff413cc22a57134cb88b
Author: David Lyle <dl...@gmail.com>
Date: Fri May 27 06:57:18 2016 -0400
Add to_json to help with dict type error.
Added kibana to services_to_start
commit c5a5e2cbdc81ec43120201cb15d4f7ea6f848c92
Author: David Lyle <dl...@gmail.com>
Date: Wed May 25 11:26:16 2016 -0400
METRON-152: Add support for ES 2.3.x and Kibana 4.5.0
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
Re: [GitHub] incubator-metron issue #135: METRON-152: Add support for
ES 2.3.x and Kibana...
Posted by Nick Allen <ni...@nickallen.org>.
You are correct that the Kibana 4 interface does not currently have a
dashboard that gets created automatically like the older Kibana 3
dashboard. I am actually working on that now per METRON-219;
https://issues.apache.org/jira/browse/METRON-219.
The link below is a fairly good introduction to using Kibana 4. Perhaps
this will help you until we have a Metron dashboard with sensible defaults.
https://www.timroes.de/2015/02/07/kibana-4-tutorial-part-1-introduction/
On Tue, Jun 14, 2016 at 2:47 AM, lizhenmxcz <gi...@git.apache.org> wrote:
> Github user lizhenmxcz commented on the issue:
>
> https://github.com/apache/incubator-metron/pull/135
>
> but how to config the kibana4 to visualize the data in the
> elasticsearch like in kibana3
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastructure@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
--
Nick Allen <ni...@nickallen.org>
[GitHub] incubator-metron issue #135: METRON-152: Add support for ES 2.3.x and Kibana...
Posted by lizhenmxcz <gi...@git.apache.org>.
Github user lizhenmxcz commented on the issue:
https://github.com/apache/incubator-metron/pull/135
but how to config the kibana4 to visualize the data in the elasticsearch like in kibana3
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-152: Add support for ES 2.3....
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the pull request:
https://github.com/apache/incubator-metron/pull/135#issuecomment-222119162
Thanks for running it up @nickwallen. I was able to test out these changes in EC2 this morning and everything worked. I'm keen to hear what happens in your setup.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-152: Add support for ES 2.3....
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/135#issuecomment-222142620
Do you understand why it worked on Vagrant and broke on EC2? Is there anything outside of your PR that we can do to bring the platforms together so that it if works on one, it works on all?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-152: Add support for ES 2.3....
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the pull request:
https://github.com/apache/incubator-metron/pull/135#issuecomment-222145541
I have no idea why it failed. It looked similar to this Ansible issue: https://github.com/ansible/ansible-modules-core/issues/265.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-152: Add support for ES 2.3....
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/135#issuecomment-222208042
+1 Success. Golden
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-152: Add support for ES 2.3....
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/135#issuecomment-221739471
+1, spun up in single node vagrant and looks good!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-152: Add support for ES 2.3....
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/135#issuecomment-222179081
This is not necessarily a deal-breaker, but just something to call out.
We don't have backwards compatibility with this change. It would be nice to maintain backwards compatibility in this case because we don't quite yet have feature parity. We are missing pcap search and comparable out-of-the-box histograms in Kibana 4.
It would be achievable by renaming the existing roles to `elasticsearch1`/`kibana3` and add the new functionality as `elasticsearch2`/`kibana4`. I haven't looked yet, but we might have to untangle some of the platform code too. The effort on the platform side may not be worth it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-152: Add support for ES 2.3....
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/135#issuecomment-221885902
Ran into this error when deploying on Amazon-EC2. Anyone else seeing this?
TLDR
```
TypeError: sendall() argument 1 must be string or buffer, not dict
```
```
TASK [elasticsearch : Add Elasticsearch templates for topologies] **************
failed: [ec2-52-32-188-202.us-west-2.compute.amazonaws.com] => (item={u'sensor': u'bro', u'file': {'mappings': {'bro_doc': {'_timestamp': {'enabled': True}, 'properties': {'enrichments:geo:ip_dst_addr:location_point': {'type': 'geo_point'}, 'timestamp': {'type': 'date', 'format': 'epoch_millis'}}}}, 'template': 'bro_index*'}}) => {"failed": true, "item": {"file": {"mappings": {"bro_doc": {"_timestamp": {"enabled": true}, "properties": {"enrichments:geo:ip_dst_addr:location_point": {"type": "geo_point"}, "timestamp": {"format": "epoch_millis", "type": "date"}}}}, "template": "bro_index*"}, "sensor": "bro"}, "module_stderr": "", "module_stdout": "Traceback (most recent call last):\r\n File \"/home/centos/.ansible/tmp/ansible-tmp-1464272005.1-272260267672829/uri\", line 2464, in <module>\r\n main()\r\n File \"/home/centos/.ansible/tmp/ansible-tmp-1464272005.1-272260267672829/uri\", line 453, in main\r\n resp, content, dest = uri(module, url, dest, user, password, body, body_
format, method, dict_headers, redirects, socket_timeout, validate_certs)\r\n File \"/home/centos/.ansible/tmp/ansible-tmp-1464272005.1-272260267672829/uri\", line 340, in uri\r\n resp, content = h.request(url, method=method, body=body, headers=headers) \r\n File \"/usr/lib/python2.6/site-packages/httplib2/__init__.py\", line 1609, in request\r\n (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)\r\n File \"/usr/lib/python2.6/site-packages/httplib2/__init__.py\", line 1351, in _request\r\n (response, content) = self._conn_request(conn, request_uri, method, body, headers)\r\n File \"/usr/lib/python2.6/site-packages/httplib2/__init__.py\", line 1273, in _conn_request\r\n conn.request(method, request_uri, body, headers)\r\n File \"/usr/lib64/python2.6/httplib.py\", line 936, in request\r\n self._send_request(method, url, body, headers)\r\n File \"/usr/lib64/python2.6/httplib.py\", line 976, in _s
end_request\r\n self.send(body)\r\n File \"/usr/lib64/python2.6/httplib.py\", line 781, in send\r\n self.sock.sendall(str)\r\n File \"<string>\", line 1, in sendall\r\nTypeError: sendall() argument 1 must be string or buffer, not dict\r\n", "msg": "MODULE FAILURE", "parsed": false}
failed: [ec2-52-32-188-202.us-west-2.compute.amazonaws.com] => (item={u'sensor': u'yaf', u'file': {'mappings': {'yaf_doc': {'_timestamp': {'enabled': True}, 'properties': {'uflags': {'type': 'string'}, 'pkt': {'type': 'string'}, 'app': {'type': 'string'}, 'rtt': {'type': 'string'}, 'tag': {'type': 'string'}, 'duration': {'type': 'string'}, 'riflags': {'type': 'string'}, 'sip': {'type': 'string'}, 'proto': {'type': 'string'}, 'rtag': {'type': 'string'}, 'oct': {'type': 'string'}, 'risn': {'type': 'string'}, 'end-time': {'type': 'string'}, 'end-reason': {'type': 'string'}, 'timestamp': {'type': 'date', 'format': 'epoch_millis'}, 'dp': {'type': 'string'}, 'enrichments:geo:ip_dst_addr:location_point': {'type': 'geo_point'}, 'roct': {'type': 'string'}, 'sp': {'type': 'string'}, 'iflags': {'type': 'string'}, 'isn': {'type': 'string'}, 'ruflags': {'type': 'string'}, 'rpkt': {'type': 'string'}, 'dip': {'type': 'string'}}}}, 'template': 'yaf_index*'}}) => {"failed": true, "item": {"file":
{"mappings": {"yaf_doc": {"_timestamp": {"enabled": true}, "properties": {"app": {"type": "string"}, "dip": {"type": "string"}, "dp": {"type": "string"}, "duration": {"type": "string"}, "end-reason": {"type": "string"}, "end-time": {"type": "string"}, "enrichments:geo:ip_dst_addr:location_point": {"type": "geo_point"}, "iflags": {"type": "string"}, "isn": {"type": "string"}, "oct": {"type": "string"}, "pkt": {"type": "string"}, "proto": {"type": "string"}, "riflags": {"type": "string"}, "risn": {"type": "string"}, "roct": {"type": "string"}, "rpkt": {"type": "string"}, "rtag": {"type": "string"}, "rtt": {"type": "string"}, "ruflags": {"type": "string"}, "sip": {"type": "string"}, "sp": {"type": "string"}, "tag": {"type": "string"}, "timestamp": {"format": "epoch_millis", "type": "date"}, "uflags": {"type": "string"}}}}, "template": "yaf_index*"}, "sensor": "yaf"}, "module_stderr": "", "module_stdout": "Traceback (most recent call last):\r\n File \"/home/centos/.ansible/tmp/ansible
-tmp-1464272006.63-119978789749599/uri\", line 2464, in <module>\r\n main()\r\n File \"/home/centos/.ansible/tmp/ansible-tmp-1464272006.63-119978789749599/uri\", line 453, in main\r\n resp, content, dest = uri(module, url, dest, user, password, body, body_format, method, dict_headers, redirects, socket_timeout, validate_certs)\r\n File \"/home/centos/.ansible/tmp/ansible-tmp-1464272006.63-119978789749599/uri\", line 340, in uri\r\n resp, content = h.request(url, method=method, body=body, headers=headers) \r\n File \"/usr/lib/python2.6/site-packages/httplib2/__init__.py\", line 1609, in request\r\n (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)\r\n File \"/usr/lib/python2.6/site-packages/httplib2/__init__.py\", line 1351, in _request\r\n (response, content) = self._conn_request(conn, request_uri, method, body, headers)\r\n File \"/usr/lib/python2.6/site-packages/httplib2/__init__.py\", line
1273, in _conn_request\r\n conn.request(method, request_uri, body, headers)\r\n File \"/usr/lib64/python2.6/httplib.py\", line 936, in request\r\n self._send_request(method, url, body, headers)\r\n File \"/usr/lib64/python2.6/httplib.py\", line 976, in _send_request\r\n self.send(body)\r\n File \"/usr/lib64/python2.6/httplib.py\", line 781, in send\r\n self.sock.sendall(str)\r\n File \"<string>\", line 1, in sendall\r\nTypeError: sendall() argument 1 must be string or buffer, not dict\r\n", "msg": "MODULE FAILURE", "parsed": false}
failed: [ec2-52-32-188-202.us-west-2.compute.amazonaws.com] => (item={u'sensor': u'snort', u'file': {'mappings': {'snort_doc': {'_timestamp': {'enabled': True}, 'properties': {'enrichments:geo:ip_dst_addr:location_point': {'type': 'geo_point'}, 'timestamp': {'type': 'date', 'format': 'epoch_millis'}}}}, 'template': 'snort_index*'}}) => {"failed": true, "item": {"file": {"mappings": {"snort_doc": {"_timestamp": {"enabled": true}, "properties": {"enrichments:geo:ip_dst_addr:location_point": {"type": "geo_point"}, "timestamp": {"format": "epoch_millis", "type": "date"}}}}, "template": "snort_index*"}, "sensor": "snort"}, "module_stderr": "", "module_stdout": "Traceback (most recent call last):\r\n File \"/home/centos/.ansible/tmp/ansible-tmp-1464272007.91-28974849171520/uri\", line 2464, in <module>\r\n main()\r\n File \"/home/centos/.ansible/tmp/ansible-tmp-1464272007.91-28974849171520/uri\", line 453, in main\r\n resp, content, dest = uri(module, url, dest, user, password,
body, body_format, method, dict_headers, redirects, socket_timeout, validate_certs)\r\n File \"/home/centos/.ansible/tmp/ansible-tmp-1464272007.91-28974849171520/uri\", line 340, in uri\r\n resp, content = h.request(url, method=method, body=body, headers=headers) \r\n File \"/usr/lib/python2.6/site-packages/httplib2/__init__.py\", line 1609, in request\r\n (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)\r\n File \"/usr/lib/python2.6/site-packages/httplib2/__init__.py\", line 1351, in _request\r\n (response, content) = self._conn_request(conn, request_uri, method, body, headers)\r\n File \"/usr/lib/python2.6/site-packages/httplib2/__init__.py\", line 1273, in _conn_request\r\n conn.request(method, request_uri, body, headers)\r\n File \"/usr/lib64/python2.6/httplib.py\", line 936, in request\r\n self._send_request(method, url, body, headers)\r\n File \"/usr/lib64/python2.6/httplib.py\", lin
e 976, in _send_request\r\n self.send(body)\r\n File \"/usr/lib64/python2.6/httplib.py\", line 781, in send\r\n self.sock.sendall(str)\r\n File \"<string>\", line 1, in sendall\r\nTypeError: sendall() argument 1 must be string or buffer, not dict\r\n", "msg": "MODULE FAILURE", "parsed": false}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-152: Add support for ES 2.3....
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/135#discussion_r64924182
--- Diff: metron-platform/metron-elasticsearch/pom.xml ---
@@ -206,6 +220,16 @@
<goal>shade</goal>
</goals>
<configuration>
+ <relocations>
+ <relocation>
+ <pattern>com.google.common</pattern>
+ <shadedPattern>org.apache.metron.guava.metron-elasticsearch</shadedPattern>
+ </relocation>
--- End diff --
Are we shading the wrong jar here? Shouldn't this be `<pattern>com.google.guava</pattern>`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-152: Add support for ES 2.3....
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/135#issuecomment-222176562
I was running on an unsupported version of Ansible. (That is my fault. I should read the docs that I write.) After switching to 2.0.0.2 this problem does not occur. In the process of running up a fresh deploy as a final validation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-152: Add support for ES 2.3....
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/135
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---