You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "Michael Miklavcic (JIRA)" <ji...@apache.org> on 2018/03/02 18:02:00 UTC
[jira] [Commented] (METRON-1466) The Elastic Search index are not
getting generated in the latest metron deployments for all the topologies
[ https://issues.apache.org/jira/browse/METRON-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383894#comment-16383894 ]
Michael Miklavcic commented on METRON-1466:
-------------------------------------------
Can you provide/paste the index listing for ES via the links in Ambari?
Also, can you check the Ambari service install logs to see if there are any errors there? If you can upload that as well that would be helpful.
> The Elastic Search index are not getting generated in the latest metron deployments for all the topologies
> ----------------------------------------------------------------------------------------------------------
>
> Key: METRON-1466
> URL: https://issues.apache.org/jira/browse/METRON-1466
> Project: Metron
> Issue Type: Bug
> Environment: Metron secured cluster
> Reporter: Mohan
> Priority: Major
> Attachments: bro_worker.log, enrichmnts_worker.log, indexing_worker.log
>
>
> The Elastic Search index are not getting generated in the latest metron deployments for all the topologies ,
> I see that the data is getting published to the respective topics and also getting enriched .
> below are the console consumer output for topics "bro" and "enrichments" I dont get anything for "indexing" topic .
> {code:java}
> ./kafka-console-consumer.sh --zookeeper ctr-e138-1518143905142-61778-01-000002.hwx.site:2181 --security-protocol PLAINTEXTSASL --topic bro {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}} {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}} {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}} {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}} {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}} ./kafka-console-consumer.sh --zookeeper ctr-e138-1518143905142-61778-01-000002.hwx.site:2181 --security-protocol PLAINTEXTSASL --topic enrichments {"bro_timestamp":"1469686182.899762","status_code":200,"method":"GET","ip_dst_port":80,"request_body_len":0,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"source.type":"bro","uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"protocol":"http","original_string":"HTTP | id.orig_p:49210 status_code:200 method:GET request_body_len:0 id.resp_p:80 uri:\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764 tags:[] uid:CRDObQRKAmoHCQq1a referrer:http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg resp_mime_types:[\"image\\\/png\"] trans_depth:3 host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158 response_body_len:1823 user_agent:Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0) ts:1469686182.899762 id.resp_h:13.163.121.204 resp_fuids:[\"FQbZZfax5TLVX6M42\"]","ip_dst_addr":"13.163.121.204","ip_src_port":49210,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","guid":"897e534c-ed6b-458c-9b2d-8749a68ed447","response_body_len":1823,"ip_src_addr":"12.172.138.158","user_agent":"Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","resp_fuids":["FQbZZfax5TLVX6M42"],"timestamp":1469686182899} {"bro_timestamp":"1469686182.899762","status_code":200,"method":"GET","ip_dst_port":80,"request_body_len":0,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"source.type":"bro","uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"protocol":"http","original_string":"HTTP | id.orig_p:49210 status_code:200 method:GET request_body_len:0 id.resp_p:80 uri:\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764 tags:[] uid:CRDObQRKAmoHCQq1a referrer:http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg resp_mime_types:[\"image\\\/png\"] trans_depth:3 host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158 response_body_len:1823 user_agent:Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0) ts:1469686182.899762 id.resp_h:13.163.121.204 resp_fuids:[\"FQbZZfax5TLVX6M42\"]","ip_dst_addr":"13.163.121.204","ip_src_port":49210,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","guid":"6caaab54-e78d-4d31-9b2c-5c2bb25d609f","response_body_len":1823,"ip_src_addr":"12.172.138.158","user_agent":"Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","resp_fuids":["FQbZZfax5TLVX6M42"],"timestamp":1469686182899} {"bro_timestamp":"1469686182.899762","status_code":200,"method":"GET","ip_dst_port":80,"request_body_len":0,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"source.type":"bro","uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"protocol":"http","original_string":"HTTP | id.orig_p:49210 status_code:200 method:GET request_body_len:0 id.resp_p:80 uri:\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764 tags:[] uid:CRDObQRKAmoHCQq1a referrer:http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg resp_mime_types:[\"image\\\/png\"] trans_depth:3 host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158 response_body_len:1823 user_agent:Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0) ts:1469686182.899762 id.resp_h:13.163.121.204 resp_fuids:[\"FQbZZfax5TLVX6M42\"]","ip_dst_addr":"13.163.121.204","ip_src_port":49210,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","guid":"a7475614-789a-4221-88cb-0734ef5fdc0a","response_body_len":1823,"ip_src_addr":"12.172.138.158","user_agent":"Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","resp_fuids":["FQbZZfax5TLVX6M42"],"timestamp":1469686182899} {"bro_timestamp":"1469686182.899762","status_code":200,"method":"GET","ip_dst_port":80,"request_body_len":0,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"source.type":"bro","uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"protocol":"http","original_string":"HTTP | id.orig_p:49210 status_code:200 method:GET request_body_len:0 id.resp_p:80 uri:\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764 tags:[] uid:CRDObQRKAmoHCQq1a referrer:http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg resp_mime_types:[\"image\\\/png\"] trans_depth:3 host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158 response_body_len:1823 user_agent:Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0) ts:1469686182.899762 id.resp_h:13.163.121.204 resp_fuids:[\"FQbZZfax5TLVX6M42\"]","ip_dst_addr":"13.163.121.204","ip_src_port":49210,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","guid":"f05fa5de-5549-4293-85cf-006c66dba5d4","response_body_len":1823,"ip_src_addr":"12.172.138.158","user_agent":"Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","resp_fuids":["FQbZZfax5TLVX6M42"],"timestamp":1469686182899} {"bro_timestamp":"1469686182.899762","status_code":200,"method":"GET","ip_dst_port":80,"request_body_len":0,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"source.type":"bro","uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"protocol":"http","original_string":"HTTP | id.orig_p:49210 status_code:200 method:GET request_body_len:0 id.resp_p:80 uri:\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764 tags:[] uid:CRDObQRKAmoHCQq1a referrer:http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg resp_mime_types:[\"image\\\/png\"] trans_depth:3 host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158 response_body_len:1823 user_agent:Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0) ts:1469686182.899762 id.resp_h:13.163.121.204 resp_fuids:[\"FQbZZfax5TLVX6M42\"]","ip_dst_addr":"13.163.121.204","ip_src_port":49210,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","guid":"116cb86a-2f41-48a8-8cc0-ee15d25d8db6","response_body_len":1823,"ip_src_addr":"12.172.138.158","user_agent":"Mozilla\/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","resp_fuids":["FQbZZfax5TLVX6M42"],"timestamp":1469686182899}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)