You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by tkg_cangkul <yu...@gmail.com> on 2017/03/22 15:38:45 UTC
Re: snort topology doesn't emitted automatically
anyone can help me to solved this?
On 22/03/17 15:24, tkg_cangkul wrote:
> hi, i've try to using snort as a sensor on metron in my ambari cluster.
> now i've a problem. the snort topology doesn't emitted the data
> automatically.
> i must send the messages to kafka manually to emitted the data.
>
> /cat /var/log/snort/alert.csv | bin/kafka-console-producer.sh
> --broker-list localhost:6667 --topic snort/
>
> any suggest about this?
Re: snort topology doesn't emitted automatically
Posted by Michael Miklavcic <mi...@gmail.com>.
Can you provide the following to us for some added context?
Hadoop version info, e.g.
# hadoop version
Hadoop 2.7.3.2.5.3.0-37
Subversion git@github.com:hortonworks/hadoop.git -r
9828acfdec41a121f0121f556b09e2d112259e92
Compiled by jenkins on 2016-11-29T18:06Z
Compiled with protoc 2.5.0
From source with checksum a6c8c64bfe734b01a46c7c480ebc7ff
This command was run using
/usr/hdp/2.5.3.0-37/hadoop/hadoop-common-2.7.3.2.5.3.0-37.jar
(Below are examples from some Kerberos testing I was doing earlier, so
yours will look a bit different.)
Log into zookeeper from the CLI with your zk host
<hadoop home>/zookeeper-client/bin/zkCli.sh -server node1:2181
Print the broker info, e.g.
[zk: node1:2181(CONNECTED) 3] get /brokers/ids/1001
{"jmx_port":-1,"timestamp":"1490275583997","endpoints":["PLAINTEXTSASL://node1:6667"],"host":null,"version":3,"port":-1}
cZxid = 0x9021
ctime = Thu Mar 23 13:26:23 UTC 2017
mZxid = 0x9021
mtime = Thu Mar 23 13:26:23 UTC 2017
pZxid = 0x9021
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x15af754455a0bae
dataLength = 120
numChildren = 0
Show advertised listener info
# cat <hadoop home>/kafka-broker/config/server.properties |grep
advertised.listeners
advertised.listeners=PLAINTEXTSASL://node1:6667
Explanation: advertised.listeners = Listeners to publish to ZooKeeper for
clients to use, if different than the listeners above. In IaaS
environments, this may need to be different from the interface to which the
broker binds. If this is not set, the value for `listeners` will be used.
Thanks,
Michael Miklavcic
On Wed, Mar 22, 2017 at 8:29 PM, tkg_cangkul <yu...@gmail.com> wrote:
> yeah my kafka broker running on that machine. is this the cause why the
> snort topology automatically emitted the data?
> when i try with *cat *command it's work normally. that error only appear
> when i use *tail -F *command.
>
>
>
> On 23/03/17 01:55, Casey Stella wrote:
>
> Also, try --broker-list 127.0.0.1:6667
>
> On Wed, Mar 22, 2017 at 2:53 PM, Casey Stella <ce...@gmail.com> wrote:
>
>> Is your kafka broker running and is it on localhost:6667?
>>
>> On Wed, Mar 22, 2017 at 2:10 PM, tkg_cangkul <yu...@gmail.com>
>> wrote:
>>
>>> sorry but there i didn't found that directory on my cluster.
>>> for your information, i'm using metron 0.3.0 now.
>>>
>>> when i try to run this command manually :
>>>
>>> tail -F /var/log/snort/alert.csv | /usr/yava/2.2.0.5/kafka/bin/ka
>>> fka-console-producer.sh --broker-list localhost:6667 --topic snort
>>>
>>> i've got an error message like this :
>>>
>>> *[2017-03-23 01:03:49,998] ERROR Error when sending message to topic
>>> snort with key: null, value: 214 bytes with error:
>>> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)*
>>> *org.apache.kafka.common.errors.TimeoutException: Failed to update
>>> metadata after 60000 ms*
>>>
>>>
>>>
>>>
>>> On 23/03/17 00:59, Otto Fowler wrote:
>>>
>>> /opt/snort-producer/start-snort-producer.sh
>>>
>>>
>>>
>>> On March 22, 2017 at 13:30:36, tkg_cangkul (yuza.rasfar@gmail.com)
>>> wrote:
>>>
>>> start_snort_producer.sh
>>>
>>>
>>>
>>
>
>
Re: snort topology doesn't emitted automatically
Posted by tkg_cangkul <yu...@gmail.com>.
yeah my kafka broker running on that machine. is this the cause why the
snort topology automatically emitted the data?
when i try with *cat *command it's work normally. that error only appear
when i use *tail -F *command.
On 23/03/17 01:55, Casey Stella wrote:
> Also, try --broker-list 127.0.0.1:6667 <http://127.0.0.1:6667>
>
> On Wed, Mar 22, 2017 at 2:53 PM, Casey Stella <cestella@gmail.com
> <ma...@gmail.com>> wrote:
>
> Is your kafka broker running and is it on localhost:6667?
>
> On Wed, Mar 22, 2017 at 2:10 PM, tkg_cangkul
> <yuza.rasfar@gmail.com <ma...@gmail.com>> wrote:
>
> sorry but there i didn't found that directory on my cluster.
> for your information, i'm using metron 0.3.0 now.
>
> when i try to run this command manually :
>
> tail -F /var/log/snort/alert.csv |
> /usr/yava/2.2.0.5/kafka/bin/kafka-console-producer.sh
> <http://2.2.0.5/kafka/bin/kafka-console-producer.sh>
> --broker-list localhost:6667 --topic snort
>
> i've got an error message like this :
>
> *[2017-03-23 01:03:49,998] ERROR Error when sending message to
> topic snort with key: null, value: 214 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)**
> **org.apache.kafka.common.errors.TimeoutException: Failed to
> update metadata after 60000 ms*
>
>
>
>
> On 23/03/17 00:59, Otto Fowler wrote:
>> /opt/snort-producer/start-snort-producer.sh
>>
>>
>> On March 22, 2017 at 13:30:36, tkg_cangkul
>> (yuza.rasfar@gmail.com <ma...@gmail.com>) wrote:
>>
>>> start_snort_producer.sh
>
>
>
Re: snort topology doesn't emitted automatically
Posted by Casey Stella <ce...@gmail.com>.
Also, try --broker-list 127.0.0.1:6667
On Wed, Mar 22, 2017 at 2:53 PM, Casey Stella <ce...@gmail.com> wrote:
> Is your kafka broker running and is it on localhost:6667?
>
> On Wed, Mar 22, 2017 at 2:10 PM, tkg_cangkul <yu...@gmail.com>
> wrote:
>
>> sorry but there i didn't found that directory on my cluster.
>> for your information, i'm using metron 0.3.0 now.
>>
>> when i try to run this command manually :
>>
>> tail -F /var/log/snort/alert.csv | /usr/yava/2.2.0.5/kafka/bin/ka
>> fka-console-producer.sh --broker-list localhost:6667 --topic snort
>>
>> i've got an error message like this :
>>
>> *[2017-03-23 01:03:49,998] ERROR Error when sending message to topic
>> snort with key: null, value: 214 bytes with error:
>> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)*
>> *org.apache.kafka.common.errors.TimeoutException: Failed to update
>> metadata after 60000 ms*
>>
>>
>>
>>
>> On 23/03/17 00:59, Otto Fowler wrote:
>>
>> /opt/snort-producer/start-snort-producer.sh
>>
>>
>>
>> On March 22, 2017 at 13:30:36, tkg_cangkul (yuza.rasfar@gmail.com) wrote:
>>
>> start_snort_producer.sh
>>
>>
>>
>
Re: snort topology doesn't emitted automatically
Posted by Casey Stella <ce...@gmail.com>.
Is your kafka broker running and is it on localhost:6667?
On Wed, Mar 22, 2017 at 2:10 PM, tkg_cangkul <yu...@gmail.com> wrote:
> sorry but there i didn't found that directory on my cluster.
> for your information, i'm using metron 0.3.0 now.
>
> when i try to run this command manually :
>
> tail -F /var/log/snort/alert.csv | /usr/yava/2.2.0.5/kafka/bin/
> kafka-console-producer.sh --broker-list localhost:6667 --topic snort
>
> i've got an error message like this :
>
> *[2017-03-23 01:03:49,998] ERROR Error when sending message to topic snort
> with key: null, value: 214 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)*
> *org.apache.kafka.common.errors.TimeoutException: Failed to update
> metadata after 60000 ms*
>
>
>
>
> On 23/03/17 00:59, Otto Fowler wrote:
>
> /opt/snort-producer/start-snort-producer.sh
>
>
>
> On March 22, 2017 at 13:30:36, tkg_cangkul (yuza.rasfar@gmail.com) wrote:
>
> start_snort_producer.sh
>
>
>
Re: snort topology doesn't emitted automatically
Posted by tkg_cangkul <yu...@gmail.com>.
sorry but there i didn't found that directory on my cluster.
for your information, i'm using metron 0.3.0 now.
when i try to run this command manually :
tail -F /var/log/snort/alert.csv |
/usr/yava/2.2.0.5/kafka/bin/kafka-console-producer.sh --broker-list
localhost:6667 --topic snort
i've got an error message like this :
*[2017-03-23 01:03:49,998] ERROR Error when sending message to topic
snort with key: null, value: 214 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)**
**org.apache.kafka.common.errors.TimeoutException: Failed to update
metadata after 60000 ms*
On 23/03/17 00:59, Otto Fowler wrote:
> /opt/snort-producer/start-snort-producer.sh
>
>
> On March 22, 2017 at 13:30:36, tkg_cangkul (yuza.rasfar@gmail.com
> <ma...@gmail.com>) wrote:
>
>> start_snort_producer.sh
Re: snort topology doesn't emitted automatically
Posted by Otto Fowler <ot...@gmail.com>.
/opt/snort-producer/start-snort-producer.sh
On March 22, 2017 at 13:30:36, tkg_cangkul (yuza.rasfar@gmail.com) wrote:
start_snort_producer.sh
Re: snort topology doesn't emitted automatically
Posted by tkg_cangkul <yu...@gmail.com>.
where i can find the start_snort_producer.sh script? i didn't see it
inside my metron_home dir
On 22/03/17 23:54, Otto Fowler wrote:
> One time, I saw an issue where the flume agent did not have the
> correct rights
> to access the csv, so died a horrible death.
>
> We don\u2019t use flume any longer however. I would want to take a look at
> the log files
> for what is reading the snort csv.
>
> I believe the start_snort_producer.sh script is used now. I am not
> sure about the logs,
> but maybe you can try to run that manually and see the output?
>
>
> On March 22, 2017 at 11:38:53, tkg_cangkul (yuza.rasfar@gmail.com
> <ma...@gmail.com>) wrote:
>
>> anyone can help me to solved this?
>>
>> On 22/03/17 15:24, tkg_cangkul wrote:
>>> hi, i've try to using snort as a sensor on metron in my ambari cluster.
>>> now i've a problem. the snort topology doesn't emitted the data
>>> automatically.
>>> i must send the messages to kafka manually to emitted the data.
>>>
>>> /cat /var/log/snort/alert.csv | bin/kafka-console-producer.sh
>>> --broker-list localhost:6667 --topic snort/
>>>
>>> any suggest about this?
>>
Re: snort topology doesn't emitted automatically
Posted by Otto Fowler <ot...@gmail.com>.
One time, I saw an issue where the flume agent did not have the correct
rights
to access the csv, so died a horrible death.
We don’t use flume any longer however. I would want to take a look at the
log files
for what is reading the snort csv.
I believe the start_snort_producer.sh script is used now. I am not sure
about the logs,
but maybe you can try to run that manually and see the output?
On March 22, 2017 at 11:38:53, tkg_cangkul (yuza.rasfar@gmail.com) wrote:
anyone can help me to solved this?
On 22/03/17 15:24, tkg_cangkul wrote:
hi, i've try to using snort as a sensor on metron in my ambari cluster.
now i've a problem. the snort topology doesn't emitted the data
automatically.
i must send the messages to kafka manually to emitted the data.
*cat /var/log/snort/alert.csv | bin/kafka-console-producer.sh --broker-list
localhost:6667 --topic snort*
any suggest about this?