You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Dima Kovalyov <Di...@sstech.us> on 2017/07/12 12:39:24 UTC
metron profiler hbase storage issue
Hello Team,
We have installed metron profiler with the help of this link:
https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
Attached my profiler.properties and profiler.json files, remote.yaml and
worker.log files in metron-artifacts.zip.
Once I place the following message into kafka topic 'indexing':
> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
It's picked by splitterBolt and passed till HBaseBolt. But data is not
getting stored to Hbase.
Attached storm-ui screenshot.storm-ui-screen.png
Can you please advise if am I missing something here?
Thanks in advance.
p.s. also posted it here:
https://community.hortonworks.com/questions/110486/metron-profiler-hbasebolt-not-storing-data-to-hbas.html
- Dima
Re: metron profiler hbase storage issue
Posted by Nick Allen <ni...@nickallen.org>.
Have you been able to follow these instructions successfully? [1] I would
start there and follow them exactly. Only then try and create your own
profiles.
I would suggest you include settings in main email body rather than an
attachment.
[1]
http://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html#Getting_Started
On Wed, Jul 12, 2017 at 8:39 AM, Dima Kovalyov <Di...@sstech.us>
wrote:
> Hello Team,
>
> We have installed metron profiler with the help of this link:
> https://metron.apache.org/current-book/metron-analytics/
> metron-profiler/index.html
>
> Attached my profiler.properties and profiler.json files, remote.yaml and
> worker.log files in metron-artifacts.zip.
>
> Once I place the following message into kafka topic 'indexing':
> > [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"
> 10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","
> protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_
> addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
> It's picked by splitterBolt and passed till HBaseBolt. But data is not
> getting stored to Hbase.
> Attached storm-ui screenshot.storm-ui-screen.png
>
> Can you please advise if am I missing something here?
> Thanks in advance.
>
> p.s. also posted it here:
> https://community.hortonworks.com/questions/110486/metron-
> profiler-hbasebolt-not-storing-data-to-hbas.html
>
> - Dima
>
Re: metron profiler hbase storage issue
Posted by Nick Allen <ni...@nickallen.org>.
The 'expire' field is actually in days not milliseconds. So that should
not be a problem here.
On Wed, Jul 12, 2017 at 8:50 AM, Simon Elliston Ball <
simon@simonellistonball.com> wrote:
> As I answered earlier today, your profile in this zip file is set to
> expire data after 30 milliseconds, so you will not be writing any data.
>
>
> > On 12 Jul 2017, at 13:40, Dima Kovalyov <Di...@sstech.us> wrote:
> >
> > Attached artifacts.
> >
> > On 07/12/2017 07:39 PM, Dima Kovalyov wrote:
> >> Hello Team,
> >>
> >> We have installed metron profiler with the help of this link:
> >> https://metron.apache.org/current-book/metron-analytics/
> metron-profiler/index.html
> >>
> >> Attached my profiler.properties and profiler.json files, remote.yaml and
> >> worker.log files in metron-artifacts.zip.
> >>
> >> Once I place the following message into kafka topic 'indexing':
> >>> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"
> 10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","
> protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_
> addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
> >> It's picked by splitterBolt and passed till HBaseBolt. But data is not
> >> getting stored to Hbase.
> >> Attached storm-ui screenshot.storm-ui-screen.png
> >>
> >> Can you please advise if am I missing something here?
> >> Thanks in advance.
> >>
> >> p.s. also posted it here:
> >> https://community.hortonworks.com/questions/110486/metron-
> profiler-hbasebolt-not-storing-data-to-hbas.html
> >>
> >> - Dima
> >
> > <metron-artifacts.zip><storm-ui-screen.png>
>
>
Re: metron profiler hbase storage issue
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
As I answered earlier today, your profile in this zip file is set to expire data after 30 milliseconds, so you will not be writing any data.
> On 12 Jul 2017, at 13:40, Dima Kovalyov <Di...@sstech.us> wrote:
>
> Attached artifacts.
>
> On 07/12/2017 07:39 PM, Dima Kovalyov wrote:
>> Hello Team,
>>
>> We have installed metron profiler with the help of this link:
>> https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
>>
>> Attached my profiler.properties and profiler.json files, remote.yaml and
>> worker.log files in metron-artifacts.zip.
>>
>> Once I place the following message into kafka topic 'indexing':
>>> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
>> It's picked by splitterBolt and passed till HBaseBolt. But data is not
>> getting stored to Hbase.
>> Attached storm-ui screenshot.storm-ui-screen.png
>>
>> Can you please advise if am I missing something here?
>> Thanks in advance.
>>
>> p.s. also posted it here:
>> https://community.hortonworks.com/questions/110486/metron-profiler-hbasebolt-not-storing-data-to-hbas.html
>>
>> - Dima
>
> <metron-artifacts.zip><storm-ui-screen.png>
Re: metron profiler hbase storage issue
Posted by Dima Kovalyov <Di...@sstech.us>.
Attached artifacts.
On 07/12/2017 07:39 PM, Dima Kovalyov wrote:
> Hello Team,
>
> We have installed metron profiler with the help of this link:
> https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
>
> Attached my profiler.properties and profiler.json files, remote.yaml and
> worker.log files in metron-artifacts.zip.
>
> Once I place the following message into kafka topic 'indexing':
>> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
> It's picked by splitterBolt and passed till HBaseBolt. But data is not
> getting stored to Hbase.
> Attached storm-ui screenshot.storm-ui-screen.png
>
> Can you please advise if am I missing something here?
> Thanks in advance.
>
> p.s. also posted it here:
> https://community.hortonworks.com/questions/110486/metron-profiler-hbasebolt-not-storing-data-to-hbas.html
>
> - Dima