You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Krishna Dhanekula <Kr...@sstech.us> on 2017/07/12 05:17:18 UTC
Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
I have an problem where profiled data is not storing to Hbase instance:
Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
Attached my profiler.properties and profiler.json files, remote.yaml and worker.log files.
Once I place the following message into kafka topic 'indexing'
[{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
It's picked by splitterBolt and passed till HBaseBolt. But data is not getting stored to Hbase.
Attached metron-artifacts.zip
Attached storm-ui screenshot.storm-ui-screen.png
Please suggest what am I missing here.
Thanks in advance.
Attachments:
storm-ui-screen.png (17.4 kB)
metron-artifacts.zip (36.6 kB)
Regards,
Balakrishna
RE: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
Instance
Posted by Krishna Dhanekula <Kr...@sstech.us>.
Here’s the screenshot with the profiler content and the PUSH command output.
[cid:image001.png@01D2FBAF.F0EA4820]
Regards,
Balakrishna Dhanekula
[http://www.sstech.org/images/logo.jpg]<http://server1.streamsend.com/streamsend/clicktracker.php?cd=2904&ld=12&md=256&ud=79d7ecb7c00913b6e28eb06831bb0553&url=http://www.sstech.us>
Plot No. 16, Sector-III, HUDA Techno Enclave,
Opp. K. Raheja IT Park, Madhapur,
Hyderabad - 500 081, India.
Mobile : +91-9000334259
E-mail: krishna.dhanekula@sstech.us<ma...@sstech.us>
"Certified Minority Business Enterprise (MBE)"
From: Nick Allen [mailto:nick@nickallen.org]
Sent: Thursday, July 13, 2017 6:36 AM
To: user@metron.apache.org
Subject: RE: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
That's your problem then. You are not getting your profile definitions into Zookeeper.
Send us a copy of your terminal session showing the location and content of your profiler.json along with the command and ouput of you running the upload script.
On Jul 12, 2017 8:41 PM, "Krishna Dhanekula" <Kr...@sstech.us>> wrote:
When I run DUMP I can see everything else under /config/zookeeper except the profile (profiler.json) created by me.
________________________________
From: Nick Allen [nick@nickallen.org<ma...@nickallen.org>]
Sent: Wednesday, July 12, 2017 2:44 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
If the Profiler definitions were uploaded to Zookeeper correctly, you should be able to see them when using the DUMP option. See example below.
[root@node1 0.4.0]# bin/zk_load_configs.sh -m DUMP -z node1:2181
...
PROFILER Config: profiler
{
"profiles": [
{
"profile": "test",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
]
}
On Wed, Jul 12, 2017 at 2:26 PM, Krishna Dhanekula <Kr...@sstech.us>> wrote:
When I run this command
bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z krishna-3:2181
(Krishna-3 is my node1 name)
[root@krishna-4 0.3.1]# bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z krishna-3:2181
log4j:WARN No appenders could be found for logger (org.apache.curator.framework.imps.CuratorFrameworkImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
I don’t see the configuration files getting uploaded to zookeeper. Should I add out directory in this command.
Regards,
Balakrishna
From: Nick Allen [mailto:nick@nickallen.org<ma...@nickallen.org>]
Sent: Wednesday, July 12, 2017 8:15 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
> Under section ‘Creating Profiles’ it’s mentioned as milli secs. In Example1 it’s mentioned as 30 days.
Yes, there is an inconsistency in the README. The sentence in the table is wrong, the rest of the docs are correct.
Wrong: https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#creating-profiles
expires<https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires>
Optional
Profile data is purged after this period of time, specified in m̶i̶l̶l̶i̶s̶e̶c̶o̶n̶d̶s̶ [days]
Correct: https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires
> Now I removed the entire field and from ‘profiler.json’ so that it profiled data never expires. I pushed and configuration and tested from kafka. I still don’t see the data storing to Hbase.
Did you follow the Getting Started guide? Can you follow that exactly and let me know if that works for you.
On Wed, Jul 12, 2017 at 9:31 AM, Krishna Dhanekula <Kr...@sstech.us>> wrote:
Under section ‘Creating Profiles’ it’s mentioned as milli secs. In Example1 it’s mentioned as 30 days.
Now I removed the entire field and from ‘profiler.json’ so that it profiled data never expires. I pushed and configuration and tested from kafka. I still don’t see the data storing to Hbase.
Regards,
Balakrishna
From: Nick Allen [mailto:nick@nickallen.org<ma...@nickallen.org>]
Sent: Wednesday, July 12, 2017 6:55 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Quick correction for posterity... The 'expire' field is actually in days not milliseconds. So that should not be a problem here.
On Wed, Jul 12, 2017 at 8:02 AM, Simon Elliston Ball <si...@simonellistonball.com>> wrote:
In your config for the individual profile though, you expire the content every 30ms (per the zip file)
Simon
On 12 Jul 2017, at 12:20, Krishna Dhanekula <Kr...@sstech.us>> wrote:
Thanks Simon for replying.
This is my ‘profiler.properties’ file content. TTL is 30 minutes and for every 30 seconds I am flushing to Hbase.
profiler.workers=1
profiler.executors=0
profiler.input.topic=indexing
profiler.period.duration=1
profiler.period.duration.units=MINUTES
profiler.ttl=30
profiler.ttl.units=MINUTES
profiler.hbase.salt.divisor=1000
profiler.hbase.table=profiler
profiler.hbase.column.family=P
profiler.hbase.batch=1
profiler.hbase.flush.interval.seconds=30
##### Kafka #####
kafka.zk=10.10.110.184:2181<http://10.10.110.184:2181>
kafka.broker=10.10.110.184:6667<http://10.10.110.184:6667>
kafka.start=WHERE_I_LEFT_OFF
Regards,
Balakrishna
From: Simon Elliston Ball [mailto:simon@simonellistonball.com]
Sent: Wednesday, July 12, 2017 3:28 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Looks like you’ve set the profile to purge (expires) every 30 ms, and your period is set to 30 minutes, so the data is being expired long before it has a change to write.
Simon
On 12 Jul 2017, at 06:17, Krishna Dhanekula <Kr...@sstech.us>> wrote:
I have an problem where profiled data is not storing to Hbase instance:
Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
Attached my profiler.properties and profiler.json files, remote.yaml and worker.log files.
Once I place the following message into kafka topic 'indexing'
[{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
It's picked by splitterBolt and passed till HBaseBolt. But data is not getting stored to Hbase.
Attached metron-artifacts.zip
Attached storm-ui screenshot.storm-ui-screen.png
Please suggest what am I missing here.
Thanks in advance.
Attachments:
storm-ui-screen.png (17.4 kB)
metron-artifacts.zip (36.6 kB)
Regards,
Balakrishna
<metron_artifacts.zip><storm_ui_screen.png>
Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Posted by Nick Allen <ni...@nickallen.org>.
Bala - I am trying to replicate the problem that you are seeing, but I have
not been able to.
I have also opened a PR to help better document the installation process.
Can you review the steps in this PR and make sure that you haven't
accidentally missed anything? Also, do let me know if any of the steps do
not make sense.
https://github.com/nickwallen/metron/blob/75c3c88a56d817fc2791302c539c3ca354484b95/metron-analytics/metron-profiler/README.md#installation
https://github.com/apache/metron/pull/653
On Thu, Jul 13, 2017 at 4:46 AM, Krishna Dhanekula <
Krishna.Dhanekula@sstech.us> wrote:
> Resending the data as a text.
>
>
>
> profiler content and the PUSH command output.
>
>
>
> [root@krishna-4 zookeeper]# pwd
>
> /usr/metron/0.3.1/config/zookeeper
>
> [root@krishna-4 zookeeper]# ls -lrt
>
> total 12
>
> drwxr-xr-x. 2 root root 4096 Apr 28 10:16 parsers
>
> drwxr-xr-x. 2 root root 89 Apr 28 10:16 enrichments
>
> drwxr-xr-x. 2 root root 89 Apr 28 10:17 indexing
>
> -rw-r--r--. 1 metron root 108 Jul 12 06:55 global.json
>
> -rw-r--r--. 1 metron hadoop 196 Jul 13 03:30 profiler.json
>
> [root@krishna-4 zookeeper]# cat profiler.json
>
> {
>
> "profiles": [
>
> {
>
> "profile": "test",
>
> "foreach": "ip_src_addr",
>
> "init": { "count": "0" },
>
> "update": { "count": "count + 1" },
>
> "result": "count"
>
> }
>
> ]
>
> }
>
> [root@krishna-4 zookeeper]# cd ../..
>
> [root@krishna-4 0.3.1]# bin/zk_load_configs.sh -m PUSH -i
> config/zookeeper/ -z krishna-3:2181
>
> log4j:WARN No appenders could be found for logger
> (org.apache.curator.framework.imps.CuratorFrameworkImpl).
>
> log4j:WARN Please initialize the log4j system properly.
>
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>
> [root@krishna-4 0.3.1]#
>
>
>
>
>
> *Regards,*
>
>
>
> Bala
>
> *From:* Nick Allen [mailto:nick@nickallen.org]
> *Sent:* Thursday, July 13, 2017 6:36 AM
> *To:* user@metron.apache.org
> *Subject:* RE: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> That's your problem then. You are not getting your profile definitions
> into Zookeeper.
>
>
>
> Send us a copy of your terminal session showing the location and content
> of your profiler.json along with the command and ouput of you running the
> upload script.
>
>
>
>
>
> On Jul 12, 2017 8:41 PM, "Krishna Dhanekula" <Kr...@sstech.us>
> wrote:
>
> When I run DUMP I can see everything else under /config/zookeeper except
> the profile (profiler.json) created by me.
>
>
> ------------------------------
>
> *From:* Nick Allen [nick@nickallen.org]
> *Sent:* Wednesday, July 12, 2017 2:44 PM
>
>
> *To:* user@metron.apache.org
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> If the Profiler definitions were uploaded to Zookeeper correctly, you
> should be able to see them when using the DUMP option. See example below.
>
>
>
>
>
> [root@node1 0.4.0]# bin/zk_load_configs.sh -m DUMP -z node1:2181
>
> ...
>
>
>
> PROFILER Config: profiler
>
> {
>
> "profiles": [
>
> {
>
> "profile": "test",
>
> "foreach": "ip_src_addr",
>
> "init": { "count": "0" },
>
> "update": { "count": "count + 1" },
>
> "result": "count"
>
> }
>
> ]
>
> }
>
>
>
>
>
> On Wed, Jul 12, 2017 at 2:26 PM, Krishna Dhanekula <
> Krishna.Dhanekula@sstech.us> wrote:
>
> When I run this command
>
>
>
> bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z krishna-3:2181
>
> (Krishna-3 is my node1 name)
>
>
>
> [root@krishna-4 0.3.1]# bin/zk_load_configs.sh -m PUSH -i
> config/zookeeper/ -z krishna-3:2181
>
> log4j:WARN No appenders could be found for logger
> (org.apache.curator.framework.imps.CuratorFrameworkImpl).
>
> log4j:WARN Please initialize the log4j system properly.
>
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>
>
>
> I don’t see the configuration files getting uploaded to zookeeper. Should
> I add out directory in this command.
>
>
>
>
>
>
>
> *Regards,*
>
>
>
> *Balakrishna*
>
>
>
> *From:* Nick Allen [mailto:nick@nickallen.org]
> *Sent:* Wednesday, July 12, 2017 8:15 PM
>
>
> *To:* user@metron.apache.org
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> > Under section ‘Creating Profiles’ it’s mentioned as milli secs. In
> Example1 it’s mentioned as 30 days.
>
>
>
> Yes, there is an inconsistency in the README. The sentence in the table
> is wrong, the rest of the docs are correct.
>
>
>
> Wrong: https://github.com/apache/metron/tree/master/
> metron-analytics/metron-profiler#creating-profiles
>
>
>
> expires
> <https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires>
>
> Optional
>
> Profile data is purged after this period of time, specified
> in m̶i̶l̶l̶i̶s̶e̶c̶o̶n̶d̶s̶ [days]
>
> Correct: https://github.com/apache/metron/tree/master/
> metron-analytics/metron-profiler#expires
>
>
>
>
>
> > Now I removed the entire field and from ‘profiler.json’ so that it
> profiled data never expires. I pushed and configuration and tested from
> kafka. I still don’t see the data storing to Hbase.
>
>
>
> Did you follow the Getting Started guide? Can you follow that exactly and
> let me know if that works for you.
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Jul 12, 2017 at 9:31 AM, Krishna Dhanekula <
> Krishna.Dhanekula@sstech.us> wrote:
>
> Under section ‘Creating Profiles’ it’s mentioned as milli secs. In
> Example1 it’s mentioned as 30 days.
>
>
>
> Now I removed the entire field and from ‘profiler.json’ so that it
> profiled data never expires. I pushed and configuration and tested from
> kafka. I still don’t see the data storing to Hbase.
>
>
>
>
>
>
>
> *Regards,*
>
>
>
> *Balakrishna *
>
> *From:* Nick Allen [mailto:nick@nickallen.org]
> *Sent:* Wednesday, July 12, 2017 6:55 PM
>
>
> *To:* user@metron.apache.org
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> Quick correction for posterity... The 'expire' field is actually in days
> not milliseconds. So that should not be a problem here.
>
>
>
> On Wed, Jul 12, 2017 at 8:02 AM, Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
> In your config for the individual profile though, you expire the content
> every 30ms (per the zip file)
>
>
>
> Simon
>
>
>
>
>
>
>
> On 12 Jul 2017, at 12:20, Krishna Dhanekula <Kr...@sstech.us>
> wrote:
>
>
>
> Thanks Simon for replying.
>
>
>
> This is my ‘profiler.properties’ file content. TTL is 30 minutes and for
> every 30 seconds I am flushing to Hbase.
>
>
>
> profiler.workers=1
>
> profiler.executors=0
>
> profiler.input.topic=indexing
>
> profiler.period.duration=1
>
> profiler.period.duration.units=MINUTES
>
> profiler.ttl=30
>
> profiler.ttl.units=MINUTES
>
> profiler.hbase.salt.divisor=1000
>
> profiler.hbase.table=profiler
>
> profiler.hbase.column.family=P
>
> profiler.hbase.batch=1
>
> profiler.hbase.flush.interval.seconds=30
>
>
>
> ##### Kafka #####
>
>
>
> kafka.zk=10.10.110.184:2181
>
> kafka.broker=10.10.110.184:6667
>
> kafka.start=WHERE_I_LEFT_OFF
>
>
>
>
>
>
>
> *Regards,*
>
>
>
> *Balakrishna*
>
>
>
>
>
> *From:* Simon Elliston Ball [mailto:simon@simonellistonball.com
> <si...@simonellistonball.com>]
> *Sent:* Wednesday, July 12, 2017 3:28 PM
> *To:* user@metron.apache.org
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> Looks like you’ve set the profile to purge (expires) every 30 ms, and your
> period is set to 30 minutes, so the data is being expired long before it
> has a change to write.
>
>
>
> Simon
>
>
>
>
>
> On 12 Jul 2017, at 06:17, Krishna Dhanekula <Kr...@sstech.us>
> wrote:
>
>
>
> I have an problem where profiled data is not storing to Hbase instance:
>
>
>
> Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
>
>
>
> https://metron.apache.org/current-book/metron-analytics/
> metron-profiler/index.html
>
>
>
> Attached my profiler.properties and profiler.json files, remote.yaml and
> worker.log files.
>
>
>
> Once I place the following message into kafka topic 'indexing'
>
>
>
> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"
> 10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","
> protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_
> addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
>
>
>
> It's picked by splitterBolt and passed till HBaseBolt. But data is not
> getting stored to Hbase.
>
>
>
> Attached metron-artifacts.zip
>
>
>
> Attached storm-ui screenshot.storm-ui-screen.png
>
>
>
> Please suggest what am I missing here.
>
>
>
> Thanks in advance.
>
>
>
> Attachments:
>
> storm-ui-screen.png (17.4 kB)
>
> metron-artifacts.zip (36.6 kB)
>
>
>
>
>
> Regards,
>
> Balakrishna
>
> <metron_artifacts.zip><storm_ui_screen.png>
>
>
>
>
>
>
>
>
>
>
>
RE: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
Instance
Posted by Krishna Dhanekula <Kr...@sstech.us>.
Resending the data as a text.
profiler content and the PUSH command output.
[root@krishna-4 zookeeper]# pwd
/usr/metron/0.3.1/config/zookeeper
[root@krishna-4 zookeeper]# ls -lrt
total 12
drwxr-xr-x. 2 root root 4096 Apr 28 10:16 parsers
drwxr-xr-x. 2 root root 89 Apr 28 10:16 enrichments
drwxr-xr-x. 2 root root 89 Apr 28 10:17 indexing
-rw-r--r--. 1 metron root 108 Jul 12 06:55 global.json
-rw-r--r--. 1 metron hadoop 196 Jul 13 03:30 profiler.json
[root@krishna-4 zookeeper]# cat profiler.json
{
"profiles": [
{
"profile": "test",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
]
}
[root@krishna-4 zookeeper]# cd ../..
[root@krishna-4 0.3.1]# bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z krishna-3:2181
log4j:WARN No appenders could be found for logger (org.apache.curator.framework.imps.CuratorFrameworkImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[root@krishna-4 0.3.1]#
Regards,
Bala
From: Nick Allen [mailto:nick@nickallen.org]
Sent: Thursday, July 13, 2017 6:36 AM
To: user@metron.apache.org
Subject: RE: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
That's your problem then. You are not getting your profile definitions into Zookeeper.
Send us a copy of your terminal session showing the location and content of your profiler.json along with the command and ouput of you running the upload script.
On Jul 12, 2017 8:41 PM, "Krishna Dhanekula" <Kr...@sstech.us>> wrote:
When I run DUMP I can see everything else under /config/zookeeper except the profile (profiler.json) created by me.
________________________________
From: Nick Allen [nick@nickallen.org<ma...@nickallen.org>]
Sent: Wednesday, July 12, 2017 2:44 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
If the Profiler definitions were uploaded to Zookeeper correctly, you should be able to see them when using the DUMP option. See example below.
[root@node1 0.4.0]# bin/zk_load_configs.sh -m DUMP -z node1:2181
...
PROFILER Config: profiler
{
"profiles": [
{
"profile": "test",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
]
}
On Wed, Jul 12, 2017 at 2:26 PM, Krishna Dhanekula <Kr...@sstech.us>> wrote:
When I run this command
bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z krishna-3:2181
(Krishna-3 is my node1 name)
[root@krishna-4 0.3.1]# bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z krishna-3:2181
log4j:WARN No appenders could be found for logger (org.apache.curator.framework.imps.CuratorFrameworkImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
I don’t see the configuration files getting uploaded to zookeeper. Should I add out directory in this command.
Regards,
Balakrishna
From: Nick Allen [mailto:nick@nickallen.org<ma...@nickallen.org>]
Sent: Wednesday, July 12, 2017 8:15 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
> Under section ‘Creating Profiles’ it’s mentioned as milli secs. In Example1 it’s mentioned as 30 days.
Yes, there is an inconsistency in the README. The sentence in the table is wrong, the rest of the docs are correct.
Wrong: https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#creating-profiles
expires<https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires>
Optional
Profile data is purged after this period of time, specified in m̶i̶l̶l̶i̶s̶e̶c̶o̶n̶d̶s̶ [days]
Correct: https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires
> Now I removed the entire field and from ‘profiler.json’ so that it profiled data never expires. I pushed and configuration and tested from kafka. I still don’t see the data storing to Hbase.
Did you follow the Getting Started guide? Can you follow that exactly and let me know if that works for you.
On Wed, Jul 12, 2017 at 9:31 AM, Krishna Dhanekula <Kr...@sstech.us>> wrote:
Under section ‘Creating Profiles’ it’s mentioned as milli secs. In Example1 it’s mentioned as 30 days.
Now I removed the entire field and from ‘profiler.json’ so that it profiled data never expires. I pushed and configuration and tested from kafka. I still don’t see the data storing to Hbase.
Regards,
Balakrishna
From: Nick Allen [mailto:nick@nickallen.org<ma...@nickallen.org>]
Sent: Wednesday, July 12, 2017 6:55 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Quick correction for posterity... The 'expire' field is actually in days not milliseconds. So that should not be a problem here.
On Wed, Jul 12, 2017 at 8:02 AM, Simon Elliston Ball <si...@simonellistonball.com>> wrote:
In your config for the individual profile though, you expire the content every 30ms (per the zip file)
Simon
On 12 Jul 2017, at 12:20, Krishna Dhanekula <Kr...@sstech.us>> wrote:
Thanks Simon for replying.
This is my ‘profiler.properties’ file content. TTL is 30 minutes and for every 30 seconds I am flushing to Hbase.
profiler.workers=1
profiler.executors=0
profiler.input.topic=indexing
profiler.period.duration=1
profiler.period.duration.units=MINUTES
profiler.ttl=30
profiler.ttl.units=MINUTES
profiler.hbase.salt.divisor=1000
profiler.hbase.table=profiler
profiler.hbase.column.family=P
profiler.hbase.batch=1
profiler.hbase.flush.interval.seconds=30
##### Kafka #####
kafka.zk=10.10.110.184:2181<http://10.10.110.184:2181>
kafka.broker=10.10.110.184:6667<http://10.10.110.184:6667>
kafka.start=WHERE_I_LEFT_OFF
Regards,
Balakrishna
From: Simon Elliston Ball [mailto:simon@simonellistonball.com]
Sent: Wednesday, July 12, 2017 3:28 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Looks like you’ve set the profile to purge (expires) every 30 ms, and your period is set to 30 minutes, so the data is being expired long before it has a change to write.
Simon
On 12 Jul 2017, at 06:17, Krishna Dhanekula <Kr...@sstech.us>> wrote:
I have an problem where profiled data is not storing to Hbase instance:
Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
Attached my profiler.properties and profiler.json files, remote.yaml and worker.log files.
Once I place the following message into kafka topic 'indexing'
[{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
It's picked by splitterBolt and passed till HBaseBolt. But data is not getting stored to Hbase.
Attached metron-artifacts.zip
Attached storm-ui screenshot.storm-ui-screen.png
Please suggest what am I missing here.
Thanks in advance.
Attachments:
storm-ui-screen.png (17.4 kB)
metron-artifacts.zip (36.6 kB)
Regards,
Balakrishna
<metron_artifacts.zip><storm_ui_screen.png>
RE: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Posted by Nick Allen <ni...@nickallen.org>.
That's your problem then. You are not getting your profile definitions
into Zookeeper.
Send us a copy of your terminal session showing the location and content of
your profiler.json along with the command and ouput of you running the
upload script.
On Jul 12, 2017 8:41 PM, "Krishna Dhanekula" <Kr...@sstech.us>
wrote:
When I run DUMP I can see everything else under /config/zookeeper except
the profile (profiler.json) created by me.
------------------------------
*From:* Nick Allen [nick@nickallen.org]
*Sent:* Wednesday, July 12, 2017 2:44 PM
*To:* user@metron.apache.org
*Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
Instance
If the Profiler definitions were uploaded to Zookeeper correctly, you
should be able to see them when using the DUMP option. See example below.
[root@node1 0.4.0]# bin/zk_load_configs.sh -m DUMP -z node1:2181
...
PROFILER Config: profiler
{
"profiles": [
{
"profile": "test",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
]
}
On Wed, Jul 12, 2017 at 2:26 PM, Krishna Dhanekula <
Krishna.Dhanekula@sstech.us> wrote:
> When I run this command
>
>
>
> bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z krishna-3:2181
>
> (Krishna-3 is my node1 name)
>
>
>
> [root@krishna-4 0.3.1]# bin/zk_load_configs.sh -m PUSH -i
> config/zookeeper/ -z krishna-3:2181
>
> log4j:WARN No appenders could be found for logger
> (org.apache.curator.framework.imps.CuratorFrameworkImpl).
>
> log4j:WARN Please initialize the log4j system properly.
>
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>
>
>
> I don’t see the configuration files getting uploaded to zookeeper. Should
> I add out directory in this command.
>
>
>
>
>
>
>
> *Regards,*
>
>
>
> *Balakrishna*
>
>
>
> *From:* Nick Allen [mailto:nick@nickallen.org]
> *Sent:* Wednesday, July 12, 2017 8:15 PM
>
> *To:* user@metron.apache.org
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> > Under section ‘Creating Profiles’ it’s mentioned as milli secs. In
> Example1 it’s mentioned as 30 days.
>
>
>
> Yes, there is an inconsistency in the README. The sentence in the table
> is wrong, the rest of the docs are correct.
>
>
>
> Wrong: https://github.com/apache/metron/tree/master/metron-
> analytics/metron-profiler#creating-profiles
>
>
>
> expires
> <https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires>
>
> Optional
>
> Profile data is purged after this period of time, specified
> in m̶i̶l̶l̶i̶s̶e̶c̶o̶n̶d̶s̶ [days]
>
> Correct: https://github.com/apache/metron/tree/master/metron
> -analytics/metron-profiler#expires
>
>
>
>
>
> > Now I removed the entire field and from ‘profiler.json’ so that it
> profiled data never expires. I pushed and configuration and tested from
> kafka. I still don’t see the data storing to Hbase.
>
>
>
> Did you follow the Getting Started guide? Can you follow that exactly and
> let me know if that works for you.
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Jul 12, 2017 at 9:31 AM, Krishna Dhanekula <
> Krishna.Dhanekula@sstech.us> wrote:
>
> Under section ‘Creating Profiles’ it’s mentioned as milli secs. In
> Example1 it’s mentioned as 30 days.
>
>
>
> Now I removed the entire field and from ‘profiler.json’ so that it
> profiled data never expires. I pushed and configuration and tested from
> kafka. I still don’t see the data storing to Hbase.
>
>
>
>
>
>
>
> *Regards,*
>
>
>
> *Balakrishna *
>
> *From:* Nick Allen [mailto:nick@nickallen.org]
> *Sent:* Wednesday, July 12, 2017 6:55 PM
>
>
> *To:* user@metron.apache.org
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> Quick correction for posterity... The 'expire' field is actually in days
> not milliseconds. So that should not be a problem here.
>
>
>
> On Wed, Jul 12, 2017 at 8:02 AM, Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
> In your config for the individual profile though, you expire the content
> every 30ms (per the zip file)
>
>
>
> Simon
>
>
>
>
>
>
>
> On 12 Jul 2017, at 12:20, Krishna Dhanekula <Kr...@sstech.us>
> wrote:
>
>
>
> Thanks Simon for replying.
>
>
>
> This is my ‘profiler.properties’ file content. TTL is 30 minutes and for
> every 30 seconds I am flushing to Hbase.
>
>
>
> profiler.workers=1
>
> profiler.executors=0
>
> profiler.input.topic=indexing
>
> profiler.period.duration=1
>
> profiler.period.duration.units=MINUTES
>
> profiler.ttl=30
>
> profiler.ttl.units=MINUTES
>
> profiler.hbase.salt.divisor=1000
>
> profiler.hbase.table=profiler
>
> profiler.hbase.column.family=P
>
> profiler.hbase.batch=1
>
> profiler.hbase.flush.interval.seconds=30
>
>
>
> ##### Kafka #####
>
>
>
> kafka.zk=10.10.110.184:2181
>
> kafka.broker=10.10.110.184:6667
>
> kafka.start=WHERE_I_LEFT_OFF
>
>
>
>
>
>
>
> *Regards,*
>
>
>
> *Balakrishna*
>
>
>
>
>
> *From:* Simon Elliston Ball [mailto:simon@simonellistonball.com
> <si...@simonellistonball.com>]
> *Sent:* Wednesday, July 12, 2017 3:28 PM
> *To:* user@metron.apache.org
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> Looks like you’ve set the profile to purge (expires) every 30 ms, and your
> period is set to 30 minutes, so the data is being expired long before it
> has a change to write.
>
>
>
> Simon
>
>
>
>
>
> On 12 Jul 2017, at 06:17, Krishna Dhanekula <Kr...@sstech.us>
> wrote:
>
>
>
> I have an problem where profiled data is not storing to Hbase instance:
>
>
>
> Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
>
>
>
> https://metron.apache.org/current-book/metron-analytics/metr
> on-profiler/index.html
>
>
>
> Attached my profiler.properties and profiler.json files, remote.yaml and
> worker.log files.
>
>
>
> Once I place the following message into kafka topic 'indexing'
>
>
>
> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10",
> "bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"
> HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.
> 0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
>
>
>
> It's picked by splitterBolt and passed till HBaseBolt. But data is not
> getting stored to Hbase.
>
>
>
> Attached metron-artifacts.zip
>
>
>
> Attached storm-ui screenshot.storm-ui-screen.png
>
>
>
> Please suggest what am I missing here.
>
>
>
> Thanks in advance.
>
>
>
> Attachments:
>
> storm-ui-screen.png (17.4 kB)
>
> metron-artifacts.zip (36.6 kB)
>
>
>
>
>
> Regards,
>
> Balakrishna
>
> <metron_artifacts.zip><storm_ui_screen.png>
>
>
>
>
>
>
>
RE: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
Instance
Posted by Krishna Dhanekula <Kr...@sstech.us>.
When I run DUMP I can see everything else under /config/zookeeper except the profile (profiler.json) created by me.
________________________________
From: Nick Allen [nick@nickallen.org]
Sent: Wednesday, July 12, 2017 2:44 PM
To: user@metron.apache.org
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
If the Profiler definitions were uploaded to Zookeeper correctly, you should be able to see them when using the DUMP option. See example below.
[root@node1 0.4.0]# bin/zk_load_configs.sh -m DUMP -z node1:2181
...
PROFILER Config: profiler
{
"profiles": [
{
"profile": "test",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
]
}
On Wed, Jul 12, 2017 at 2:26 PM, Krishna Dhanekula <Kr...@sstech.us>> wrote:
When I run this command
bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z krishna-3:2181
(Krishna-3 is my node1 name)
[root@krishna-4 0.3.1]# bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z krishna-3:2181
log4j:WARN No appenders could be found for logger (org.apache.curator.framework.imps.CuratorFrameworkImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
I don’t see the configuration files getting uploaded to zookeeper. Should I add out directory in this command.
Regards,
Balakrishna
From: Nick Allen [mailto:nick@nickallen.org<ma...@nickallen.org>]
Sent: Wednesday, July 12, 2017 8:15 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
> Under section ‘Creating Profiles’ it’s mentioned as milli secs. In Example1 it’s mentioned as 30 days.
Yes, there is an inconsistency in the README. The sentence in the table is wrong, the rest of the docs are correct.
Wrong: https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#creating-profiles
expires<https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires>
Optional
Profile data is purged after this period of time, specified in m̶i̶l̶l̶i̶s̶e̶c̶o̶n̶d̶s̶ [days]
Correct: https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires
> Now I removed the entire field and from ‘profiler.json’ so that it profiled data never expires. I pushed and configuration and tested from kafka. I still don’t see the data storing to Hbase.
Did you follow the Getting Started guide? Can you follow that exactly and let me know if that works for you.
On Wed, Jul 12, 2017 at 9:31 AM, Krishna Dhanekula <Kr...@sstech.us>> wrote:
Under section ‘Creating Profiles’ it’s mentioned as milli secs. In Example1 it’s mentioned as 30 days.
Now I removed the entire field and from ‘profiler.json’ so that it profiled data never expires. I pushed and configuration and tested from kafka. I still don’t see the data storing to Hbase.
Regards,
Balakrishna
From: Nick Allen [mailto:nick@nickallen.org<ma...@nickallen.org>]
Sent: Wednesday, July 12, 2017 6:55 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Quick correction for posterity... The 'expire' field is actually in days not milliseconds. So that should not be a problem here.
On Wed, Jul 12, 2017 at 8:02 AM, Simon Elliston Ball <si...@simonellistonball.com>> wrote:
In your config for the individual profile though, you expire the content every 30ms (per the zip file)
Simon
On 12 Jul 2017, at 12:20, Krishna Dhanekula <Kr...@sstech.us>> wrote:
Thanks Simon for replying.
This is my ‘profiler.properties’ file content. TTL is 30 minutes and for every 30 seconds I am flushing to Hbase.
profiler.workers=1
profiler.executors=0
profiler.input.topic=indexing
profiler.period.duration=1
profiler.period.duration.units=MINUTES
profiler.ttl=30
profiler.ttl.units=MINUTES
profiler.hbase.salt.divisor=1000
profiler.hbase.table=profiler
profiler.hbase.column.family=P
profiler.hbase.batch=1
profiler.hbase.flush.interval.seconds=30
##### Kafka #####
kafka.zk=10.10.110.184:2181<http://10.10.110.184:2181>
kafka.broker=10.10.110.184:6667<http://10.10.110.184:6667>
kafka.start=WHERE_I_LEFT_OFF
Regards,
Balakrishna
From: Simon Elliston Ball [mailto:simon@simonellistonball.com]
Sent: Wednesday, July 12, 2017 3:28 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Looks like you’ve set the profile to purge (expires) every 30 ms, and your period is set to 30 minutes, so the data is being expired long before it has a change to write.
Simon
On 12 Jul 2017, at 06:17, Krishna Dhanekula <Kr...@sstech.us>> wrote:
I have an problem where profiled data is not storing to Hbase instance:
Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
Attached my profiler.properties and profiler.json files, remote.yaml and worker.log files.
Once I place the following message into kafka topic 'indexing'
[{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
It's picked by splitterBolt and passed till HBaseBolt. But data is not getting stored to Hbase.
Attached metron-artifacts.zip
Attached storm-ui screenshot.storm-ui-screen.png
Please suggest what am I missing here.
Thanks in advance.
Attachments:
storm-ui-screen.png (17.4 kB)
metron-artifacts.zip (36.6 kB)
Regards,
Balakrishna
<metron_artifacts.zip><storm_ui_screen.png>
Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Posted by Nick Allen <ni...@nickallen.org>.
If the Profiler definitions were uploaded to Zookeeper correctly, you
should be able to see them when using the DUMP option. See example below.
[root@node1 0.4.0]# bin/zk_load_configs.sh -m DUMP -z node1:2181
...
PROFILER Config: profiler
{
"profiles": [
{
"profile": "test",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
]
}
On Wed, Jul 12, 2017 at 2:26 PM, Krishna Dhanekula <
Krishna.Dhanekula@sstech.us> wrote:
> When I run this command
>
>
>
> bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z krishna-3:2181
>
> (Krishna-3 is my node1 name)
>
>
>
> [root@krishna-4 0.3.1]# bin/zk_load_configs.sh -m PUSH -i
> config/zookeeper/ -z krishna-3:2181
>
> log4j:WARN No appenders could be found for logger
> (org.apache.curator.framework.imps.CuratorFrameworkImpl).
>
> log4j:WARN Please initialize the log4j system properly.
>
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>
>
>
> I don’t see the configuration files getting uploaded to zookeeper. Should
> I add out directory in this command.
>
>
>
>
>
>
>
> *Regards,*
>
>
>
> *Balakrishna*
>
>
>
> *From:* Nick Allen [mailto:nick@nickallen.org]
> *Sent:* Wednesday, July 12, 2017 8:15 PM
>
> *To:* user@metron.apache.org
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> > Under section ‘Creating Profiles’ it’s mentioned as milli secs. In
> Example1 it’s mentioned as 30 days.
>
>
>
> Yes, there is an inconsistency in the README. The sentence in the table
> is wrong, the rest of the docs are correct.
>
>
>
> Wrong: https://github.com/apache/metron/tree/master/
> metron-analytics/metron-profiler#creating-profiles
>
>
>
> expires
> <https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires>
>
> Optional
>
> Profile data is purged after this period of time, specified
> in m̶i̶l̶l̶i̶s̶e̶c̶o̶n̶d̶s̶ [days]
>
> Correct: https://github.com/apache/metron/tree/master/
> metron-analytics/metron-profiler#expires
>
>
>
>
>
> > Now I removed the entire field and from ‘profiler.json’ so that it
> profiled data never expires. I pushed and configuration and tested from
> kafka. I still don’t see the data storing to Hbase.
>
>
>
> Did you follow the Getting Started guide? Can you follow that exactly and
> let me know if that works for you.
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Jul 12, 2017 at 9:31 AM, Krishna Dhanekula <
> Krishna.Dhanekula@sstech.us> wrote:
>
> Under section ‘Creating Profiles’ it’s mentioned as milli secs. In
> Example1 it’s mentioned as 30 days.
>
>
>
> Now I removed the entire field and from ‘profiler.json’ so that it
> profiled data never expires. I pushed and configuration and tested from
> kafka. I still don’t see the data storing to Hbase.
>
>
>
>
>
>
>
> *Regards,*
>
>
>
> *Balakrishna *
>
> *From:* Nick Allen [mailto:nick@nickallen.org]
> *Sent:* Wednesday, July 12, 2017 6:55 PM
>
>
> *To:* user@metron.apache.org
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> Quick correction for posterity... The 'expire' field is actually in days
> not milliseconds. So that should not be a problem here.
>
>
>
> On Wed, Jul 12, 2017 at 8:02 AM, Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
> In your config for the individual profile though, you expire the content
> every 30ms (per the zip file)
>
>
>
> Simon
>
>
>
>
>
>
>
> On 12 Jul 2017, at 12:20, Krishna Dhanekula <Kr...@sstech.us>
> wrote:
>
>
>
> Thanks Simon for replying.
>
>
>
> This is my ‘profiler.properties’ file content. TTL is 30 minutes and for
> every 30 seconds I am flushing to Hbase.
>
>
>
> profiler.workers=1
>
> profiler.executors=0
>
> profiler.input.topic=indexing
>
> profiler.period.duration=1
>
> profiler.period.duration.units=MINUTES
>
> profiler.ttl=30
>
> profiler.ttl.units=MINUTES
>
> profiler.hbase.salt.divisor=1000
>
> profiler.hbase.table=profiler
>
> profiler.hbase.column.family=P
>
> profiler.hbase.batch=1
>
> profiler.hbase.flush.interval.seconds=30
>
>
>
> ##### Kafka #####
>
>
>
> kafka.zk=10.10.110.184:2181
>
> kafka.broker=10.10.110.184:6667
>
> kafka.start=WHERE_I_LEFT_OFF
>
>
>
>
>
>
>
> *Regards,*
>
>
>
> *Balakrishna*
>
>
>
>
>
> *From:* Simon Elliston Ball [mailto:simon@simonellistonball.com
> <si...@simonellistonball.com>]
> *Sent:* Wednesday, July 12, 2017 3:28 PM
> *To:* user@metron.apache.org
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> Looks like you’ve set the profile to purge (expires) every 30 ms, and your
> period is set to 30 minutes, so the data is being expired long before it
> has a change to write.
>
>
>
> Simon
>
>
>
>
>
> On 12 Jul 2017, at 06:17, Krishna Dhanekula <Kr...@sstech.us>
> wrote:
>
>
>
> I have an problem where profiled data is not storing to Hbase instance:
>
>
>
> Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
>
>
>
> https://metron.apache.org/current-book/metron-analytics/
> metron-profiler/index.html
>
>
>
> Attached my profiler.properties and profiler.json files, remote.yaml and
> worker.log files.
>
>
>
> Once I place the following message into kafka topic 'indexing'
>
>
>
> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"
> 10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","
> protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_
> addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
>
>
>
> It's picked by splitterBolt and passed till HBaseBolt. But data is not
> getting stored to Hbase.
>
>
>
> Attached metron-artifacts.zip
>
>
>
> Attached storm-ui screenshot.storm-ui-screen.png
>
>
>
> Please suggest what am I missing here.
>
>
>
> Thanks in advance.
>
>
>
> Attachments:
>
> storm-ui-screen.png (17.4 kB)
>
> metron-artifacts.zip (36.6 kB)
>
>
>
>
>
> Regards,
>
> Balakrishna
>
> <metron_artifacts.zip><storm_ui_screen.png>
>
>
>
>
>
>
>
RE: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
Instance
Posted by Krishna Dhanekula <Kr...@sstech.us>.
When I run this command
bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z krishna-3:2181
(Krishna-3 is my node1 name)
[root@krishna-4 0.3.1]# bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z krishna-3:2181
log4j:WARN No appenders could be found for logger (org.apache.curator.framework.imps.CuratorFrameworkImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
I don’t see the configuration files getting uploaded to zookeeper. Should I add out directory in this command.
Regards,
Balakrishna
From: Nick Allen [mailto:nick@nickallen.org]
Sent: Wednesday, July 12, 2017 8:15 PM
To: user@metron.apache.org
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
> Under section ‘Creating Profiles’ it’s mentioned as milli secs. In Example1 it’s mentioned as 30 days.
Yes, there is an inconsistency in the README. The sentence in the table is wrong, the rest of the docs are correct.
Wrong: https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#creating-profiles
expires<https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires>
Optional
Profile data is purged after this period of time, specified in m̶i̶l̶l̶i̶s̶e̶c̶o̶n̶d̶s̶ [days]
Correct: https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires
> Now I removed the entire field and from ‘profiler.json’ so that it profiled data never expires. I pushed and configuration and tested from kafka. I still don’t see the data storing to Hbase.
Did you follow the Getting Started guide? Can you follow that exactly and let me know if that works for you.
On Wed, Jul 12, 2017 at 9:31 AM, Krishna Dhanekula <Kr...@sstech.us>> wrote:
Under section ‘Creating Profiles’ it’s mentioned as milli secs. In Example1 it’s mentioned as 30 days.
Now I removed the entire field and from ‘profiler.json’ so that it profiled data never expires. I pushed and configuration and tested from kafka. I still don’t see the data storing to Hbase.
Regards,
Balakrishna
From: Nick Allen [mailto:nick@nickallen.org<ma...@nickallen.org>]
Sent: Wednesday, July 12, 2017 6:55 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Quick correction for posterity... The 'expire' field is actually in days not milliseconds. So that should not be a problem here.
On Wed, Jul 12, 2017 at 8:02 AM, Simon Elliston Ball <si...@simonellistonball.com>> wrote:
In your config for the individual profile though, you expire the content every 30ms (per the zip file)
Simon
On 12 Jul 2017, at 12:20, Krishna Dhanekula <Kr...@sstech.us>> wrote:
Thanks Simon for replying.
This is my ‘profiler.properties’ file content. TTL is 30 minutes and for every 30 seconds I am flushing to Hbase.
profiler.workers=1
profiler.executors=0
profiler.input.topic=indexing
profiler.period.duration=1
profiler.period.duration.units=MINUTES
profiler.ttl=30
profiler.ttl.units=MINUTES
profiler.hbase.salt.divisor=1000
profiler.hbase.table=profiler
profiler.hbase.column.family=P
profiler.hbase.batch=1
profiler.hbase.flush.interval.seconds=30
##### Kafka #####
kafka.zk=10.10.110.184:2181<http://10.10.110.184:2181>
kafka.broker=10.10.110.184:6667<http://10.10.110.184:6667>
kafka.start=WHERE_I_LEFT_OFF
Regards,
Balakrishna
From: Simon Elliston Ball [mailto:simon@simonellistonball.com]
Sent: Wednesday, July 12, 2017 3:28 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Looks like you’ve set the profile to purge (expires) every 30 ms, and your period is set to 30 minutes, so the data is being expired long before it has a change to write.
Simon
On 12 Jul 2017, at 06:17, Krishna Dhanekula <Kr...@sstech.us>> wrote:
I have an problem where profiled data is not storing to Hbase instance:
Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
Attached my profiler.properties and profiler.json files, remote.yaml and worker.log files.
Once I place the following message into kafka topic 'indexing'
[{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
It's picked by splitterBolt and passed till HBaseBolt. But data is not getting stored to Hbase.
Attached metron-artifacts.zip
Attached storm-ui screenshot.storm-ui-screen.png
Please suggest what am I missing here.
Thanks in advance.
Attachments:
storm-ui-screen.png (17.4 kB)
metron-artifacts.zip (36.6 kB)
Regards,
Balakrishna
<metron_artifacts.zip><storm_ui_screen.png>
Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Posted by Nick Allen <ni...@nickallen.org>.
> Under section ‘Creating Profiles’ it’s mentioned as milli secs. In
Example1 it’s mentioned as 30 days.
Yes, there is an inconsistency in the README. The sentence in the table is
wrong, the rest of the docs are correct.
Wrong:
https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#creating-profiles
expires
<https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires>
Optional Profile data is purged after this period of time, specified
in m̶i̶l̶l̶i̶s̶e̶c̶o̶n̶d̶s̶ [days]
Correct:
https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires
> Now I removed the entire field and from ‘profiler.json’ so that it
profiled data never expires. I pushed and configuration and tested from
kafka. I still don’t see the data storing to Hbase.
Did you follow the Getting Started guide? Can you follow that exactly and
let me know if that works for you.
On Wed, Jul 12, 2017 at 9:31 AM, Krishna Dhanekula <
Krishna.Dhanekula@sstech.us> wrote:
> Under section ‘Creating Profiles’ it’s mentioned as milli secs. In
> Example1 it’s mentioned as 30 days.
>
>
>
> Now I removed the entire field and from ‘profiler.json’ so that it
> profiled data never expires. I pushed and configuration and tested from
> kafka. I still don’t see the data storing to Hbase.
>
>
>
>
>
>
>
> *Regards,*
>
>
>
> *Balakrishna *
>
> *From:* Nick Allen [mailto:nick@nickallen.org]
> *Sent:* Wednesday, July 12, 2017 6:55 PM
>
> *To:* user@metron.apache.org
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> Quick correction for posterity... The 'expire' field is actually in days
> not milliseconds. So that should not be a problem here.
>
>
>
> On Wed, Jul 12, 2017 at 8:02 AM, Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
> In your config for the individual profile though, you expire the content
> every 30ms (per the zip file)
>
>
>
> Simon
>
>
>
>
>
>
>
> On 12 Jul 2017, at 12:20, Krishna Dhanekula <Kr...@sstech.us>
> wrote:
>
>
>
> Thanks Simon for replying.
>
>
>
> This is my ‘profiler.properties’ file content. TTL is 30 minutes and for
> every 30 seconds I am flushing to Hbase.
>
>
>
> profiler.workers=1
>
> profiler.executors=0
>
> profiler.input.topic=indexing
>
> profiler.period.duration=1
>
> profiler.period.duration.units=MINUTES
>
> profiler.ttl=30
>
> profiler.ttl.units=MINUTES
>
> profiler.hbase.salt.divisor=1000
>
> profiler.hbase.table=profiler
>
> profiler.hbase.column.family=P
>
> profiler.hbase.batch=1
>
> profiler.hbase.flush.interval.seconds=30
>
>
>
> ##### Kafka #####
>
>
>
> kafka.zk=10.10.110.184:2181
>
> kafka.broker=10.10.110.184:6667
>
> kafka.start=WHERE_I_LEFT_OFF
>
>
>
>
>
>
>
> *Regards,*
>
>
>
> *Balakrishna*
>
>
>
>
>
> *From:* Simon Elliston Ball [mailto:simon@simonellistonball.com
> <si...@simonellistonball.com>]
> *Sent:* Wednesday, July 12, 2017 3:28 PM
> *To:* user@metron.apache.org
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> Looks like you’ve set the profile to purge (expires) every 30 ms, and your
> period is set to 30 minutes, so the data is being expired long before it
> has a change to write.
>
>
>
> Simon
>
>
>
>
>
> On 12 Jul 2017, at 06:17, Krishna Dhanekula <Kr...@sstech.us>
> wrote:
>
>
>
> I have an problem where profiled data is not storing to Hbase instance:
>
>
>
> Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
>
>
>
> https://metron.apache.org/current-book/metron-analytics/
> metron-profiler/index.html
>
>
>
> Attached my profiler.properties and profiler.json files, remote.yaml and
> worker.log files.
>
>
>
> Once I place the following message into kafka topic 'indexing'
>
>
>
> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"
> 10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","
> protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_
> addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
>
>
>
> It's picked by splitterBolt and passed till HBaseBolt. But data is not
> getting stored to Hbase.
>
>
>
> Attached metron-artifacts.zip
>
>
>
> Attached storm-ui screenshot.storm-ui-screen.png
>
>
>
> Please suggest what am I missing here.
>
>
>
> Thanks in advance.
>
>
>
> Attachments:
>
> storm-ui-screen.png (17.4 kB)
>
> metron-artifacts.zip (36.6 kB)
>
>
>
>
>
> Regards,
>
> Balakrishna
>
> <metron_artifacts.zip><storm_ui_screen.png>
>
>
>
>
>
RE: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
Instance
Posted by Krishna Dhanekula <Kr...@sstech.us>.
Under section ‘Creating Profiles’ it’s mentioned as milli secs. In Example1 it’s mentioned as 30 days.
Now I removed the entire field and from ‘profiler.json’ so that it profiled data never expires. I pushed and configuration and tested from kafka. I still don’t see the data storing to Hbase.
Regards,
Balakrishna
From: Nick Allen [mailto:nick@nickallen.org]
Sent: Wednesday, July 12, 2017 6:55 PM
To: user@metron.apache.org
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Quick correction for posterity... The 'expire' field is actually in days not milliseconds. So that should not be a problem here.
On Wed, Jul 12, 2017 at 8:02 AM, Simon Elliston Ball <si...@simonellistonball.com>> wrote:
In your config for the individual profile though, you expire the content every 30ms (per the zip file)
Simon
On 12 Jul 2017, at 12:20, Krishna Dhanekula <Kr...@sstech.us>> wrote:
Thanks Simon for replying.
This is my ‘profiler.properties’ file content. TTL is 30 minutes and for every 30 seconds I am flushing to Hbase.
profiler.workers=1
profiler.executors=0
profiler.input.topic=indexing
profiler.period.duration=1
profiler.period.duration.units=MINUTES
profiler.ttl=30
profiler.ttl.units=MINUTES
profiler.hbase.salt.divisor=1000
profiler.hbase.table=profiler
profiler.hbase.column.family=P
profiler.hbase.batch=1
profiler.hbase.flush.interval.seconds=30
##### Kafka #####
kafka.zk=10.10.110.184:2181<http://10.10.110.184:2181>
kafka.broker=10.10.110.184:6667<http://10.10.110.184:6667>
kafka.start=WHERE_I_LEFT_OFF
Regards,
Balakrishna
From: Simon Elliston Ball [mailto:simon@simonellistonball.com]
Sent: Wednesday, July 12, 2017 3:28 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Looks like you’ve set the profile to purge (expires) every 30 ms, and your period is set to 30 minutes, so the data is being expired long before it has a change to write.
Simon
On 12 Jul 2017, at 06:17, Krishna Dhanekula <Kr...@sstech.us>> wrote:
I have an problem where profiled data is not storing to Hbase instance:
Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
Attached my profiler.properties and profiler.json files, remote.yaml and worker.log files.
Once I place the following message into kafka topic 'indexing'
[{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
It's picked by splitterBolt and passed till HBaseBolt. But data is not getting stored to Hbase.
Attached metron-artifacts.zip
Attached storm-ui screenshot.storm-ui-screen.png
Please suggest what am I missing here.
Thanks in advance.
Attachments:
storm-ui-screen.png (17.4 kB)
metron-artifacts.zip (36.6 kB)
Regards,
Balakrishna
<metron_artifacts.zip><storm_ui_screen.png>
Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Posted by Nick Allen <ni...@nickallen.org>.
Quick correction for posterity... The 'expire' field is actually in days
not milliseconds. So that should not be a problem here.
On Wed, Jul 12, 2017 at 8:02 AM, Simon Elliston Ball <
simon@simonellistonball.com> wrote:
> In your config for the individual profile though, you expire the content
> every 30ms (per the zip file)
>
> Simon
>
>
>
>
> On 12 Jul 2017, at 12:20, Krishna Dhanekula <Kr...@sstech.us>
> wrote:
>
> Thanks Simon for replying.
>
> This is my ‘profiler.properties’ file content. TTL is 30 minutes and for
> every 30 seconds I am flushing to Hbase.
>
> profiler.workers=1
> profiler.executors=0
> profiler.input.topic=indexing
> profiler.period.duration=1
> profiler.period.duration.units=MINUTES
> profiler.ttl=30
> profiler.ttl.units=MINUTES
> profiler.hbase.salt.divisor=1000
> profiler.hbase.table=profiler
> profiler.hbase.column.family=P
> profiler.hbase.batch=1
> profiler.hbase.flush.interval.seconds=30
>
> ##### Kafka #####
>
> kafka.zk=10.10.110.184:2181
> kafka.broker=10.10.110.184:6667
> kafka.start=WHERE_I_LEFT_OFF
>
>
>
> *Regards,*
>
> *Balakrishna*
>
>
> *From:* Simon Elliston Ball [mailto:simon@simonellistonball.com
> <si...@simonellistonball.com>]
> *Sent:* Wednesday, July 12, 2017 3:28 PM
> *To:* user@metron.apache.org
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
> Looks like you’ve set the profile to purge (expires) every 30 ms, and your
> period is set to 30 minutes, so the data is being expired long before it
> has a change to write.
>
> Simon
>
>
>
> On 12 Jul 2017, at 06:17, Krishna Dhanekula <Kr...@sstech.us>
> wrote:
>
> I have an problem where profiled data is not storing to Hbase instance:
>
> Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
>
> https://metron.apache.org/current-book/metron-analytics/
> metron-profiler/index.html
>
> Attached my profiler.properties and profiler.json files, remote.yaml and
> worker.log files.
>
> Once I place the following message into kafka topic 'indexing'
>
> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"
> 10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","
> protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_
> addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
>
> It's picked by splitterBolt and passed till HBaseBolt. But data is not
> getting stored to Hbase.
>
> Attached metron-artifacts.zip
>
> Attached storm-ui screenshot.storm-ui-screen.png
>
> Please suggest what am I missing here.
>
> Thanks in advance.
>
> Attachments:
> storm-ui-screen.png (17.4 kB)
> metron-artifacts.zip (36.6 kB)
>
>
> Regards,
> Balakrishna
> <metron_artifacts.zip><storm_ui_screen.png>
>
>
>
Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Posted by Nick Allen <ni...@nickallen.org>.
Hi Krishna -
I did not look at your Profile definition. Please include it in the body
of your email the next time you might need help. I am a little hesitant to
open attachments on a public mailing list. My apologies.
Keep in mind there are two places for what you might call 'settings'.
- There are global settings for the "Profiler" (aka
profiler.properties). These are settings that apply globally to the Storm
topology that consumes profile definitions and generates profile data.
- There are also 'settings' that are unique to each Profile. These are
contained in what I call the Profile definition. This defines how each
individual Profile produces data.
Based on Simon's feedback, it seems that you are confusing some of these
settings. Simon is referring to the 'expires' field in your Profile
definition that is documented here [1].
Please see the following links for more information. If anything is not
clear in these links, please let us know.
- Profile Definitions [2]
- Configuring the Profiler [3]
--
[1]
http://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html#expires
[2]
http://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html#Creating_Profiles
[3]
http://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html#Configuring_the_Profiler
On Wed, Jul 12, 2017 at 8:02 AM, Simon Elliston Ball <
simon@simonellistonball.com> wrote:
> In your config for the individual profile though, you expire the content
> every 30ms (per the zip file)
>
> Simon
>
>
>
>
> On 12 Jul 2017, at 12:20, Krishna Dhanekula <Kr...@sstech.us>
> wrote:
>
> Thanks Simon for replying.
>
> This is my ‘profiler.properties’ file content. TTL is 30 minutes and for
> every 30 seconds I am flushing to Hbase.
>
> profiler.workers=1
> profiler.executors=0
> profiler.input.topic=indexing
> profiler.period.duration=1
> profiler.period.duration.units=MINUTES
> profiler.ttl=30
> profiler.ttl.units=MINUTES
> profiler.hbase.salt.divisor=1000
> profiler.hbase.table=profiler
> profiler.hbase.column.family=P
> profiler.hbase.batch=1
> profiler.hbase.flush.interval.seconds=30
>
> ##### Kafka #####
>
> kafka.zk=10.10.110.184:2181
> kafka.broker=10.10.110.184:6667
> kafka.start=WHERE_I_LEFT_OFF
>
>
>
> *Regards,*
>
> *Balakrishna*
>
>
> *From:* Simon Elliston Ball [mailto:simon@simonellistonball.com
> <si...@simonellistonball.com>]
> *Sent:* Wednesday, July 12, 2017 3:28 PM
> *To:* user@metron.apache.org
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
> Looks like you’ve set the profile to purge (expires) every 30 ms, and your
> period is set to 30 minutes, so the data is being expired long before it
> has a change to write.
>
> Simon
>
>
>
> On 12 Jul 2017, at 06:17, Krishna Dhanekula <Kr...@sstech.us>
> wrote:
>
> I have an problem where profiled data is not storing to Hbase instance:
>
> Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
>
> https://metron.apache.org/current-book/metron-analytics/
> metron-profiler/index.html
>
> Attached my profiler.properties and profiler.json files, remote.yaml and
> worker.log files.
>
> Once I place the following message into kafka topic 'indexing'
>
> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"
> 10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","
> protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_
> addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
>
> It's picked by splitterBolt and passed till HBaseBolt. But data is not
> getting stored to Hbase.
>
> Attached metron-artifacts.zip
>
> Attached storm-ui screenshot.storm-ui-screen.png
>
> Please suggest what am I missing here.
>
> Thanks in advance.
>
> Attachments:
> storm-ui-screen.png (17.4 kB)
> metron-artifacts.zip (36.6 kB)
>
>
> Regards,
> Balakrishna
> <metron_artifacts.zip><storm_ui_screen.png>
>
>
>
Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
Instance
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
In your config for the individual profile though, you expire the content every 30ms (per the zip file)
Simon
> On 12 Jul 2017, at 12:20, Krishna Dhanekula <Kr...@sstech.us> wrote:
>
> Thanks Simon for replying.
>
> This is my ‘profiler.properties’ file content. TTL is 30 minutes and for every 30 seconds I am flushing to Hbase.
>
> profiler.workers=1
> profiler.executors=0
> profiler.input.topic=indexing
> profiler.period.duration=1
> profiler.period.duration.units=MINUTES
> profiler.ttl=30
> profiler.ttl.units=MINUTES
> profiler.hbase.salt.divisor=1000
> profiler.hbase.table=profiler
> profiler.hbase.column.family=P
> profiler.hbase.batch=1
> profiler.hbase.flush.interval.seconds=30
>
> ##### Kafka #####
>
> kafka.zk=10.10.110.184:2181
> kafka.broker=10.10.110.184:6667
> kafka.start=WHERE_I_LEFT_OFF
>
>
>
> Regards,
>
> Balakrishna
>
>
> From: Simon Elliston Ball [mailto:simon@simonellistonball.com]
> Sent: Wednesday, July 12, 2017 3:28 PM
> To: user@metron.apache.org
> Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
>
> Looks like you’ve set the profile to purge (expires) every 30 ms, and your period is set to 30 minutes, so the data is being expired long before it has a change to write.
>
> Simon
>
>
> On 12 Jul 2017, at 06:17, Krishna Dhanekula <Krishna.Dhanekula@sstech.us <ma...@sstech.us>> wrote:
>
> I have an problem where profiled data is not storing to Hbase instance:
>
> Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
>
> https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html <https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html>
>
> Attached my profiler.properties and profiler.json files, remote.yaml and worker.log files.
>
> Once I place the following message into kafka topic 'indexing'
>
> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
>
> It's picked by splitterBolt and passed till HBaseBolt. But data is not getting stored to Hbase.
>
> Attached metron-artifacts.zip
>
> Attached storm-ui screenshot.storm-ui-screen.png
>
> Please suggest what am I missing here.
>
> Thanks in advance.
>
> Attachments:
> storm-ui-screen.png (17.4 kB)
> metron-artifacts.zip (36.6 kB)
>
>
> Regards,
> Balakrishna
> <metron_artifacts.zip><storm_ui_screen.png>
RE: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
Instance
Posted by Krishna Dhanekula <Kr...@sstech.us>.
Thanks Simon for replying.
This is my ‘profiler.properties’ file content. TTL is 30 minutes and for every 30 seconds I am flushing to Hbase.
profiler.workers=1
profiler.executors=0
profiler.input.topic=indexing
profiler.period.duration=1
profiler.period.duration.units=MINUTES
profiler.ttl=30
profiler.ttl.units=MINUTES
profiler.hbase.salt.divisor=1000
profiler.hbase.table=profiler
profiler.hbase.column.family=P
profiler.hbase.batch=1
profiler.hbase.flush.interval.seconds=30
##### Kafka #####
kafka.zk=10.10.110.184:2181
kafka.broker=10.10.110.184:6667
kafka.start=WHERE_I_LEFT_OFF
Regards,
Balakrishna
From: Simon Elliston Ball [mailto:simon@simonellistonball.com]
Sent: Wednesday, July 12, 2017 3:28 PM
To: user@metron.apache.org
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Looks like you’ve set the profile to purge (expires) every 30 ms, and your period is set to 30 minutes, so the data is being expired long before it has a change to write.
Simon
On 12 Jul 2017, at 06:17, Krishna Dhanekula <Kr...@sstech.us>> wrote:
I have an problem where profiled data is not storing to Hbase instance:
Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
Attached my profiler.properties and profiler.json files, remote.yaml and worker.log files.
Once I place the following message into kafka topic 'indexing'
[{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
It's picked by splitterBolt and passed till HBaseBolt. But data is not getting stored to Hbase.
Attached metron-artifacts.zip
Attached storm-ui screenshot.storm-ui-screen.png
Please suggest what am I missing here.
Thanks in advance.
Attachments:
storm-ui-screen.png (17.4 kB)
metron-artifacts.zip (36.6 kB)
Regards,
Balakrishna
<metron_artifacts.zip><storm_ui_screen.png>
Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
Instance
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
Looks like you’ve set the profile to purge (expires) every 30 ms, and your period is set to 30 minutes, so the data is being expired long before it has a change to write.
Simon
> On 12 Jul 2017, at 06:17, Krishna Dhanekula <Kr...@sstech.us> wrote:
>
> I have an problem where profiled data is not storing to Hbase instance:
>
> Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
>
> https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html <https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html>
>
> Attached my profiler.properties and profiler.json files, remote.yaml and worker.log files.
>
> Once I place the following message into kafka topic 'indexing'
>
> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
>
> It's picked by splitterBolt and passed till HBaseBolt. But data is not getting stored to Hbase.
>
> Attached metron-artifacts.zip
>
> Attached storm-ui screenshot.storm-ui-screen.png
>
> Please suggest what am I missing here.
>
> Thanks in advance.
>
> Attachments:
> storm-ui-screen.png (17.4 kB)
> metron-artifacts.zip (36.6 kB)
>
>
> Regards,
> Balakrishna
> <metron_artifacts.zip><storm_ui_screen.png>