You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by "Yerex, Tom" <to...@ubc.ca> on 2019/11/06 00:46:59 UTC
Error stemming from hbaseBolt
Good evening,
I am working with metron 0.7.2. I have tried to implement a variation on the solution at hxxps://metron.apache.org/current-book/use-cases/geographic_login_outliers/index.html, with the modification that data is coming into a Kafka topic (JSON format) instead of an import from a CSV.
No data has appeared in my Hbase profiler table. I discovered in the Storm UI an error in the hbaseBolt for the profiler topology (see below). I tried an insert at the command prompt, everything seemed okay but my experience with HBase is still limited. My plan tomorrow is to try and debug things using Stellar, but perhaps someone has seen this error somewhere and knows what is wrong?
Thank you,
Tom.
java.lang.Error: Unresolved compilation problem: at org.apache.metron.common.utils.SerDeUtils.toBytes(SerDeUtils.java:235) at org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder.columns(ValueOnlyColumnBuilder.java:52) at org.apache.metron.profiler.storm.ProfileHBaseMapper.columns(ProfileHBaseMapper.java:82) at org.apache.metron.hbase.bolt.HBaseBolt.save(HBaseBolt.java:183) at org.apache.metron.hbase.bolt.HBaseBolt.execute(HBaseBolt.java:164) at org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735) at org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855) at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
(parser configuration)
{
"parserClassName": "org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic": "radius",
"outputTopic": "indexing",
"fieldTransformations": [{
"transformation": "STELLAR",
"output": [
"geohash"
],
"config": {
"geohash": "GEOHASH_FROM_LOC(GEO_GET(client.ip))"
}
}]
}
(enrichment configuration)
{
"enrichment": {
"fieldMap": {
"stellar" : {
"config" : [
"geo_locations := MULTISET_MERGE( PROFILE_GET( 'locations_by_user', user.id, PROFILE_FIXED( 15, ‘MINUTES’)))",
"geo_centroid := GEOHASH_CENTROID(geo_locations)",
"geo_distance := TO_INTEGER(GEOHASH_DIST(geo_centroid, hash))",
"geo_locations := null"
]
}
}
,"fieldToTypeMap": { }
},
"threatIntel": {
"fieldMap": {
"stellar" : {
"config" : [
"geo_distance_distr:= STATS_MERGE( PROFILE_GET( 'geo_distribution_from_centroid', 'global', PROFILE_FIXED( 15, ‘MINUTES’)))",
"dist_median := STATS_PERCENTILE(geo_distance_distr, 50.0)",
"dist_sd := STATS_SD(geo_distance_distr)",
"geo_outlier := ABS(dist_median - geo_distance) >= 5*dist_sd",
"is_alert := is_alert || (geo_outlier != null && geo_outlier == true)",
"geo_distance_distr := null"
]
}
},
"fieldToTypeMap": { },
"triageConfig" : {
"riskLevelRules" : [
{
"name" : "Geographic Outlier",
"comment" : "Determine if the user's geographic distance from the centroid of the historic logins is an outlier as compared to all users.",
"rule" : "geo_outlier != null && geo_outlier",
"score" : 10,
"reason" : "FORMAT('user %s has a distance (%d) from the centroid of their last login is 5 std deviations (%f) from the median (%f)', user.id, geo_distance, dist_sd, dist_median)"
}
],
"aggregator" : "MAX"
}
}
}
(profiler.json)
{
"profiles": [
{
"profile": "geo_distribution_from_centroid",
"foreach": "'global'",
"onlyif": "geo_distance != null",
"init": {
"s": "STATS_INIT()"
},
"update": {
"s": "STATS_ADD(s, geo_distance)"
},
"result": {
"profile":"s"
}
},
{
"profile": "locations_by_user",
"foreach": "user.id",
"onlyif": "geohash != null && LENGTH(geohash) > 0",
"init": {
"s": "MULTISET_INIT()"
},
"update": {
"s": "MULTISET_ADD(s, geohash)"
},
"result": {
"profile":"s"
}
}
]
}
Re: Error stemming from hbaseBolt
Posted by "Yerex, Tom" <to...@ubc.ca>.
Thank you Mike, I’ll tackle it this morning. I must have built from HEAD so that would be my first mistake when deploying this with an eye to production.
Cheers,
Tom.
From: Michael Miklavcic <mi...@gmail.com>
Reply-To: "user@metron.apache.org" <us...@metron.apache.org>
Date: Wednesday, November 6, 2019 at 9:39 AM
To: "user@metron.apache.org" <us...@metron.apache.org>
Subject: Re: Error stemming from hbaseBolt
Hi Tom,
How did you build Metron? Our latest official release is 0.7.1 -> https://archive.apache.org/dist/metron/. Are you building via latest HEAD in the master branch? You might see a bland/vague exception like this if using an IDE (like Eclipse) for compilation. If you're not doing so already, I'd recommend building from the project root using "mvn clean install -DskipTests". For additional guidance, please run <your git project root dir>/metron-deployment/scripts/platform-info.sh and copy the output of running that in a reply. That should give the community some better detail to help.
Thanks,
Mike
On Tue, Nov 5, 2019 at 5:47 PM Yerex, Tom <to...@ubc.ca> wrote:
Good evening,
I am working with metron 0.7.2. I have tried to implement a variation on the solution at hxxps://metron.apache.org/current-book/use-cases/geographic_login_outliers/index.html, with the modification that data is coming into a Kafka topic (JSON format) instead of an import from a CSV.
No data has appeared in my Hbase profiler table. I discovered in the Storm UI an error in the hbaseBolt for the profiler topology (see below). I tried an insert at the command prompt, everything seemed okay but my experience with HBase is still limited. My plan tomorrow is to try and debug things using Stellar, but perhaps someone has seen this error somewhere and knows what is wrong?
Thank you,
Tom.
java.lang.Error: Unresolved compilation problem: at org.apache.metron.common.utils.SerDeUtils.toBytes(SerDeUtils.java:235) at org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder.columns(ValueOnlyColumnBuilder.java:52) at org.apache.metron.profiler.storm.ProfileHBaseMapper.columns(ProfileHBaseMapper.java:82) at org.apache.metron.hbase.bolt.HBaseBolt.save(HBaseBolt.java:183) at org.apache.metron.hbase.bolt.HBaseBolt.execute(HBaseBolt.java:164) at org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735) at org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855) at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
(parser configuration)
{
"parserClassName": "org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic": "radius",
"outputTopic": "indexing",
"fieldTransformations": [{
"transformation": "STELLAR",
"output": [
"geohash"
],
"config": {
"geohash": "GEOHASH_FROM_LOC(GEO_GET(client.ip))"
}
}]
}
(enrichment configuration)
{
"enrichment": {
"fieldMap": {
"stellar" : {
"config" : [
"geo_locations := MULTISET_MERGE( PROFILE_GET( 'locations_by_user', user.id, PROFILE_FIXED( 15, ‘MINUTES’)))",
"geo_centroid := GEOHASH_CENTROID(geo_locations)",
"geo_distance := TO_INTEGER(GEOHASH_DIST(geo_centroid, hash))",
"geo_locations := null"
]
}
}
,"fieldToTypeMap": { }
},
"threatIntel": {
"fieldMap": {
"stellar" : {
"config" : [
"geo_distance_distr:= STATS_MERGE( PROFILE_GET( 'geo_distribution_from_centroid', 'global', PROFILE_FIXED( 15, ‘MINUTES’)))",
"dist_median := STATS_PERCENTILE(geo_distance_distr, 50.0)",
"dist_sd := STATS_SD(geo_distance_distr)",
"geo_outlier := ABS(dist_median - geo_distance) >= 5*dist_sd",
"is_alert := is_alert || (geo_outlier != null && geo_outlier == true)",
"geo_distance_distr := null"
]
}
},
"fieldToTypeMap": { },
"triageConfig" : {
"riskLevelRules" : [
{
"name" : "Geographic Outlier",
"comment" : "Determine if the user's geographic distance from the centroid of the historic logins is an outlier as compared to all users.",
"rule" : "geo_outlier != null && geo_outlier",
"score" : 10,
"reason" : "FORMAT('user %s has a distance (%d) from the centroid of their last login is 5 std deviations (%f) from the median (%f)', user.id, geo_distance, dist_sd, dist_median)"
}
],
"aggregator" : "MAX"
}
}
}
(profiler.json)
{
"profiles": [
{
"profile": "geo_distribution_from_centroid",
"foreach": "'global'",
"onlyif": "geo_distance != null",
"init": {
"s": "STATS_INIT()"
},
"update": {
"s": "STATS_ADD(s, geo_distance)"
},
"result": {
"profile":"s"
}
},
{
"profile": "locations_by_user",
"foreach": "user.id",
"onlyif": "geohash != null && LENGTH(geohash) > 0",
"init": {
"s": "MULTISET_INIT()"
},
"update": {
"s": "MULTISET_ADD(s, geohash)"
},
"result": {
"profile":"s"
}
}
]
}
Re: Error stemming from hbaseBolt
Posted by Michael Miklavcic <mi...@gmail.com>.
Hi Tom,
How did you build Metron? Our latest official release is 0.7.1 ->
https://archive.apache.org/dist/metron/. Are you building via latest HEAD
in the master branch? You might see a bland/vague exception like this if
using an IDE (like Eclipse) for compilation. If you're not doing so
already, I'd recommend building from the project root using "mvn clean
install -DskipTests". For additional guidance, please run <your git project
root dir>/metron-deployment/scripts/platform-info.sh and copy the output of
running that in a reply. That should give the community some better detail
to help.
Thanks,
Mike
On Tue, Nov 5, 2019 at 5:47 PM Yerex, Tom <to...@ubc.ca> wrote:
> Good evening,
>
>
>
> I am working with metron 0.7.2. I have tried to implement a variation on
> the solution at hxxps://
> metron.apache.org/current-book/use-cases/geographic_login_outliers/index.html,
> with the modification that data is coming into a Kafka topic (JSON format)
> instead of an import from a CSV.
>
>
>
> No data has appeared in my Hbase profiler table. I discovered in the Storm
> UI an error in the hbaseBolt for the profiler topology (see below). I tried
> an insert at the command prompt, everything seemed okay but my experience
> with HBase is still limited. My plan tomorrow is to try and debug things
> using Stellar, but perhaps someone has seen this error somewhere and knows
> what is wrong?
>
>
>
> Thank you,
>
>
>
> Tom.
>
>
>
> java.lang.Error: Unresolved compilation problem: at
> org.apache.metron.common.utils.SerDeUtils.toBytes(SerDeUtils.java:235) at
> org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder.columns(ValueOnlyColumnBuilder.java:52)
> at
> org.apache.metron.profiler.storm.ProfileHBaseMapper.columns(ProfileHBaseMapper.java:82)
> at org.apache.metron.hbase.bolt.HBaseBolt.save(HBaseBolt.java:183) at
> org.apache.metron.hbase.bolt.HBaseBolt.execute(HBaseBolt.java:164) at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> at
> org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855)
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484) at
> clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
>
>
>
> (parser configuration)
>
> {
>
> "parserClassName":
> "org.apache.metron.parsers.json.JSONMapParser",
>
> "sensorTopic": "radius",
>
> "outputTopic": "indexing",
>
> "fieldTransformations": [{
>
> "transformation": "STELLAR",
>
> "output": [
>
> "geohash"
>
> ],
>
> "config": {
>
> "geohash":
> "GEOHASH_FROM_LOC(GEO_GET(client.ip))"
>
> }
>
> }]
>
> }
>
>
>
> (enrichment configuration)
>
> {
>
> "enrichment": {
>
> "fieldMap": {
>
> "stellar" : {
>
> "config" : [
>
> "geo_locations := MULTISET_MERGE( PROFILE_GET(
> 'locations_by_user', user.id, PROFILE_FIXED( 15, ‘MINUTES’)))",
>
> "geo_centroid := GEOHASH_CENTROID(geo_locations)",
>
> "geo_distance := TO_INTEGER(GEOHASH_DIST(geo_centroid, hash))",
>
> "geo_locations := null"
>
> ]
>
> }
>
> }
>
> ,"fieldToTypeMap": { }
>
> },
>
> "threatIntel": {
>
> "fieldMap": {
>
> "stellar" : {
>
> "config" : [
>
> "geo_distance_distr:= STATS_MERGE( PROFILE_GET(
> 'geo_distribution_from_centroid', 'global', PROFILE_FIXED( 15,
> ‘MINUTES’)))",
>
> "dist_median := STATS_PERCENTILE(geo_distance_distr, 50.0)",
>
> "dist_sd := STATS_SD(geo_distance_distr)",
>
> "geo_outlier := ABS(dist_median - geo_distance) >= 5*dist_sd",
>
> "is_alert := is_alert || (geo_outlier != null && geo_outlier ==
> true)",
>
> "geo_distance_distr := null"
>
> ]
>
> }
>
> },
>
> "fieldToTypeMap": { },
>
> "triageConfig" : {
>
> "riskLevelRules" : [
>
> {
>
> "name" : "Geographic Outlier",
>
> "comment" : "Determine if the user's geographic distance from
> the centroid of the historic logins is an outlier as compared to all
> users.",
>
> "rule" : "geo_outlier != null && geo_outlier",
>
> "score" : 10,
>
> "reason" : "FORMAT('user %s has a distance (%d) from the
> centroid of their last login is 5 std deviations (%f) from the median
> (%f)', user.id, geo_distance, dist_sd, dist_median)"
>
> }
>
> ],
>
> "aggregator" : "MAX"
>
> }
>
> }
>
> }
>
>
>
>
>
>
>
> (profiler.json)
>
> {
>
> "profiles": [
>
> {
>
> "profile": "geo_distribution_from_centroid",
>
> "foreach": "'global'",
>
> "onlyif": "geo_distance != null",
>
> "init": {
>
> "s": "STATS_INIT()"
>
> },
>
> "update": {
>
> "s": "STATS_ADD(s, geo_distance)"
>
> },
>
> "result": {
>
> "profile":"s"
>
> }
>
> },
>
> {
>
> "profile": "locations_by_user",
>
> "foreach": "user.id",
>
> "onlyif": "geohash != null && LENGTH(geohash) > 0",
>
> "init": {
>
> "s": "MULTISET_INIT()"
>
> },
>
> "update": {
>
> "s": "MULTISET_ADD(s, geohash)"
>
> },
>
> "result": {
>
> "profile":"s"
>
> }
>
> }
>
> ]
>
> }
>