You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Ali Nazemian <al...@gmail.com> on 2017/05/24 02:34:32 UTC
Metron HBase conditional enrichment
Hi,
I was wondering how I can manage Stellar syntax to be aligned with the
following structure for the HBase enrichment:
HBase_row_key: tenant_name+device_type+device_name
At the high-level, I need to create a separate field via a post-parse
Stellar function to be a concatenation of tenan_name, device_type and
device_name. Let's call this field "key". Basically, I need to do the
enrichment on the "key" which would be corresponding to the HBase row key.
My first question is *can I do the concatenation on the fly at the
enrichment level, so I don't need to store this temp field in
Elasticsearch/HDFS*.
Moreover, I need to have a conditional enrichment to say if you couldn't
find any match for "tenant_name+device_type+device_name" lookup for
"tenant_name+device_type+default_device". The second question would be *how
can I manage conditional enrichment like this one*. I would be really
grateful if you can provide some example.
Regards,
Ali
Re: Metron HBase conditional enrichment
Posted by Matt Foley <mf...@hortonworks.com>.
Hmm. I just refreshed a current copy of master, ran “mvn clean install –DskipTests”, and found:
java -cp metron-platform/metron-common/target/metron-common-0.4.0.jar org.apache.metron.common.stellar.shell.StellarShell
fails with “ClassNotFoundException: org.slf4j.LoggerFactory”, because this jar does not include the logging stuff. However,
java -cp metron-platform/metron-common/target/metron-common-0.4.0.jar:metron-platform/metron-enrichment/target/metron-enrichment-0.4.0-uber.jar org.apache.metron.common.stellar.shell.StellarShell
comes up fine, because the metron-enrichment uber-jar includes logging as well as a lot of other stuff. Of course, using the maven exec plugin as you suggest also “knows” how to gather the dependency jars. Using the full list:
java -cp metron-platform/metron-enrichment/target/metron-enrichment-0.4.0-uber.jar:metron-platform/metron-data-management/target/metron-data-management-0.4.0.jar:metron-platform/metron-common/target/metron-common-0.4.0.jar:metron-platform/metron-management/target/metron-management-0.4.0.jar:metron-platform/metron-parsers/target/metron-parsers-0.4.0-uber.jar org.apache.metron.common.stellar.shell.StellarShell
also worked fine for me.
Thanks,
--Matt
From: Nick Allen <ni...@nickallen.org>
Reply-To: "user@metron.apache.org" <us...@metron.apache.org>
Date: Thursday, May 25, 2017 at 10:58 AM
To: "user@metron.apache.org" <us...@metron.apache.org>
Subject: Re: Metron HBase conditional enrichment
Thanks, Matt. That command-line doesn't work for me; even after updating version number. Not sure why. This is what I tend to run after a build and install of Metron. Maybe something is different in my environment.
mvn clean install -DskipTests
mvn exec:java -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" -pl metron-platform/metron-common
For example...
$ mvn exec:java -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" -pl metron-platform/metron-common
...
Stellar, Go!
Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
[Stellar]>>> %functions
APPEND_IF_MISSING, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, PREPEND_IF_MISSING, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
[Stellar]>>>
On Thu, May 25, 2017 at 1:31 PM, Matt Foley <mf...@hortonworks.com>> wrote:
Hi Ali,
When writing Stellar statements, it is convenient to test them out in the REPL, which can be invoked via some variant of the commands in https://github.com/apache/metron/blob/master/metron-platform/metron-common/src/main/scripts/stellar , depending on the particular environment you’re working in . For instance, from the root of a git clone (that has already been successfully compiled), you can run:
java -cp metron-platform/metron-enrichment/target/metron-enrichment-0.3.1-uber.jar:metron-platform/metron-data-management/target/metron-data-management-0.3.1.jar:metron-platform/metron-common/target/metron-common-0.3.1.jar:metron-platform/metron-management/target/metron-management-0.3.1.jar:metron-platform/metron-parsers/target/metron-parsers-0.3.1-uber.jar org.apache.metron.common.stellar.shell.StellarShell
As you probably infer from the above, any jar that has @Stellar annotated classes has to be included in the cp, or those stellar operators won’t be available. The main jar is metron-common, which will give you the REPL and the “base” operator set, but many interesting ops are in those other jars. (If I missed one, just add it to the cp.)
Once in the REPL you can type `%functions` to get a list of all available stellar operators. Basically, any class annotated with @Stellar will automatically be loaded, both in the REPL and in the installed runtime environment.
Cheers,
--Matt
From: Otto Fowler <ot...@gmail.com>>
Reply-To: "user@metron.apache.org<ma...@metron.apache.org>" <us...@metron.apache.org>>
Date: Thursday, May 25, 2017 at 5:23 AM
To: Nick Allen <ni...@nickallen.org>>, "user@metron.apache.org<ma...@metron.apache.org>" <us...@metron.apache.org>>
Subject: Re: Metron HBase conditional enrichment
I think most of those restricted functions are in the metron-managment section.
On May 25, 2017 at 07:27:24, Nick Allen (nick@nickallen.org<ma...@nickallen.org>) wrote:
> everywhere I can use Stellar DSL, all of the functions have been implemented and ready to use?
Generally, yes, you are right.
I vaguely remember a couple instances of functions that are useful in the REPL only, but I cannot remember what those are right now. Hopefully we have those doc'd appropriately.
On Wed, May 24, 2017 at 10:38 PM, Ali Nazemian <al...@gmail.com>> wrote:
Hi Nick,
I was not sure about the implementation, so does it generally mean everywhere I can use Stellar DSL, all of the functions have been implemented and ready to use?
Cheers,
Ali
On Thu, May 25, 2017 at 2:52 AM, Nick Allen <ni...@nickallen.org>> wrote:
> can I do the concatenation on the fly at the enrichment level, so I don't need to store this temp field in Elasticsearch/HDFS.
Sure, absolutely.
> Moreover, I need to have a conditional enrichment to say if you couldn't find any match for "tenant_name+device_type+device_name" lookup for "tenant_name+device_type+default_device".
Yes, you can. You've got if/else, JOIN, IS_EMPTY, and others that should make implementing this logic pretty easy.
On Tue, May 23, 2017 at 10:34 PM, Ali Nazemian <al...@gmail.com>> wrote:
Hi,
I was wondering how I can manage Stellar syntax to be aligned with the following structure for the HBase enrichment:
HBase_row_key: tenant_name+device_type+device_name
At the high-level, I need to create a separate field via a post-parse Stellar function to be a concatenation of tenan_name, device_type and device_name. Let's call this field "key". Basically, I need to do the enrichment on the "key" which would be corresponding to the HBase row key. My first question is can I do the concatenation on the fly at the enrichment level, so I don't need to store this temp field in Elasticsearch/HDFS.
Moreover, I need to have a conditional enrichment to say if you couldn't find any match for "tenant_name+device_type+device_name" lookup for "tenant_name+device_type+default_device". The second question would be how can I manage conditional enrichment like this one. I would be really grateful if you can provide some example.
Regards,
Ali
--
A.Nazemian
Re: Metron HBase conditional enrichment
Posted by Matt Foley <mf...@hortonworks.com>.
That sheds light. Thanks, Nick.
--Matt
From: Nick Allen <ni...@nickallen.org>
Reply-To: "user@metron.apache.org" <us...@metron.apache.org>
Date: Thursday, May 25, 2017 at 3:06 PM
To: "user@metron.apache.org" <us...@metron.apache.org>
Subject: Re: Metron HBase conditional enrichment
Each topology has its own uber-jar that is built from all of it's dependencies. It's classpath is basically whatever is in the uber-jar.
That's why running with -pl against the project from which the uber jar is built should identify the Stellar functions available to each topology.
When running the REPL from a deployed instance of Metron it pulls in all of the jars deployed to /usr/metron/(version)/lib.
On May 25, 2017 3:29 PM, "Matt Foley" <mf...@hortonworks.com>> wrote:
Nick is correct that in any given environment, only Stellar functions defined in jars on the current classpath will be available.
When running with the maven exec:java plugin, as below, this means only jars declared (or transitively required) as dependencies to the given project.
In the installed environment, however, it is strictly a matter of the configured classpath. I thought (I could be wrong), that all the metron jars are installed together (in /usr/metron/<version>/lib/* , in CentOS 7), and that all those jars will be added to the classpath for all topologies.
Nick, you’re much more familiar with the install stuff than I, please clarify if you can how the classpath is configured for the running topologies in the installed env.
At any rate, it is the classpath currently being run under that determines the availability of Stellar functions.
Thanks,
--Matt
From: Nick Allen <ni...@nickallen.org>>
Reply-To: "user@metron.apache.org<ma...@metron.apache.org>" <us...@metron.apache.org>>
Date: Thursday, May 25, 2017 at 12:05 PM
To: "user@metron.apache.org<ma...@metron.apache.org>" <us...@metron.apache.org>>
Subject: Re: Metron HBase conditional enrichment
Correct me if I am wrong, Matt, but I believe that changing the project that you pass to the -pl switch will allow you to see exactly what Stellar functions would be available in each topology. You just have to refer to whichever project drives the topology.
This might help answer Ali's previous question as to what functions are available where. And of course, if some function is not available where it is needed, then it is a simply matter of changing the dependencies to make it available.
For example, these functions are available from the Profiler.
$ mvn exec:java \
-Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \
-pl metron-analytics/metron-profiler
...
Stellar, Go!
Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
%functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
And these functions are available in the Enrichment topology.
$ mvn exec:java \
-Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \
-pl metron-platform/metron-enrichment/
...
Stellar, Go!
Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
[Stellar]>>> %functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GEO_GET, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
For example, I can see from this that `GEO_GET` during Enrichment, but is not available in the Profiler right now. This makes sense because `GEO_GET` is defined in metron-enrichment.
On Thu, May 25, 2017 at 1:58 PM, Nick Allen <ni...@nickallen.org>> wrote:
Thanks, Matt. That command-line doesn't work for me; even after updating version number. Not sure why. This is what I tend to run after a build and install of Metron. Maybe something is different in my environment.
mvn clean install -DskipTests
mvn exec:java -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" -pl metron-platform/metron-common
For example...
$ mvn exec:java -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" -pl metron-platform/metron-common
...
Stellar, Go!
Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
[Stellar]>>> %functions
APPEND_IF_MISSING, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, PREPEND_IF_MISSING, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
[Stellar]>>>
On Thu, May 25, 2017 at 1:31 PM, Matt Foley <mf...@hortonworks.com>> wrote:
Hi Ali,
When writing Stellar statements, it is convenient to test them out in the REPL, which can be invoked via some variant of the commands in https://github.com/apache/metron/blob/master/metron-platform/metron-common/src/main/scripts/stellar , depending on the particular environment you’re working in . For instance, from the root of a git clone (that has already been successfully compiled), you can run:
java -cp metron-platform/metron-enrichment/target/metron-enrichment-0.3.1-uber.jar:metron-platform/metron-data-management/target/metron-data-management-0.3.1.jar:metron-platform/metron-common/target/metron-common-0.3.1.jar:metron-platform/metron-management/target/metron-management-0.3.1.jar:metron-platform/metron-parsers/target/metron-parsers-0.3.1-uber.jar org.apache.metron.common.stellar.shell.StellarShell
As you probably infer from the above, any jar that has @Stellar annotated classes has to be included in the cp, or those stellar operators won’t be available. The main jar is metron-common, which will give you the REPL and the “base” operator set, but many interesting ops are in those other jars. (If I missed one, just add it to the cp.)
Once in the REPL you can type `%functions` to get a list of all available stellar operators. Basically, any class annotated with @Stellar will automatically be loaded, both in the REPL and in the installed runtime environment.
Cheers,
--Matt
From: Otto Fowler <ot...@gmail.com>>
Reply-To: "user@metron.apache.org<ma...@metron.apache.org>" <us...@metron.apache.org>>
Date: Thursday, May 25, 2017 at 5:23 AM
To: Nick Allen <ni...@nickallen.org>>, "user@metron.apache.org<ma...@metron.apache.org>" <us...@metron.apache.org>>
Subject: Re: Metron HBase conditional enrichment
I think most of those restricted functions are in the metron-managment section.
On May 25, 2017 at 07:27:24, Nick Allen (nick@nickallen.org<ma...@nickallen.org>) wrote:
> everywhere I can use Stellar DSL, all of the functions have been implemented and ready to use?
Generally, yes, you are right.
I vaguely remember a couple instances of functions that are useful in the REPL only, but I cannot remember what those are right now. Hopefully we have those doc'd appropriately.
On Wed, May 24, 2017 at 10:38 PM, Ali Nazemian <al...@gmail.com>> wrote:
Hi Nick,
I was not sure about the implementation, so does it generally mean everywhere I can use Stellar DSL, all of the functions have been implemented and ready to use?
Cheers,
Ali
On Thu, May 25, 2017 at 2:52 AM, Nick Allen <ni...@nickallen.org>> wrote:
> can I do the concatenation on the fly at the enrichment level, so I don't need to store this temp field in Elasticsearch/HDFS.
Sure, absolutely.
> Moreover, I need to have a conditional enrichment to say if you couldn't find any match for "tenant_name+device_type+device_name" lookup for "tenant_name+device_type+default_device".
Yes, you can. You've got if/else, JOIN, IS_EMPTY, and others that should make implementing this logic pretty easy.
On Tue, May 23, 2017 at 10:34 PM, Ali Nazemian <al...@gmail.com>> wrote:
Hi,
I was wondering how I can manage Stellar syntax to be aligned with the following structure for the HBase enrichment:
HBase_row_key: tenant_name+device_type+device_name
At the high-level, I need to create a separate field via a post-parse Stellar function to be a concatenation of tenan_name, device_type and device_name. Let's call this field "key". Basically, I need to do the enrichment on the "key" which would be corresponding to the HBase row key. My first question is can I do the concatenation on the fly at the enrichment level, so I don't need to store this temp field in Elasticsearch/HDFS.
Moreover, I need to have a conditional enrichment to say if you couldn't find any match for "tenant_name+device_type+device_name" lookup for "tenant_name+device_type+default_device". The second question would be how can I manage conditional enrichment like this one. I would be really grateful if you can provide some example.
Regards,
Ali
--
A.Nazemian
Re: Metron HBase conditional enrichment
Posted by Nick Allen <ni...@nickallen.org>.
Each topology has its own uber-jar that is built from all of it's
dependencies. It's classpath is basically whatever is in the uber-jar.
That's why running with -pl against the project from which the uber jar is
built should identify the Stellar functions available to each topology.
When running the REPL from a deployed instance of Metron it pulls in all of
the jars deployed to /usr/metron/(version)/lib.
On May 25, 2017 3:29 PM, "Matt Foley" <mf...@hortonworks.com> wrote:
Nick is correct that in any given environment, only Stellar functions
defined in jars on the current classpath will be available.
When running with the maven exec:java plugin, as below, this means only
jars declared (or transitively required) as dependencies to the given
project.
In the installed environment, however, it is strictly a matter of the
configured classpath. I thought (I could be wrong), that all the metron
jars are installed together (in /usr/metron/<version>/lib/* , in CentOS 7),
and that all those jars will be added to the classpath for all topologies.
Nick, you’re much more familiar with the install stuff than I, please
clarify if you can how the classpath is configured for the running
topologies in the installed env.
At any rate, it is the classpath currently being run under that determines
the availability of Stellar functions.
Thanks,
--Matt
*From: *Nick Allen <ni...@nickallen.org>
*Reply-To: *"user@metron.apache.org" <us...@metron.apache.org>
*Date: *Thursday, May 25, 2017 at 12:05 PM
*To: *"user@metron.apache.org" <us...@metron.apache.org>
*Subject: *Re: Metron HBase conditional enrichment
Correct me if I am wrong, Matt, but I believe that changing the project
that you pass to the -pl switch will allow you to see exactly what Stellar
functions would be available in each topology. You just have to refer to
whichever project drives the topology.
This might help answer Ali's previous question as to what functions are
available where. And of course, if some function is not available where it
is needed, then it is a simply matter of changing the dependencies to make
it available.
For example, these functions are available from the Profiler.
$ mvn exec:java \
-Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \
-pl metron-analytics/metron-profiler
...
Stellar, Go!
Please note that functions are loading lazily in the background and will be
unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
%functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT,
BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK,
DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD,
ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST,
HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE,
IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH,
LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET,
MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE,
PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW,
PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD,
STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS,
STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE,
STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS,
STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE,
STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE,
TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING,
TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL,
WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
And these functions are available in the Enrichment topology.
$ mvn exec:java \
-Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \
-pl metron-platform/metron-enrichment/
...
Stellar, Go!
Please note that functions are loading lazily in the background and will be
unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
[Stellar]>>> %functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT,
BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK,
DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD,
ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, FILL_LEFT, FILL_RIGHT,
FILTER, FORMAT, GEO_GET, GET, GET_FIRST, GET_LAST, HLLP_ADD,
HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN,
IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD,
MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH,
OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE,
PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW,
PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD,
STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS,
STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE,
STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS,
STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE,
STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE,
TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING,
TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL,
WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
For example, I can see from this that `GEO_GET` during Enrichment, but is
not available in the Profiler right now. This makes sense because
`GEO_GET` is defined in metron-enrichment.
On Thu, May 25, 2017 at 1:58 PM, Nick Allen <ni...@nickallen.org> wrote:
Thanks, Matt. That command-line doesn't work for me; even after updating
version number. Not sure why. This is what I tend to run after a build
and install of Metron. Maybe something is different in my environment.
mvn clean install -DskipTests
mvn exec:java -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell"
-pl metron-platform/metron-common
For example...
$ mvn exec:java
-Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell"
-pl metron-platform/metron-common
...
Stellar, Go!
Please note that functions are loading lazily in the background and will be
unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
[Stellar]>>> %functions
APPEND_IF_MISSING, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP,
CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR,
DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH,
FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, IN_SUBNET,
IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN,
LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS,
MAP_GET, MONTH, PREPEND_IF_MISSING, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH,
SPLIT, STARTS_WITH, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET,
TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER,
TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT,
URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
[Stellar]>>>
On Thu, May 25, 2017 at 1:31 PM, Matt Foley <mf...@hortonworks.com> wrote:
Hi Ali,
When writing Stellar statements, it is convenient to test them out in the
REPL, which can be invoked via some variant of the commands in
https://github.com/apache/metron/blob/master/metron-
platform/metron-common/src/main/scripts/stellar , depending on the
particular environment you’re working in . For instance, from the root of
a git clone (that has already been successfully compiled), you can run:
java -cp metron-platform/metron-enrichment/target/metron-
enrichment-0.3.1-uber.jar:metron-platform/metron-data-
management/target/metron-data-management-0.3.1.jar:metron-
platform/metron-common/target/metron-common-0.3.1.jar:
metron-platform/metron-management/target/metron-management-0.3.1.jar:metron-
platform/metron-parsers/target/metron-parsers-0.3.1-uber.jar
org.apache.metron.common.stellar.shell.StellarShell
As you probably infer from the above, any jar that has @Stellar annotated
classes has to be included in the cp, or those stellar operators won’t be
available. The main jar is metron-common, which will give you the REPL and
the “base” operator set, but many interesting ops are in those other jars.
(If I missed one, just add it to the cp.)
Once in the REPL you can type `%functions` to get a list of all available
stellar operators. Basically, any class annotated with @Stellar will
automatically be loaded, both in the REPL and in the installed runtime
environment.
Cheers,
--Matt
*From: *Otto Fowler <ot...@gmail.com>
*Reply-To: *"user@metron.apache.org" <us...@metron.apache.org>
*Date: *Thursday, May 25, 2017 at 5:23 AM
*To: *Nick Allen <ni...@nickallen.org>, "user@metron.apache.org" <
user@metron.apache.org>
*Subject: *Re: Metron HBase conditional enrichment
I think most of those restricted functions are in the metron-managment
section.
On May 25, 2017 at 07:27:24, Nick Allen (nick@nickallen.org) wrote:
> everywhere I can use Stellar DSL, all of the functions have been
implemented and ready to use?
Generally, yes, you are right.
I vaguely remember a couple instances of functions that are useful in the
REPL only, but I cannot remember what those are right now. Hopefully we
have those doc'd appropriately.
On Wed, May 24, 2017 at 10:38 PM, Ali Nazemian <al...@gmail.com>
wrote:
Hi Nick,
I was not sure about the implementation, so does it generally mean
everywhere I can use Stellar DSL, all of the functions have been
implemented and ready to use?
Cheers,
Ali
On Thu, May 25, 2017 at 2:52 AM, Nick Allen <ni...@nickallen.org> wrote:
> can I do the concatenation on the fly at the enrichment level, so I don't
need to store this temp field in Elasticsearch/HDFS.
Sure, absolutely.
> Moreover, I need to have a conditional enrichment to say if you couldn't
find any match for "tenant_name+device_type+device_name" lookup for
"tenant_name+device_type+default_device".
Yes, you can. You've got if/else, JOIN, IS_EMPTY, and others that should
make implementing this logic pretty easy.
On Tue, May 23, 2017 at 10:34 PM, Ali Nazemian <al...@gmail.com>
wrote:
Hi,
I was wondering how I can manage Stellar syntax to be aligned with the
following structure for the HBase enrichment:
HBase_row_key: tenant_name+device_type+device_name
At the high-level, I need to create a separate field via a post-parse
Stellar function to be a concatenation of tenan_name, device_type and
device_name. Let's call this field "key". Basically, I need to do the
enrichment on the "key" which would be corresponding to the HBase row key.
My first question is *can I do the concatenation on the fly at the
enrichment level, so I don't need to store this temp field in
Elasticsearch/HDFS*.
Moreover, I need to have a conditional enrichment to say if you couldn't
find any match for "tenant_name+device_type+device_name" lookup for
"tenant_name+device_type+default_device". The second question would be *how
can I manage conditional enrichment like this one*. I would be really
grateful if you can provide some example.
Regards,
Ali
--
A.Nazemian
Re: Metron HBase conditional enrichment
Posted by Matt Foley <mf...@hortonworks.com>.
Nick is correct that in any given environment, only Stellar functions defined in jars on the current classpath will be available.
When running with the maven exec:java plugin, as below, this means only jars declared (or transitively required) as dependencies to the given project.
In the installed environment, however, it is strictly a matter of the configured classpath. I thought (I could be wrong), that all the metron jars are installed together (in /usr/metron/<version>/lib/* , in CentOS 7), and that all those jars will be added to the classpath for all topologies.
Nick, you’re much more familiar with the install stuff than I, please clarify if you can how the classpath is configured for the running topologies in the installed env.
At any rate, it is the classpath currently being run under that determines the availability of Stellar functions.
Thanks,
--Matt
From: Nick Allen <ni...@nickallen.org>
Reply-To: "user@metron.apache.org" <us...@metron.apache.org>
Date: Thursday, May 25, 2017 at 12:05 PM
To: "user@metron.apache.org" <us...@metron.apache.org>
Subject: Re: Metron HBase conditional enrichment
Correct me if I am wrong, Matt, but I believe that changing the project that you pass to the -pl switch will allow you to see exactly what Stellar functions would be available in each topology. You just have to refer to whichever project drives the topology.
This might help answer Ali's previous question as to what functions are available where. And of course, if some function is not available where it is needed, then it is a simply matter of changing the dependencies to make it available.
For example, these functions are available from the Profiler.
$ mvn exec:java \
-Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \
-pl metron-analytics/metron-profiler
...
Stellar, Go!
Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
%functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
And these functions are available in the Enrichment topology.
$ mvn exec:java \
-Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \
-pl metron-platform/metron-enrichment/
...
Stellar, Go!
Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
[Stellar]>>> %functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GEO_GET, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
For example, I can see from this that `GEO_GET` during Enrichment, but is not available in the Profiler right now. This makes sense because `GEO_GET` is defined in metron-enrichment.
On Thu, May 25, 2017 at 1:58 PM, Nick Allen <ni...@nickallen.org>> wrote:
Thanks, Matt. That command-line doesn't work for me; even after updating version number. Not sure why. This is what I tend to run after a build and install of Metron. Maybe something is different in my environment.
mvn clean install -DskipTests
mvn exec:java -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" -pl metron-platform/metron-common
For example...
$ mvn exec:java -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" -pl metron-platform/metron-common
...
Stellar, Go!
Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
[Stellar]>>> %functions
APPEND_IF_MISSING, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, PREPEND_IF_MISSING, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
[Stellar]>>>
On Thu, May 25, 2017 at 1:31 PM, Matt Foley <mf...@hortonworks.com>> wrote:
Hi Ali,
When writing Stellar statements, it is convenient to test them out in the REPL, which can be invoked via some variant of the commands in https://github.com/apache/metron/blob/master/metron-platform/metron-common/src/main/scripts/stellar , depending on the particular environment you’re working in . For instance, from the root of a git clone (that has already been successfully compiled), you can run:
java -cp metron-platform/metron-enrichment/target/metron-enrichment-0.3.1-uber.jar:metron-platform/metron-data-management/target/metron-data-management-0.3.1.jar:metron-platform/metron-common/target/metron-common-0.3.1.jar:metron-platform/metron-management/target/metron-management-0.3.1.jar:metron-platform/metron-parsers/target/metron-parsers-0.3.1-uber.jar org.apache.metron.common.stellar.shell.StellarShell
As you probably infer from the above, any jar that has @Stellar annotated classes has to be included in the cp, or those stellar operators won’t be available. The main jar is metron-common, which will give you the REPL and the “base” operator set, but many interesting ops are in those other jars. (If I missed one, just add it to the cp.)
Once in the REPL you can type `%functions` to get a list of all available stellar operators. Basically, any class annotated with @Stellar will automatically be loaded, both in the REPL and in the installed runtime environment.
Cheers,
--Matt
From: Otto Fowler <ot...@gmail.com>>
Reply-To: "user@metron.apache.org<ma...@metron.apache.org>" <us...@metron.apache.org>>
Date: Thursday, May 25, 2017 at 5:23 AM
To: Nick Allen <ni...@nickallen.org>>, "user@metron.apache.org<ma...@metron.apache.org>" <us...@metron.apache.org>>
Subject: Re: Metron HBase conditional enrichment
I think most of those restricted functions are in the metron-managment section.
On May 25, 2017 at 07:27:24, Nick Allen (nick@nickallen.org<ma...@nickallen.org>) wrote:
> everywhere I can use Stellar DSL, all of the functions have been implemented and ready to use?
Generally, yes, you are right.
I vaguely remember a couple instances of functions that are useful in the REPL only, but I cannot remember what those are right now. Hopefully we have those doc'd appropriately.
On Wed, May 24, 2017 at 10:38 PM, Ali Nazemian <al...@gmail.com>> wrote:
Hi Nick,
I was not sure about the implementation, so does it generally mean everywhere I can use Stellar DSL, all of the functions have been implemented and ready to use?
Cheers,
Ali
On Thu, May 25, 2017 at 2:52 AM, Nick Allen <ni...@nickallen.org>> wrote:
> can I do the concatenation on the fly at the enrichment level, so I don't need to store this temp field in Elasticsearch/HDFS.
Sure, absolutely.
> Moreover, I need to have a conditional enrichment to say if you couldn't find any match for "tenant_name+device_type+device_name" lookup for "tenant_name+device_type+default_device".
Yes, you can. You've got if/else, JOIN, IS_EMPTY, and others that should make implementing this logic pretty easy.
On Tue, May 23, 2017 at 10:34 PM, Ali Nazemian <al...@gmail.com>> wrote:
Hi,
I was wondering how I can manage Stellar syntax to be aligned with the following structure for the HBase enrichment:
HBase_row_key: tenant_name+device_type+device_name
At the high-level, I need to create a separate field via a post-parse Stellar function to be a concatenation of tenan_name, device_type and device_name. Let's call this field "key". Basically, I need to do the enrichment on the "key" which would be corresponding to the HBase row key. My first question is can I do the concatenation on the fly at the enrichment level, so I don't need to store this temp field in Elasticsearch/HDFS.
Moreover, I need to have a conditional enrichment to say if you couldn't find any match for "tenant_name+device_type+device_name" lookup for "tenant_name+device_type+default_device". The second question would be how can I manage conditional enrichment like this one. I would be really grateful if you can provide some example.
Regards,
Ali
--
A.Nazemian
Re: Metron HBase conditional enrichment
Posted by Nick Allen <ni...@nickallen.org>.
Correct me if I am wrong, Matt, but I believe that changing the project
that you pass to the -pl switch will allow you to see exactly what Stellar
functions would be available in each topology. You just have to refer to
whichever project drives the topology.
This might help answer Ali's previous question as to what functions are
available where. And of course, if some function is not available where it
is needed, then it is a simply matter of changing the dependencies to make
it available.
For example, these functions are available from the Profiler.
$ mvn exec:java \
-Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \
-pl metron-analytics/metron-profiler
...
Stellar, Go!
Please note that functions are loading lazily in the background and will be
unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
%functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT,
BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK,
DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD,
ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST,
HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE,
IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH,
LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET,
MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE,
PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW,
PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD,
STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS,
STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE,
STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS,
STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE,
STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE,
TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING,
TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL,
WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
And these functions are available in the Enrichment topology.
$ mvn exec:java \
-Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \
-pl metron-platform/metron-enrichment/
...
Stellar, Go!
Please note that functions are loading lazily in the background and will be
unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
[Stellar]>>> %functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT,
BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK,
DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD,
ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, FILL_LEFT, FILL_RIGHT,
FILTER, FORMAT, GEO_GET, GET, GET_FIRST, GET_LAST, HLLP_ADD,
HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN,
IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD,
MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH,
OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE,
PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW,
PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD,
STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS,
STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE,
STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS,
STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE,
STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE,
TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING,
TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL,
WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
For example, I can see from this that `GEO_GET` during Enrichment, but is
not available in the Profiler right now. This makes sense because
`GEO_GET` is defined in metron-enrichment.
On Thu, May 25, 2017 at 1:58 PM, Nick Allen <ni...@nickallen.org> wrote:
> Thanks, Matt. That command-line doesn't work for me; even after updating
> version number. Not sure why. This is what I tend to run after a build
> and install of Metron. Maybe something is different in my environment.
>
> mvn clean install -DskipTests
>
>
> mvn exec:java -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell"
> -pl metron-platform/metron-common
>
>
> For example...
>
> $ mvn exec:java -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell"
> -pl metron-platform/metron-common
> ...
>
> Stellar, Go!
> Please note that functions are loading lazily in the background and will
> be unavailable until loaded fully.
> [Stellar]>>> Functions loaded, you may refer to functions now...
>
> [Stellar]>>> %functions
> APPEND_IF_MISSING, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE,
> CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR,
> DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH,
> FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, IN_SUBNET,
> IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN,
> LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS,
> MAP_GET, MONTH, PREPEND_IF_MISSING, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH,
> SPLIT, STARTS_WITH, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET,
> TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER,
> TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT,
> URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
> [Stellar]>>>
>
>
> On Thu, May 25, 2017 at 1:31 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
>
>> Hi Ali,
>>
>> When writing Stellar statements, it is convenient to test them out in the
>> REPL, which can be invoked via some variant of the commands in
>> https://github.com/apache/metron/blob/master/metron-platform
>> /metron-common/src/main/scripts/stellar , depending on the particular
>> environment you’re working in . For instance, from the root of a git clone
>> (that has already been successfully compiled), you can run:
>>
>>
>>
>> java -cp metron-platform/metron-enrichment/target/metron-enrichment-
>> 0.3.1-uber.jar:metron-platform/metron-data-managemen
>> t/target/metron-data-management-0.3.1.jar:metron-platform/
>> metron-common/target/metron-common-0.3.1.jar:metron-
>> platform/metron-management/target/metron-management-0.3.
>> 1.jar:metron-platform/metron-parsers/target/metron-parsers-0.3.1-uber.jar
>> org.apache.metron.common.stellar.shell.StellarShell
>>
>>
>>
>> As you probably infer from the above, any jar that has @Stellar annotated
>> classes has to be included in the cp, or those stellar operators won’t be
>> available. The main jar is metron-common, which will give you the REPL and
>> the “base” operator set, but many interesting ops are in those other jars.
>> (If I missed one, just add it to the cp.)
>>
>>
>>
>> Once in the REPL you can type `%functions` to get a list of all available
>> stellar operators. Basically, any class annotated with @Stellar will
>> automatically be loaded, both in the REPL and in the installed runtime
>> environment.
>>
>>
>>
>> Cheers,
>>
>> --Matt
>>
>>
>>
>> *From: *Otto Fowler <ot...@gmail.com>
>> *Reply-To: *"user@metron.apache.org" <us...@metron.apache.org>
>> *Date: *Thursday, May 25, 2017 at 5:23 AM
>> *To: *Nick Allen <ni...@nickallen.org>, "user@metron.apache.org" <
>> user@metron.apache.org>
>> *Subject: *Re: Metron HBase conditional enrichment
>>
>>
>>
>> I think most of those restricted functions are in the metron-managment
>> section.
>>
>>
>>
>>
>>
>> On May 25, 2017 at 07:27:24, Nick Allen (nick@nickallen.org) wrote:
>>
>> > everywhere I can use Stellar DSL, all of the functions have been
>> implemented and ready to use?
>>
>>
>>
>> Generally, yes, you are right.
>>
>>
>>
>> I vaguely remember a couple instances of functions that are useful in the
>> REPL only, but I cannot remember what those are right now. Hopefully we
>> have those doc'd appropriately.
>>
>>
>>
>>
>>
>>
>>
>> On Wed, May 24, 2017 at 10:38 PM, Ali Nazemian <al...@gmail.com>
>> wrote:
>>
>> Hi Nick,
>>
>>
>>
>> I was not sure about the implementation, so does it generally mean
>> everywhere I can use Stellar DSL, all of the functions have been
>> implemented and ready to use?
>>
>>
>>
>> Cheers,
>>
>> Ali
>>
>>
>>
>> On Thu, May 25, 2017 at 2:52 AM, Nick Allen <ni...@nickallen.org> wrote:
>>
>> > can I do the concatenation on the fly at the enrichment level, so I
>> don't need to store this temp field in Elasticsearch/HDFS.
>>
>>
>>
>> Sure, absolutely.
>>
>>
>>
>> > Moreover, I need to have a conditional enrichment to say if you
>> couldn't find any match for "tenant_name+device_type+device_name" lookup
>> for "tenant_name+device_type+default_device".
>>
>>
>>
>> Yes, you can. You've got if/else, JOIN, IS_EMPTY, and others that should
>> make implementing this logic pretty easy.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, May 23, 2017 at 10:34 PM, Ali Nazemian <al...@gmail.com>
>> wrote:
>>
>> Hi,
>>
>>
>>
>> I was wondering how I can manage Stellar syntax to be aligned with the
>> following structure for the HBase enrichment:
>>
>>
>>
>> HBase_row_key: tenant_name+device_type+device_name
>>
>>
>>
>> At the high-level, I need to create a separate field via a post-parse
>> Stellar function to be a concatenation of tenan_name, device_type and
>> device_name. Let's call this field "key". Basically, I need to do the
>> enrichment on the "key" which would be corresponding to the HBase row key.
>> My first question is *can I do the concatenation on the fly at the
>> enrichment level, so I don't need to store this temp field in
>> Elasticsearch/HDFS*.
>>
>>
>>
>> Moreover, I need to have a conditional enrichment to say if you couldn't
>> find any match for "tenant_name+device_type+device_name" lookup for
>> "tenant_name+device_type+default_device". The second question would be *how
>> can I manage conditional enrichment like this one*. I would be really
>> grateful if you can provide some example.
>>
>>
>>
>> Regards,
>>
>> Ali
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> A.Nazemian
>>
>>
>>
>>
>
Re: Metron HBase conditional enrichment
Posted by Nick Allen <ni...@nickallen.org>.
Thanks, Matt. That command-line doesn't work for me; even after updating
version number. Not sure why. This is what I tend to run after a build
and install of Metron. Maybe something is different in my environment.
mvn clean install -DskipTests
mvn exec:java
-Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" -pl
metron-platform/metron-common
For example...
$ mvn exec:java
-Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" -pl
metron-platform/metron-common
...
Stellar, Go!
Please note that functions are loading lazily in the background and will be
unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
[Stellar]>>> %functions
APPEND_IF_MISSING, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP,
CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR,
DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH,
FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, IN_SUBNET,
IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN,
LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS,
MAP_GET, MONTH, PREPEND_IF_MISSING, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH,
SPLIT, STARTS_WITH, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET,
TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER,
TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT,
URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
[Stellar]>>>
On Thu, May 25, 2017 at 1:31 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Hi Ali,
>
> When writing Stellar statements, it is convenient to test them out in the
> REPL, which can be invoked via some variant of the commands in
> https://github.com/apache/metron/blob/master/metron-
> platform/metron-common/src/main/scripts/stellar , depending on the
> particular environment you’re working in . For instance, from the root of
> a git clone (that has already been successfully compiled), you can run:
>
>
>
> java -cp metron-platform/metron-enrichment/target/metron-
> enrichment-0.3.1-uber.jar:metron-platform/metron-data-
> management/target/metron-data-management-0.3.1.jar:metron-
> platform/metron-common/target/metron-common-0.3.1.jar:
> metron-platform/metron-management/target/metron-
> management-0.3.1.jar:metron-platform/metron-parsers/
> target/metron-parsers-0.3.1-uber.jar org.apache.metron.common.
> stellar.shell.StellarShell
>
>
>
> As you probably infer from the above, any jar that has @Stellar annotated
> classes has to be included in the cp, or those stellar operators won’t be
> available. The main jar is metron-common, which will give you the REPL and
> the “base” operator set, but many interesting ops are in those other jars.
> (If I missed one, just add it to the cp.)
>
>
>
> Once in the REPL you can type `%functions` to get a list of all available
> stellar operators. Basically, any class annotated with @Stellar will
> automatically be loaded, both in the REPL and in the installed runtime
> environment.
>
>
>
> Cheers,
>
> --Matt
>
>
>
> *From: *Otto Fowler <ot...@gmail.com>
> *Reply-To: *"user@metron.apache.org" <us...@metron.apache.org>
> *Date: *Thursday, May 25, 2017 at 5:23 AM
> *To: *Nick Allen <ni...@nickallen.org>, "user@metron.apache.org" <
> user@metron.apache.org>
> *Subject: *Re: Metron HBase conditional enrichment
>
>
>
> I think most of those restricted functions are in the metron-managment
> section.
>
>
>
>
>
> On May 25, 2017 at 07:27:24, Nick Allen (nick@nickallen.org) wrote:
>
> > everywhere I can use Stellar DSL, all of the functions have been
> implemented and ready to use?
>
>
>
> Generally, yes, you are right.
>
>
>
> I vaguely remember a couple instances of functions that are useful in the
> REPL only, but I cannot remember what those are right now. Hopefully we
> have those doc'd appropriately.
>
>
>
>
>
>
>
> On Wed, May 24, 2017 at 10:38 PM, Ali Nazemian <al...@gmail.com>
> wrote:
>
> Hi Nick,
>
>
>
> I was not sure about the implementation, so does it generally mean
> everywhere I can use Stellar DSL, all of the functions have been
> implemented and ready to use?
>
>
>
> Cheers,
>
> Ali
>
>
>
> On Thu, May 25, 2017 at 2:52 AM, Nick Allen <ni...@nickallen.org> wrote:
>
> > can I do the concatenation on the fly at the enrichment level, so I
> don't need to store this temp field in Elasticsearch/HDFS.
>
>
>
> Sure, absolutely.
>
>
>
> > Moreover, I need to have a conditional enrichment to say if you couldn't
> find any match for "tenant_name+device_type+device_name" lookup for
> "tenant_name+device_type+default_device".
>
>
>
> Yes, you can. You've got if/else, JOIN, IS_EMPTY, and others that should
> make implementing this logic pretty easy.
>
>
>
>
>
>
>
>
>
> On Tue, May 23, 2017 at 10:34 PM, Ali Nazemian <al...@gmail.com>
> wrote:
>
> Hi,
>
>
>
> I was wondering how I can manage Stellar syntax to be aligned with the
> following structure for the HBase enrichment:
>
>
>
> HBase_row_key: tenant_name+device_type+device_name
>
>
>
> At the high-level, I need to create a separate field via a post-parse
> Stellar function to be a concatenation of tenan_name, device_type and
> device_name. Let's call this field "key". Basically, I need to do the
> enrichment on the "key" which would be corresponding to the HBase row key.
> My first question is *can I do the concatenation on the fly at the
> enrichment level, so I don't need to store this temp field in
> Elasticsearch/HDFS*.
>
>
>
> Moreover, I need to have a conditional enrichment to say if you couldn't
> find any match for "tenant_name+device_type+device_name" lookup for
> "tenant_name+device_type+default_device". The second question would be *how
> can I manage conditional enrichment like this one*. I would be really
> grateful if you can provide some example.
>
>
>
> Regards,
>
> Ali
>
>
>
>
>
>
>
> --
>
> A.Nazemian
>
>
>
>
Re: Metron HBase conditional enrichment
Posted by Matt Foley <mf...@hortonworks.com>.
Hi Ali,
When writing Stellar statements, it is convenient to test them out in the REPL, which can be invoked via some variant of the commands in https://github.com/apache/metron/blob/master/metron-platform/metron-common/src/main/scripts/stellar , depending on the particular environment you’re working in . For instance, from the root of a git clone (that has already been successfully compiled), you can run:
java -cp metron-platform/metron-enrichment/target/metron-enrichment-0.3.1-uber.jar:metron-platform/metron-data-management/target/metron-data-management-0.3.1.jar:metron-platform/metron-common/target/metron-common-0.3.1.jar:metron-platform/metron-management/target/metron-management-0.3.1.jar:metron-platform/metron-parsers/target/metron-parsers-0.3.1-uber.jar org.apache.metron.common.stellar.shell.StellarShell
As you probably infer from the above, any jar that has @Stellar annotated classes has to be included in the cp, or those stellar operators won’t be available. The main jar is metron-common, which will give you the REPL and the “base” operator set, but many interesting ops are in those other jars. (If I missed one, just add it to the cp.)
Once in the REPL you can type `%functions` to get a list of all available stellar operators. Basically, any class annotated with @Stellar will automatically be loaded, both in the REPL and in the installed runtime environment.
Cheers,
--Matt
From: Otto Fowler <ot...@gmail.com>
Reply-To: "user@metron.apache.org" <us...@metron.apache.org>
Date: Thursday, May 25, 2017 at 5:23 AM
To: Nick Allen <ni...@nickallen.org>, "user@metron.apache.org" <us...@metron.apache.org>
Subject: Re: Metron HBase conditional enrichment
I think most of those restricted functions are in the metron-managment section.
On May 25, 2017 at 07:27:24, Nick Allen (nick@nickallen.org<ma...@nickallen.org>) wrote:
> everywhere I can use Stellar DSL, all of the functions have been implemented and ready to use?
Generally, yes, you are right.
I vaguely remember a couple instances of functions that are useful in the REPL only, but I cannot remember what those are right now. Hopefully we have those doc'd appropriately.
On Wed, May 24, 2017 at 10:38 PM, Ali Nazemian <al...@gmail.com>> wrote:
Hi Nick,
I was not sure about the implementation, so does it generally mean everywhere I can use Stellar DSL, all of the functions have been implemented and ready to use?
Cheers,
Ali
On Thu, May 25, 2017 at 2:52 AM, Nick Allen <ni...@nickallen.org>> wrote:
> can I do the concatenation on the fly at the enrichment level, so I don't need to store this temp field in Elasticsearch/HDFS.
Sure, absolutely.
> Moreover, I need to have a conditional enrichment to say if you couldn't find any match for "tenant_name+device_type+device_name" lookup for "tenant_name+device_type+default_device".
Yes, you can. You've got if/else, JOIN, IS_EMPTY, and others that should make implementing this logic pretty easy.
On Tue, May 23, 2017 at 10:34 PM, Ali Nazemian <al...@gmail.com>> wrote:
Hi,
I was wondering how I can manage Stellar syntax to be aligned with the following structure for the HBase enrichment:
HBase_row_key: tenant_name+device_type+device_name
At the high-level, I need to create a separate field via a post-parse Stellar function to be a concatenation of tenan_name, device_type and device_name. Let's call this field "key". Basically, I need to do the enrichment on the "key" which would be corresponding to the HBase row key. My first question is can I do the concatenation on the fly at the enrichment level, so I don't need to store this temp field in Elasticsearch/HDFS.
Moreover, I need to have a conditional enrichment to say if you couldn't find any match for "tenant_name+device_type+device_name" lookup for "tenant_name+device_type+default_device". The second question would be how can I manage conditional enrichment like this one. I would be really grateful if you can provide some example.
Regards,
Ali
--
A.Nazemian
Re: Metron HBase conditional enrichment
Posted by Otto Fowler <ot...@gmail.com>.
I think most of those restricted functions are in the metron-managment
section.
On May 25, 2017 at 07:27:24, Nick Allen (nick@nickallen.org) wrote:
> everywhere I can use Stellar DSL, all of the functions have been
implemented and ready to use?
Generally, yes, you are right.
I vaguely remember a couple instances of functions that are useful in the
REPL only, but I cannot remember what those are right now. Hopefully we
have those doc'd appropriately.
On Wed, May 24, 2017 at 10:38 PM, Ali Nazemian <al...@gmail.com>
wrote:
> Hi Nick,
>
> I was not sure about the implementation, so does it generally mean
> everywhere I can use Stellar DSL, all of the functions have been
> implemented and ready to use?
>
> Cheers,
> Ali
>
> On Thu, May 25, 2017 at 2:52 AM, Nick Allen <ni...@nickallen.org> wrote:
>
>> > can I do the concatenation on the fly at the enrichment level, so I
>> don't need to store this temp field in Elasticsearch/HDFS.
>>
>> Sure, absolutely.
>>
>> > Moreover, I need to have a conditional enrichment to say if you
>> couldn't find any match for "tenant_name+device_type+device_name" lookup
>> for "tenant_name+device_type+default_device".
>>
>> Yes, you can. You've got if/else, JOIN, IS_EMPTY, and others that should
>> make implementing this logic pretty easy.
>>
>>
>>
>>
>> On Tue, May 23, 2017 at 10:34 PM, Ali Nazemian <al...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I was wondering how I can manage Stellar syntax to be aligned with the
>>> following structure for the HBase enrichment:
>>>
>>> HBase_row_key: tenant_name+device_type+device_name
>>>
>>> At the high-level, I need to create a separate field via a post-parse
>>> Stellar function to be a concatenation of tenan_name, device_type and
>>> device_name. Let's call this field "key". Basically, I need to do the
>>> enrichment on the "key" which would be corresponding to the HBase row key.
>>> My first question is *can I do the concatenation on the fly at the
>>> enrichment level, so I don't need to store this temp field in
>>> Elasticsearch/HDFS*.
>>>
>>> Moreover, I need to have a conditional enrichment to say if you couldn't
>>> find any match for "tenant_name+device_type+device_name" lookup for
>>> "tenant_name+device_type+default_device". The second question would be *how
>>> can I manage conditional enrichment like this one*. I would be really
>>> grateful if you can provide some example.
>>>
>>> Regards,
>>> Ali
>>>
>>
>>
>
>
> --
> A.Nazemian
>
Re: Metron HBase conditional enrichment
Posted by Nick Allen <ni...@nickallen.org>.
> everywhere I can use Stellar DSL, all of the functions have been
implemented and ready to use?
Generally, yes, you are right.
I vaguely remember a couple instances of functions that are useful in the
REPL only, but I cannot remember what those are right now. Hopefully we
have those doc'd appropriately.
On Wed, May 24, 2017 at 10:38 PM, Ali Nazemian <al...@gmail.com>
wrote:
> Hi Nick,
>
> I was not sure about the implementation, so does it generally mean
> everywhere I can use Stellar DSL, all of the functions have been
> implemented and ready to use?
>
> Cheers,
> Ali
>
> On Thu, May 25, 2017 at 2:52 AM, Nick Allen <ni...@nickallen.org> wrote:
>
>> > can I do the concatenation on the fly at the enrichment level, so I
>> don't need to store this temp field in Elasticsearch/HDFS.
>>
>> Sure, absolutely.
>>
>> > Moreover, I need to have a conditional enrichment to say if you
>> couldn't find any match for "tenant_name+device_type+device_name" lookup
>> for "tenant_name+device_type+default_device".
>>
>> Yes, you can. You've got if/else, JOIN, IS_EMPTY, and others that should
>> make implementing this logic pretty easy.
>>
>>
>>
>>
>> On Tue, May 23, 2017 at 10:34 PM, Ali Nazemian <al...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I was wondering how I can manage Stellar syntax to be aligned with the
>>> following structure for the HBase enrichment:
>>>
>>> HBase_row_key: tenant_name+device_type+device_name
>>>
>>> At the high-level, I need to create a separate field via a post-parse
>>> Stellar function to be a concatenation of tenan_name, device_type and
>>> device_name. Let's call this field "key". Basically, I need to do the
>>> enrichment on the "key" which would be corresponding to the HBase row key.
>>> My first question is *can I do the concatenation on the fly at the
>>> enrichment level, so I don't need to store this temp field in
>>> Elasticsearch/HDFS*.
>>>
>>> Moreover, I need to have a conditional enrichment to say if you couldn't
>>> find any match for "tenant_name+device_type+device_name" lookup for
>>> "tenant_name+device_type+default_device". The second question would be *how
>>> can I manage conditional enrichment like this one*. I would be really
>>> grateful if you can provide some example.
>>>
>>> Regards,
>>> Ali
>>>
>>
>>
>
>
> --
> A.Nazemian
>
Re: Metron HBase conditional enrichment
Posted by Ali Nazemian <al...@gmail.com>.
Hi Nick,
I was not sure about the implementation, so does it generally mean
everywhere I can use Stellar DSL, all of the functions have been
implemented and ready to use?
Cheers,
Ali
On Thu, May 25, 2017 at 2:52 AM, Nick Allen <ni...@nickallen.org> wrote:
> > can I do the concatenation on the fly at the enrichment level, so I
> don't need to store this temp field in Elasticsearch/HDFS.
>
> Sure, absolutely.
>
> > Moreover, I need to have a conditional enrichment to say if you
> couldn't find any match for "tenant_name+device_type+device_name" lookup
> for "tenant_name+device_type+default_device".
>
> Yes, you can. You've got if/else, JOIN, IS_EMPTY, and others that should
> make implementing this logic pretty easy.
>
>
>
>
> On Tue, May 23, 2017 at 10:34 PM, Ali Nazemian <al...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I was wondering how I can manage Stellar syntax to be aligned with the
>> following structure for the HBase enrichment:
>>
>> HBase_row_key: tenant_name+device_type+device_name
>>
>> At the high-level, I need to create a separate field via a post-parse
>> Stellar function to be a concatenation of tenan_name, device_type and
>> device_name. Let's call this field "key". Basically, I need to do the
>> enrichment on the "key" which would be corresponding to the HBase row key.
>> My first question is *can I do the concatenation on the fly at the
>> enrichment level, so I don't need to store this temp field in
>> Elasticsearch/HDFS*.
>>
>> Moreover, I need to have a conditional enrichment to say if you couldn't
>> find any match for "tenant_name+device_type+device_name" lookup for
>> "tenant_name+device_type+default_device". The second question would be *how
>> can I manage conditional enrichment like this one*. I would be really
>> grateful if you can provide some example.
>>
>> Regards,
>> Ali
>>
>
>
--
A.Nazemian
Re: Metron HBase conditional enrichment
Posted by Nick Allen <ni...@nickallen.org>.
> can I do the concatenation on the fly at the enrichment level, so I don't
need to store this temp field in Elasticsearch/HDFS.
Sure, absolutely.
> Moreover, I need to have a conditional enrichment to say if you couldn't
find any match for "tenant_name+device_type+device_name" lookup for
"tenant_name+device_type+default_device".
Yes, you can. You've got if/else, JOIN, IS_EMPTY, and others that should
make implementing this logic pretty easy.
On Tue, May 23, 2017 at 10:34 PM, Ali Nazemian <al...@gmail.com>
wrote:
> Hi,
>
> I was wondering how I can manage Stellar syntax to be aligned with the
> following structure for the HBase enrichment:
>
> HBase_row_key: tenant_name+device_type+device_name
>
> At the high-level, I need to create a separate field via a post-parse
> Stellar function to be a concatenation of tenan_name, device_type and
> device_name. Let's call this field "key". Basically, I need to do the
> enrichment on the "key" which would be corresponding to the HBase row key.
> My first question is *can I do the concatenation on the fly at the
> enrichment level, so I don't need to store this temp field in
> Elasticsearch/HDFS*.
>
> Moreover, I need to have a conditional enrichment to say if you couldn't
> find any match for "tenant_name+device_type+device_name" lookup for
> "tenant_name+device_type+default_device". The second question would be *how
> can I manage conditional enrichment like this one*. I would be really
> grateful if you can provide some example.
>
> Regards,
> Ali
>