You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Otto Fowler <ot...@gmail.com> on 2016/09/27 18:12:06 UTC
Removing and renaming fields with STELLAR
Is it possible to remove and rename fields with stellar?
I have “foreign” json coming out of the JSONParser, and I need to rename
some fields to the core metron json object names and possibly remove some
other fields.
Re: Removing and renaming fields with STELLAR
Posted by Otto Fowler <ot...@gmail.com>.
Thanks James,
I took a shot at it today. It looks like everything I did worked, but I
won’t be sure until I fix the storm slots issue. It would be good to know
why things had to be configured as opposed to just winging it.
--
On September 28, 2016 at 17:40:15, James Sirota (jsirota@apache.org) wrote:
We'll try to throw some docs together tomorrow to help you out. We need to
make sure we are clear enough in our docs so everyone knows how to do what
you are trying to do. We'll put more clarity around creating monit scripts
and doing HDFS and ES pruning.
28.09.2016, 06:54, "Otto Fowler" <ot...@gmail.com>:
> I’ve written the rules in the configuration, I’m going through the parser
> deployment integration that I know about ( configuration for parser,
> enrichment, es-templates ), but I still have to do the monit stuff, hdfs
> pruning etc that I’ve been asking about
>
> --
>
> Sent with Airmail
>
> On September 28, 2016 at 01:02:56, James Sirota (jsirota@apache.org)
wrote:
>
> Do you still need help? Or were you able to do what you wanted?
>
> 27.09.2016, 12:34, "Otto Fowler" <ot...@gmail.com>:
>> I’ll get there
>>
>> --
>>
>> On September 27, 2016 at 15:23:10, Casey Stella (cestella@gmail.com)
>
> wrote:
>> Yeah, it's confusing, so I feel your pain. I made a "Navigating the
>> Architecture" (which should probably be named "Navigating the Developer
>> Docs" ;) section in the top-level doc here
>> <https://github.com/apache/incubator-metron#navigating-the-architecture>
>> that
>> links to the docs for various parts of the architecture.
>>
>> Hope that's helpful in the future! :)
>>
>> On Tue, Sep 27, 2016 at 3:05 PM, Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>>> Ok, I was going by the readme in metron-common and the wiki.
>>> I’ll look in both readme files next time
>>>
>>> --
>>>
>>> Sent with Airmail
>>>
>>> On September 27, 2016 at 14:49:29, Casey Stella (cestella@gmail.com)
>>> wrote:
>>>
>>> It's described in
>
> https://github.com/apache/incubator-metron/tree/master/
>>> metron-platform/metron-parsers#fieldtransformation-configuration
under
>>> "REMOVE". The wiki tends to lag; the README.md docs are mostly
>>
>> up-to-date.
>>> Casey
>>>
>>> On Tue, Sep 27, 2016 at 2:47 PM, Otto Fowler <ot...@gmail.com>
>>> wrote:
>>>
>>> > I’m going by the wiki and REMOVE is not in there. Where should I
look
>>
>> for
>>> > the most up to date stuff? The unit tests?
>>> >
>>> >
>>> >
>>> > On September 27, 2016 at 14:33:08, Casey Stella (cestella@gmail.com)
>>> > wrote:
>>> >
>>> > Yes, it is in the parser. You can assign a field to NULL and it
will
>>> > remove it.
>>> >
>>> > Consider the following example parser config which takes the foo
>
> field
>>> and
>>> > renames it to bar:
>>> >
>>> > {
>>> > "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>>> > "sensorTopic":"custom",
>>> > "parserConfig" : {
>>> > "mapStrategy" : "UNFOLD"
>>> > },"fieldTransformations" : [
>>> > {
>>> > "transformation" : "STELLAR"
>>> > ,"output" : [ "bar", "foo" ]
>>> > ,"config" : {
>>> > "bar" : "foo"
>>> > ,"foo" : "NULL"
>>> > }
>>> > }
>>> > ]
>>> > }
>>> >
>>> > Note the output section, that defines the order in which the
stellar
>>> > statements are executed. You first set bar to whatever is in foo.
>
> Then
>>> > you set foo to NULL, which will delete it. You will be left with
just
>>> bar,
>>> > which will contain what is in foo.
>>> >
>>> > Another way to do this is to use the REMOVE transformation
>>> > <
>>> > https://github.com/apache/incubator-metron/tree/master/
>>> > metron-platform/metron-parsers#fieldtransformation-configuration>
>>> >
>>> > in
>>> > combination with the stellar transformation:
>>> > {
>>> > "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>>> > "sensorTopic":"custom",
>>> > "parserConfig" : {
>>> > "mapStrategy" : "UNFOLD"
>>> > },"fieldTransformations" : [
>>> > {
>>> > "transformation" : "STELLAR"
>>> > ,"output" : [ "bar", "foo" ]
>>> > ,"config" : {
>>> > "bar" : "foo"
>>> > }
>>> > },
>>> > {
>>> > "input" : "foo"
>>> > , "transformation" : "REMOVE"
>>> > }
>>> > ]
>>> > }
>>> >
>>> > Best,
>>> >
>>> > Casey
>>> >
>>> > On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler <
ottobackwards@gmail.com>
>
>>> > wrote:
>>> >
>>> > > Is it possible to remove and rename fields with stellar?
>>> > > I have “foreign” json coming out of the JSONParser, and I need to
>>> rename
>>> > > some fields to the core metron json object names and possibly
>
> remove
>>> some
>>> > > other fields.
>>> > >
>>> >
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
-------------------
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: Removing and renaming fields with STELLAR
Posted by James Sirota <js...@apache.org>.
We'll try to throw some docs together tomorrow to help you out. We need to make sure we are clear enough in our docs so everyone knows how to do what you are trying to do. We'll put more clarity around creating monit scripts and doing HDFS and ES pruning.
28.09.2016, 06:54, "Otto Fowler" <ot...@gmail.com>:
> I\u2019ve written the rules in the configuration, I\u2019m going through the parser
> deployment integration that I know about ( configuration for parser,
> enrichment, es-templates ), but I still have to do the monit stuff, hdfs
> pruning etc that I\u2019ve been asking about
>
> --
>
> Sent with Airmail
>
> On September 28, 2016 at 01:02:56, James Sirota (jsirota@apache.org) wrote:
>
> Do you still need help? Or were you able to do what you wanted?
>
> 27.09.2016, 12:34, "Otto Fowler" <ot...@gmail.com>:
>> �I\u2019ll get there
>>
>> �--
>>
>> �On September 27, 2016 at 15:23:10, Casey Stella (cestella@gmail.com)
>
> wrote:
>> �Yeah, it's confusing, so I feel your pain. I made a "Navigating the
>> �Architecture" (which should probably be named "Navigating the Developer
>> �Docs" ;) section in the top-level doc here
>> �<https://github.com/apache/incubator-metron#navigating-the-architecture>
>> �that
>> �links to the docs for various parts of the architecture.
>>
>> �Hope that's helpful in the future! :)
>>
>> �On Tue, Sep 27, 2016 at 3:05 PM, Otto Fowler <ot...@gmail.com>
>> �wrote:
>>
>>> ��Ok, I was going by the readme in metron-common and the wiki.
>>> ��I\u2019ll look in both readme files next time
>>>
>>> ��--
>>>
>>> ��Sent with Airmail
>>>
>>> ��On September 27, 2016 at 14:49:29, Casey Stella (cestella@gmail.com)
>>> ��wrote:
>>>
>>> ��It's described in
>
> https://github.com/apache/incubator-metron/tree/master/
>>> ��metron-platform/metron-parsers#fieldtransformation-configuration under
>>> ��"REMOVE". The wiki tends to lag; the README.md docs are mostly
>>
>> �up-to-date.
>>> ��Casey
>>>
>>> ��On Tue, Sep 27, 2016 at 2:47 PM, Otto Fowler <ot...@gmail.com>
>>> ��wrote:
>>>
>>> ��> I\u2019m going by the wiki and REMOVE is not in there. Where should I look
>>
>> �for
>>> ��> the most up to date stuff? The unit tests?
>>> ��>
>>> ��>
>>> ��>
>>> ��> On September 27, 2016 at 14:33:08, Casey Stella (cestella@gmail.com)
>>> ��> wrote:
>>> ��>
>>> ��> Yes, it is in the parser. You can assign a field to NULL and it will
>>> ��> remove it.
>>> ��>
>>> ��> Consider the following example parser config which takes the foo
>
> field
>>> ��and
>>> ��> renames it to bar:
>>> ��>
>>> ��> {
>>> ��> "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>>> ��> "sensorTopic":"custom",
>>> ��> "parserConfig" : {
>>> ��> "mapStrategy" : "UNFOLD"
>>> ��> },"fieldTransformations" : [
>>> ��> {
>>> ��> "transformation" : "STELLAR"
>>> ��> ,"output" : [ "bar", "foo" ]
>>> ��> ,"config" : {
>>> ��> "bar" : "foo"
>>> ��> ,"foo" : "NULL"
>>> ��> }
>>> ��> }
>>> ��> ]
>>> ��> }
>>> ��>
>>> ��> Note the output section, that defines the order in which the stellar
>>> ��> statements are executed. You first set bar to whatever is in foo.
>
> Then
>>> ��> you set foo to NULL, which will delete it. You will be left with just
>>> ��bar,
>>> ��> which will contain what is in foo.
>>> ��>
>>> ��> Another way to do this is to use the REMOVE transformation
>>> ��> <
>>> ��> https://github.com/apache/incubator-metron/tree/master/
>>> ��> metron-platform/metron-parsers#fieldtransformation-configuration>
>>> ��>
>>> ��> in
>>> ��> combination with the stellar transformation:
>>> ��> {
>>> ��> "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>>> ��> "sensorTopic":"custom",
>>> ��> "parserConfig" : {
>>> ��> "mapStrategy" : "UNFOLD"
>>> ��> },"fieldTransformations" : [
>>> ��> {
>>> ��> "transformation" : "STELLAR"
>>> ��> ,"output" : [ "bar", "foo" ]
>>> ��> ,"config" : {
>>> ��> "bar" : "foo"
>>> ��> }
>>> ��> },
>>> ��> {
>>> ��> "input" : "foo"
>>> ��> , "transformation" : "REMOVE"
>>> ��> }
>>> ��> ]
>>> ��> }
>>> ��>
>>> ��> Best,
>>> ��>
>>> ��> Casey
>>> ��>
>>> ��> On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler <ot...@gmail.com>
>
>>> ��> wrote:
>>> ��>
>>> ��> > Is it possible to remove and rename fields with stellar?
>>> ��> > I have \u201cforeign\u201d json coming out of the JSONParser, and I need to
>>> ��rename
>>> ��> > some fields to the core metron json object names and possibly
>
> remove
>>> ��some
>>> ��> > other fields.
>>> ��> >
>>> ��>
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
-------------------�
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: Removing and renaming fields with STELLAR
Posted by Otto Fowler <ot...@gmail.com>.
I’ve written the rules in the configuration, I’m going through the parser
deployment integration that I know about ( configuration for parser,
enrichment, es-templates ), but I still have to do the monit stuff, hdfs
pruning etc that I’ve been asking about
--
Sent with Airmail
On September 28, 2016 at 01:02:56, James Sirota (jsirota@apache.org) wrote:
Do you still need help? Or were you able to do what you wanted?
27.09.2016, 12:34, "Otto Fowler" <ot...@gmail.com>:
> I’ll get there
>
> --
>
> On September 27, 2016 at 15:23:10, Casey Stella (cestella@gmail.com)
wrote:
>
> Yeah, it's confusing, so I feel your pain. I made a "Navigating the
> Architecture" (which should probably be named "Navigating the Developer
> Docs" ;) section in the top-level doc here
> <https://github.com/apache/incubator-metron#navigating-the-architecture>
> that
> links to the docs for various parts of the architecture.
>
> Hope that's helpful in the future! :)
>
> On Tue, Sep 27, 2016 at 3:05 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
>> Ok, I was going by the readme in metron-common and the wiki.
>> I’ll look in both readme files next time
>>
>> --
>>
>> Sent with Airmail
>>
>> On September 27, 2016 at 14:49:29, Casey Stella (cestella@gmail.com)
>> wrote:
>>
>> It's described in
https://github.com/apache/incubator-metron/tree/master/
>> metron-platform/metron-parsers#fieldtransformation-configuration under
>> "REMOVE". The wiki tends to lag; the README.md docs are mostly
>
> up-to-date.
>> Casey
>>
>> On Tue, Sep 27, 2016 at 2:47 PM, Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>> > I’m going by the wiki and REMOVE is not in there. Where should I look
>
> for
>> > the most up to date stuff? The unit tests?
>> >
>> >
>> >
>> > On September 27, 2016 at 14:33:08, Casey Stella (cestella@gmail.com)
>> > wrote:
>> >
>> > Yes, it is in the parser. You can assign a field to NULL and it will
>> > remove it.
>> >
>> > Consider the following example parser config which takes the foo
field
>> and
>> > renames it to bar:
>> >
>> > {
>> > "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>> > "sensorTopic":"custom",
>> > "parserConfig" : {
>> > "mapStrategy" : "UNFOLD"
>> > },"fieldTransformations" : [
>> > {
>> > "transformation" : "STELLAR"
>> > ,"output" : [ "bar", "foo" ]
>> > ,"config" : {
>> > "bar" : "foo"
>> > ,"foo" : "NULL"
>> > }
>> > }
>> > ]
>> > }
>> >
>> > Note the output section, that defines the order in which the stellar
>> > statements are executed. You first set bar to whatever is in foo.
Then
>> > you set foo to NULL, which will delete it. You will be left with just
>> bar,
>> > which will contain what is in foo.
>> >
>> > Another way to do this is to use the REMOVE transformation
>> > <
>> > https://github.com/apache/incubator-metron/tree/master/
>> > metron-platform/metron-parsers#fieldtransformation-configuration>
>> >
>> > in
>> > combination with the stellar transformation:
>> > {
>> > "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>> > "sensorTopic":"custom",
>> > "parserConfig" : {
>> > "mapStrategy" : "UNFOLD"
>> > },"fieldTransformations" : [
>> > {
>> > "transformation" : "STELLAR"
>> > ,"output" : [ "bar", "foo" ]
>> > ,"config" : {
>> > "bar" : "foo"
>> > }
>> > },
>> > {
>> > "input" : "foo"
>> > , "transformation" : "REMOVE"
>> > }
>> > ]
>> > }
>> >
>> > Best,
>> >
>> > Casey
>> >
>> > On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler <ot...@gmail.com>
>> > wrote:
>> >
>> > > Is it possible to remove and rename fields with stellar?
>> > > I have “foreign” json coming out of the JSONParser, and I need to
>> rename
>> > > some fields to the core metron json object names and possibly
remove
>> some
>> > > other fields.
>> > >
>> >
-------------------
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: Removing and renaming fields with STELLAR
Posted by James Sirota <js...@apache.org>.
Do you still need help? Or were you able to do what you wanted?
27.09.2016, 12:34, "Otto Fowler" <ot...@gmail.com>:
> I\u2019ll get there
>
> --
>
> On September 27, 2016 at 15:23:10, Casey Stella (cestella@gmail.com) wrote:
>
> Yeah, it's confusing, so I feel your pain. I made a "Navigating the
> Architecture" (which should probably be named "Navigating the Developer
> Docs" ;) section in the top-level doc here
> <https://github.com/apache/incubator-metron#navigating-the-architecture>
> that
> links to the docs for various parts of the architecture.
>
> Hope that's helpful in the future! :)
>
> On Tue, Sep 27, 2016 at 3:05 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
>> �Ok, I was going by the readme in metron-common and the wiki.
>> �I\u2019ll look in both readme files next time
>>
>> �--
>>
>> �Sent with Airmail
>>
>> �On September 27, 2016 at 14:49:29, Casey Stella (cestella@gmail.com)
>> �wrote:
>>
>> �It's described in https://github.com/apache/incubator-metron/tree/master/
>> �metron-platform/metron-parsers#fieldtransformation-configuration under
>> �"REMOVE". The wiki tends to lag; the README.md docs are mostly
>
> up-to-date.
>> �Casey
>>
>> �On Tue, Sep 27, 2016 at 2:47 PM, Otto Fowler <ot...@gmail.com>
>> �wrote:
>>
>> �> I\u2019m going by the wiki and REMOVE is not in there. Where should I look
>
> for
>> �> the most up to date stuff? The unit tests?
>> �>
>> �>
>> �>
>> �> On September 27, 2016 at 14:33:08, Casey Stella (cestella@gmail.com)
>> �> wrote:
>> �>
>> �> Yes, it is in the parser. You can assign a field to NULL and it will
>> �> remove it.
>> �>
>> �> Consider the following example parser config which takes the foo field
>> �and
>> �> renames it to bar:
>> �>
>> �> {
>> �> "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>> �> "sensorTopic":"custom",
>> �> "parserConfig" : {
>> �> "mapStrategy" : "UNFOLD"
>> �> },"fieldTransformations" : [
>> �> {
>> �> "transformation" : "STELLAR"
>> �> ,"output" : [ "bar", "foo" ]
>> �> ,"config" : {
>> �> "bar" : "foo"
>> �> ,"foo" : "NULL"
>> �> }
>> �> }
>> �> ]
>> �> }
>> �>
>> �> Note the output section, that defines the order in which the stellar
>> �> statements are executed. You first set bar to whatever is in foo. Then
>> �> you set foo to NULL, which will delete it. You will be left with just
>> �bar,
>> �> which will contain what is in foo.
>> �>
>> �> Another way to do this is to use the REMOVE transformation
>> �> <
>> �> https://github.com/apache/incubator-metron/tree/master/
>> �> metron-platform/metron-parsers#fieldtransformation-configuration>
>> �>
>> �> in
>> �> combination with the stellar transformation:
>> �> {
>> �> "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>> �> "sensorTopic":"custom",
>> �> "parserConfig" : {
>> �> "mapStrategy" : "UNFOLD"
>> �> },"fieldTransformations" : [
>> �> {
>> �> "transformation" : "STELLAR"
>> �> ,"output" : [ "bar", "foo" ]
>> �> ,"config" : {
>> �> "bar" : "foo"
>> �> }
>> �> },
>> �> {
>> �> "input" : "foo"
>> �> , "transformation" : "REMOVE"
>> �> }
>> �> ]
>> �> }
>> �>
>> �> Best,
>> �>
>> �> Casey
>> �>
>> �> On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler <ot...@gmail.com>
>> �> wrote:
>> �>
>> �> > Is it possible to remove and rename fields with stellar?
>> �> > I have \u201cforeign\u201d json coming out of the JSONParser, and I need to
>> �rename
>> �> > some fields to the core metron json object names and possibly remove
>> �some
>> �> > other fields.
>> �> >
>> �>
-------------------�
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: Removing and renaming fields with STELLAR
Posted by Otto Fowler <ot...@gmail.com>.
I’ll get there
--
On September 27, 2016 at 15:23:10, Casey Stella (cestella@gmail.com) wrote:
Yeah, it's confusing, so I feel your pain. I made a "Navigating the
Architecture" (which should probably be named "Navigating the Developer
Docs" ;) section in the top-level doc here
<https://github.com/apache/incubator-metron#navigating-the-architecture>
that
links to the docs for various parts of the architecture.
Hope that's helpful in the future! :)
On Tue, Sep 27, 2016 at 3:05 PM, Otto Fowler <ot...@gmail.com>
wrote:
> Ok, I was going by the readme in metron-common and the wiki.
> I’ll look in both readme files next time
>
> --
>
> Sent with Airmail
>
> On September 27, 2016 at 14:49:29, Casey Stella (cestella@gmail.com)
> wrote:
>
> It's described in https://github.com/apache/incubator-metron/tree/master/
> metron-platform/metron-parsers#fieldtransformation-configuration under
> "REMOVE". The wiki tends to lag; the README.md docs are mostly
up-to-date.
>
> Casey
>
> On Tue, Sep 27, 2016 at 2:47 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
> > I’m going by the wiki and REMOVE is not in there. Where should I look
for
> > the most up to date stuff? The unit tests?
> >
> >
> >
> > On September 27, 2016 at 14:33:08, Casey Stella (cestella@gmail.com)
> > wrote:
> >
> > Yes, it is in the parser. You can assign a field to NULL and it will
> > remove it.
> >
> > Consider the following example parser config which takes the foo field
> and
> > renames it to bar:
> >
> > {
> > "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
> > "sensorTopic":"custom",
> > "parserConfig" : {
> > "mapStrategy" : "UNFOLD"
> > },"fieldTransformations" : [
> > {
> > "transformation" : "STELLAR"
> > ,"output" : [ "bar", "foo" ]
> > ,"config" : {
> > "bar" : "foo"
> > ,"foo" : "NULL"
> > }
> > }
> > ]
> > }
> >
> > Note the output section, that defines the order in which the stellar
> > statements are executed. You first set bar to whatever is in foo. Then
> > you set foo to NULL, which will delete it. You will be left with just
> bar,
> > which will contain what is in foo.
> >
> > Another way to do this is to use the REMOVE transformation
> > <
> > https://github.com/apache/incubator-metron/tree/master/
> > metron-platform/metron-parsers#fieldtransformation-configuration>
> >
> > in
> > combination with the stellar transformation:
> > {
> > "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
> > "sensorTopic":"custom",
> > "parserConfig" : {
> > "mapStrategy" : "UNFOLD"
> > },"fieldTransformations" : [
> > {
> > "transformation" : "STELLAR"
> > ,"output" : [ "bar", "foo" ]
> > ,"config" : {
> > "bar" : "foo"
> > }
> > },
> > {
> > "input" : "foo"
> > , "transformation" : "REMOVE"
> > }
> > ]
> > }
> >
> > Best,
> >
> > Casey
> >
> > On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler <ot...@gmail.com>
> > wrote:
> >
> > > Is it possible to remove and rename fields with stellar?
> > > I have “foreign” json coming out of the JSONParser, and I need to
> rename
> > > some fields to the core metron json object names and possibly remove
> some
> > > other fields.
> > >
> >
>
Re: Removing and renaming fields with STELLAR
Posted by Casey Stella <ce...@gmail.com>.
Yeah, it's confusing, so I feel your pain. I made a "Navigating the
Architecture" (which should probably be named "Navigating the Developer
Docs" ;) section in the top-level doc here
<https://github.com/apache/incubator-metron#navigating-the-architecture> that
links to the docs for various parts of the architecture.
Hope that's helpful in the future! :)
On Tue, Sep 27, 2016 at 3:05 PM, Otto Fowler <ot...@gmail.com>
wrote:
> Ok, I was going by the readme in metron-common and the wiki.
> I’ll look in both readme files next time
>
> --
>
> Sent with Airmail
>
> On September 27, 2016 at 14:49:29, Casey Stella (cestella@gmail.com)
> wrote:
>
> It's described in https://github.com/apache/incubator-metron/tree/master/
> metron-platform/metron-parsers#fieldtransformation-configuration under
> "REMOVE". The wiki tends to lag; the README.md docs are mostly up-to-date.
>
> Casey
>
> On Tue, Sep 27, 2016 at 2:47 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
> > I’m going by the wiki and REMOVE is not in there. Where should I look for
> > the most up to date stuff? The unit tests?
> >
> >
> >
> > On September 27, 2016 at 14:33:08, Casey Stella (cestella@gmail.com)
> > wrote:
> >
> > Yes, it is in the parser. You can assign a field to NULL and it will
> > remove it.
> >
> > Consider the following example parser config which takes the foo field
> and
> > renames it to bar:
> >
> > {
> > "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
> > "sensorTopic":"custom",
> > "parserConfig" : {
> > "mapStrategy" : "UNFOLD"
> > },"fieldTransformations" : [
> > {
> > "transformation" : "STELLAR"
> > ,"output" : [ "bar", "foo" ]
> > ,"config" : {
> > "bar" : "foo"
> > ,"foo" : "NULL"
> > }
> > }
> > ]
> > }
> >
> > Note the output section, that defines the order in which the stellar
> > statements are executed. You first set bar to whatever is in foo. Then
> > you set foo to NULL, which will delete it. You will be left with just
> bar,
> > which will contain what is in foo.
> >
> > Another way to do this is to use the REMOVE transformation
> > <
> > https://github.com/apache/incubator-metron/tree/master/
> > metron-platform/metron-parsers#fieldtransformation-configuration>
> >
> > in
> > combination with the stellar transformation:
> > {
> > "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
> > "sensorTopic":"custom",
> > "parserConfig" : {
> > "mapStrategy" : "UNFOLD"
> > },"fieldTransformations" : [
> > {
> > "transformation" : "STELLAR"
> > ,"output" : [ "bar", "foo" ]
> > ,"config" : {
> > "bar" : "foo"
> > }
> > },
> > {
> > "input" : "foo"
> > , "transformation" : "REMOVE"
> > }
> > ]
> > }
> >
> > Best,
> >
> > Casey
> >
> > On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler <ot...@gmail.com>
> > wrote:
> >
> > > Is it possible to remove and rename fields with stellar?
> > > I have “foreign” json coming out of the JSONParser, and I need to
> rename
> > > some fields to the core metron json object names and possibly remove
> some
> > > other fields.
> > >
> >
>
Re: Removing and renaming fields with STELLAR
Posted by Otto Fowler <ot...@gmail.com>.
Ok, I was going by the readme in metron-common and the wiki.
I’ll look in both readme files next time
--
Sent with Airmail
On September 27, 2016 at 14:49:29, Casey Stella (cestella@gmail.com) wrote:
It's described in https://github.com/apache/incubator-metron/tree/master/
metron-platform/metron-parsers#fieldtransformation-configuration under
"REMOVE". The wiki tends to lag; the README.md docs are mostly up-to-date.
Casey
On Tue, Sep 27, 2016 at 2:47 PM, Otto Fowler <ot...@gmail.com>
wrote:
> I’m going by the wiki and REMOVE is not in there. Where should I look for
> the most up to date stuff? The unit tests?
>
>
>
> On September 27, 2016 at 14:33:08, Casey Stella (cestella@gmail.com)
> wrote:
>
> Yes, it is in the parser. You can assign a field to NULL and it will
> remove it.
>
> Consider the following example parser config which takes the foo field
and
> renames it to bar:
>
> {
> "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
> "sensorTopic":"custom",
> "parserConfig" : {
> "mapStrategy" : "UNFOLD"
> },"fieldTransformations" : [
> {
> "transformation" : "STELLAR"
> ,"output" : [ "bar", "foo" ]
> ,"config" : {
> "bar" : "foo"
> ,"foo" : "NULL"
> }
> }
> ]
> }
>
> Note the output section, that defines the order in which the stellar
> statements are executed. You first set bar to whatever is in foo. Then
> you set foo to NULL, which will delete it. You will be left with just
bar,
> which will contain what is in foo.
>
> Another way to do this is to use the REMOVE transformation
> <
> https://github.com/apache/incubator-metron/tree/master/
> metron-platform/metron-parsers#fieldtransformation-configuration>
>
> in
> combination with the stellar transformation:
> {
> "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
> "sensorTopic":"custom",
> "parserConfig" : {
> "mapStrategy" : "UNFOLD"
> },"fieldTransformations" : [
> {
> "transformation" : "STELLAR"
> ,"output" : [ "bar", "foo" ]
> ,"config" : {
> "bar" : "foo"
> }
> },
> {
> "input" : "foo"
> , "transformation" : "REMOVE"
> }
> ]
> }
>
> Best,
>
> Casey
>
> On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
> > Is it possible to remove and rename fields with stellar?
> > I have “foreign” json coming out of the JSONParser, and I need to
rename
> > some fields to the core metron json object names and possibly remove
some
> > other fields.
> >
>
Re: Removing and renaming fields with STELLAR
Posted by Casey Stella <ce...@gmail.com>.
It's described in https://github.com/apache/incubator-metron/tree/master/
metron-platform/metron-parsers#fieldtransformation-configuration under
"REMOVE". The wiki tends to lag; the README.md docs are mostly up-to-date.
Casey
On Tue, Sep 27, 2016 at 2:47 PM, Otto Fowler <ot...@gmail.com>
wrote:
> I’m going by the wiki and REMOVE is not in there. Where should I look for
> the most up to date stuff? The unit tests?
>
>
>
> On September 27, 2016 at 14:33:08, Casey Stella (cestella@gmail.com)
> wrote:
>
> Yes, it is in the parser. You can assign a field to NULL and it will
> remove it.
>
> Consider the following example parser config which takes the foo field and
> renames it to bar:
>
> {
> "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
> "sensorTopic":"custom",
> "parserConfig" : {
> "mapStrategy" : "UNFOLD"
> },"fieldTransformations" : [
> {
> "transformation" : "STELLAR"
> ,"output" : [ "bar", "foo" ]
> ,"config" : {
> "bar" : "foo"
> ,"foo" : "NULL"
> }
> }
> ]
> }
>
> Note the output section, that defines the order in which the stellar
> statements are executed. You first set bar to whatever is in foo. Then
> you set foo to NULL, which will delete it. You will be left with just bar,
> which will contain what is in foo.
>
> Another way to do this is to use the REMOVE transformation
> <
> https://github.com/apache/incubator-metron/tree/master/
> metron-platform/metron-parsers#fieldtransformation-configuration>
>
> in
> combination with the stellar transformation:
> {
> "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
> "sensorTopic":"custom",
> "parserConfig" : {
> "mapStrategy" : "UNFOLD"
> },"fieldTransformations" : [
> {
> "transformation" : "STELLAR"
> ,"output" : [ "bar", "foo" ]
> ,"config" : {
> "bar" : "foo"
> }
> },
> {
> "input" : "foo"
> , "transformation" : "REMOVE"
> }
> ]
> }
>
> Best,
>
> Casey
>
> On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
> > Is it possible to remove and rename fields with stellar?
> > I have “foreign” json coming out of the JSONParser, and I need to rename
> > some fields to the core metron json object names and possibly remove some
> > other fields.
> >
>
Re: Removing and renaming fields with STELLAR
Posted by Otto Fowler <ot...@gmail.com>.
I’m going by the wiki and REMOVE is not in there. Where should I look for
the most up to date stuff? The unit tests?
On September 27, 2016 at 14:33:08, Casey Stella (cestella@gmail.com) wrote:
Yes, it is in the parser. You can assign a field to NULL and it will
remove it.
Consider the following example parser config which takes the foo field and
renames it to bar:
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"custom",
"parserConfig" : {
"mapStrategy" : "UNFOLD"
},"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : [ "bar", "foo" ]
,"config" : {
"bar" : "foo"
,"foo" : "NULL"
}
}
]
}
Note the output section, that defines the order in which the stellar
statements are executed. You first set bar to whatever is in foo. Then
you set foo to NULL, which will delete it. You will be left with just bar,
which will contain what is in foo.
Another way to do this is to use the REMOVE transformation
<
https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-parsers#fieldtransformation-configuration>
in
combination with the stellar transformation:
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"custom",
"parserConfig" : {
"mapStrategy" : "UNFOLD"
},"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : [ "bar", "foo" ]
,"config" : {
"bar" : "foo"
}
},
{
"input" : "foo"
, "transformation" : "REMOVE"
}
]
}
Best,
Casey
On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler <ot...@gmail.com>
wrote:
> Is it possible to remove and rename fields with stellar?
> I have “foreign” json coming out of the JSONParser, and I need to rename
> some fields to the core metron json object names and possibly remove some
> other fields.
>
Re: Removing and renaming fields with STELLAR
Posted by Otto Fowler <ot...@gmail.com>.
Thanks Casey!
--
Sent with Airmail
On September 27, 2016 at 14:33:08, Casey Stella (cestella@gmail.com) wrote:
Yes, it is in the parser. You can assign a field to NULL and it will
remove it.
Consider the following example parser config which takes the foo field and
renames it to bar:
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"custom",
"parserConfig" : {
"mapStrategy" : "UNFOLD"
},"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : [ "bar", "foo" ]
,"config" : {
"bar" : "foo"
,"foo" : "NULL"
}
}
]
}
Note the output section, that defines the order in which the stellar
statements are executed. You first set bar to whatever is in foo. Then
you set foo to NULL, which will delete it. You will be left with just bar,
which will contain what is in foo.
Another way to do this is to use the REMOVE transformation
<
https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-parsers#fieldtransformation-configuration>
in
combination with the stellar transformation:
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"custom",
"parserConfig" : {
"mapStrategy" : "UNFOLD"
},"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : [ "bar", "foo" ]
,"config" : {
"bar" : "foo"
}
},
{
"input" : "foo"
, "transformation" : "REMOVE"
}
]
}
Best,
Casey
On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler <ot...@gmail.com>
wrote:
> Is it possible to remove and rename fields with stellar?
> I have “foreign” json coming out of the JSONParser, and I need to rename
> some fields to the core metron json object names and possibly remove some
> other fields.
>
Re: Removing and renaming fields with STELLAR
Posted by James Sirota <js...@apache.org>.
i think it would be great if we had something like this. I can see lots of people needing this
27.09.2016, 11:39, "Casey Stella" <ce...@gmail.com>:
> Just as an aside, I was thinking that it might be nice to have a special
> THIS object in stellar which would return the current message being
> processed, so you could do stuff like MAP_REMOVE('foo', this) to remove a
> field.
>
> On Tue, Sep 27, 2016 at 2:33 PM, Casey Stella <ce...@gmail.com> wrote:
>
>> �Yes, it is in the parser. You can assign a field to NULL and it will
>> �remove it.
>>
>> �Consider the following example parser config which takes the foo field and
>> �renames it to bar:
>>
>> �{
>> ���"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>> ���"sensorTopic":"custom",
>> ���"parserConfig" : {
>> �����"mapStrategy" : "UNFOLD"
>> ���},"fieldTransformations" : [
>> �����{
>> �������"transformation" : "STELLAR"
>> �����,"output" : [ "bar", "foo" ]
>> �����,"config" : {
>> �������"bar" : "foo"
>> �������,"foo" : "NULL"
>> �����������������}
>> �����}
>> ����������������������������]
>> �}
>>
>> �Note the output section, that defines the order in which the stellar
>> �statements are executed. You first set bar to whatever is in foo. Then
>> �you set foo to NULL, which will delete it. You will be left with just bar,
>> �which will contain what is in foo.
>>
>> �Another way to do this is to use the REMOVE transformation
>> �<https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-parsers#fieldtransformation-configuration> in
>> �combination with the stellar transformation:
>> �{
>> ���"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>> ���"sensorTopic":"custom",
>> ���"parserConfig" : {
>> �����"mapStrategy" : "UNFOLD"
>> ���},"fieldTransformations" : [
>> �����{
>> �������"transformation" : "STELLAR"
>> �����,"output" : [ "bar", "foo" ]
>> �����,"config" : {
>> �������"bar" : "foo"
>> �����������������}
>> �����},
>> �����{
>> �������"input" : "foo"
>> ������, "transformation" : "REMOVE"
>> �����}
>> ����������������������������]
>> �}
>>
>> �Best,
>>
>> �Casey
>>
>> �On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler <ot...@gmail.com>
>> �wrote:
>>
>>> �Is it possible to remove and rename fields with stellar?
>>> �I have \u201cforeign\u201d json coming out of the JSONParser, and I need to rename
>>> �some fields to the core metron json object names and possibly remove some
>>> �other fields.
-------------------�
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: Removing and renaming fields with STELLAR
Posted by Casey Stella <ce...@gmail.com>.
Just as an aside, I was thinking that it might be nice to have a special
THIS object in stellar which would return the current message being
processed, so you could do stuff like MAP_REMOVE('foo', this) to remove a
field.
On Tue, Sep 27, 2016 at 2:33 PM, Casey Stella <ce...@gmail.com> wrote:
> Yes, it is in the parser. You can assign a field to NULL and it will
> remove it.
>
> Consider the following example parser config which takes the foo field and
> renames it to bar:
>
> {
> "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
> "sensorTopic":"custom",
> "parserConfig" : {
> "mapStrategy" : "UNFOLD"
> },"fieldTransformations" : [
> {
> "transformation" : "STELLAR"
> ,"output" : [ "bar", "foo" ]
> ,"config" : {
> "bar" : "foo"
> ,"foo" : "NULL"
> }
> }
> ]
> }
>
> Note the output section, that defines the order in which the stellar
> statements are executed. You first set bar to whatever is in foo. Then
> you set foo to NULL, which will delete it. You will be left with just bar,
> which will contain what is in foo.
>
> Another way to do this is to use the REMOVE transformation
> <https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-parsers#fieldtransformation-configuration> in
> combination with the stellar transformation:
> {
> "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
> "sensorTopic":"custom",
> "parserConfig" : {
> "mapStrategy" : "UNFOLD"
> },"fieldTransformations" : [
> {
> "transformation" : "STELLAR"
> ,"output" : [ "bar", "foo" ]
> ,"config" : {
> "bar" : "foo"
> }
> },
> {
> "input" : "foo"
> , "transformation" : "REMOVE"
> }
> ]
> }
>
> Best,
>
> Casey
>
> On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
>> Is it possible to remove and rename fields with stellar?
>> I have “foreign” json coming out of the JSONParser, and I need to rename
>> some fields to the core metron json object names and possibly remove some
>> other fields.
>>
>
>
Re: Removing and renaming fields with STELLAR
Posted by Casey Stella <ce...@gmail.com>.
Yes, it is in the parser. You can assign a field to NULL and it will
remove it.
Consider the following example parser config which takes the foo field and
renames it to bar:
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"custom",
"parserConfig" : {
"mapStrategy" : "UNFOLD"
},"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : [ "bar", "foo" ]
,"config" : {
"bar" : "foo"
,"foo" : "NULL"
}
}
]
}
Note the output section, that defines the order in which the stellar
statements are executed. You first set bar to whatever is in foo. Then
you set foo to NULL, which will delete it. You will be left with just bar,
which will contain what is in foo.
Another way to do this is to use the REMOVE transformation
<https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-parsers#fieldtransformation-configuration>
in
combination with the stellar transformation:
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"custom",
"parserConfig" : {
"mapStrategy" : "UNFOLD"
},"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : [ "bar", "foo" ]
,"config" : {
"bar" : "foo"
}
},
{
"input" : "foo"
, "transformation" : "REMOVE"
}
]
}
Best,
Casey
On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler <ot...@gmail.com>
wrote:
> Is it possible to remove and rename fields with stellar?
> I have “foreign” json coming out of the JSONParser, and I need to rename
> some fields to the core metron json object names and possibly remove some
> other fields.
>