You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by "Yerex, Tom" <to...@ubc.ca> on 2020/02/01 00:00:04 UTC
Using something other than colons in field names?
Good afternoon,
Our Metron installation uses colons in the field names. For example, geo ip enriched data appears as “enrichments:geo:ip_dst_addr:country”. Under Kibana (and from what I read Banana), the colon cannot be properly escaped for use with Timelion.
My question: has anyone figured out a way to escape colons in their query or another work around in general? Is there a setting somewhere that can be used to change the default from a colon to a period or another character?
Thank you,
Tom.
Re: Using something other than colons in field names?
Posted by "Yerex, Tom" <to...@ubc.ca>.
Thanks Simon.
--T.
From: Simon Elliston Ball <si...@simonellistonball.com>
Reply-To: "user@metron.apache.org" <us...@metron.apache.org>
Date: Monday, February 3, 2020 at 2:02 PM
To: "user@metron.apache.org" <us...@metron.apache.org>
Subject: Re: Using something other than colons in field names?
The colons were originally required to avoid poor performance in sub documents in elastic search 2.x. It’s really a legacy thing now, and the NOOP should be considered the best path going forward.
Simon
On Mon, 3 Feb 2020 at 12:41, Yerex, Tom <to...@ubc.ca> wrote:
Thank you Vladimir.
Before I go diving into making a lot of changes from the default, does anyone happen to recall why the colon was selected as the default? I'm curious if it works better for analysis in HDFS or Zeppelin?
Cheers,
Tom.
On 2020-02-02, 8:53 PM, "Vladimir Mikhailov" <v....@content-media.ru> wrote:
Hi
There is a parameter "fieldNameConverter" in the parser indexing configuration:
fieldNameConverter
"Defines how field names are transformed before being written to the index. Only applicable to elasticsearch.
Defaults to DEDOT. Acceptable values are DEDOT that replaces all '.' with ':' or NOOP that does not change the field names."
https://github.com/apache/metron/blob/master/metron-platform/metron-indexing/metron-indexing-common/README.md#sensor-indexing-configuration
Usage example:
"elasticsearch": {
"batchSize": 100,
"enabled": true,
"index": "myindex",
"fieldNameConverter": "NOOP"
},
On 2020/02/01 00:00:04, "Yerex, Tom" <to...@ubc.ca> wrote:
> Good afternoon,
>
>
>
> Our Metron installation uses colons in the field names. For example, geo ip enriched data appears as “enrichments:geo:ip_dst_addr:country”. Under Kibana (and from what I read Banana), the colon cannot be properly escaped for use with Timelion.
>
>
>
> My question: has anyone figured out a way to escape colons in their query or another work around in general? Is there a setting somewhere that can be used to change the default from a colon to a period or another character?
>
>
>
> Thank you,
>
>
>
> Tom.
>
>
>
>
--
--
simon elliston ball
@sireb
Re: Using something other than colons in field names?
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
The colons were originally required to avoid poor performance in sub
documents in elastic search 2.x. It’s really a legacy thing now, and the
NOOP should be considered the best path going forward.
Simon
On Mon, 3 Feb 2020 at 12:41, Yerex, Tom <to...@ubc.ca> wrote:
> Thank you Vladimir.
>
> Before I go diving into making a lot of changes from the default, does
> anyone happen to recall why the colon was selected as the default? I'm
> curious if it works better for analysis in HDFS or Zeppelin?
>
> Cheers,
>
> Tom.
>
> On 2020-02-02, 8:53 PM, "Vladimir Mikhailov" <
> v.mikhailov@content-media.ru> wrote:
>
> Hi
>
> There is a parameter "fieldNameConverter" in the parser indexing
> configuration:
>
> fieldNameConverter
>
> "Defines how field names are transformed before being written to the
> index. Only applicable to elasticsearch.
>
> Defaults to DEDOT. Acceptable values are DEDOT that replaces all '.'
> with ':' or NOOP that does not change the field names."
>
>
> https://github.com/apache/metron/blob/master/metron-platform/metron-indexing/metron-indexing-common/README.md#sensor-indexing-configuration
>
> Usage example:
>
> "elasticsearch": {
> "batchSize": 100,
> "enabled": true,
> "index": "myindex",
> "fieldNameConverter": "NOOP"
> },
>
> On 2020/02/01 00:00:04, "Yerex, Tom" <to...@ubc.ca> wrote:
> > Good afternoon,
> >
> >
> >
> > Our Metron installation uses colons in the field names. For example,
> geo ip enriched data appears as “enrichments:geo:ip_dst_addr:country”.
> Under Kibana (and from what I read Banana), the colon cannot be properly
> escaped for use with Timelion.
> >
> >
> >
> > My question: has anyone figured out a way to escape colons in their
> query or another work around in general? Is there a setting somewhere that
> can be used to change the default from a colon to a period or another
> character?
> >
> >
> >
> > Thank you,
> >
> >
> >
> > Tom.
> >
> >
> >
> >
>
> --
--
simon elliston ball
@sireb
Re: Using something other than colons in field names?
Posted by "Yerex, Tom" <to...@ubc.ca>.
Thank you Vladimir.
Before I go diving into making a lot of changes from the default, does anyone happen to recall why the colon was selected as the default? I'm curious if it works better for analysis in HDFS or Zeppelin?
Cheers,
Tom.
On 2020-02-02, 8:53 PM, "Vladimir Mikhailov" <v....@content-media.ru> wrote:
Hi
There is a parameter "fieldNameConverter" in the parser indexing configuration:
fieldNameConverter
"Defines how field names are transformed before being written to the index. Only applicable to elasticsearch.
Defaults to DEDOT. Acceptable values are DEDOT that replaces all '.' with ':' or NOOP that does not change the field names."
https://github.com/apache/metron/blob/master/metron-platform/metron-indexing/metron-indexing-common/README.md#sensor-indexing-configuration
Usage example:
"elasticsearch": {
"batchSize": 100,
"enabled": true,
"index": "myindex",
"fieldNameConverter": "NOOP"
},
On 2020/02/01 00:00:04, "Yerex, Tom" <to...@ubc.ca> wrote:
> Good afternoon,
>
>
>
> Our Metron installation uses colons in the field names. For example, geo ip enriched data appears as “enrichments:geo:ip_dst_addr:country”. Under Kibana (and from what I read Banana), the colon cannot be properly escaped for use with Timelion.
>
>
>
> My question: has anyone figured out a way to escape colons in their query or another work around in general? Is there a setting somewhere that can be used to change the default from a colon to a period or another character?
>
>
>
> Thank you,
>
>
>
> Tom.
>
>
>
>
Re: Using something other than colons in field names?
Posted by Vladimir Mikhailov <v....@content-media.ru>.
Hi
There is a parameter "fieldNameConverter" in the parser indexing configuration:
fieldNameConverter
"Defines how field names are transformed before being written to the index. Only applicable to elasticsearch.
Defaults to DEDOT. Acceptable values are DEDOT that replaces all '.' with ':' or NOOP that does not change the field names."
https://github.com/apache/metron/blob/master/metron-platform/metron-indexing/metron-indexing-common/README.md#sensor-indexing-configuration
Usage example:
"elasticsearch": {
"batchSize": 100,
"enabled": true,
"index": "myindex",
"fieldNameConverter": "NOOP"
},
On 2020/02/01 00:00:04, "Yerex, Tom" <to...@ubc.ca> wrote:
> Good afternoon,
>
>
>
> Our Metron installation uses colons in the field names. For example, geo ip enriched data appears as “enrichments:geo:ip_dst_addr:country”. Under Kibana (and from what I read Banana), the colon cannot be properly escaped for use with Timelion.
>
>
>
> My question: has anyone figured out a way to escape colons in their query or another work around in general? Is there a setting somewhere that can be used to change the default from a colon to a period or another character?
>
>
>
> Thank you,
>
>
>
> Tom.
>
>
>
>