You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@metron.apache.org by Justin Leet <ju...@gmail.com> on 2017/09/29 13:59:48 UTC

[DISCUSS] Meta alert Elasticsearch new template requirement ramifications

As part of building a backend for meta-alerts (
https://github.com/apache/metron/pull/734), there's an additional
requirement for the Elasticsearch templates for new sensors. Although
seemingly minor, this should be called out explicitly because of the wider
implications of leaving it out of ANY sensor. Specifically, this can
result in the UI and other queries not returning results, because
Elasticsearch throws an error.

A nested "alert" field must be added in the form:

"alert": {
"type": "nested"
},

This results from Elasticsearch 2.x requiring the type of searches that
meta alerts wrap to have the fields exist in all indices or the query fails.

In Elasticsearch 5.x, there is a per query parameter that can be set to
avoid this: see
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields
.

The obvious short-term thing that needs to happen with this is improved
documentation. A ticket for documenting this (with some more specific
details and what the error looks like is at
https://issues.apache.org/jira/browse/METRON-1220). Where should that
documentation live? It seems like our documentation in general around this
type of stuff is a little lacking. Right now, I'm putting a prelim version
into the metron-indexing README, but either now or as a followon we should
have a more robust version that lays requirements for things like
templating.

There are a couple options I see to address this more robustly.
1) Just upgrade to ES 5.x and modify the meta alert query appropriately. A
beginning basis for this change exists in
https://github.com/apache/metron/pull/619. More works needs to happen
there to finalize it
2) Add in ZK hooks for when a new sensor is added. The DAOs could receive
word that a new sensor has been added in ZK and then build and submit the
modified template itself. This (or a variant) is probably something that
should happen anyway, in order to be more consistent with the other pieces
that monitor and act on ZK updates.
3) There may be some mitigating that can be done here, e.g. if a query
fails with the relevant error, rerun a different variant that may not hit
the meta alerts, but doesn't fail as extravagantly.

Is there a preference on either where the new documentation lives? And is
there a preference on how we address this going forward?

Justin

Re: [DISCUSS] Meta alert Elasticsearch new template requirement ramifications

Posted by Rita McKissick <rm...@hortonworks.com>.

Hi Justin,

I am adding the nested “alert" field requirement to the product documentation, and I had a question:

* If the user opts to use the default configuration, will they still need to add the nested “alert” field?

When I’m finished with the documentation, I’ll send you a link to make sure what I’ve documented is correct.

Thanks,

Rita


Rita McKissick ! Sr. Technical Writer
rmckissick@hortonworks.com
(mobile) 831-234-3676








On 9/29/17, 6:59 AM, "Justin Leet" <ju...@gmail.com> wrote:

>As part of building a backend for meta-alerts (
>https://github.com/apache/metron/pull/734), there's an additional
>requirement for the Elasticsearch templates for new sensors.  Although
>seemingly minor, this should be called out explicitly because of the wider
>implications of leaving it out of ANY sensor.  Specifically, this can
>result in the UI and other queries not returning results, because
>Elasticsearch throws an error.
>
>A nested "alert" field must be added in the form:
>
>"alert": {
>   "type": "nested"
>},
>
>This results from Elasticsearch 2.x requiring the type of searches that
>meta alerts wrap to have the fields exist in all indices or the query fails.
>
>In Elasticsearch 5.x, there is a per query parameter that can be set to
>avoid this: see
>https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields
>.
>
>The obvious short-term thing that needs to happen with this is improved
>documentation.  A ticket for documenting this (with some more specific
>details and what the error looks like is at
>https://issues.apache.org/jira/browse/METRON-1220).  Where should that
>documentation live?  It seems like our documentation in general around this
>type of stuff is a little lacking.  Right now, I'm putting a prelim version
>into the metron-indexing README, but either now or as a followon we should
>have a more robust version that lays requirements for things like
>templating.
>
>There are a couple options I see to address this more robustly.
>1) Just upgrade to ES 5.x and modify the meta alert query appropriately. A
>beginning basis for this change exists in
>https://github.com/apache/metron/pull/619.  More works needs to happen
>there to finalize it
>2) Add in ZK hooks for when a new sensor is added.  The DAOs could receive
>word that a new sensor has been added in ZK and then build and submit the
>modified template itself.  This (or a variant) is probably something that
>should happen anyway, in order to be more consistent with the other pieces
>that monitor and act on ZK updates.
>3) There may be some mitigating that can be done here, e.g. if a query
>fails with the relevant error, rerun a different variant that may not hit
>the meta alerts, but doesn't fail as extravagantly.
>
>Is there a preference on either where the new documentation lives?  And is
>there a preference on how we address this going forward?
>
>Justin

Re: [DISCUSS] Meta alert Elasticsearch new template requirement ramifications

Posted by Justin Leet <ju...@gmail.com>.

I put up a preliminary PR at https://github.com/apache/metron/pull/780. As
noted there, this should almost certainly be under a different heading, and
possibly a different README, so feel free to chime in on that.  Primary
goal is to make sure the content makes sense and get adjustments as needed.

On Fri, Sep 29, 2017 at 10:10 AM, Otto Fowler <ot...@gmail.com>
wrote:

> We can also consider this when thinking about creating parsers with
> archetypes that contain ‘default’
> elasticsearch templates.
>
>
>
> On September 29, 2017 at 10:00:03, Justin Leet (justinjleet@gmail.com)
> wrote:
>
> As part of building a backend for meta-alerts (
> https://github.com/apache/metron/pull/734), there's an additional
> requirement for the Elasticsearch templates for new sensors. Although
> seemingly minor, this should be called out explicitly because of the wider
> implications of leaving it out of ANY sensor. Specifically, this can
> result in the UI and other queries not returning results, because
> Elasticsearch throws an error.
>
> A nested "alert" field must be added in the form:
>
> "alert": {
> "type": "nested"
> },
>
> This results from Elasticsearch 2.x requiring the type of searches that
> meta alerts wrap to have the fields exist in all indices or the query
> fails.
>
> In Elasticsearch 5.x, there is a per query parameter that can be set to
> avoid this: see
> https://www.elastic.co/guide/en/elasticsearch/reference/
> current/search-request-sort.html#_ignoring_unmapped_fields
> .
>
> The obvious short-term thing that needs to happen with this is improved
> documentation. A ticket for documenting this (with some more specific
> details and what the error looks like is at
> https://issues.apache.org/jira/browse/METRON-1220). Where should that
> documentation live? It seems like our documentation in general around this
> type of stuff is a little lacking. Right now, I'm putting a prelim version
> into the metron-indexing README, but either now or as a followon we should
> have a more robust version that lays requirements for things like
> templating.
>
> There are a couple options I see to address this more robustly.
> 1) Just upgrade to ES 5.x and modify the meta alert query appropriately. A
> beginning basis for this change exists in
> https://github.com/apache/metron/pull/619. More works needs to happen
> there to finalize it
> 2) Add in ZK hooks for when a new sensor is added. The DAOs could receive
> word that a new sensor has been added in ZK and then build and submit the
> modified template itself. This (or a variant) is probably something that
> should happen anyway, in order to be more consistent with the other pieces
> that monitor and act on ZK updates.
> 3) There may be some mitigating that can be done here, e.g. if a query
> fails with the relevant error, rerun a different variant that may not hit
> the meta alerts, but doesn't fail as extravagantly.
>
> Is there a preference on either where the new documentation lives? And is
> there a preference on how we address this going forward?
>
> Justin
>
>

Re: [DISCUSS] Meta alert Elasticsearch new template requirement ramifications

Posted by Otto Fowler <ot...@gmail.com>.

We can also consider this when thinking about creating parsers with
archetypes that contain ‘default’
elasticsearch templates.

On September 29, 2017 at 10:00:03, Justin Leet (justinjleet@gmail.com)
wrote:

A nested "alert" field must be added in the form:

"alert": {
"type": "nested"
},

This results from Elasticsearch 2.x requiring the type of searches that
meta alerts wrap to have the fields exist in all indices or the query
fails.

Is there a preference on either where the new documentation lives? And is
there a preference on how we address this going forward?

Justin