Records duplicating in alerts ui when commenting

[Euan Hope <ho...@gmail.com>]

Hello Metron community.

My users have encountered a duplication of records in the alerts ui when
the user places a comment for that specific record.

I’m not sure why this is happening.

Could anyone advise and provide some guidance?

Thanking you in advance for your assistance

Regards

Re: Records duplicating in alerts ui when commenting

[Euan Hope <ho...@gmail.com>]

Apologies for my regular posts on this topics. As I continue to look into
this, I am providing more insights I am coming across and hopefully this
helps to provide context as to what we are facing challenges with.

What I have recently observed is that when I initially search for records,
I see that each has a document id assigned.

When a comment is added to a recorded, the document id for the record
remains the same following to submission of this comment.

When the status is changed (say from “NEW” to “OPEN”, a patch is done using
/api/v1/update/patch. When viewing the content submitted, the document id
is null.

When another search is done, a new document id is assigned to the record
that had a null document id. And so in the ui there is a record where the
status shows as “NEW” in the ui (but this field is not visible in the
payload) and also has the comment submitted.

There is also a replica of the same record but it has a different document
id, it has an alert_status of “OPEN” and also has the comment.

What seems to then happen is that each time the status is changed, and
subsequently a patch is submitted, the document id assigned is null and
then a new document id is created.

After the above steps, I also noticed that when I add another comment, the
same behavior of assigning a null value to document id occurs. And this was
not the case when the first comment was submitted.

To determine, if this same behavior occurs when comments are not submitted,
I search for another record. When I change the status on this record, the
document id remains the same each time the status is updated. At no time is
there a document id of null.

Hoping that this might help to shed light on whether this is expected
behavior or if we have configured something incorrectly on our side.

Thank you.


On Tue, 19 May 2020 at 16:00, Euan Hope <ho...@gmail.com> wrote:

> Hello Metron community,
>
> I have been delving further into the replication of records that our users
> are experiencing in the Metron Alerts ui.
>
> What we have picked up is that when there is a status change, ie from
> “NEW” to “OPEN”. We do not get a replication of the record.
>
> However, when the user inserts a comment, the record is replicated. When
> looking at the payload, the replicated record has a different document id
> to the original record.
>
> Then, when either another comment is added or if the status is changed
> (say from “OPEN” to “RESOLVE”) either of these operations will result in
> more replications of the record.
>
> After searching around, I came across this pull request (METRON-1677). I
> was wondering if what we are facing is related?
>
> Thanking you in advance for any assistance on this.
>
> On Tue, 12 May 2020 at 15:44, Euan Hope <ho...@gmail.com> wrote:
>
>> My sincerest apologies for the very late response to this.
>>
>> We haven’t changed any of the default settings.
>>
>> We did define the elasticsearch index ourselves based on the data we are
>> consuming with the sensor.
>>
>> It does occur on other sensors as well.
>>
>> It seems to replicate the original record. Further, when the action is
>> changed from new to open, then another duplicate is create but with the
>> updated alert_status (in this example with the open alert_status). I am not
>> sure if this is expected behavior?
>>
>> Apologies once again for my late response and thank you for your time and
>> assistance.
>>
>>
>>
>> On Fri, 13 Mar 2020 at 23:15, Nick Allen <ni...@nickallen.org> wrote:
>>
>>> Have you changed any default settings? Have you changed the
>>> Elasticsearch index templates at all?  Does the duplication occur for only
>>> one sensor type or for all sensor types?
>>>
>>> On Wed, Mar 11, 2020 at 7:20 AM Euan Hope <ho...@gmail.com> wrote:
>>>
>>>> Hello Metron community.
>>>>
>>>> My users have encountered a duplication of records in the alerts ui
>>>> when the user places a comment for that specific record.
>>>>
>>>> I’m not sure why this is happening.
>>>>
>>>> Could anyone advise and provide some guidance?
>>>>
>>>> Thanking you in advance for your assistance
>>>>
>>>> Regards
>>>>
>>>

Re: Records duplicating in alerts ui when commenting

[Euan Hope <ho...@gmail.com>]

Hello Metron community,

I have been delving further into the replication of records that our users
are experiencing in the Metron Alerts ui.

What we have picked up is that when there is a status change, ie from “NEW”
to “OPEN”. We do not get a replication of the record.

However, when the user inserts a comment, the record is replicated. When
looking at the payload, the replicated record has a different document id
to the original record.

Then, when either another comment is added or if the status is changed (say
from “OPEN” to “RESOLVE”) either of these operations will result in more
replications of the record.

After searching around, I came across this pull request (METRON-1677). I
was wondering if what we are facing is related?

Thanking you in advance for any assistance on this.

On Tue, 12 May 2020 at 15:44, Euan Hope <ho...@gmail.com> wrote:

> My sincerest apologies for the very late response to this.
>
> We haven’t changed any of the default settings.
>
> We did define the elasticsearch index ourselves based on the data we are
> consuming with the sensor.
>
> It does occur on other sensors as well.
>
> It seems to replicate the original record. Further, when the action is
> changed from new to open, then another duplicate is create but with the
> updated alert_status (in this example with the open alert_status). I am not
> sure if this is expected behavior?
>
> Apologies once again for my late response and thank you for your time and
> assistance.
>
>
>
> On Fri, 13 Mar 2020 at 23:15, Nick Allen <ni...@nickallen.org> wrote:
>
>> Have you changed any default settings? Have you changed the Elasticsearch
>> index templates at all?  Does the duplication occur for only one sensor
>> type or for all sensor types?
>>
>> On Wed, Mar 11, 2020 at 7:20 AM Euan Hope <ho...@gmail.com> wrote:
>>
>>> Hello Metron community.
>>>
>>> My users have encountered a duplication of records in the alerts ui when
>>> the user places a comment for that specific record.
>>>
>>> I’m not sure why this is happening.
>>>
>>> Could anyone advise and provide some guidance?
>>>
>>> Thanking you in advance for your assistance
>>>
>>> Regards
>>>
>>

Re: Records duplicating in alerts ui when commenting

[Euan Hope <ho...@gmail.com>]

My sincerest apologies for the very late response to this.

We haven’t changed any of the default settings.

We did define the elasticsearch index ourselves based on the data we are
consuming with the sensor.

It does occur on other sensors as well.

It seems to replicate the original record. Further, when the action is
changed from new to open, then another duplicate is create but with the
updated alert_status (in this example with the open alert_status). I am not
sure if this is expected behavior?

Apologies once again for my late response and thank you for your time and
assistance.



On Fri, 13 Mar 2020 at 23:15, Nick Allen <ni...@nickallen.org> wrote:

> Have you changed any default settings? Have you changed the Elasticsearch
> index templates at all?  Does the duplication occur for only one sensor
> type or for all sensor types?
>
> On Wed, Mar 11, 2020 at 7:20 AM Euan Hope <ho...@gmail.com> wrote:
>
>> Hello Metron community.
>>
>> My users have encountered a duplication of records in the alerts ui when
>> the user places a comment for that specific record.
>>
>> I’m not sure why this is happening.
>>
>> Could anyone advise and provide some guidance?
>>
>> Thanking you in advance for your assistance
>>
>> Regards
>>
>

Re: Records duplicating in alerts ui when commenting

[Nick Allen <ni...@nickallen.org>]

Have you changed any default settings? Have you changed the Elasticsearch
index templates at all?  Does the duplication occur for only one sensor
type or for all sensor types?

On Wed, Mar 11, 2020 at 7:20 AM Euan Hope <ho...@gmail.com> wrote:

> Hello Metron community.
>
> My users have encountered a duplication of records in the alerts ui when
> the user places a comment for that specific record.
>
> I’m not sure why this is happening.
>
> Could anyone advise and provide some guidance?
>
> Thanking you in advance for your assistance
>
> Regards
>