[jira] [Commented] (METRON-1301) Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/METRON-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240758#comment-16240758 ] 

ASF GitHub Bot commented on METRON-1301:
----------------------------------------

GitHub user nickwallen opened a pull request:

    https://github.com/apache/metron/pull/832

    METRON-1301 Sorting on Triage Score Unexpectedly Filters Some Recods

    This PR makes the following changes.
    
    * Adds logic to sort fields that are not present in all of the underlying indices.  For example, threat triage score is only present in the records that have been triaged.  Without this PR, sorting on the threat triage score would unexpectedly filter the search results.
        * When sorted descending, missing fields are sorted last.
        * When sorted ascending, missing fields are sorted first.
    
    * Adds validation and error checking of the response received from Elasticsearch.  Previously there was little to no validation of the response.  This also handles partial shard failures; errors that occur only on a subset of the data.  Without this change some errors that occurred within Elasticsearch were not visible.
    
    * Adds additional logging in the ElasticsearchDAO so that errors can be traced and debugged.
    
    * Refactored the ElasticsearchDAO and extracted some logic into two new interfaces.  This was done so that I could write tests for the new logic.  It was not possible to specifically test this change without refactoring due to limitations with mocking.
        * The column metadata logic is now contained in `ColumnMetadataDao` and `ElasticsearchColumnMetadataDao`.  Without pulling this logic out, it was not possible to test the new sorting logic.
        * The logic for submitting a request to Elasticsearch and validating the response is now contained within `ElasticsearchSearchSubmitter` class.  Without refactoring this it was not possible to test the  new response handling logic.
    
    ## Testing
    
    1. Launch Full Dev
    2. Allow some alerts to accumulate so that when you open the Alerts UI there are a fair number of alerts to work with.
    3. By default it will sort by timestamp.  Take note of the number of records returned.  
    4. Click on the "Score" field to sort the results by threat triage score.
    5. Note the total number of records returned after sorting.  This should be equal to or greater than what you saw before.
    6. The highest scores should be sorted first.  Records that do not contain a threat triage score should be sorted last.
    7. Click on the "Score" field to reverse the sort order.  Records without a threat triage score should now be sorted first.
    
    ## Pull Request Checklist
    
    - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). 
    - [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
    - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
    - [ ] Have you included steps or a guide to how the change may be verified and tested manually?
    - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
    - [ ] Have you written or updated unit tests and or integration tests to verify your changes?
    - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? 
    - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
    - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nickwallen/metron METRON-1301

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/832.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #832
    
----
commit de7eba97331442fac293c5c3ca72993f7a431200
Author: Nick Allen <ni...@nickallen.org>
Date:   2017-10-30T16:02:46Z

    METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records

----


> Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records
> ---------------------------------------------------------------------
>
>                 Key: METRON-1301
>                 URL: https://issues.apache.org/jira/browse/METRON-1301
>             Project: Metron
>          Issue Type: Bug
>    Affects Versions: 0.4.1
>            Reporter: Nick Allen
>            Assignee: Nick Allen
>             Fix For: Next + 1
>
>         Attachments: 01-Alerts-UI-default-view.png, 02-Sort-on-Score-field.png
>
>
> Sorting on a field like threat triage score in the Alerts UI removes any records that do not have a threat triage score defined from the search results.
> For example, I have 7 records when sorted by timestamp.  All 7 records have a timestamp field.
> After sorting by score (threat triage score) there are only 5 records.  The 2 records missing a threat triage score are no longer included.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)