You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by iraghumitra <gi...@git.apache.org> on 2017/10/17 19:02:27 UTC

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

GitHub user iraghumitra opened a pull request:

    https://github.com/apache/metron/pull/803

    Metron-1252: Build ui for grouping alerts into meta alerts

    ## Contributor Comments
    The purpose of the PR is to provide GUI for grouping multiple alerts into a meta alert, the rest api for this is already available via [METRON-1158](https://issues.apache.org/jira/browse/METRON-1158)
    
    The current implementation has following features
    
    -  Meta alert can be created from the tree view of alerts 
    -  Meta alerts can be viewed in the table view 
       -  The meta alert has a link icon on the left to denote that it is a meta alert
       -  The meta alert has a expand/collapse icon to see all the alerts within it
       -  The meta alert has a cumulative score (more on how this score is calculated is available in the original ticket)
    -  Support to add/remove an alert to the meta alert from the table view
    -  Change the state of meta alert
    -  Add comments to a meta alert
    -  Option to name a meta alert (This is just a convenience to refer to alerts)
    
    **Limitations**
    
    - Meta alerts cannot be viewed in the tree view
    - Adding comments/Status change is restricted to just meta alerts. You cannot perform these actions on alerts contained in meta alert
    - A meta alert can contain only 999 alerts
    - Delete of the entire meta alert is not supported yet
    
    **Next**
     - It will be nice to have a notification when entities are changed in the UI
    
    I noticed that search on GUID was not working before I fixed it in this PR.
    
    E2E tests are incoming I wanted to check if the community has any suggestion on this.
    
    PS: I had to comment one of the test cases since sort by guid is broke. I will raise a ticket for it*
    
    ![image](https://user-images.githubusercontent.com/15019012/31682771-b9deb6da-b398-11e7-8681-6696fdcd1b6c.png)
    
    ![image](https://user-images.githubusercontent.com/15019012/31682796-cc5236f2-b398-11e7-8db0-7233b35ba7b8.png)
    
    ![image](https://user-images.githubusercontent.com/15019012/31682824-dda3c02e-b398-11e7-9369-119587041262.png)
    
    ![image](https://user-images.githubusercontent.com/15019012/31682846-ee2451fc-b398-11e7-998e-c4f441048ffe.png)
    
    ## Pull Request Checklist
    
    Thank you for submitting a contribution to Apache Metron.  
    Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.  
    Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.  
    
    
    In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
    
    ### For all changes:
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). 
    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
    
    ### For code changes:
    - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
    - [x] Have you included steps or a guide to how the change may be verified and tested manually?
    - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
      ```
      mvn -q clean integration-test install && build_utils/verify_licenses.sh 
      ```
    
    - [ ] Have you written or updated unit tests and or integration tests to verify your changes?
    - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? 
    - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
    
    ### For documentation related changes:
    - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
    
      ```
      cd site-book
      mvn site
      ```
    
    #### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
    It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/iraghumitra/incubator-metron METRON-1252

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/803.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #803
    
----
commit 52d721e50b2136165d89195978d110c3541b046f
Author: iraghumitra <ra...@gmail.com>
Date:   2017-10-17T13:30:36Z

    Initial commit for grouping alerts into a meta alert

commit 09747a739db12741f26b0f14acb5073986c3d369
Author: iraghumitra <ra...@gmail.com>
Date:   2017-10-17T16:04:38Z

    Disallow comments for a alert in meta alert

commit dba5493d9836047df6411d583b9c12327a524624
Author: iraghumitra <ra...@gmail.com>
Date:   2017-10-17T18:22:18Z

    Fixed the existing tests

----


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @iraghumitra Are all these e2e tests passing for you?  I am still getting failures.  I manually truncated the `metron_updates` table before running them also.
    ```
    **************************************************
    *                    Failures                    *
    **************************************************
    
    1) metron-alerts tree view drag and drop should change group order
      - Expected '0,alerts_ui_e2e,ALERTS,91' to equal '0,alerts_ui_e2e,ALERTS,169' First Dash Row should be correct
      - Expected '' to equal '0 US (22)' Dash Group Values should be correct for US
      - Expected '' to equal '0 RU (44)' Dash Group Values should be present for RU
      - Expected '' to equal '0 FR (25)' Dash Group Values should be present for FR
    
    2) metron-alerts tree view should have sort working for group details for multiple sub groups
      - Failed: stale element reference: element is not attached to the page document
        (Session info: chrome=62.0.3202.94)
        (Driver info: chromedriver=2.33.506106 (8a06c39c4582fbfbab6966dbb1c38a9173bfb1a2),platform=Mac OS X 10.13.1 x86_64)
    
    3) metron-alerts tree view should have search working for group details for multiple sub groups
      - Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
      - Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
    
    4) meta-alerts workflow should create a meta alert from nesting of more than one level
      - Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
      - Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
    
    Executed 44 of 44 specs (4 FAILED) in 8 mins 46 secs.
    ```


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r147852935
  
    --- Diff: metron-interface/metron-alerts/e2e/alerts-list/meta-alerts/meta-alert.po.ts ---
    @@ -0,0 +1,43 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +import {browser, element, by} from 'protractor';
    +
    +export class MetaAlertPage {
    +
    +  getPageTitle() {
    +    return element(by.css('app-meta-alerts .form-title')).getText();
    +  }
    +
    +  getMetaAlertsTitle() {
    +    return element(by.css('app-meta-alerts .title')).getText();
    +  }
    +
    +  getAvailableMetaAlerts() {
    +    return element(by.css('app-meta-alerts .guid-name-container div')).getText();
    +  }
    +
    +  selectRadio() {
    +    return element.all(by.css('app-meta-alerts .checkmark')).click();
    +  }
    +
    +  addToMetaAlert() {
    +    element.all(by.css('app-meta-alerts')).get(0).element(by.buttonText('ADD')).click();
    +    browser.sleep(2000);
    --- End diff --
    
    Is this sleep statement absolutely necessary?


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:

    https://github.com/apache/metron/pull/803
  
    I filed the following follow-on PRs per your comments:
    
    https://issues.apache.org/jira/browse/METRON-1268
    https://issues.apache.org/jira/browse/METRON-1269
    



---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r151964471
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.scss ---
    @@ -143,3 +167,12 @@ textarea {
     .comment-container:hover i {
       display: block;
     }
    +
    +.input-group-addon {
    +  cursor: pointer;
    +}
    +
    +.disabled {
    +  opacity: 0.5;
    +  cursor: not-allowed;
    +}
    --- End diff --
    
    Added newline


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @nickwallen can you raise a jira to track this. I guess even if we are able to reproduce I highly doubt if it is bcoz of changes in this PR.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @nickwallen I agree, unless I'm missing something, it seems like a bug with recent searches, since you didn't say searches were weird during the initial attempts.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @merrimanr Looks like the following rest-api's are not working as expected
     - When a meta alert is created and when we try to group by an IP address(ip_dst_addr/ip_src_addr) the IP address is shown as string instead of IP
     - When a meta alert is created the alert details of an alert under meta-alert are missing when we query using findone
    
    The spec that is failing is unable to find state field in the details view bcoz of findone issue  that i mentioned hence the failure. (The alert-detail-status.e2e-spec.ts set's the status of the alert that it is trying to test to [NEW](https://github.com/apache/metron/blob/383d798c7827e916068f89ff362a6ecccb550241/metron-interface/metron-alerts/e2e/alert-details/alert-status/alert-details-status.e2e-spec.ts#L55) before starting. So I feel the issue might not be bcoz of #796. Anyhow, I will check this again)
    
    I am trying to address the usage of sleep in specs by removing the 'fade' transition for dialogs. We don't need this since the dialog comes on top of the window and the css transition is not needed.
    
    Also, i noticed that sometimes when we delete a meta alert not all the alerts in the meta alert are returned back the alerts pool.Ex: If we have 100 alerts and create a meta-alert with 50 alerts and then delete the meta alert. The alert search shows only 60 alerts.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145816832
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/table-view/table-view.component.scss ---
    @@ -24,4 +24,12 @@
     .configure-table-icon {
       font-size: 16px;
       cursor: pointer;
    +}
    +
    +.fa-chain-broken {
    +  color: $piction-blue;
    +}
    +
    +.dropdown-cell {
    +  padding-left: 0.6rem;
     }
    --- End diff --
    
    need newline


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @nickwallen This is unfortunate  I am not seeing these failures
    Spec started
    
      login to application
        ✓ should display error message for invalid credentials
        ✓ should login for valid credentials
        ✓ should logout
    
      metron-alerts App
        ✓ should have all the UI elements
        ✓ should have all pagination controls and they should be working
        ✓ should have all settings controls and they should be working
        ✓ play pause should start polling and stop polling
        ✓ should select columns from table configuration
        ✓ should have all time-range controls
        ✓ should have all time range values populated - 1
        ✓ should have all time range values populated - 2
        ✓ should have all time range values populated - 3
        ✓ should have all time range values populated - 4
        ✓ should disable date picker when timestamp is present in search
        ✓ should have now included when to date is empty
        ✓ should have all time-range included while searching
    
      metron-alerts configure table
        ✓ should select columns from table configuration
        ✓ should rename columns from table configuration
    
      metron-alerts Search
        ✓ should display all the default values for saved searches
        ✓ should have all save search controls and they save search should be working
        ✓ should populate search items when selected on table
        ✓ should delete search items from search box
        ✓ should delete first search items from search box having multiple search fields
        ✓ manually entering search queries to search box and pressing enter key should search
    
      metron-alerts tree view
        ✓ should have all group by elements
        ✓ drag and drop should change group order
        ✓ should have group details for single group by
        ✓ should have group details for multiple group by
        ✓ should have sort working for group details for multiple sub groups
        ✓ should have search working for group details for multiple sub groups
    
      metron-alerts facets
        ✓ should display facets data
        ✓ should expand all facets
        ✓ should have all facet  values
        ✓ should collapse all facets
    
      metron-alerts alert status
        ✓ should change alert status for multiple alerts to OPEN
        ✓ should change alert status for multiple alerts to DISMISS
        ✓ should change alert status for multiple alerts to ESCALATE
        ✓ should change alert status for multiple alerts to RESOLVE
        ✓ should change alert status for multiple alerts to OPEN in tree view
    
      metron-alerts alert status
        ✓ should change alert statuses
        ✓ should add comments for table view
        ✓ should add comments for tree view
    
      meta-alerts workflow
        ✓ should have all the steps for meta alerts workflow
        ✓ should create a meta alert from nesting of more than one level
    
    Executed 44 of 44 specs SUCCESS in 6 mins 31 secs.
    [20:11:19] I/launcher - 0 instance(s) of WebDriver still running
    [20:11:19] I/launcher - chrome #01 passed
    ------------------------------------------------------------
    ~/incubator-metron/metron-interface/metron-alerts(METRON-1252*) » Date                                                                                                                
    Thu Nov 23 00:36:54 IST 2017`
    
    Can you try them running again, UI tests are sometimes flaky. There seems to be timing issue. If you run them independently they work all the time, when we run them together they are flaky.
    I am currently working on stabilizing them.  If you notice some of the test cases have nothing to do with current PR :(


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/803
  
    +1  
    
    I'd like to see sign-off from at least one other committer (if not more) before this gets merged.
    
    I previously outlined the manual functional testing that I performed.  All the core functionality is there.  We also have sufficient test coverage for the functionality that was added.
    
    I have been able to get the e2e tests to run good enough (IMO) to move forward with the PR.  Based on the experimentation that I describe below, I believe that the test failures are due to the tests and test infrastructure and not the core functionality in the PR.  
    
    The intermittent failures are a pre-existing condition to this PR.  This makes me comfortable enough to merge this PR and then follow-on with test fixes.  @iraghumitra mentioned that he is currently working on improving the tests and will open a separate PR.  
    
    1. I used Brew to install the following versions of Node/NPM.
        ```
        $ node --version
        v8.9.1
    
        $ npm --version
        5.5.1
        ```
    
    2. Before running the tests, truncate the `metron_update` table.  The tests cannot currently clear out old test data stored in HBase.
    
    3. Enabling/disabling sets of tests in `metron-interface/metron-alerts/protractor.conf.js` also allowed me to see that all the tests can pass on an intermittent basis.  It seems that earlier test failures may cause later tests to fail unnecessarily.
    
    4. Run the e2e tests using the following steps.
    
        ```
        cd metron-interface/metron-alerts
        npm install
        ./node_modules/.bin/webdriver-manager update
        ./node_modules/.bin/protractor
        ```
    
    Thanks for working through this with me @iraghumitra.



---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145816388
  
    --- Diff: metron-interface/metron-alerts/src/app/service/update.service.ts ---
    @@ -38,22 +41,27 @@ export class UpdateService {
     
       constructor(private http: Http) { }
     
    -  public patch(patchRequest: PatchRequest): Observable<{}> {
    +  public patch(patchRequest: PatchRequest, fireChangeListner = true): Observable<{}> {
    --- End diff --
    
    Could we fix the typo in Listner?


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145816267
  
    --- Diff: metron-interface/metron-alerts/src/app/service/update.service.ts ---
    @@ -38,22 +41,27 @@ export class UpdateService {
     
       constructor(private http: Http) { }
     
    -  public patch(patchRequest: PatchRequest): Observable<{}> {
    +  public patch(patchRequest: PatchRequest, fireChangeListner = true): Observable<{}> {
         let url = '/api/v1/update/patch';
         return this.http.patch(url, patchRequest, new RequestOptions({headers: new Headers(this.defaultHeaders)}))
         .catch(HttpUtil.handleError)
         .map(result => {
    -      this.alertChangedSource.next(patchRequest);
    +      if (fireChangeListner) {
    +        this.alertChangedSource.next(patchRequest);
    +      }
           return result;
         });
       }
     
    -  public updateAlertState(alerts: Alert[], state: string): Observable<{}> {
    +  public updateAlertState(alerts: Alert[], state: string, fireChangeListner = true): Observable<{}> {
    --- End diff --
    
    is fireChangeListner used in this function?


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra closed the pull request at:

    https://github.com/apache/metron/pull/803


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r146964503
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts ---
    @@ -337,12 +343,67 @@ export class TreeViewComponent extends TableViewComponent implements OnChanges {
         });
       }
     
    +  canCreateMetaAlert(count: number) {
    +    if (count > 999) {
    --- End diff --
    
    Given the practical restriction detailed at https://issues.apache.org/jira/browse/METRON-1279, does this limit need to be lowered?  I've seen ~350 work without issues, personally.  Obviously, given the way the update works, I'm not sure we can ever guarantee a fixed limit, but even 500 seems really likely to blow up.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r146267005
  
    --- Diff: metron-interface/metron-alerts/src/app/service/update.service.ts ---
    @@ -38,22 +41,27 @@ export class UpdateService {
     
       constructor(private http: Http) { }
     
    -  public patch(patchRequest: PatchRequest): Observable<{}> {
    +  public patch(patchRequest: PatchRequest, fireChangeListner = true): Observable<{}> {
         let url = '/api/v1/update/patch';
         return this.http.patch(url, patchRequest, new RequestOptions({headers: new Headers(this.defaultHeaders)}))
         .catch(HttpUtil.handleError)
         .map(result => {
    -      this.alertChangedSource.next(patchRequest);
    +      if (fireChangeListner) {
    +        this.alertChangedSource.next(patchRequest);
    +      }
           return result;
         });
       }
     
    -  public updateAlertState(alerts: Alert[], state: string): Observable<{}> {
    +  public updateAlertState(alerts: Alert[], state: string, fireChangeListner = true): Observable<{}> {
    --- End diff --
    
    I agree with you and I think that's a good approach.  The problem is you're not actually using that variable in this method.  Maybe you meant to pass it to `this.patch(patchRequest)` on line 69?


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r147853067
  
    --- Diff: metron-interface/metron-alerts/e2e/alerts-list/tree-view/tree-view.po.ts ---
    @@ -159,4 +159,34 @@ export class TreeViewPage {
           return column.getText();
         });
       }
    +
    +  clickOnMergeAlerts(groupName: string) {
    +    return element(by.css('[data-name="' + groupName + '"] .fa-link')).click();
    +  }
    +
    +  getConfirmationText() {
    +    browser.sleep(1000);
    +    let dialogElement = element(by.css('.metron-dialog .modal-header .close'));
    +    return waitForElementVisibility(dialogElement).then(() =>  element(by.css('.metron-dialog .modal-body')).getText());
    +  }
    +
    +  clickNoForConfirmation() {
    +    browser.sleep(1000);
    --- End diff --
    
    Is this sleep statement absolutely necessary?


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by iraghumitra <gi...@git.apache.org>.
GitHub user iraghumitra reopened a pull request:

    https://github.com/apache/metron/pull/803

    Metron-1252: Build ui for grouping alerts into meta alerts

    ## Contributor Comments
    The purpose of the PR is to provide GUI for grouping multiple alerts into a meta alert, the rest api for this is already available via [METRON-1158](https://issues.apache.org/jira/browse/METRON-1158)
    
    The current implementation has following features
    
    -  Meta alert can be created from the tree view of alerts 
    -  Meta alerts can be viewed in the table view 
       -  The meta alert has a link icon on the left to denote that it is a meta alert
       -  The meta alert has a expand/collapse icon to see all the alerts within it
       -  The meta alert has a cumulative score (more on how this score is calculated is available in the original ticket)
    -  Support to add/remove an alert to the meta alert from the table view
    -  Change the state of meta alert
    -  Add comments to a meta alert
    -  Option to name a meta alert (This is just a convenience to refer to alerts)
    
    **Limitations**
    
    - Meta alerts cannot be viewed in the tree view
    - Adding comments/Status change is restricted to just meta alerts. You cannot perform these actions on alerts contained in meta alert
    - A meta alert can contain only 999 alerts
    - Delete of the entire meta alert is not supported yet
    
    **Next**
     - It will be nice to have a notification when entities are changed in the UI
    
    I noticed that search on GUID was not working before I fixed it in this PR.
    
    ~E2E tests are incoming I wanted to check if the community has any suggestion on this.~
    
    ~PS: I had to comment one of the test cases since sort by guid is broke. I will raise a ticket for it~
    
    ![image](https://user-images.githubusercontent.com/15019012/31682771-b9deb6da-b398-11e7-8681-6696fdcd1b6c.png)
    
    ![image](https://user-images.githubusercontent.com/15019012/31682796-cc5236f2-b398-11e7-8db0-7233b35ba7b8.png)
    
    ![image](https://user-images.githubusercontent.com/15019012/31682824-dda3c02e-b398-11e7-9369-119587041262.png)
    
    ![image](https://user-images.githubusercontent.com/15019012/31682846-ee2451fc-b398-11e7-998e-c4f441048ffe.png)
    
    ## Pull Request Checklist
    
    Thank you for submitting a contribution to Apache Metron.  
    Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.  
    Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.  
    
    
    In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
    
    ### For all changes:
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). 
    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
    
    ### For code changes:
    - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
    - [x] Have you included steps or a guide to how the change may be verified and tested manually?
    - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
      ```
      mvn -q clean integration-test install && build_utils/verify_licenses.sh 
      ```
    
    - [ ] Have you written or updated unit tests and or integration tests to verify your changes?
    - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? 
    - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
    
    ### For documentation related changes:
    - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
    
      ```
      cd site-book
      mvn site
      ```
    
    #### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
    It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/iraghumitra/incubator-metron METRON-1252

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/803.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #803
    
----
commit 52d721e50b2136165d89195978d110c3541b046f
Author: iraghumitra <ra...@gmail.com>
Date:   2017-10-17T13:30:36Z

    Initial commit for grouping alerts into a meta alert

commit 09747a739db12741f26b0f14acb5073986c3d369
Author: iraghumitra <ra...@gmail.com>
Date:   2017-10-17T16:04:38Z

    Disallow comments for a alert in meta alert

commit dba5493d9836047df6411d583b9c12327a524624
Author: iraghumitra <ra...@gmail.com>
Date:   2017-10-17T18:22:18Z

    Fixed the existing tests

commit 3b48f7166057ae3db12fbc602e3a8b9427026f51
Author: merrimanr <me...@gmail.com>
Date:   2017-10-19T17:13:58Z

    Merge branch 'METRON-1252' of https://github.com/iraghumitra/incubator-metron into METRON-1252-review

commit 18c15f8147dd14fec8ae3eee9c1816f4ed8ab8a9
Author: merrimanr <me...@gmail.com>
Date:   2017-10-20T18:56:21Z

    Merge remote-tracking branch 'mirror/master' into METRON-1252-review

commit 30cc762c5894b43f0de29f87e26db982b730d3c9
Author: merrimanr <me...@gmail.com>
Date:   2017-10-21T13:43:19Z

    fixed several bugs

commit d96915cf9c5915fd69c200451e665e2747870bbd
Author: RaghuMitra <ra...@gmail.com>
Date:   2017-10-22T13:41:31Z

    Merge pull request #3 from merrimanr/METRON-1252-review
    
    METRON-1252 bug fixes

commit f704a1f0ea7f51fb3e410e72bf2811d76f332fec
Author: iraghumitra <ra...@gmail.com>
Date:   2017-10-22T13:47:54Z

    Merge branch 'master' into METRON-1252

commit c68ce2163374aa9fdf1323103d61d2885b26acad
Author: iraghumitra <ra...@gmail.com>
Date:   2017-10-23T09:52:18Z

    Fixed review comments

commit 51580d4bf78f1d505b2ec8041f859fb61c16915c
Author: iraghumitra <ra...@gmail.com>
Date:   2017-10-23T20:08:05Z

    E2E tests for Meta Alerts

commit c1a142e010edf3a183b5d9176b7386b25604bdf1
Author: iraghumitra <ra...@gmail.com>
Date:   2017-10-23T21:18:04Z

    Added call to refresh group view after mata-alert is created

commit 0cfa8abd9318601ba73fa12c1df8daa0775414d4
Author: iraghumitra <ra...@gmail.com>
Date:   2017-10-26T11:02:19Z

    Merge master

commit fe54e101347f4a8ae9fac54089883e54f540b1f0
Author: iraghumitra <ra...@gmail.com>
Date:   2017-10-26T13:22:28Z

    Fixes for e2e tests, limited the alerts in meta alerts to 350

----


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145809831
  
    --- Diff: metron-interface/metron-alerts/src/app/utils/constants.ts ---
    @@ -18,6 +18,9 @@
     
     import {environment} from '../../environments/environment';
     
    +export const META_ALERTS_SENSOR_TYPE = 'metaalert';
    +export const META_ALERTS_INDEX = 'metaalerts';
    --- End diff --
    
    should be `export const META_ALERTS_INDEX = 'metaalert';`


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r146216800
  
    --- Diff: metron-interface/metron-alerts/src/styles.scss ---
    @@ -259,4 +259,61 @@ hr {
       padding: 0;
     }
     
    -
    +/** Custom Radio box **/
    +$background_color_1: #eee;
    +$background_color_2: #ccc;
    +$background_color_3: #2196F3;
    +.radio-container {
    +  display: block;
    +  position: relative;
    +  padding-left: 35px;
    +  margin-bottom: 12px;
    +  cursor: pointer;
    +  font-size: 22px;
    +  -webkit-user-select: none;
    +  -moz-user-select: none;
    +  -ms-user-select: none;
    +  user-select: none;
    +  input {
    +    position: absolute;
    +    opacity: 0;
    +    &:checked {
    +      &~.checkmark {
    +        background-color: $eastern-blue-2;
    +        &:after {
    +          display: block;
    +        }
    +      }
    +    }
    +  }
    +  &:hover {
    +    input {
    +      &~.checkmark {
    +        background-color: $eastern-blue-2;
    +      }
    +    }
    +  }
    +  .checkmark {
    +    position: absolute;
    +    top: 0;
    +    left: 0;
    +    height: 12px;
    +    width: 12px;
    +    background-color: $mine-shaft-2;
    +    border: 1px solid $tundora;
    +    border-radius: 50%;
    +
    +    &:after {
    +      top: 2px;
    +      left: 2px;
    +      width: 6px;
    +      height: 6px;
    +      border-radius: 50%;
    +      background: $white;
    +      content: "";
    +      position: absolute;
    +      display: none;
    +    }
    +  }
    +}
    +/**********************/
    --- End diff --
    
    fixed


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145814896
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/meta-alerts/meta-alerts.component.ts ---
    @@ -0,0 +1,101 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * 'License'); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an 'AS IS' BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +import { Component, OnInit } from '@angular/core';
    +import {Router} from '@angular/router';
    +
    +import {MetaAlertService} from '../../service/meta-alert.service';
    +import {UpdateService} from '../../service/update.service';
    +import {SearchRequest} from '../../model/search-request';
    +import {SearchService} from '../../service/search.service';
    +import {SearchResponse} from '../../model/search-response';
    +import {SortField} from '../../model/sort-field';
    +import {META_ALERTS_INDEX} from '../../utils/constants';
    +import {AlertSource} from '../../model/alert-source';
    +import {PatchRequest} from '../../model/patch-request';
    +import {Patch} from '../../model/patch';
    +
    +@Component({
    +  selector: 'app-meta-alerts',
    +  templateUrl: './meta-alerts.component.html',
    +  styleUrls: ['./meta-alerts.component.scss']
    +})
    +export class MetaAlertsComponent implements OnInit {
    +
    +  selectedMetaAlert = '';
    +  searchResponse: SearchResponse = new SearchResponse();
    +
    +  constructor(private router: Router,
    +              private metaAlertService: MetaAlertService,
    +              private updateService: UpdateService,
    +              private searchService: SearchService) {
    +  }
    +
    +  goBack() {
    +    this.router.navigateByUrl('/alerts-list');
    +    return false;
    +  }
    +
    +  ngOnInit() {
    +    let searchRequest = new SearchRequest();
    +    searchRequest.query = '*';
    +    searchRequest.from = 0;
    +    searchRequest.size = 999;
    +    searchRequest.facetFields = [];
    +    searchRequest.indices =  [META_ALERTS_INDEX];
    +    searchRequest.sort = [new SortField('threat:triage:score', 'desc')];
    +    searchRequest.fields = ['id', 'alert_status', 'threat:triage:score', 'count', 'guid', 'name'];
    +
    +    this.searchService.search(searchRequest).subscribe(resp => this.searchResponse = resp);
    +  }
    +
    +  doAddAlertToMetaAlert(alertSources: AlertSource[]) {
    +    let patchRequest = new PatchRequest();
    +    patchRequest.guid = this.selectedMetaAlert;
    +    patchRequest.sensorType = 'metaalert';
    +    patchRequest.index = META_ALERTS_INDEX;
    +    patchRequest.patch = [new Patch('replace', 'alert', alertSources)];
    +
    +    this.updateService.patch(patchRequest).subscribe(rep => {
    +      console.log('Meta alert saved');
    +      this.goBack();
    +    });
    +  }
    +
    +  addAlertToMetaAlert() {
    +    let searchRequest = new SearchRequest();
    +    searchRequest.query = 'guid:"' + this.selectedMetaAlert + '"';
    +    searchRequest.from = 0;
    +    searchRequest.size = 1;
    +    searchRequest.facetFields = [];
    +    searchRequest.indices =  [META_ALERTS_INDEX];
    +    searchRequest.sort = [];
    +    searchRequest.fields = [];
    +
    +    this.searchService.search(searchRequest).subscribe((searchResponse: SearchResponse) => {
    --- End diff --
    
    I think this should be a findOne call instead of a search.  Seems inefficient to search and filter down to a single record using a guid.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145811869
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/meta-alerts/meta-alerts.module.ts ---
    @@ -0,0 +1,14 @@
    +import { NgModule } from '@angular/core';
    +
    +import {routing} from './meta-alerts.routing';
    +import { MetaAlertsComponent } from './meta-alerts.component';
    +import {UpdateService} from '../../service/update.service';
    +import {MetaAlertService} from '../../service/meta-alert.service';
    +import {SharedModule} from '../../shared/shared.module';
    +
    +@NgModule({
    +  imports: [ routing,  SharedModule ],
    +  declarations: [ MetaAlertsComponent ],
    +  providers: [ UpdateService, MetaAlertService ],
    --- End diff --
    
    Should UpdateService be in the list of providers here?  Isn't it application-wide and provided at the app root level as well?


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:

    https://github.com/apache/metron/pull/803
  
    Potentially we want to expose some abstraction for the ES options for missing field sorting (that I admittedly don't know exist in Solr).
    
    https://www.elastic.co/guide/en/elasticsearch/reference/2.3/search-request-sort.html#_missing_values


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @iraghumitra Can you describe how you install Node/NPM on your development box?  I want to install using the same mechanism (and versions) and see if I can get the e2e tests all working like you.  Thanks


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:

    https://github.com/apache/metron/pull/803
  
    Most of the functional tests worked for me as well.  I was able to get the e2e tests to pass after several runs with the exception of a time picker test that I believe is due to a timezone issue and not related to this PR.
    
    I was also able to test that meta alerts show up in search results when a search matches a child alert field.  However I don't see this case covered in the e2e tests.  I think this is a pretty critical function and needs a test.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145809660
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/meta-alerts/meta-alerts.component.ts ---
    @@ -0,0 +1,101 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * 'License'); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an 'AS IS' BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +import { Component, OnInit } from '@angular/core';
    +import {Router} from '@angular/router';
    +
    +import {MetaAlertService} from '../../service/meta-alert.service';
    +import {UpdateService} from '../../service/update.service';
    +import {SearchRequest} from '../../model/search-request';
    +import {SearchService} from '../../service/search.service';
    +import {SearchResponse} from '../../model/search-response';
    +import {SortField} from '../../model/sort-field';
    +import {META_ALERTS_INDEX} from '../../utils/constants';
    +import {AlertSource} from '../../model/alert-source';
    +import {PatchRequest} from '../../model/patch-request';
    +import {Patch} from '../../model/patch';
    +
    +@Component({
    +  selector: 'app-meta-alerts',
    +  templateUrl: './meta-alerts.component.html',
    +  styleUrls: ['./meta-alerts.component.scss']
    +})
    +export class MetaAlertsComponent implements OnInit {
    +
    +  selectedMetaAlert = '';
    +  searchResponse: SearchResponse = new SearchResponse();
    +
    +  constructor(private router: Router,
    +              private metaAlertService: MetaAlertService,
    +              private updateService: UpdateService,
    +              private searchService: SearchService) {
    +  }
    +
    +  goBack() {
    +    this.router.navigateByUrl('/alerts-list');
    +    return false;
    +  }
    +
    +  ngOnInit() {
    +    let searchRequest = new SearchRequest();
    +    searchRequest.query = '*';
    +    searchRequest.from = 0;
    +    searchRequest.size = 999;
    +    searchRequest.facetFields = [];
    +    searchRequest.indices =  [META_ALERTS_INDEX];
    +    searchRequest.sort = [new SortField('threat:triage:score', 'desc')];
    +    searchRequest.fields = ['id', 'alert_status', 'threat:triage:score', 'count', 'guid', 'name'];
    +
    +    this.searchService.search(searchRequest).subscribe(resp => this.searchResponse = resp);
    +  }
    +
    +  doAddAlertToMetaAlert(alertSources: AlertSource[]) {
    +    let patchRequest = new PatchRequest();
    +    patchRequest.guid = this.selectedMetaAlert;
    +    patchRequest.sensorType = 'metaalert';
    +    patchRequest.index = META_ALERTS_INDEX;
    --- End diff --
    
    should be `patchRequest.index = META_ALERTS_INDEX + '_index';`


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/803
  
    Hi @iraghumitra - I'd like to see your work get in ASAP.  Can you merge with master when you get a chance?  
    
    Also, I think you need to make some updates based on recent PRs that have gone in.  @justinleet pointed out that at a minimum the UI has to change for #824.  There may be others too.
    



---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:

    https://github.com/apache/metron/pull/803
  
    I made a metaalert with four entries
    
    ```
    957f20a3-d67b-407a-a593-09bdcbca19df
    b18e0949-9ac5-48e2-945f-74f9609667db
    8fb8f6cf-861f-4337-8d34-1becc9cecad9
    0c6543c8-c5b3-4540-ba83-b338b1aa52f0
    ```
    
    When I delete an alert, e.g. `0c6543c8-c5b3-4540-ba83-b338b1aa52f0`, the wrong alert is removed (in this case `957f20a3-d67b-407a-a593-09bdcbca19df`).
    
    The wrong alert is passed
    ```
    {
      "metaAlertGuid": "f5cd050c-7a7d-4562-bfc1-1ca5796fa1e4",
      "alerts": [
        {
          "guid": "957f20a3-d67b-407a-a593-09bdcbca19df",
          "sensorType": "snort",
          "index": ""
        }
      ]
    }
    ```
    
    It appears to always submit a request to remove the first alert, not the alert I choose in the UI (although I haven't extensively validated that)


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @iraghumitra looks like the new API isn't being used quite right.
    
    Sample from the dev tools
    ```
    {
      "alerts": [
        {
          "guid": "50a0c1f6-8a55-4cdd-a031-81c53174ad7b",
          "sensorType": [
            "snort"
          ],
          "index": "snort_index_2017.11.20.15"
        },
    ...
    ```
    
    This should be, I believe,
    ```
    {
      "alerts": [
        {
          "guid": "50a0c1f6-8a55-4cdd-a031-81c53174ad7b",
          "sensorType": "snort",
          "index": "snort_index_2017.11.20.15"
        },
    ...
      ],
      "groups": [
        "source:type"
      ]
    }
    ```


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145810720
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/meta-alerts/meta-alerts.component.ts ---
    @@ -0,0 +1,101 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * 'License'); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an 'AS IS' BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +import { Component, OnInit } from '@angular/core';
    +import {Router} from '@angular/router';
    +
    +import {MetaAlertService} from '../../service/meta-alert.service';
    +import {UpdateService} from '../../service/update.service';
    +import {SearchRequest} from '../../model/search-request';
    +import {SearchService} from '../../service/search.service';
    +import {SearchResponse} from '../../model/search-response';
    +import {SortField} from '../../model/sort-field';
    +import {META_ALERTS_INDEX} from '../../utils/constants';
    +import {AlertSource} from '../../model/alert-source';
    +import {PatchRequest} from '../../model/patch-request';
    +import {Patch} from '../../model/patch';
    +
    +@Component({
    +  selector: 'app-meta-alerts',
    +  templateUrl: './meta-alerts.component.html',
    +  styleUrls: ['./meta-alerts.component.scss']
    +})
    +export class MetaAlertsComponent implements OnInit {
    +
    +  selectedMetaAlert = '';
    +  searchResponse: SearchResponse = new SearchResponse();
    +
    +  constructor(private router: Router,
    +              private metaAlertService: MetaAlertService,
    +              private updateService: UpdateService,
    +              private searchService: SearchService) {
    +  }
    +
    +  goBack() {
    +    this.router.navigateByUrl('/alerts-list');
    +    return false;
    +  }
    +
    +  ngOnInit() {
    +    let searchRequest = new SearchRequest();
    +    searchRequest.query = '*';
    +    searchRequest.from = 0;
    +    searchRequest.size = 999;
    +    searchRequest.facetFields = [];
    +    searchRequest.indices =  [META_ALERTS_INDEX];
    +    searchRequest.sort = [new SortField('threat:triage:score', 'desc')];
    +    searchRequest.fields = ['id', 'alert_status', 'threat:triage:score', 'count', 'guid', 'name'];
    +
    +    this.searchService.search(searchRequest).subscribe(resp => this.searchResponse = resp);
    +  }
    +
    +  doAddAlertToMetaAlert(alertSources: AlertSource[]) {
    +    let patchRequest = new PatchRequest();
    +    patchRequest.guid = this.selectedMetaAlert;
    +    patchRequest.sensorType = 'metaalert';
    +    patchRequest.index = META_ALERTS_INDEX;
    +    patchRequest.patch = [new Patch('replace', 'alert', alertSources)];
    +
    +    this.updateService.patch(patchRequest).subscribe(rep => {
    +      console.log('Meta alert saved');
    --- End diff --
    
    should this be here?


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r146204676
  
    --- Diff: metron-interface/metron-alerts/e2e/alerts-list/tree-view/tree-view.e2e-spec.ts ---
    @@ -175,7 +175,7 @@ describe('metron-alerts tree view', function () {
       });
     
     
    -  it('should have sort working for group details for multiple sub groups', () => {
    +  xit('should have sort working for group details for multiple sub groups', () => {
    --- End diff --
    
    x skips the execution. Since sort on guid is not working anymore, I wanted to skip the test


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r147152861
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts ---
    @@ -337,12 +343,67 @@ export class TreeViewComponent extends TableViewComponent implements OnChanges {
         });
       }
     
    +  canCreateMetaAlert(count: number) {
    +    if (count > 999) {
    --- End diff --
    
    @james-sirota You'd know a bit more about the practical usage pattern than I would.  Do you have any input on this limit?  I'm guessing we could/should probably drop it a bit to give ourselves a bit more safety factor (e.g. 200?), assuming that fits a practical pattern.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r152310478
  
    --- Diff: metron-interface/metron-alerts/e2e/alerts-list/meta-alerts/meta-alert.po.ts ---
    @@ -0,0 +1,43 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +import {browser, element, by} from 'protractor';
    +
    +export class MetaAlertPage {
    +
    +  getPageTitle() {
    +    return element(by.css('app-meta-alerts .form-title')).getText();
    +  }
    +
    +  getMetaAlertsTitle() {
    +    return element(by.css('app-meta-alerts .title')).getText();
    +  }
    +
    +  getAvailableMetaAlerts() {
    +    return element(by.css('app-meta-alerts .guid-name-container div')).getText();
    +  }
    +
    +  selectRadio() {
    +    return element.all(by.css('app-meta-alerts .checkmark')).click();
    +  }
    +
    +  addToMetaAlert() {
    +    element.all(by.css('app-meta-alerts')).get(0).element(by.buttonText('ADD')).click();
    +    browser.sleep(2000);
    --- End diff --
    
    @iraghumitra any thoughts on this?


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @nickwallen That looks wicked can you share your manual steps that lead to this?


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @justinleet my bad. The search query to fetch all the alerts in a group was returning a nested object since I was passing 'source: type' twice in the fields. I don't know why I was getting a nested object if I pass the same column name twice but for now, I fixed it in the UI.
    
    The test spec for the above issue is in flight...


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r146205835
  
    --- Diff: metron-interface/metron-alerts/src/app/service/update.service.ts ---
    @@ -38,22 +41,27 @@ export class UpdateService {
     
       constructor(private http: Http) { }
     
    -  public patch(patchRequest: PatchRequest): Observable<{}> {
    +  public patch(patchRequest: PatchRequest, fireChangeListner = true): Observable<{}> {
         let url = '/api/v1/update/patch';
         return this.http.patch(url, patchRequest, new RequestOptions({headers: new Headers(this.defaultHeaders)}))
         .catch(HttpUtil.handleError)
         .map(result => {
    -      this.alertChangedSource.next(patchRequest);
    +      if (fireChangeListner) {
    +        this.alertChangedSource.next(patchRequest);
    +      }
           return result;
         });
       }
     
    -  public updateAlertState(alerts: Alert[], state: string): Observable<{}> {
    +  public updateAlertState(alerts: Alert[], state: string, fireChangeListner = true): Observable<{}> {
    --- End diff --
    
    It is passed to patch function and is used there. This is specifically needed in the case where we select multiple rows on the table to update the status. There are too many unnecessary find one calls fired.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/803
  
    >  I am not refreshing on purpose. If I refresh the UI after any operation users would lose the current context and see new alerts (If available).
    
    Ok @iraghumitra.  Since you have a valid concern about whether this is the right approach, then let's NOT change this in your PR.  We can track this separately and come to the right resolution after your PR goes in.  Thanks.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @justinleet my bad. The search query to fetch all the alerts in a group was returning a nested object since I was passing 'source: type' twice in the fields. I don't know why I was getting a nested object if I pass the same column name twice but for now, I fixed it in the UI.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @nickwallen I will raise a jira to track this. Even if we are able to reproduce this issue it is unlikely bcoz this PR.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @iraghumitra I see that you merged some changes.  Is this ready to test?


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145816633
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/meta-alerts/meta-alerts.component.html ---
    @@ -0,0 +1,48 @@
    +<!--
    +  Licensed to the Apache Software
    +	Foundation (ASF) under one or more contributor license agreements. See the
    +	NOTICE file distributed with this work for additional information regarding
    +	copyright ownership. The ASF licenses this file to You under the Apache License,
    +	Version 2.0 (the "License"); you may not use this file except in compliance
    +	with the License. You may obtain a copy of the License at
    +  http://www.apache.org/licenses/LICENSE-2.0
    +  Unless required by applicable law or agreed to in writing, software distributed
    +	under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
    +	OR CONDITIONS OF ANY KIND, either express or implied. See the License for
    +  the specific language governing permissions and limitations under the License.
    +  -->
    +<div class="metron-slider-pane-details load-right-to-left dialog1x">
    +  <div class="container-fluid my-3">
    +    <div class="row mb-3">
    +      <div class="col-md-10">
    +        <div class="form-title">Add to Alert</div>
    +      </div>
    +      <div class="col-md-2"><i class="fa fa-times pull-right close-button" aria-hidden="true" (click)="goBack()"></i></div>
    +    </div>
    +    <div>
    +      <div class="title"> SELECT OPEN ALERT </div>
    +
    +      <div class="container-fluid">
    +        <div class="row meta-alert-row" *ngFor="let alert of searchResponse.results" >
    +          <div class="col-1 px-0">
    +            <label class="radio-container">
    +              <input type="radio"  name="radio"><span class="checkmark" (click)="selectedMetaAlert=alert.source.guid"></span>
    +            </label>
    +          </div>
    +          <div class="col-11 px-0">
    +            <span class="severity" appAlertSeverity [severity]="alert.source['threat:triage:score']"></span><sup> {{ alert.source['threat:triage:score'] }} </sup>
    +            <div class="px-0 guid-name-container">
    +              <div [ngClass]="{'selected': selectedMetaAlert===alert.source.guid}"> {{(alert.source.name && alert.source.name.length > 0) ? alert.source.name : alert.source.guid | centerEllipses:20 }}({{ alert.source.count }}) </div>
    +              <span class="pull-left sub-text"> {{ (alert.source.alert_status && alert.source.alert_status.length > 0) ? alert.source.alert_status : 'NEW' }} </span>
    +              <span class="pull-right sub-text"> {{ alert.source._timestamp | timeLapse }} </span>
    +            </div>
    +          </div>
    +      </div>
    +
    +      <div class="my-4">
    +        <button type="submit" class="btn btn-all_ports" (click)="addAlertToMetaAlert()" [disabled]="selectedMetaAlert.length===0" title="Please select a meta alert">ADD</button>
    +        <button class="btn btn-mine_shaft_2" (click)="goBack()">CANCEL</button>
    +      </div>
    +    </div>
    +  </div>
    +</div>
    --- End diff --
    
    need newline


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/metron/pull/803


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r146216580
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/meta-alerts/meta-alerts.component.ts ---
    @@ -0,0 +1,101 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * 'License'); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an 'AS IS' BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +import { Component, OnInit } from '@angular/core';
    +import {Router} from '@angular/router';
    +
    +import {MetaAlertService} from '../../service/meta-alert.service';
    +import {UpdateService} from '../../service/update.service';
    +import {SearchRequest} from '../../model/search-request';
    +import {SearchService} from '../../service/search.service';
    +import {SearchResponse} from '../../model/search-response';
    +import {SortField} from '../../model/sort-field';
    +import {META_ALERTS_INDEX} from '../../utils/constants';
    +import {AlertSource} from '../../model/alert-source';
    +import {PatchRequest} from '../../model/patch-request';
    +import {Patch} from '../../model/patch';
    +
    +@Component({
    +  selector: 'app-meta-alerts',
    +  templateUrl: './meta-alerts.component.html',
    +  styleUrls: ['./meta-alerts.component.scss']
    +})
    +export class MetaAlertsComponent implements OnInit {
    +
    +  selectedMetaAlert = '';
    +  searchResponse: SearchResponse = new SearchResponse();
    +
    +  constructor(private router: Router,
    +              private metaAlertService: MetaAlertService,
    +              private updateService: UpdateService,
    +              private searchService: SearchService) {
    +  }
    +
    +  goBack() {
    +    this.router.navigateByUrl('/alerts-list');
    +    return false;
    +  }
    +
    +  ngOnInit() {
    +    let searchRequest = new SearchRequest();
    +    searchRequest.query = '*';
    +    searchRequest.from = 0;
    +    searchRequest.size = 999;
    +    searchRequest.facetFields = [];
    +    searchRequest.indices =  [META_ALERTS_INDEX];
    +    searchRequest.sort = [new SortField('threat:triage:score', 'desc')];
    +    searchRequest.fields = ['id', 'alert_status', 'threat:triage:score', 'count', 'guid', 'name'];
    +
    +    this.searchService.search(searchRequest).subscribe(resp => this.searchResponse = resp);
    +  }
    +
    +  doAddAlertToMetaAlert(alertSources: AlertSource[]) {
    +    let patchRequest = new PatchRequest();
    +    patchRequest.guid = this.selectedMetaAlert;
    +    patchRequest.sensorType = 'metaalert';
    +    patchRequest.index = META_ALERTS_INDEX;
    +    patchRequest.patch = [new Patch('replace', 'alert', alertSources)];
    +
    +    this.updateService.patch(patchRequest).subscribe(rep => {
    +      console.log('Meta alert saved');
    --- End diff --
    
    As mentioned in the Next Section of CC, we need some kind of notification once actions like add/remove are done otherwise it is confusing. These are placeholders till we have a notification mechanism. If it is not a problem I would suggest we leave them else I can remove them.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/803
  
    Hi @iraghumitra - The UI does not refresh itself after a metaalert is deleted.  If this is something easy to fix, let's tackle it.  Otherwise, I can create a JIRA to track this and we can fix it as a separate PR.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/803
  
    Something seems to be wrong with saved searches.  When I run through a manual test script of creating metaalerts, deleting and navigating, most (if not all) of my saved searches are these giant query strings that are incorrect.  
    
    ![screen shot 2017-11-21 at 10 52 47 am](https://user-images.githubusercontent.com/2475409/33082288-68416002-ceaa-11e7-98ec-62dd5065b047.png)



---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:

    https://github.com/apache/metron/pull/803
  
    I've verified the bug reported by Justin happens when you create a meta alert from a group that is nested by more than 1 level.  Creating a meta alert from a top level group works.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145816548
  
    --- Diff: metron-interface/metron-alerts/src/styles.scss ---
    @@ -259,4 +259,61 @@ hr {
       padding: 0;
     }
     
    -
    +/** Custom Radio box **/
    +$background_color_1: #eee;
    +$background_color_2: #ccc;
    +$background_color_3: #2196F3;
    +.radio-container {
    +  display: block;
    +  position: relative;
    +  padding-left: 35px;
    +  margin-bottom: 12px;
    +  cursor: pointer;
    +  font-size: 22px;
    +  -webkit-user-select: none;
    +  -moz-user-select: none;
    +  -ms-user-select: none;
    +  user-select: none;
    +  input {
    +    position: absolute;
    +    opacity: 0;
    +    &:checked {
    +      &~.checkmark {
    +        background-color: $eastern-blue-2;
    +        &:after {
    +          display: block;
    +        }
    +      }
    +    }
    +  }
    +  &:hover {
    +    input {
    +      &~.checkmark {
    +        background-color: $eastern-blue-2;
    +      }
    +    }
    +  }
    +  .checkmark {
    +    position: absolute;
    +    top: 0;
    +    left: 0;
    +    height: 12px;
    +    width: 12px;
    +    background-color: $mine-shaft-2;
    +    border: 1px solid $tundora;
    +    border-radius: 50%;
    +
    +    &:after {
    +      top: 2px;
    +      left: 2px;
    +      width: 6px;
    +      height: 6px;
    +      border-radius: 50%;
    +      background: $white;
    +      content: "";
    +      position: absolute;
    +      display: none;
    +    }
    +  }
    +}
    +/**********************/
    --- End diff --
    
    need newline


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r146991821
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts ---
    @@ -337,12 +343,67 @@ export class TreeViewComponent extends TableViewComponent implements OnChanges {
         });
       }
     
    +  canCreateMetaAlert(count: number) {
    +    if (count > 999) {
    --- End diff --
    
    Ah ok, I was able to add 999 from my full-dev. What is your suggestion on the number of meta alerts 350 or 500?.
    
    We are not validating the size when users try's to add few more alerts to meta alerts. If you confirm the possible number I will add it too.
    
    Can I limit it to 350 for now?


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/803
  
    IMHO I think the functionality is good on this PR.  There are 3 functional tests that failed, but they are minor and can be addressed after we get this PR in.
    
    My only hold-up right now is that I cannot get the e2e tests to pass.  I am going to rebuild Full Dev and try again.  It might have something to do with stale data.
    - [ ] Run e2e tests.
    
    The functional tests that I am working with...
    - [ ] Validate initial state; No metaalerts exist.
    - [ ] Create a metaalert from 1 level grouping
    - [ ] Create a metaalert from 2 level grouping
    - [ ] Create a metaalert from 3 level grouping
    - [ ] Create a metaalert from 4 level grouping
    - [ ] Delete a metaalert containing child alerts
    - [ ] Delete one child alert from an existing metaalert
    - [ ] Delete child alerts until a metaalert contains no child alerts
    - [ ] Delete a metaalert with no child alerts.  (FAIL)
    - [ ] Add an alert to a metaalert with no child alerts. (FAIL)
    - [ ] Add one child alert to existing metaalert 
    - [ ] Should not be able to add alert to a metaalert that is resolved (FAIL)
    - [ ] Add multiple child alerts to an existing metaalert
    - [ ] Change status of one metaalert
    - [ ] Change status of multiple metaalerts 
    - [ ] Rename a metaalert
    - [ ] Add a comment to a metaalert
    - [ ] Delete a comment from a metaalert



---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r152311299
  
    --- Diff: metron-interface/metron-alerts/e2e/alerts-list/meta-alerts/meta-alert.po.ts ---
    @@ -0,0 +1,43 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +import {browser, element, by} from 'protractor';
    +
    +export class MetaAlertPage {
    +
    +  getPageTitle() {
    +    return element(by.css('app-meta-alerts .form-title')).getText();
    +  }
    +
    +  getMetaAlertsTitle() {
    +    return element(by.css('app-meta-alerts .title')).getText();
    +  }
    +
    +  getAvailableMetaAlerts() {
    +    return element(by.css('app-meta-alerts .guid-name-container div')).getText();
    +  }
    +
    +  selectRadio() {
    +    return element.all(by.css('app-meta-alerts .checkmark')).click();
    +  }
    +
    +  addToMetaAlert() {
    +    element.all(by.css('app-meta-alerts')).get(0).element(by.buttonText('ADD')).click();
    +    browser.sleep(2000);
    --- End diff --
    
    There are several sleep statements throughout the e2e tests.  I would be ok with cleaning them all up in follow-on PR.  


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145809162
  
    --- Diff: metron-interface/metron-alerts/e2e/alerts-list/tree-view/tree-view.e2e-spec.ts ---
    @@ -175,7 +175,7 @@ describe('metron-alerts tree view', function () {
       });
     
     
    -  it('should have sort working for group details for multiple sub groups', () => {
    +  xit('should have sort working for group details for multiple sub groups', () => {
    --- End diff --
    
    What does xit mean here?  Just curious.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r146265870
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/meta-alerts/meta-alerts.component.ts ---
    @@ -0,0 +1,101 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * 'License'); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an 'AS IS' BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +import { Component, OnInit } from '@angular/core';
    +import {Router} from '@angular/router';
    +
    +import {MetaAlertService} from '../../service/meta-alert.service';
    +import {UpdateService} from '../../service/update.service';
    +import {SearchRequest} from '../../model/search-request';
    +import {SearchService} from '../../service/search.service';
    +import {SearchResponse} from '../../model/search-response';
    +import {SortField} from '../../model/sort-field';
    +import {META_ALERTS_INDEX} from '../../utils/constants';
    +import {AlertSource} from '../../model/alert-source';
    +import {PatchRequest} from '../../model/patch-request';
    +import {Patch} from '../../model/patch';
    +
    +@Component({
    +  selector: 'app-meta-alerts',
    +  templateUrl: './meta-alerts.component.html',
    +  styleUrls: ['./meta-alerts.component.scss']
    +})
    +export class MetaAlertsComponent implements OnInit {
    +
    +  selectedMetaAlert = '';
    +  searchResponse: SearchResponse = new SearchResponse();
    +
    +  constructor(private router: Router,
    +              private metaAlertService: MetaAlertService,
    +              private updateService: UpdateService,
    +              private searchService: SearchService) {
    +  }
    +
    +  goBack() {
    +    this.router.navigateByUrl('/alerts-list');
    +    return false;
    +  }
    +
    +  ngOnInit() {
    +    let searchRequest = new SearchRequest();
    +    searchRequest.query = '*';
    +    searchRequest.from = 0;
    +    searchRequest.size = 999;
    +    searchRequest.facetFields = [];
    +    searchRequest.indices =  [META_ALERTS_INDEX];
    +    searchRequest.sort = [new SortField('threat:triage:score', 'desc')];
    +    searchRequest.fields = ['id', 'alert_status', 'threat:triage:score', 'count', 'guid', 'name'];
    +
    +    this.searchService.search(searchRequest).subscribe(resp => this.searchResponse = resp);
    +  }
    +
    +  doAddAlertToMetaAlert(alertSources: AlertSource[]) {
    +    let patchRequest = new PatchRequest();
    +    patchRequest.guid = this.selectedMetaAlert;
    +    patchRequest.sensorType = 'metaalert';
    +    patchRequest.index = META_ALERTS_INDEX;
    +    patchRequest.patch = [new Patch('replace', 'alert', alertSources)];
    +
    +    this.updateService.patch(patchRequest).subscribe(rep => {
    +      console.log('Meta alert saved');
    --- End diff --
    
    If you are planning on replace that with something in the future then I'm ok with it.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r146979225
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts ---
    @@ -337,12 +343,67 @@ export class TreeViewComponent extends TableViewComponent implements OnChanges {
         });
       }
     
    +  canCreateMetaAlert(count: number) {
    +    if (count > 999) {
    --- End diff --
    
    Do we prevent users from adding enough alerts to go over the limit?  E.g. if I have 995 alerts and I try to add 20 more, will the UI prevent it?


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:

    https://github.com/apache/metron/pull/803
  
    I am consistently getting these e2e test failures:
    ```
    1) meta-alerts workflow should have all the steps for meta alerts workflow
      - Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
      - Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
    
    2) metron-alerts alert status should change alert statuses
      - Expected 'RESOLVE' to equal 'OPEN'.
      - Expected 'NEW' to equal 'OPEN'.
      - Expected 'RESOLVE' to equal 'DISMISS'.
      - Expected 'NEW' to equal 'DISMISS'.
      - Expected 'RESOLVE' to equal 'ESCALATE'.
      - Expected 'NEW' to equal 'ESCALATE'.
      - Expected 'NEW' to equal 'RESOLVE'.
    
    3) metron-alerts alert status should add comments for table view
      - Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
      - Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
    
    4) metron-alerts alert status should add comments for tree view
      - Failed: unknown error: Element <i..._ngcontent-c7="" aria-expanded="false" class="down-arrow" data-animation="false" data-placement="left" data-toggle="tooltip" title="Open Group" href="#body-0" aria-controls="body-0"> </i> is not clickable at point (2485, 311). Other element would receive the click: <textarea..._ngcontent-c11="" class="form-control ng-valid ng-dirty ng-touched" ng-reflect-model="This is a sample comment"> </textarea>
        (Session info: chrome=61.0.3163.100)
        (Driver info: chromedriver=2.33.506106 (8a06c39c4582fbfbab6966dbb1c38a9173bfb1a2),platform=Mac OS X 10.11.6 x86_64)
    ```
    
    Not sure if 3) and 4) are cascading errors caused by previous errors.  I suspect 2) was introduced by https://github.com/apache/metron/pull/796.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:

    https://github.com/apache/metron/pull/803
  
    I did an initial review of this and I see several things we need to work through.  It's a pretty significant feature so that's not surprising.  I know there are some other PRs being worked on that this PR depends on (https://github.com/apache/metron/pull/806).  Some questions I have:
    - In this PR description, what does 'Change the state of meta alert' mean?
    - Can I create a new empty metaalert?
    - Can I create a metaalert by manually selecting alerts?
    - How do I create a metaalert from a group that is >1 levels deep? 
    - How do I see all metaalerts? I was able to query with "_exists_:alert.*" but that is not intuitive.
    - Can I only remove 1 alert from a metaalert at a time?
    - Would it be useful to assign a name to your metaalert in the confirmation form rather than having to find it after you create it and rename in the detail view?
    - Not a fan of vertical scrolling through alerts in metaalert detail, would it be possible to add pagination?
    
    Some initial bugs I found (commented on some of these):
    - metaalert index name throughout code is wrong, should be 'metaalert'
    - metaalerts have alert_status set to 'NEW'
    - when you group by ip address fields, then expand group, count goes to 0 and corrects after the next search (not sure if this existed before this PR)
    - clicking on a metaalert fires this findOne call:
    ```
    {
      "guid": "ca80f4fc-0cdb-431c-b972-c460dad022ee",
      "sensorType": "undefined"
    }
    ```
    - not obvious from a user perspective what happens when I remove all alerts from a metaalert (I can see rest call that sets metaalert status to inactive)
    - missing space in alert merge confirmation and 1 alert displays as '1alerts'
    - need to add metaalert index to e2e npm environment
    - when metaalerts and alerts are in the same result set, sorting on fields other than timestamp causes metaalerts to be excluded
    - when I select an alert and then select Add to Alert in bulk actions, metaalerts display 0 alerts  ie.
     (0) 
    - adding a comment to metaalert is failing because sensorType is undefined in patch request:
    ```
    {
      "patch": [
        {
          "op": "add",
          "path": "/comments",
          "value": [
            {
              "comment": "test",
              "username": "user",
              "timestamp": 1508446386940
            }
          ]
        }
      ],
      "guid": "b9479340-316b-46db-baa5-0a0376ff015a",
      "index": "metaalert_index",
      "sensorType": "undefined"
    }
    ```
    - when I change/assign a metaalert name, reverts back to id in search result list on next search
    - comment icons in the list view are not appearing after I add a comment even though comments are displayed in the metaalert default view (likely related to failed add comment call, being written only to hbase and not ES)
    
    I'm also assuming more e2e tests are coming soon.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145809924
  
    --- Diff: metron-interface/metron-alerts/src/app/utils/constants.ts ---
    @@ -27,5 +30,5 @@ export const ALERTS_COLUMN_NAMES = 'metron-alerts-column-names';
     export let TREE_SUB_GROUP_SIZE = 5;
     export let DEFAULT_FACETS = ['source:type', 'ip_src_addr', 'ip_dst_addr', 'host', 'enrichments:geo:ip_dst_addr:country'];
     export let DEFAULT_GROUPS = ['source:type', 'ip_src_addr', 'ip_dst_addr', 'host', 'enrichments:geo:ip_dst_addr:country'];
    -export let INDEXES =  environment.indices ? environment.indices.split(',') : ['websphere', 'snort', 'asa', 'bro', 'yaf'];
    +export let INDEXES =  environment.indices ? environment.indices.split(',') : ['websphere', 'snort', 'asa', 'bro', 'yaf',  'metaalerts'];
    --- End diff --
    
    should be `export let INDEXES =  environment.indices ? environment.indices.split(',') : ['websphere', 'snort', 'asa', 'bro', 'yaf',  'metaalert'];`


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @justinleet Thanks for taking time to test the PR. I was using an incorrect index to find the alert to delete when I implemented the new API's I fixed it now. Also added tests for all the issues I fixed so far.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r146205224
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/meta-alerts/meta-alerts.component.ts ---
    @@ -0,0 +1,101 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * 'License'); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an 'AS IS' BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +import { Component, OnInit } from '@angular/core';
    +import {Router} from '@angular/router';
    +
    +import {MetaAlertService} from '../../service/meta-alert.service';
    +import {UpdateService} from '../../service/update.service';
    +import {SearchRequest} from '../../model/search-request';
    +import {SearchService} from '../../service/search.service';
    +import {SearchResponse} from '../../model/search-response';
    +import {SortField} from '../../model/sort-field';
    +import {META_ALERTS_INDEX} from '../../utils/constants';
    +import {AlertSource} from '../../model/alert-source';
    +import {PatchRequest} from '../../model/patch-request';
    +import {Patch} from '../../model/patch';
    +
    +@Component({
    +  selector: 'app-meta-alerts',
    +  templateUrl: './meta-alerts.component.html',
    +  styleUrls: ['./meta-alerts.component.scss']
    +})
    +export class MetaAlertsComponent implements OnInit {
    +
    +  selectedMetaAlert = '';
    +  searchResponse: SearchResponse = new SearchResponse();
    +
    +  constructor(private router: Router,
    +              private metaAlertService: MetaAlertService,
    +              private updateService: UpdateService,
    +              private searchService: SearchService) {
    +  }
    +
    +  goBack() {
    +    this.router.navigateByUrl('/alerts-list');
    +    return false;
    +  }
    +
    +  ngOnInit() {
    +    let searchRequest = new SearchRequest();
    +    searchRequest.query = '*';
    +    searchRequest.from = 0;
    +    searchRequest.size = 999;
    +    searchRequest.facetFields = [];
    +    searchRequest.indices =  [META_ALERTS_INDEX];
    +    searchRequest.sort = [new SortField('threat:triage:score', 'desc')];
    +    searchRequest.fields = ['id', 'alert_status', 'threat:triage:score', 'count', 'guid', 'name'];
    +
    +    this.searchService.search(searchRequest).subscribe(resp => this.searchResponse = resp);
    +  }
    +
    +  doAddAlertToMetaAlert(alertSources: AlertSource[]) {
    +    let patchRequest = new PatchRequest();
    +    patchRequest.guid = this.selectedMetaAlert;
    +    patchRequest.sensorType = 'metaalert';
    +    patchRequest.index = META_ALERTS_INDEX;
    +    patchRequest.patch = [new Patch('replace', 'alert', alertSources)];
    +
    +    this.updateService.patch(patchRequest).subscribe(rep => {
    +      console.log('Meta alert saved');
    +      this.goBack();
    +    });
    +  }
    +
    +  addAlertToMetaAlert() {
    +    let searchRequest = new SearchRequest();
    +    searchRequest.query = 'guid:"' + this.selectedMetaAlert + '"';
    +    searchRequest.from = 0;
    +    searchRequest.size = 1;
    +    searchRequest.facetFields = [];
    +    searchRequest.indices =  [META_ALERTS_INDEX];
    +    searchRequest.sort = [];
    +    searchRequest.fields = [];
    +
    +    this.searchService.search(searchRequest).subscribe((searchResponse: SearchResponse) => {
    +      if (searchResponse.results.length === 1) {
    +        searchResponse.results[0].source.alert = [...searchResponse.results[0].source.alert,
    +                                                  ...this.metaAlertService.selectedAlerts.map(alert => alert.source)];
    +        this.doAddAlertToMetaAlert(searchResponse.results[0].source.alert);
    +      } else {
    +        console.log('Unable to get a single meta alert');
    +      }
    +    });
    +    console.log(this.selectedMetaAlert);
    --- End diff --
    
    As mentioned in the Next Section of CC, we need some kind of notification once actions like add/remove are done otherwise it is confusing. These are placeholders till we have a notification mechanism. If it is not a problem I would suggest we leave them else I can remove them.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @nickwallen I haven't been able to get into the same sort of state, but I know you've used/tested the UI more than me.  Do you know a repeatable way to reproduce this?  Or have you been able to determine if it's preexisting, assuming you've looked into it.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:

    https://github.com/apache/metron/pull/803
  
    I agree with @nickwallen.  I think we're good to merge this as long as e2e tests are being addressed in a separate PR.  +1


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145814193
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/meta-alerts/meta-alerts.component.ts ---
    @@ -0,0 +1,101 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * 'License'); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an 'AS IS' BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +import { Component, OnInit } from '@angular/core';
    +import {Router} from '@angular/router';
    +
    +import {MetaAlertService} from '../../service/meta-alert.service';
    +import {UpdateService} from '../../service/update.service';
    +import {SearchRequest} from '../../model/search-request';
    +import {SearchService} from '../../service/search.service';
    +import {SearchResponse} from '../../model/search-response';
    +import {SortField} from '../../model/sort-field';
    +import {META_ALERTS_INDEX} from '../../utils/constants';
    +import {AlertSource} from '../../model/alert-source';
    +import {PatchRequest} from '../../model/patch-request';
    +import {Patch} from '../../model/patch';
    +
    +@Component({
    +  selector: 'app-meta-alerts',
    +  templateUrl: './meta-alerts.component.html',
    +  styleUrls: ['./meta-alerts.component.scss']
    +})
    +export class MetaAlertsComponent implements OnInit {
    +
    +  selectedMetaAlert = '';
    +  searchResponse: SearchResponse = new SearchResponse();
    +
    +  constructor(private router: Router,
    +              private metaAlertService: MetaAlertService,
    +              private updateService: UpdateService,
    +              private searchService: SearchService) {
    +  }
    +
    +  goBack() {
    +    this.router.navigateByUrl('/alerts-list');
    +    return false;
    +  }
    +
    +  ngOnInit() {
    +    let searchRequest = new SearchRequest();
    +    searchRequest.query = '*';
    +    searchRequest.from = 0;
    +    searchRequest.size = 999;
    +    searchRequest.facetFields = [];
    +    searchRequest.indices =  [META_ALERTS_INDEX];
    +    searchRequest.sort = [new SortField('threat:triage:score', 'desc')];
    +    searchRequest.fields = ['id', 'alert_status', 'threat:triage:score', 'count', 'guid', 'name'];
    +
    +    this.searchService.search(searchRequest).subscribe(resp => this.searchResponse = resp);
    +  }
    +
    +  doAddAlertToMetaAlert(alertSources: AlertSource[]) {
    +    let patchRequest = new PatchRequest();
    +    patchRequest.guid = this.selectedMetaAlert;
    +    patchRequest.sensorType = 'metaalert';
    +    patchRequest.index = META_ALERTS_INDEX;
    +    patchRequest.patch = [new Patch('replace', 'alert', alertSources)];
    +
    +    this.updateService.patch(patchRequest).subscribe(rep => {
    +      console.log('Meta alert saved');
    +      this.goBack();
    +    });
    +  }
    +
    +  addAlertToMetaAlert() {
    +    let searchRequest = new SearchRequest();
    +    searchRequest.query = 'guid:"' + this.selectedMetaAlert + '"';
    +    searchRequest.from = 0;
    +    searchRequest.size = 1;
    +    searchRequest.facetFields = [];
    +    searchRequest.indices =  [META_ALERTS_INDEX];
    +    searchRequest.sort = [];
    +    searchRequest.fields = [];
    +
    +    this.searchService.search(searchRequest).subscribe((searchResponse: SearchResponse) => {
    +      if (searchResponse.results.length === 1) {
    +        searchResponse.results[0].source.alert = [...searchResponse.results[0].source.alert,
    --- End diff --
    
    Why are you reassigning old and new alerts back to `searchResponse.results[0].source.alert`?  Makes it a little confusing to read.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    I added a single test case which covers the entire workflow of meta-alert. I couldn't think of a way to decompose it into smaller cases without making them dependent on each other. Any suggestions on the existing tests or for any more test case are most welcomed.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r146265654
  
    --- Diff: metron-interface/metron-alerts/e2e/alerts-list/tree-view/tree-view.e2e-spec.ts ---
    @@ -175,7 +175,7 @@ describe('metron-alerts tree view', function () {
       });
     
     
    -  it('should have sort working for group details for multiple sub groups', () => {
    +  xit('should have sort working for group details for multiple sub groups', () => {
    --- End diff --
    
    Can you sort on a different field instead?  Timestamp maybe?  I don't think the right answer is to just disable the test.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r147853082
  
    --- Diff: metron-interface/metron-alerts/e2e/alerts-list/tree-view/tree-view.po.ts ---
    @@ -159,4 +159,34 @@ export class TreeViewPage {
           return column.getText();
         });
       }
    +
    +  clickOnMergeAlerts(groupName: string) {
    +    return element(by.css('[data-name="' + groupName + '"] .fa-link')).click();
    +  }
    +
    +  getConfirmationText() {
    +    browser.sleep(1000);
    +    let dialogElement = element(by.css('.metron-dialog .modal-header .close'));
    +    return waitForElementVisibility(dialogElement).then(() =>  element(by.css('.metron-dialog .modal-body')).getText());
    +  }
    +
    +  clickNoForConfirmation() {
    +    browser.sleep(1000);
    +    let dialogElement = element(by.css('.metron-dialog .modal-header .close'));
    +    let maskElement = element(by.css('.modal-backdrop.fade'));
    +    waitForElementVisibility(dialogElement).then(() => element(by.css('.metron-dialog')).element(by.buttonText('Cancel')).click())
    +    .then(() => waitForElementInVisibility(maskElement));
    +  }
    +
    +  clickYesForConfirmation() {
    +    browser.sleep(1000);
    --- End diff --
    
    Is this sleep statement absolutely necessary?


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    I had to force push the last merge since I was facing issues with github. 


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    Merged the PR with master and used new API's for creating meta-alerts. Please feel free to review and let me know the feedback.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/803
  
    Great.  I will get this PR merged.  I am glad to see that this one is ready to go.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145813076
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/meta-alerts/meta-alerts.component.ts ---
    @@ -0,0 +1,101 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * 'License'); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an 'AS IS' BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +import { Component, OnInit } from '@angular/core';
    +import {Router} from '@angular/router';
    +
    +import {MetaAlertService} from '../../service/meta-alert.service';
    +import {UpdateService} from '../../service/update.service';
    +import {SearchRequest} from '../../model/search-request';
    +import {SearchService} from '../../service/search.service';
    +import {SearchResponse} from '../../model/search-response';
    +import {SortField} from '../../model/sort-field';
    +import {META_ALERTS_INDEX} from '../../utils/constants';
    +import {AlertSource} from '../../model/alert-source';
    +import {PatchRequest} from '../../model/patch-request';
    +import {Patch} from '../../model/patch';
    +
    +@Component({
    +  selector: 'app-meta-alerts',
    +  templateUrl: './meta-alerts.component.html',
    +  styleUrls: ['./meta-alerts.component.scss']
    +})
    +export class MetaAlertsComponent implements OnInit {
    +
    +  selectedMetaAlert = '';
    +  searchResponse: SearchResponse = new SearchResponse();
    +
    +  constructor(private router: Router,
    +              private metaAlertService: MetaAlertService,
    +              private updateService: UpdateService,
    +              private searchService: SearchService) {
    +  }
    +
    +  goBack() {
    +    this.router.navigateByUrl('/alerts-list');
    +    return false;
    +  }
    +
    +  ngOnInit() {
    +    let searchRequest = new SearchRequest();
    +    searchRequest.query = '*';
    +    searchRequest.from = 0;
    +    searchRequest.size = 999;
    +    searchRequest.facetFields = [];
    +    searchRequest.indices =  [META_ALERTS_INDEX];
    +    searchRequest.sort = [new SortField('threat:triage:score', 'desc')];
    +    searchRequest.fields = ['id', 'alert_status', 'threat:triage:score', 'count', 'guid', 'name'];
    --- End diff --
    
    should 'alert_status' be 'status'?


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/803
  
    I was testing #811 along with the changes in this PR and noted an issue.  It appears that the UI does not refresh itself after a meta-alert is created.
    
    1. First, I isolate 10 alerts with a specific host name that I want to work with.  I have turned off ingest so that no additional alerts should appear during the running of this example.  I can see that the facet counts are what I would expect.
    
    ![screen shot 2017-10-23 at 3 01 45 pm](https://user-images.githubusercontent.com/2475409/31908592-6de23998-b805-11e7-8201-c9983bfdc476.png)
    
    2. Next, I group by host so that I can create my meta-alert.
    
    ![screen shot 2017-10-23 at 3 01 58 pm](https://user-images.githubusercontent.com/2475409/31908646-97cf304e-b805-11e7-9ca4-14bb0269fc0c.png)
    
    ![screen shot 2017-10-23 at 3 02 09 pm](https://user-images.githubusercontent.com/2475409/31908665-a1bea3e6-b805-11e7-96e1-572faaf38e7a.png)
    
    3. Immediately after creating the meta-alert, I do not immediately see it.  I think this is a problem with the UI itself not refreshing after creating the alert.  
    
    ![screen shot 2017-10-23 at 3 02 37 pm](https://user-images.githubusercontent.com/2475409/31908754-edf3aebe-b805-11e7-86ff-838a1572a27d.png)
    
    4. If I then trigger another search, I do see the meta-alert.  I would expect to see it directly without having to manually trigger another search.  Ignore the facet counts being 0.  I think that is a problem with #811.
    
    ![screen shot 2017-10-23 at 3 03 14 pm](https://user-images.githubusercontent.com/2475409/31908976-a74e13cc-b806-11e7-860b-8a86aac8d36a.png)



---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:

    https://github.com/apache/metron/pull/803
  
    I just submitted a [PR](https://github.com/iraghumitra/incubator-metron/pull/3) against this PR that addresses all of the bugs reported above except 1:
    
    - when metaalerts and alerts are in the same result set, sorting on fields other than timestamp causes metaalerts to be excluded
    
    I think we need more discussion on what should happen in that scenario (sorting on fields that doesn't exist in metaalerts).  Let me know what you think.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145816904
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.scss ---
    @@ -143,3 +167,12 @@ textarea {
     .comment-container:hover i {
       display: block;
     }
    +
    +.input-group-addon {
    +  cursor: pointer;
    +}
    +
    +.disabled {
    +  opacity: 0.5;
    +  cursor: not-allowed;
    +}
    --- End diff --
    
    need newline


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/803
  
    Just a quick update... I was working with @iraghumitra yesterday on getting the e2e tests to pass for me.  It seems that with older versions of NPM the tests do pass, but then only various shades of success with newer versions.  I don't totally understand why that is.  Still working through it.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r147853051
  
    --- Diff: metron-interface/metron-alerts/e2e/alerts-list/tree-view/tree-view.po.ts ---
    @@ -159,4 +159,34 @@ export class TreeViewPage {
           return column.getText();
         });
       }
    +
    +  clickOnMergeAlerts(groupName: string) {
    +    return element(by.css('[data-name="' + groupName + '"] .fa-link')).click();
    +  }
    +
    +  getConfirmationText() {
    +    browser.sleep(1000);
    --- End diff --
    
    Is this sleep statement absolutely necessary?


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @merrimanr Thanks for https://github.com/iraghumitra/incubator-metron/pull/5. It was a pretty significant issue. I merged the fix 👍 


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @merrimanr is HBase running these errors are typically seen when HBase is down.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:

    https://github.com/apache/metron/pull/803
  
    You should not have empty meta alerts.  That does not make sense 


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:

    https://github.com/apache/metron/pull/803
  
    Yes HBase is running.  The first set of status tests are working (they would not if HBase were down).


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @nickwallen  your observation is right. As mentioned in the Limitations section of CC, as of now we are not displaying the meta-alerts in the group view. Hence I didn't see a need to fire a refresh. 
    
    However looking at the screenshot it occurred to me that we need to refresh the UI to at least get the group numbers correct. This is a simple change, I will add this functionality once de-duping of alerts in meta alert is available otherwise you will see the group back which might cause even more confusion for now.



---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/803
  
    That's weird @justinleet .  The create request is working for me.  I'll mess with it some more and try to replicate what you are seeing.
    
    I am seeing a separate issue on the REST UI side.
    ```
    17/11/20 16:55:40 ERROR dao.ElasticsearchColumnMetadataDao: Field type mismatch: snort_index_2017.11.20.16.threat:triage:score has type float while metaalert_index.threat:triage:score has type double.  Defaulting type to other.
    17/11/20 16:55:40 ERROR dao.ElasticsearchColumnMetadataDao: Field type mismatch: bro_index_2017.11.20.16.id has type string while snort_index_2017.11.20.16.id has type integer.  Defaulting type to other.
    17/11/20 16:55:40 ERROR dao.ElasticsearchRequestSubmitter: Failed to execute search; error='NotSerializableExceptionWrapper: class_cast_exception: java.lang.Double cannot be cast to java.lang.Float', search='{"from":0,"size":25,"query":{"constant_score":{"filter":{"bool":{"must":[{"bool":{"should":[{"query_string":{"query":"*"}},{"nested":{"query":{"query_string":{"query":"*"}},"path":"alert"}}]}},{"bool":{"should":[{"term":{"status":"active"}},{"bool":{"must_not":{"exists":{"field":"status"}}}}]}}],"must_not":{"exists":{"field":"metaalerts"}}}}}},"_source":{"includes":[],"excludes":[]},"sort":[{"threat:triage:score":{"order":"desc","missing":"_last","unmapped_type":"other"}}],"track_scores":true,"aggregations":{"source:type_count":{"terms":{"field":"source:type"}},"ip_src_addr_count":{"terms":{"field":"ip_src_addr"}},"ip_dst_addr_count":{"terms":{"field":"ip_dst_addr"}},"host_count":{"terms":{"field":"host"}},"enrichments:geo:ip_dst_addr:country_count":{"terms":{"field":"enrichm
 ents:geo:ip_dst_addr:country"}}}}'
    Failed to execute phase [query], [reduce] ; shardFailures {[0KqVPgyOT2KKCjTKZYIl3Q][bro_index_2017.11.20.16][0]: RemoteTransportException[[node1][192.168.66.121:9300][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [{"from":0,"size":25,"query":{"constant_score":{"filter":{"bool":{"must":[{"bool":{"should":[{"query_string":{"query":"*"}},{"nested":{"query":{"query_string":{"query":"*"}},"path":"alert"}}]}},{"bool":{"should":[{"term":{"status":"active"}},{"bool":{"must_not":{"exists":{"field":"status"}}}}]}}],"must_not":{"exists":{"field":"metaalerts"}}}}}},"_source":{"includes":[],"excludes":[]},"sort":[{"threat:triage:score":{"order":"desc","missing":"_last","unmapped_type":"other"}}],"track_scores":true,"aggregations":{"source:type_count":{"terms":{"field":"source:type"}},"ip_src_addr_count":{"terms":{"field":"ip_src_addr"}},"ip_dst_addr_count":{"terms":{"field":"ip_dst_addr"}},"host_count":{"terms":{"field":"host"}},"enrichmen
 ts:geo:ip_dst_addr:country_count":{"terms":{"field":"enrichments:geo:ip_dst_addr:country"}}}}]]; nested: IllegalArgumentException[No mapper found for type [other]]; }
            at org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:176)
            at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onResponse(AbstractSearchAsyncAction.java:147)
            at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onResponse(AbstractSearchAsyncAction.java:144)
            at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:41)
            at org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:819)
            at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:803)
            at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:793)
            at org.elasticsearch.transport.DelegatingTransportChannel.sendResponse(DelegatingTransportChannel.java:58)
            at org.elasticsearch.transport.RequestHandlerRegistry$TransportChannelWrapper.sendResponse(RequestHandlerRegistry.java:134)
            at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:369)
            at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
            at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
            at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
            at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
            at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)
    Caused by: NotSerializableExceptionWrapper[class_cast_exception: java.lang.Double cannot be cast to java.lang.Float]
            at java.lang.Float.compareTo(Float.java:49)
            at org.apache.lucene.search.FieldComparator.compareValues(FieldComparator.java:132)
            at org.apache.lucene.search.TopDocs$MergeSortQueue.lessThan(TopDocs.java:170)
            at org.apache.lucene.search.TopDocs$MergeSortQueue.lessThan(TopDocs.java:119)
            at org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:263)
            at org.apache.lucene.util.PriorityQueue.add(PriorityQueue.java:140)
            at org.apache.lucene.search.TopDocs.mergeAux(TopDocs.java:255)
            at org.apache.lucene.search.TopDocs.merge(TopDocs.java:232)
            at org.elasticsearch.search.controller.SearchPhaseController.sortDocs(SearchPhaseController.java:253)
            at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.moveToSecondPhase(SearchQueryThenFetchAsyncAction.java:72)
            at org.elasticsearch.action.search.AbstractSearchAsyncAction.innerMoveToSecondPhase(AbstractSearchAsyncAction.java:374)
            at org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:171)
            ... 17 more
    ```


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @justinleet @iraghumitra I am glad that no one else can reproduce.  I will work on trying to get a set of steps to reproduce this.  Let's not worry about this on the current PR.


---

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/803#discussion_r145810773
  
    --- Diff: metron-interface/metron-alerts/src/app/alerts/meta-alerts/meta-alerts.component.ts ---
    @@ -0,0 +1,101 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * 'License'); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an 'AS IS' BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +import { Component, OnInit } from '@angular/core';
    +import {Router} from '@angular/router';
    +
    +import {MetaAlertService} from '../../service/meta-alert.service';
    +import {UpdateService} from '../../service/update.service';
    +import {SearchRequest} from '../../model/search-request';
    +import {SearchService} from '../../service/search.service';
    +import {SearchResponse} from '../../model/search-response';
    +import {SortField} from '../../model/sort-field';
    +import {META_ALERTS_INDEX} from '../../utils/constants';
    +import {AlertSource} from '../../model/alert-source';
    +import {PatchRequest} from '../../model/patch-request';
    +import {Patch} from '../../model/patch';
    +
    +@Component({
    +  selector: 'app-meta-alerts',
    +  templateUrl: './meta-alerts.component.html',
    +  styleUrls: ['./meta-alerts.component.scss']
    +})
    +export class MetaAlertsComponent implements OnInit {
    +
    +  selectedMetaAlert = '';
    +  searchResponse: SearchResponse = new SearchResponse();
    +
    +  constructor(private router: Router,
    +              private metaAlertService: MetaAlertService,
    +              private updateService: UpdateService,
    +              private searchService: SearchService) {
    +  }
    +
    +  goBack() {
    +    this.router.navigateByUrl('/alerts-list');
    +    return false;
    +  }
    +
    +  ngOnInit() {
    +    let searchRequest = new SearchRequest();
    +    searchRequest.query = '*';
    +    searchRequest.from = 0;
    +    searchRequest.size = 999;
    +    searchRequest.facetFields = [];
    +    searchRequest.indices =  [META_ALERTS_INDEX];
    +    searchRequest.sort = [new SortField('threat:triage:score', 'desc')];
    +    searchRequest.fields = ['id', 'alert_status', 'threat:triage:score', 'count', 'guid', 'name'];
    +
    +    this.searchService.search(searchRequest).subscribe(resp => this.searchResponse = resp);
    +  }
    +
    +  doAddAlertToMetaAlert(alertSources: AlertSource[]) {
    +    let patchRequest = new PatchRequest();
    +    patchRequest.guid = this.selectedMetaAlert;
    +    patchRequest.sensorType = 'metaalert';
    +    patchRequest.index = META_ALERTS_INDEX;
    +    patchRequest.patch = [new Patch('replace', 'alert', alertSources)];
    +
    +    this.updateService.patch(patchRequest).subscribe(rep => {
    +      console.log('Meta alert saved');
    +      this.goBack();
    +    });
    +  }
    +
    +  addAlertToMetaAlert() {
    +    let searchRequest = new SearchRequest();
    +    searchRequest.query = 'guid:"' + this.selectedMetaAlert + '"';
    +    searchRequest.from = 0;
    +    searchRequest.size = 1;
    +    searchRequest.facetFields = [];
    +    searchRequest.indices =  [META_ALERTS_INDEX];
    +    searchRequest.sort = [];
    +    searchRequest.fields = [];
    +
    +    this.searchService.search(searchRequest).subscribe((searchResponse: SearchResponse) => {
    +      if (searchResponse.results.length === 1) {
    +        searchResponse.results[0].source.alert = [...searchResponse.results[0].source.alert,
    +                                                  ...this.metaAlertService.selectedAlerts.map(alert => alert.source)];
    +        this.doAddAlertToMetaAlert(searchResponse.results[0].source.alert);
    +      } else {
    +        console.log('Unable to get a single meta alert');
    +      }
    +    });
    +    console.log(this.selectedMetaAlert);
    --- End diff --
    
    should this be here?


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @merrimanr METRON-1272 has broken few of our assertions and these are cascaded failures bcoz of them. I am working on fixing them.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:

    https://github.com/apache/metron/pull/803
  
    I agree. I'm fine with going ahead with this, but I'd like to see end to end stability being addressed as the next UI priority, which I believe @iraghumitra is already doing some work on.
    
    +1


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @merrimanr Added the test case for searching an alert inside meta-alert. Also took the liberty to fix the timezone issue it was a one-liner.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    I noticed that when I am running e2e test's, the comments that were added in the previous run are still visible at times. The e2e tests delete's  all the comments that it added and the only way I can see this happening is if in the previous run of tests add comment was successful and remove comment fails. The best fix for this would be to clean the metron_update table before running the tests. Since we don't have a rest API's for Hbase I am looking at finding alternatives. For the time being, if  you bump into this issue I request you to truncate the meton_update table manually.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:

    https://github.com/apache/metron/pull/803
  
    @nickwallen I am not refreshing on purpose. If I refresh the UI after any operation users would lose the current context and see new alerts (If available). If you think this is not a major concern I can add a call to refresh the UI.


---