You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by iraghumitra <gi...@git.apache.org> on 2017/09/21 11:51:22 UTC
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
GitHub user iraghumitra opened a pull request:
https://github.com/apache/metron/pull/768
Metron 1123: Add group by option using faceted search capabilities of metron-rest-api
## Contributor Comments
This PR extends the Metron Alerts GUI capabilities to do group by on Alerts Data
The PR is on top of METRON-1189
The group by buttons appear on top of the table. When user selects a group, the table view changes to a tree view with the values of the groups as the root of the tree. The individual alerts would be shown as the leaf nodes of the tree.
Tree view has all the following features
- Search will search through all the groups
- Users can click on the data inside the table to search
- Users can select and table inside tree to view the details
- Users can escalate the alerts from the table inside the tree
- Tree view is refreshed automatically depending on the polling setting in the settings
- A cumulative score and number of alerts in a group is shown next to group name
- Table inside a group can be paginated
- The column names for the alerts table in the tree can be configured via configure column pane
The table config settings is moved next to settings on the GUI as it makes more meaning there.
## Testing
E2E test cases are available to test groups functionality. From root of the project
```
cd metron-interface/metron-alerts
sh ./scripts/start-server-for-e2e.sh
```
In othere shell run the test cases
```
npm run e2e
```
All the features listed in contributor comments can be verified manually
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/iraghumitra/incubator-metron METRON-1123
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/768.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #768
----
commit b89d2e4698b5fd32a85e078b55718682f02ac161
Author: iraghumitra <ra...@gmail.com>
Date: 2017-09-19T09:43:13Z
Initial check-in for alerts grouping
commit 60b979427ba76af3c664f42b9f94e021ff4b715d
Author: iraghumitra <ra...@gmail.com>
Date: 2017-09-20T13:24:49Z
e2e tests for groups and suport for escalation action in tree view
----
---
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/768#discussion_r142459136
--- Diff: metron-interface/metron-alerts/e2e/alerts-list/tree-view/tree-view.e2e-spec.ts ---
@@ -0,0 +1,183 @@
+/// <reference path="../../matchers/custom-matchers.d.ts"/>
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {customMatchers} from '../../matchers/custom-matchers';
+import {LoginPage} from '../../login/login.po';
+import {TreeViewPage} from './tree-view.po';
+import {loadTestData, deleteTestData} from '../../utils/e2e_util';
+import {MetronAlertsPage} from '../alerts-list.po';
+
+describe('metron-alerts tree view', function () {
+ let page: TreeViewPage;
+ let listPage: MetronAlertsPage;
+ let loginPage: LoginPage;
+
+ beforeAll(() => {
+ loadTestData();
+ loginPage = new LoginPage();
+ page = new TreeViewPage();
+ listPage = new MetronAlertsPage();
+ loginPage.login();
+ page.navigateToAlertsList();
+ });
+
+ afterAll(() => {
+ loginPage.logout();
+ deleteTestData();
+ });
+
+ beforeEach(() => {
+ jasmine.addMatchers(customMatchers);
+ });
+
+ it('should have all group by elements', () => {
+ let groupByItems = {
+ 'source:type': '1',
+ 'ip_dst_addr': '8',
+ 'host': '9',
+ 'enrichm...:country': '3',
+ 'ip_src_addr': '2'
+ };
+ expect(page.getGroupByCount()).toEqualBcoz(Object.keys(groupByItems).length, '5 Group By Elements should be present');
+ expect(page.getGroupByItemNames()).toEqualBcoz(Object.keys(groupByItems), 'Group By Elements names should be present');
+ expect(page.getGroupByItemCounts()).toEqualBcoz(Object.keys(groupByItems).map(key => groupByItems[key]),
+ '5 Group By Elements values should be present');
+ });
+
+ it('should have group details for single group by', () => {
+ let dashRowValues = ['0', 'alerts_ui_e2e', 'ALERTS', '169'];
+ let row1_page1 = ['-', 'dcda4423-7...0962fafc47', '2017-09-13 17:59:32', 'alerts_ui_e2e',
+ '192.168.138.158', 'US', '72.34.49.86', 'comarksecurity.com', 'NEW', ''];
+ let row1_page2 = ['-', '07b29c29-9...ff19eaa888', '2017-09-13 17:59:37', 'alerts_ui_e2e',
+ '192.168.138.158', 'FR', '62.75.195.236', '62.75.195.236', 'NEW', ''];
+
+ page.selectGroup('source:type');
+ expect(page.getActiveGroups()).toEqualBcoz(['source:type'], 'only source type group should be selected');
+ expect(page.getDashGroupValues('alerts_ui_e2e')).toEqualBcoz(dashRowValues, 'Dash Group Values should be present');
+
+ page.expandDashGroup('alerts_ui_e2e');
+ expect(page.getDashGroupTableValuesForRow('alerts_ui_e2e', 0)).toEqualBcoz(row1_page1, 'Dash Group Values should be present');
+
+ page.clickOnNextPage('alerts_ui_e2e');
+ expect(page.getTableValuesByRowId('alerts_ui_e2e', 0, 'FR')).toEqualBcoz(row1_page2, 'Dash Group Values should be present');
+
+ page.unGroup();
+ expect(page.getActiveGroups()).toEqualBcoz([], 'no groups should be selected');
+ });
+
+ it('should have group details for multiple group by', () => {
+
+ let dashRow_runLoveUs = {
+ 'dashRow': ['0', 'runlove.us', 'ALERTS', '13'],
+ 'firstSubGroup': '0 US (13)',
+ 'firstSubGroupIdCol': ['9a969c64-b...001cb011a3', 'a651f7c3-1...a97d4966c9', 'afc36901-3...d931231ab2',
+ 'd860ac35-1...f9e282d571', '04a5c3d0-9...af17c06fbc']
+ };
+
+ let dashRow_62_75_195_236 = {
+ 'dashRow': ['0', '62.75.195.236', 'ALERTS', '18'],
+ 'firstSubGroup': '0 FR (18)',
+ 'firstSubGroupIdCol': ['07b29c29-9...ff19eaa888', '7cd91565-1...de5be54a6e', 'ca5bde58-a...f3a88d2df4',
+ '5d6faf83-8...b88a407647', 'e2883424-f...79bb8b0606']
+ };
+
+ page.selectGroup('host');
+ page.selectGroup('enrichments:geo:ip_dst_addr:country');
+ expect(page.getActiveGroups()).toEqualBcoz(['host', 'enrichments:geo:ip_dst_addr:country'], 'two groups should be selected');
+
+ expect(page.getDashGroupValues('runlove.us')).toEqualBcoz(dashRow_runLoveUs.dashRow,
+ 'Dash Group Values should be present for runlove.us');
--- End diff --
Shouldn't each expect message be different? Looks like there are 3 with the exact same message.
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:
https://github.com/apache/metron/pull/768
@nickwallen I kind of agree with @justinleet here. The rest api is should handle empty values in a better way.
Though we can circumvent the empty values issue in groups by assuming all the missing records have an empty value for the selected group but the current search api doesn't support searching on empty value ie we cannot fire a search query that says 'give me all rows where host is empty'. Hence we will not be able to show the individual alerts under the group. This need to be handled in rest.
Do you think this is a must-have for this PR or can we handle it in a follow-up PR ?.
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:
https://github.com/apache/metron/pull/768
@ottobackwards & @merrimanr Fixed the following
- Disabling the ungroup button when no groups are selected
- Fixed issue with select all alerts in tree view
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/768
To handle the missing grouping aggregation. https://www.elastic.co/guide/en/elasticsearch/reference/2.3/search-aggregations-bucket-missing-aggregation.html
I *think* this is a backend change (probably just to add a param to the request and carry it through)
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/768
I spun this up in full dev and it's working pretty well. The only functional issue I could find was that the bulk actions are disabled when alerts are selected in group/tree view. This is likely due to the fact that tree view inherits from list view and selectedAlerts are part of list view.
Other than that I think this is pretty close. We should add an e2e test for bulk actions in tree view (would have caught the issue above). Also is there a test for group reordering (drag and drop)?
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:
https://github.com/apache/metron/pull/768
Looks like the problem is not with the build but with starting metron-config ui on the VM. Are you using fedora instead of centos by any chance?.
AFAIU some script is missing shell invocation. Can you try manually installing the rpm with verbose options that might give us some more help.
I just tried a full-dev-setup on my dev machine it is working fine for me.
---
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on a diff in the pull request:
https://github.com/apache/metron/pull/768#discussion_r143715850
--- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts ---
@@ -0,0 +1,319 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Component, Input, OnChanges, SimpleChanges } from '@angular/core';
+import {Router} from '@angular/router';
+import {Subscription} from 'rxjs/Rx';
+
+import {TableViewComponent} from '../table-view/table-view.component';
+import {SearchResponse} from '../../../model/search-response';
+import {SearchService} from '../../../service/search.service';
+import {TreeGroupData, TreeAlertsSubscription} from './tree-group-data';
+import {GroupResponse} from '../../../model/group-response';
+import {GroupResult} from '../../../model/group-result';
+import {Group} from '../../../model/group';
+import {SortField} from '../../../model/sort-field';
+import {Sort} from '../../../utils/enums';
+import {MetronDialogBox, DialogType} from '../../../shared/metron-dialog-box';
+import {ElasticsearchUtils} from '../../../utils/elasticsearch-utils';
+import {SearchRequest} from '../../../model/search-request';
+
+@Component({
+ selector: 'app-tree-view',
+ templateUrl: './tree-view.component.html',
+ styleUrls: ['./tree-view.component.scss']
+})
+
+export class TreeViewComponent extends TableViewComponent implements OnChanges {
+
+ groupByFields: string[] = [];
+ topGroups: TreeGroupData[] = [];
+ groupResponse: GroupResponse = new GroupResponse();
+ treeGroupSubscriptionMap: {[key: string]: TreeAlertsSubscription } = {};
+
+ constructor(router: Router,
+ searchService: SearchService,
+ metronDialogBox: MetronDialogBox) {
+ super(router, searchService, metronDialogBox);
+ }
+
+ collapseGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level > (level)) {
+ groupArray[i].show = false;
+ groupArray[i].expand = false;
+ } else {
+ break;
+ }
+ }
+ }
+
+ createQuery(selectedGroup: TreeGroupData) {
+ let searchQuery = this.queryBuilder.generateSelect();
+ let groupQery = Object.keys(selectedGroup.groupQueryMap).map(key => {
+ return key.replace(/:/g, '\\:') +
+ ':' +
+ String(selectedGroup.groupQueryMap[key])
+ .replace(/[\*\+\-=~><\"\?^\${}\(\)\:\!\/[\]\\\s]/g, '\\$&') // replace single special characters
+ .replace(/\|\|/g, '\\||') // replace ||
+ .replace(/\&\&/g, '\\&&'); // replace &&
+ }).join(' AND ');
+
+ groupQery += searchQuery === '*' ? '' : (' AND ' + searchQuery);
+ return groupQery;
+ }
+
+ expandGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level === (level + 1)) {
+ groupArray[i].show = true;
+ } else {
+ break;
+ }
+ }
+ }
+
+ getAlerts(selectedGroup: TreeGroupData): Subscription {
+ let searchRequest = new SearchRequest();
+ searchRequest.query = this.createQuery(selectedGroup);
+ searchRequest.from = selectedGroup.pagingData.from;
+ searchRequest.size = selectedGroup.pagingData.size;
+ searchRequest.sort = selectedGroup.sortField ? [selectedGroup.sortField] : [];
+
+ return this.searchGroup(selectedGroup, searchRequest);
--- End diff --
@nickwallen that's a really nice catch, fixed it
---
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/768#discussion_r143729308
--- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts ---
@@ -0,0 +1,319 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Component, Input, OnChanges, SimpleChanges } from '@angular/core';
+import {Router} from '@angular/router';
+import {Subscription} from 'rxjs/Rx';
+
+import {TableViewComponent} from '../table-view/table-view.component';
+import {SearchResponse} from '../../../model/search-response';
+import {SearchService} from '../../../service/search.service';
+import {TreeGroupData, TreeAlertsSubscription} from './tree-group-data';
+import {GroupResponse} from '../../../model/group-response';
+import {GroupResult} from '../../../model/group-result';
+import {Group} from '../../../model/group';
+import {SortField} from '../../../model/sort-field';
+import {Sort} from '../../../utils/enums';
+import {MetronDialogBox, DialogType} from '../../../shared/metron-dialog-box';
+import {ElasticsearchUtils} from '../../../utils/elasticsearch-utils';
+import {SearchRequest} from '../../../model/search-request';
+
+@Component({
+ selector: 'app-tree-view',
+ templateUrl: './tree-view.component.html',
+ styleUrls: ['./tree-view.component.scss']
+})
+
+export class TreeViewComponent extends TableViewComponent implements OnChanges {
+
+ groupByFields: string[] = [];
+ topGroups: TreeGroupData[] = [];
+ groupResponse: GroupResponse = new GroupResponse();
+ treeGroupSubscriptionMap: {[key: string]: TreeAlertsSubscription } = {};
+
+ constructor(router: Router,
+ searchService: SearchService,
+ metronDialogBox: MetronDialogBox) {
+ super(router, searchService, metronDialogBox);
+ }
+
+ collapseGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level > (level)) {
+ groupArray[i].show = false;
+ groupArray[i].expand = false;
+ } else {
+ break;
+ }
+ }
+ }
+
+ createQuery(selectedGroup: TreeGroupData) {
+ let searchQuery = this.queryBuilder.generateSelect();
+ let groupQery = Object.keys(selectedGroup.groupQueryMap).map(key => {
+ return key.replace(/:/g, '\\:') +
+ ':' +
+ String(selectedGroup.groupQueryMap[key])
+ .replace(/[\*\+\-=~><\"\?^\${}\(\)\:\!\/[\]\\\s]/g, '\\$&') // replace single special characters
+ .replace(/\|\|/g, '\\||') // replace ||
+ .replace(/\&\&/g, '\\&&'); // replace &&
+ }).join(' AND ');
+
+ groupQery += searchQuery === '*' ? '' : (' AND ' + searchQuery);
+ return groupQery;
+ }
+
+ expandGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level === (level + 1)) {
+ groupArray[i].show = true;
+ } else {
+ break;
+ }
+ }
+ }
+
+ getAlerts(selectedGroup: TreeGroupData): Subscription {
+ let searchRequest = new SearchRequest();
+ searchRequest.query = this.createQuery(selectedGroup);
+ searchRequest.from = selectedGroup.pagingData.from;
+ searchRequest.size = selectedGroup.pagingData.size;
+ searchRequest.sort = selectedGroup.sortField ? [selectedGroup.sortField] : [];
+
+ return this.searchGroup(selectedGroup, searchRequest);
--- End diff --
Do we have a test case for this? I didn't see one in the commit that addressed this issue.
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/768
https://issues.apache.org/jira/browse/METRON-1242
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/768
I would like to review this, can you resolve conflicts?
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:
https://github.com/apache/metron/pull/768
Fixed conflicts and added test cases for comments in the tree view.
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/768
@iraghumitra In case you missed it, there are some new merge conflicts based on one of your other PRs being merged into master today.
---
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/768#discussion_r143565157
--- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts ---
@@ -0,0 +1,319 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Component, Input, OnChanges, SimpleChanges } from '@angular/core';
+import {Router} from '@angular/router';
+import {Subscription} from 'rxjs/Rx';
+
+import {TableViewComponent} from '../table-view/table-view.component';
+import {SearchResponse} from '../../../model/search-response';
+import {SearchService} from '../../../service/search.service';
+import {TreeGroupData, TreeAlertsSubscription} from './tree-group-data';
+import {GroupResponse} from '../../../model/group-response';
+import {GroupResult} from '../../../model/group-result';
+import {Group} from '../../../model/group';
+import {SortField} from '../../../model/sort-field';
+import {Sort} from '../../../utils/enums';
+import {MetronDialogBox, DialogType} from '../../../shared/metron-dialog-box';
+import {ElasticsearchUtils} from '../../../utils/elasticsearch-utils';
+import {SearchRequest} from '../../../model/search-request';
+
+@Component({
+ selector: 'app-tree-view',
+ templateUrl: './tree-view.component.html',
+ styleUrls: ['./tree-view.component.scss']
+})
+
+export class TreeViewComponent extends TableViewComponent implements OnChanges {
+
+ groupByFields: string[] = [];
+ topGroups: TreeGroupData[] = [];
+ groupResponse: GroupResponse = new GroupResponse();
+ treeGroupSubscriptionMap: {[key: string]: TreeAlertsSubscription } = {};
+
+ constructor(router: Router,
+ searchService: SearchService,
+ metronDialogBox: MetronDialogBox) {
+ super(router, searchService, metronDialogBox);
+ }
+
+ collapseGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level > (level)) {
+ groupArray[i].show = false;
+ groupArray[i].expand = false;
+ } else {
+ break;
+ }
+ }
+ }
+
+ createQuery(selectedGroup: TreeGroupData) {
+ let searchQuery = this.queryBuilder.generateSelect();
+ let groupQery = Object.keys(selectedGroup.groupQueryMap).map(key => {
+ return key.replace(/:/g, '\\:') +
+ ':' +
+ String(selectedGroup.groupQueryMap[key])
+ .replace(/[\*\+\-=~><\"\?^\${}\(\)\:\!\/[\]\\\s]/g, '\\$&') // replace single special characters
+ .replace(/\|\|/g, '\\||') // replace ||
+ .replace(/\&\&/g, '\\&&'); // replace &&
+ }).join(' AND ');
+
+ groupQery += searchQuery === '*' ? '' : (' AND ' + searchQuery);
+ return groupQery;
+ }
+
+ expandGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level === (level + 1)) {
+ groupArray[i].show = true;
+ } else {
+ break;
+ }
+ }
+ }
+
+ getAlerts(selectedGroup: TreeGroupData): Subscription {
+ let searchRequest = new SearchRequest();
+ searchRequest.query = this.createQuery(selectedGroup);
+ searchRequest.from = selectedGroup.pagingData.from;
+ searchRequest.size = selectedGroup.pagingData.size;
+ searchRequest.sort = selectedGroup.sortField ? [selectedGroup.sortField] : [];
+
+ return this.searchGroup(selectedGroup, searchRequest);
--- End diff --
@iraghumitra I noticed that when grouping there are cases when the two locations where we display total counts will differ. See the attached screenshots. I underlined in yellow the counts that I am talking about.
It seems that the data used to build the total count in the group header is stale in comparison with the data used to display the individual alerts. We initially call `/group` to display the group header. When a user expands the group we issue a `/search`, which retrieves fresher data. Data that likely has a different total from what is currently displayed in the group header.
Can we update the count in the group header somewhere? Maybe around here?
---
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/768#discussion_r142454353
--- Diff: metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/MetaAlertStatus.java ---
@@ -31,4 +31,4 @@
public String getStatusString() {
return statusString;
}
-}
+}
--- End diff --
Was this intentional?
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/768
Thanks for the thoughts and links @justinleet and @iraghumitra .
> @iraghumitra: but the current search api doesn't support searching on empty value ie we cannot fire a search query that says 'give me all rows where host is empty'
Right. We would have to change the backend to do something like this.
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html#_literal_missing_literal_query
> @iraghumitra: Do you think this is a must-have for this PR or can we handle it in a follow-up PR ?
Based on the level of work to get this done, I think this is definitely a separate effort. I will create a JIRA for this.
I will run up your latest code. It's looking really solid.
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/768
Thanks @iraghumitra the bulk actions are working now. Just need to address the e2e test suggestions and I think this will be good to go.
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:
https://github.com/apache/metron/pull/768
@merrimanr oops missed the comment thanks for letting me know again. I will add a test case for tree bulk actions. Looks like e2e for drag-drop is going to be tough I tried a bit but could not get it to work. The dragula community says it is going to be tough having protractor cases for it too you can check it [here](https://github.com/valor-software/ng2-dragula/issues/720)
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:
https://github.com/apache/metron/pull/768
@ottobackwards can u check it now I merged master again looks like mpack is in now.
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/768
When grouping, we are implicitly filtering any alerts that do not have a value defined for the 'group by' field.
For example, in this screenshot there are roughly 44k alarms. I have grouped by host and all my groups add up to about 17k. The other 27k alarms are actually from Snort where the 'host' field is not defined.
Would it be better to treat undefined, blank or missing fields as their own group? So in this example, we would have another group that has no value and would have an alert count of roughly 27k.
The advantages I see with this...
* My group subtotals will always add up to the total number of alarms, which eliminates the mystery of missing alarms.
* Digging into alarms where a field is not defined or blank is a reasonable thing that a user would want to do.
This may very well be something to fix on the backend.
---
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/768#discussion_r142454390
--- Diff: metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/metaalert/MetaAlertCreateResponse.java ---
@@ -28,4 +28,4 @@ public boolean isCreated() {
public void setCreated(boolean created) {
this.created = created;
}
-}
+}
--- End diff --
Was this intentional?
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/768
The cut off text for the enrichment, both in the group and the column header will make it unusable if there is more than one enrichment I think
<img width="197" alt="screen shot 2017-10-03 at 10 40 51" src="https://user-images.githubusercontent.com/1111551/31131211-790611a2-a827-11e7-94ca-66c174034042.png">
<img width="263" alt="screen shot 2017-10-03 at 10 40 43" src="https://user-images.githubusercontent.com/1111551/31131213-7aa6a29c-a827-11e7-9a38-67a561f52d29.png">
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:
https://github.com/apache/metron/pull/768
@ottobackwards If you dont mind can you give me exact steps to simulate the installation issue. I seem to miss something here.
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/768
I am not sure. For multi field names "foo:bar:baz" maybe just "baz". or maybe like exceptions "f:b:baz". But that may not be great either. Just wanted to bring up the point for discussion, not a blocker.
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/768
I don't understand how this is not picking up the management pack integration for alerts? I don't see it in ambari
---
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/768#discussion_r144395087
--- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts ---
@@ -0,0 +1,319 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Component, Input, OnChanges, SimpleChanges } from '@angular/core';
+import {Router} from '@angular/router';
+import {Subscription} from 'rxjs/Rx';
+
+import {TableViewComponent} from '../table-view/table-view.component';
+import {SearchResponse} from '../../../model/search-response';
+import {SearchService} from '../../../service/search.service';
+import {TreeGroupData, TreeAlertsSubscription} from './tree-group-data';
+import {GroupResponse} from '../../../model/group-response';
+import {GroupResult} from '../../../model/group-result';
+import {Group} from '../../../model/group';
+import {SortField} from '../../../model/sort-field';
+import {Sort} from '../../../utils/enums';
+import {MetronDialogBox, DialogType} from '../../../shared/metron-dialog-box';
+import {ElasticsearchUtils} from '../../../utils/elasticsearch-utils';
+import {SearchRequest} from '../../../model/search-request';
+
+@Component({
+ selector: 'app-tree-view',
+ templateUrl: './tree-view.component.html',
+ styleUrls: ['./tree-view.component.scss']
+})
+
+export class TreeViewComponent extends TableViewComponent implements OnChanges {
+
+ groupByFields: string[] = [];
+ topGroups: TreeGroupData[] = [];
+ groupResponse: GroupResponse = new GroupResponse();
+ treeGroupSubscriptionMap: {[key: string]: TreeAlertsSubscription } = {};
+
+ constructor(router: Router,
+ searchService: SearchService,
+ metronDialogBox: MetronDialogBox) {
+ super(router, searchService, metronDialogBox);
+ }
+
+ collapseGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level > (level)) {
+ groupArray[i].show = false;
+ groupArray[i].expand = false;
+ } else {
+ break;
+ }
+ }
+ }
+
+ createQuery(selectedGroup: TreeGroupData) {
+ let searchQuery = this.queryBuilder.generateSelect();
+ let groupQery = Object.keys(selectedGroup.groupQueryMap).map(key => {
+ return key.replace(/:/g, '\\:') +
+ ':' +
+ String(selectedGroup.groupQueryMap[key])
+ .replace(/[\*\+\-=~><\"\?^\${}\(\)\:\!\/[\]\\\s]/g, '\\$&') // replace single special characters
+ .replace(/\|\|/g, '\\||') // replace ||
+ .replace(/\&\&/g, '\\&&'); // replace &&
+ }).join(' AND ');
+
+ groupQery += searchQuery === '*' ? '' : (' AND ' + searchQuery);
+ return groupQery;
+ }
+
+ expandGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level === (level + 1)) {
+ groupArray[i].show = true;
+ } else {
+ break;
+ }
+ }
+ }
+
+ getAlerts(selectedGroup: TreeGroupData): Subscription {
+ let searchRequest = new SearchRequest();
+ searchRequest.query = this.createQuery(selectedGroup);
+ searchRequest.from = selectedGroup.pagingData.from;
+ searchRequest.size = selectedGroup.pagingData.size;
+ searchRequest.sort = selectedGroup.sortField ? [selectedGroup.sortField] : [];
+
+ return this.searchGroup(selectedGroup, searchRequest);
--- End diff --
I am also noticing another issue with the group header counts that may or may not be related.
In this example, I am grouping by 'source:type' then 'host'. Since 'host' is specific to bro, there should be no alerts under the "snort" group. If I expand the Snort group, I correctly see no alerts.
The Snort group count is reported as 17,300 whereas I would expect that to be 0. The count of 17,300 is tallying how many Snort alerts there are. But adding a sub-group like 'host' can filter what the user will see when drilling down in the alerts.
IMHO We should always expect the group count to reflect how many alerts the user will see when drilling down.
---
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/768#discussion_r144633129
--- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts ---
@@ -0,0 +1,319 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Component, Input, OnChanges, SimpleChanges } from '@angular/core';
+import {Router} from '@angular/router';
+import {Subscription} from 'rxjs/Rx';
+
+import {TableViewComponent} from '../table-view/table-view.component';
+import {SearchResponse} from '../../../model/search-response';
+import {SearchService} from '../../../service/search.service';
+import {TreeGroupData, TreeAlertsSubscription} from './tree-group-data';
+import {GroupResponse} from '../../../model/group-response';
+import {GroupResult} from '../../../model/group-result';
+import {Group} from '../../../model/group';
+import {SortField} from '../../../model/sort-field';
+import {Sort} from '../../../utils/enums';
+import {MetronDialogBox, DialogType} from '../../../shared/metron-dialog-box';
+import {ElasticsearchUtils} from '../../../utils/elasticsearch-utils';
+import {SearchRequest} from '../../../model/search-request';
+
+@Component({
+ selector: 'app-tree-view',
+ templateUrl: './tree-view.component.html',
+ styleUrls: ['./tree-view.component.scss']
+})
+
+export class TreeViewComponent extends TableViewComponent implements OnChanges {
+
+ groupByFields: string[] = [];
+ topGroups: TreeGroupData[] = [];
+ groupResponse: GroupResponse = new GroupResponse();
+ treeGroupSubscriptionMap: {[key: string]: TreeAlertsSubscription } = {};
+
+ constructor(router: Router,
+ searchService: SearchService,
+ metronDialogBox: MetronDialogBox) {
+ super(router, searchService, metronDialogBox);
+ }
+
+ collapseGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level > (level)) {
+ groupArray[i].show = false;
+ groupArray[i].expand = false;
+ } else {
+ break;
+ }
+ }
+ }
+
+ createQuery(selectedGroup: TreeGroupData) {
+ let searchQuery = this.queryBuilder.generateSelect();
+ let groupQery = Object.keys(selectedGroup.groupQueryMap).map(key => {
+ return key.replace(/:/g, '\\:') +
+ ':' +
+ String(selectedGroup.groupQueryMap[key])
+ .replace(/[\*\+\-=~><\"\?^\${}\(\)\:\!\/[\]\\\s]/g, '\\$&') // replace single special characters
+ .replace(/\|\|/g, '\\||') // replace ||
+ .replace(/\&\&/g, '\\&&'); // replace &&
+ }).join(' AND ');
+
+ groupQery += searchQuery === '*' ? '' : (' AND ' + searchQuery);
+ return groupQery;
+ }
+
+ expandGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level === (level + 1)) {
+ groupArray[i].show = true;
+ } else {
+ break;
+ }
+ }
+ }
+
+ getAlerts(selectedGroup: TreeGroupData): Subscription {
+ let searchRequest = new SearchRequest();
+ searchRequest.query = this.createQuery(selectedGroup);
+ searchRequest.from = selectedGroup.pagingData.from;
+ searchRequest.size = selectedGroup.pagingData.size;
+ searchRequest.sort = selectedGroup.sortField ? [selectedGroup.sortField] : [];
+
+ return this.searchGroup(selectedGroup, searchRequest);
--- End diff --
@iraghumitra That doesn't explain this one, right? https://github.com/apache/metron/pull/768#discussion_r144387064
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/768
All of this happens in docker
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/768
What does the ungroup button **do** when you hit it and it is still in table view?
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/768
With respect to displaying field names, I think it's going to be tough coming up with a strategy that solves every case. This will likely depend on a user's preference and is subjective anyways. I agree with @iraghumitra, exposing a way to rename fields solves it.
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:
https://github.com/apache/metron/pull/768
@merrimanr Added e2e tests for escalation actions in tree view
---
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on a diff in the pull request:
https://github.com/apache/metron/pull/768#discussion_r144592599
--- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts ---
@@ -0,0 +1,319 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Component, Input, OnChanges, SimpleChanges } from '@angular/core';
+import {Router} from '@angular/router';
+import {Subscription} from 'rxjs/Rx';
+
+import {TableViewComponent} from '../table-view/table-view.component';
+import {SearchResponse} from '../../../model/search-response';
+import {SearchService} from '../../../service/search.service';
+import {TreeGroupData, TreeAlertsSubscription} from './tree-group-data';
+import {GroupResponse} from '../../../model/group-response';
+import {GroupResult} from '../../../model/group-result';
+import {Group} from '../../../model/group';
+import {SortField} from '../../../model/sort-field';
+import {Sort} from '../../../utils/enums';
+import {MetronDialogBox, DialogType} from '../../../shared/metron-dialog-box';
+import {ElasticsearchUtils} from '../../../utils/elasticsearch-utils';
+import {SearchRequest} from '../../../model/search-request';
+
+@Component({
+ selector: 'app-tree-view',
+ templateUrl: './tree-view.component.html',
+ styleUrls: ['./tree-view.component.scss']
+})
+
+export class TreeViewComponent extends TableViewComponent implements OnChanges {
+
+ groupByFields: string[] = [];
+ topGroups: TreeGroupData[] = [];
+ groupResponse: GroupResponse = new GroupResponse();
+ treeGroupSubscriptionMap: {[key: string]: TreeAlertsSubscription } = {};
+
+ constructor(router: Router,
+ searchService: SearchService,
+ metronDialogBox: MetronDialogBox) {
+ super(router, searchService, metronDialogBox);
+ }
+
+ collapseGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level > (level)) {
+ groupArray[i].show = false;
+ groupArray[i].expand = false;
+ } else {
+ break;
+ }
+ }
+ }
+
+ createQuery(selectedGroup: TreeGroupData) {
+ let searchQuery = this.queryBuilder.generateSelect();
+ let groupQery = Object.keys(selectedGroup.groupQueryMap).map(key => {
+ return key.replace(/:/g, '\\:') +
+ ':' +
+ String(selectedGroup.groupQueryMap[key])
+ .replace(/[\*\+\-=~><\"\?^\${}\(\)\:\!\/[\]\\\s]/g, '\\$&') // replace single special characters
+ .replace(/\|\|/g, '\\||') // replace ||
+ .replace(/\&\&/g, '\\&&'); // replace &&
+ }).join(' AND ');
+
+ groupQery += searchQuery === '*' ? '' : (' AND ' + searchQuery);
+ return groupQery;
+ }
+
+ expandGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level === (level + 1)) {
+ groupArray[i].show = true;
+ } else {
+ break;
+ }
+ }
+ }
+
+ getAlerts(selectedGroup: TreeGroupData): Subscription {
+ let searchRequest = new SearchRequest();
+ searchRequest.query = this.createQuery(selectedGroup);
+ searchRequest.from = selectedGroup.pagingData.from;
+ searchRequest.size = selectedGroup.pagingData.size;
+ searchRequest.sort = selectedGroup.sortField ? [selectedGroup.sortField] : [];
+
+ return this.searchGroup(selectedGroup, searchRequest);
--- End diff --
@nickwallen Again the problem is with how we are handling null values in the rest api. The host name for all the alerts is emty so we are not seeing any alerts there.
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/768
I think we should disable the button if there are no groups. I *does* do something, I don't know if it is resorting or something else, but the table refreshes and seems to have different content.
---
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/768#discussion_r144381219
--- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts ---
@@ -0,0 +1,319 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Component, Input, OnChanges, SimpleChanges } from '@angular/core';
+import {Router} from '@angular/router';
+import {Subscription} from 'rxjs/Rx';
+
+import {TableViewComponent} from '../table-view/table-view.component';
+import {SearchResponse} from '../../../model/search-response';
+import {SearchService} from '../../../service/search.service';
+import {TreeGroupData, TreeAlertsSubscription} from './tree-group-data';
+import {GroupResponse} from '../../../model/group-response';
+import {GroupResult} from '../../../model/group-result';
+import {Group} from '../../../model/group';
+import {SortField} from '../../../model/sort-field';
+import {Sort} from '../../../utils/enums';
+import {MetronDialogBox, DialogType} from '../../../shared/metron-dialog-box';
+import {ElasticsearchUtils} from '../../../utils/elasticsearch-utils';
+import {SearchRequest} from '../../../model/search-request';
+
+@Component({
+ selector: 'app-tree-view',
+ templateUrl: './tree-view.component.html',
+ styleUrls: ['./tree-view.component.scss']
+})
+
+export class TreeViewComponent extends TableViewComponent implements OnChanges {
+
+ groupByFields: string[] = [];
+ topGroups: TreeGroupData[] = [];
+ groupResponse: GroupResponse = new GroupResponse();
+ treeGroupSubscriptionMap: {[key: string]: TreeAlertsSubscription } = {};
+
+ constructor(router: Router,
+ searchService: SearchService,
+ metronDialogBox: MetronDialogBox) {
+ super(router, searchService, metronDialogBox);
+ }
+
+ collapseGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level > (level)) {
+ groupArray[i].show = false;
+ groupArray[i].expand = false;
+ } else {
+ break;
+ }
+ }
+ }
+
+ createQuery(selectedGroup: TreeGroupData) {
+ let searchQuery = this.queryBuilder.generateSelect();
+ let groupQery = Object.keys(selectedGroup.groupQueryMap).map(key => {
+ return key.replace(/:/g, '\\:') +
+ ':' +
+ String(selectedGroup.groupQueryMap[key])
+ .replace(/[\*\+\-=~><\"\?^\${}\(\)\:\!\/[\]\\\s]/g, '\\$&') // replace single special characters
+ .replace(/\|\|/g, '\\||') // replace ||
+ .replace(/\&\&/g, '\\&&'); // replace &&
+ }).join(' AND ');
+
+ groupQery += searchQuery === '*' ? '' : (' AND ' + searchQuery);
+ return groupQery;
+ }
+
+ expandGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level === (level + 1)) {
+ groupArray[i].show = true;
+ } else {
+ break;
+ }
+ }
+ }
+
+ getAlerts(selectedGroup: TreeGroupData): Subscription {
+ let searchRequest = new SearchRequest();
+ searchRequest.query = this.createQuery(selectedGroup);
+ searchRequest.from = selectedGroup.pagingData.from;
+ searchRequest.size = selectedGroup.pagingData.size;
+ searchRequest.sort = selectedGroup.sortField ? [selectedGroup.sortField] : [];
+
+ return this.searchGroup(selectedGroup, searchRequest);
--- End diff --
I confirmed manually that the latest does indeed fix this issue.
@merrimanr does have a good point. A quick unit test of this would be great.
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/768
I am going to try again
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:
https://github.com/apache/metron/pull/768
@merrimanr I will check the escalation actions, I guess I missed something there.
---
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on a diff in the pull request:
https://github.com/apache/metron/pull/768#discussion_r142671246
--- Diff: metron-interface/metron-alerts/e2e/alerts-list/tree-view/tree-view.e2e-spec.ts ---
@@ -0,0 +1,183 @@
+/// <reference path="../../matchers/custom-matchers.d.ts"/>
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {customMatchers} from '../../matchers/custom-matchers';
+import {LoginPage} from '../../login/login.po';
+import {TreeViewPage} from './tree-view.po';
+import {loadTestData, deleteTestData} from '../../utils/e2e_util';
+import {MetronAlertsPage} from '../alerts-list.po';
+
+describe('metron-alerts tree view', function () {
+ let page: TreeViewPage;
+ let listPage: MetronAlertsPage;
+ let loginPage: LoginPage;
+
+ beforeAll(() => {
+ loadTestData();
+ loginPage = new LoginPage();
+ page = new TreeViewPage();
+ listPage = new MetronAlertsPage();
+ loginPage.login();
+ page.navigateToAlertsList();
+ });
+
+ afterAll(() => {
+ loginPage.logout();
+ deleteTestData();
+ });
+
+ beforeEach(() => {
+ jasmine.addMatchers(customMatchers);
+ });
+
+ it('should have all group by elements', () => {
+ let groupByItems = {
+ 'source:type': '1',
+ 'ip_dst_addr': '8',
+ 'host': '9',
+ 'enrichm...:country': '3',
+ 'ip_src_addr': '2'
+ };
+ expect(page.getGroupByCount()).toEqualBcoz(Object.keys(groupByItems).length, '5 Group By Elements should be present');
+ expect(page.getGroupByItemNames()).toEqualBcoz(Object.keys(groupByItems), 'Group By Elements names should be present');
+ expect(page.getGroupByItemCounts()).toEqualBcoz(Object.keys(groupByItems).map(key => groupByItems[key]),
+ '5 Group By Elements values should be present');
+ });
+
+ it('should have group details for single group by', () => {
+ let dashRowValues = ['0', 'alerts_ui_e2e', 'ALERTS', '169'];
+ let row1_page1 = ['-', 'dcda4423-7...0962fafc47', '2017-09-13 17:59:32', 'alerts_ui_e2e',
+ '192.168.138.158', 'US', '72.34.49.86', 'comarksecurity.com', 'NEW', ''];
+ let row1_page2 = ['-', '07b29c29-9...ff19eaa888', '2017-09-13 17:59:37', 'alerts_ui_e2e',
+ '192.168.138.158', 'FR', '62.75.195.236', '62.75.195.236', 'NEW', ''];
+
+ page.selectGroup('source:type');
+ expect(page.getActiveGroups()).toEqualBcoz(['source:type'], 'only source type group should be selected');
+ expect(page.getDashGroupValues('alerts_ui_e2e')).toEqualBcoz(dashRowValues, 'Dash Group Values should be present');
+
+ page.expandDashGroup('alerts_ui_e2e');
+ expect(page.getDashGroupTableValuesForRow('alerts_ui_e2e', 0)).toEqualBcoz(row1_page1, 'Dash Group Values should be present');
+
+ page.clickOnNextPage('alerts_ui_e2e');
+ expect(page.getTableValuesByRowId('alerts_ui_e2e', 0, 'FR')).toEqualBcoz(row1_page2, 'Dash Group Values should be present');
+
+ page.unGroup();
+ expect(page.getActiveGroups()).toEqualBcoz([], 'no groups should be selected');
+ });
+
+ it('should have group details for multiple group by', () => {
+
+ let dashRow_runLoveUs = {
+ 'dashRow': ['0', 'runlove.us', 'ALERTS', '13'],
+ 'firstSubGroup': '0 US (13)',
+ 'firstSubGroupIdCol': ['9a969c64-b...001cb011a3', 'a651f7c3-1...a97d4966c9', 'afc36901-3...d931231ab2',
+ 'd860ac35-1...f9e282d571', '04a5c3d0-9...af17c06fbc']
+ };
+
+ let dashRow_62_75_195_236 = {
+ 'dashRow': ['0', '62.75.195.236', 'ALERTS', '18'],
+ 'firstSubGroup': '0 FR (18)',
+ 'firstSubGroupIdCol': ['07b29c29-9...ff19eaa888', '7cd91565-1...de5be54a6e', 'ca5bde58-a...f3a88d2df4',
+ '5d6faf83-8...b88a407647', 'e2883424-f...79bb8b0606']
+ };
+
+ page.selectGroup('host');
+ page.selectGroup('enrichments:geo:ip_dst_addr:country');
+ expect(page.getActiveGroups()).toEqualBcoz(['host', 'enrichments:geo:ip_dst_addr:country'], 'two groups should be selected');
+
+ expect(page.getDashGroupValues('runlove.us')).toEqualBcoz(dashRow_runLoveUs.dashRow,
+ 'Dash Group Values should be present for runlove.us');
--- End diff --
I am just holding the expected values in a map for easy understanding.
The actual values that we are expecting are dashRow, firstSubGroup, firstSubGroupIdCol which are present in the map named dashRow_runLoveUs.
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:
https://github.com/apache/metron/pull/768
@ottobackwards Done please have a look ...
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/768
Let me be more clear, sorry.
Vagrant up doesn't work.
If I do
```bash
% metron-deployment> mvn package -DskipTests -P build-rpms
```
I get these errors in the RPM's.
Any idea?
---
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/768#discussion_r144387064
--- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts ---
@@ -0,0 +1,319 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Component, Input, OnChanges, SimpleChanges } from '@angular/core';
+import {Router} from '@angular/router';
+import {Subscription} from 'rxjs/Rx';
+
+import {TableViewComponent} from '../table-view/table-view.component';
+import {SearchResponse} from '../../../model/search-response';
+import {SearchService} from '../../../service/search.service';
+import {TreeGroupData, TreeAlertsSubscription} from './tree-group-data';
+import {GroupResponse} from '../../../model/group-response';
+import {GroupResult} from '../../../model/group-result';
+import {Group} from '../../../model/group';
+import {SortField} from '../../../model/sort-field';
+import {Sort} from '../../../utils/enums';
+import {MetronDialogBox, DialogType} from '../../../shared/metron-dialog-box';
+import {ElasticsearchUtils} from '../../../utils/elasticsearch-utils';
+import {SearchRequest} from '../../../model/search-request';
+
+@Component({
+ selector: 'app-tree-view',
+ templateUrl: './tree-view.component.html',
+ styleUrls: ['./tree-view.component.scss']
+})
+
+export class TreeViewComponent extends TableViewComponent implements OnChanges {
+
+ groupByFields: string[] = [];
+ topGroups: TreeGroupData[] = [];
+ groupResponse: GroupResponse = new GroupResponse();
+ treeGroupSubscriptionMap: {[key: string]: TreeAlertsSubscription } = {};
+
+ constructor(router: Router,
+ searchService: SearchService,
+ metronDialogBox: MetronDialogBox) {
+ super(router, searchService, metronDialogBox);
+ }
+
+ collapseGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level > (level)) {
+ groupArray[i].show = false;
+ groupArray[i].expand = false;
+ } else {
+ break;
+ }
+ }
+ }
+
+ createQuery(selectedGroup: TreeGroupData) {
+ let searchQuery = this.queryBuilder.generateSelect();
+ let groupQery = Object.keys(selectedGroup.groupQueryMap).map(key => {
+ return key.replace(/:/g, '\\:') +
+ ':' +
+ String(selectedGroup.groupQueryMap[key])
+ .replace(/[\*\+\-=~><\"\?^\${}\(\)\:\!\/[\]\\\s]/g, '\\$&') // replace single special characters
+ .replace(/\|\|/g, '\\||') // replace ||
+ .replace(/\&\&/g, '\\&&'); // replace &&
+ }).join(' AND ');
+
+ groupQery += searchQuery === '*' ? '' : (' AND ' + searchQuery);
+ return groupQery;
+ }
+
+ expandGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level === (level + 1)) {
+ groupArray[i].show = true;
+ } else {
+ break;
+ }
+ }
+ }
+
+ getAlerts(selectedGroup: TreeGroupData): Subscription {
+ let searchRequest = new SearchRequest();
+ searchRequest.query = this.createQuery(selectedGroup);
+ searchRequest.from = selectedGroup.pagingData.from;
+ searchRequest.size = selectedGroup.pagingData.size;
+ searchRequest.sort = selectedGroup.sortField ? [selectedGroup.sortField] : [];
+
+ return this.searchGroup(selectedGroup, searchRequest);
--- End diff --
Actually, I spoke too soon. @iraghumitra Your fix works great for a single group. But if you are grouping by multiple fields, we still get in a state where the group headers are showing stale data.
In this screen shot, I am grouping by source-type, ip_src_addr, then country.
* The count for the "country" group was correctly updated as soon as I expanded the header.
* The count for the "ip_src_addr" group was NOT updated. It still displays the stale count of 1568.
* The count for the "source:type" group also was correctly updated.
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by iraghumitra <gi...@git.apache.org>.
Github user iraghumitra commented on the issue:
https://github.com/apache/metron/pull/768
@ottobackwards Good to hear the build issue is out of the way I was going nowhere in figuring out the build failure.
- The ungroup button does nothing when alerts are not grouped. I feel we can disable the ungroup button when no groups are selected. That might be less confusing
- Most of the field names in ES are too long for an aesthetic display, we have an option to rename filed's names to make them look meaningful. We can increase the number of chars but I feel there is no comprehensive way to make them look meaningful. I would suggest using renaming field names in UI. If you have a better suggestion I am open for that too.
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/768
I cannot get this to vagrant up:
----> MANY OF THESE ----->
metron-config.noarch: E: script-without-shebang /usr/metron/0.4.1/web/expressjs/node_modules/repeat-string/package.json
This text file has executable bits set or is located in a path dedicated for
executables, but lacks a shebang and cannot thus be executed. If the file is
meant to be an executable script, add the shebang, otherwise remove the
executable bits or move the file elsewhere.
metron-config.noarch: E: script-without-shebang /usr/metron/0.4.1/web/expressjs/node_modules/lodash/fp/pullAll.js
This text file has executable bits set or is located in a path dedicated for
executables, but lacks a shebang and cannot thus be executed. If the file is
meant to be an executable script, add the shebang, otherwise remove the
executable bits or move the file elsewhere.
metron-config.noarch: E: script-without-shebang /usr/metron/0.4.1/web/expressjs/node_modules/lodash/_LazyWrapper.js
This text file has executable bits set or is located in a path dedicated for
executables, but lacks a shebang and cannot thus be executed. If the file is
meant to be an executable script, add the shebang, otherwise remove the
executable bits or move the file elsewhere.
------
snip
------
(none): E: error while reading SRPMS/metron: 'SRPMS/metron'
14 packages and 1 specfiles checked; 1822 errors, 90 warnings.
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:copy-resources (copy-rpms-target) @ metron-rpm ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 14 resources
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] metron-deployment .................................. SUCCESS [ 0.672 s]
[INFO] metron-rpm ......................................... SUCCESS [07:45 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 07:45 min
[INFO] Finished at: 2017-09-29T18:10:52-04:00
[INFO] Final Memory: 15M/220M
[INFO] ------------------------------------------------------------------------
ottofowler@Winterfell [18:10:52] [~/tmp/metron-pr-768/metron-deployment] [pr-768]
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/768
It looks like it is working now. Maybe this error was an artifact or manifestation of the dependency issues that 'resolved' themselves. I'll get going
---
[GitHub] metron issue #768: Metron 1123: Add group by option using faceted search cap...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/768
I run this standard [checkout-pr](https://github.com/ottobackwards/Metron-and-Nifi-Scripts/blob/master/metron/checkout-pr) script, to download in to ~/tmp.
cd into the directory created 's metron-deployment/vagrant/full-dev-platform
vagrant up
---
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/768
---
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/768#discussion_r144644294
--- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts ---
@@ -0,0 +1,319 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Component, Input, OnChanges, SimpleChanges } from '@angular/core';
+import {Router} from '@angular/router';
+import {Subscription} from 'rxjs/Rx';
+
+import {TableViewComponent} from '../table-view/table-view.component';
+import {SearchResponse} from '../../../model/search-response';
+import {SearchService} from '../../../service/search.service';
+import {TreeGroupData, TreeAlertsSubscription} from './tree-group-data';
+import {GroupResponse} from '../../../model/group-response';
+import {GroupResult} from '../../../model/group-result';
+import {Group} from '../../../model/group';
+import {SortField} from '../../../model/sort-field';
+import {Sort} from '../../../utils/enums';
+import {MetronDialogBox, DialogType} from '../../../shared/metron-dialog-box';
+import {ElasticsearchUtils} from '../../../utils/elasticsearch-utils';
+import {SearchRequest} from '../../../model/search-request';
+
+@Component({
+ selector: 'app-tree-view',
+ templateUrl: './tree-view.component.html',
+ styleUrls: ['./tree-view.component.scss']
+})
+
+export class TreeViewComponent extends TableViewComponent implements OnChanges {
+
+ groupByFields: string[] = [];
+ topGroups: TreeGroupData[] = [];
+ groupResponse: GroupResponse = new GroupResponse();
+ treeGroupSubscriptionMap: {[key: string]: TreeAlertsSubscription } = {};
+
+ constructor(router: Router,
+ searchService: SearchService,
+ metronDialogBox: MetronDialogBox) {
+ super(router, searchService, metronDialogBox);
+ }
+
+ collapseGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level > (level)) {
+ groupArray[i].show = false;
+ groupArray[i].expand = false;
+ } else {
+ break;
+ }
+ }
+ }
+
+ createQuery(selectedGroup: TreeGroupData) {
+ let searchQuery = this.queryBuilder.generateSelect();
+ let groupQery = Object.keys(selectedGroup.groupQueryMap).map(key => {
+ return key.replace(/:/g, '\\:') +
+ ':' +
+ String(selectedGroup.groupQueryMap[key])
+ .replace(/[\*\+\-=~><\"\?^\${}\(\)\:\!\/[\]\\\s]/g, '\\$&') // replace single special characters
+ .replace(/\|\|/g, '\\||') // replace ||
+ .replace(/\&\&/g, '\\&&'); // replace &&
+ }).join(' AND ');
+
+ groupQery += searchQuery === '*' ? '' : (' AND ' + searchQuery);
+ return groupQery;
+ }
+
+ expandGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level === (level + 1)) {
+ groupArray[i].show = true;
+ } else {
+ break;
+ }
+ }
+ }
+
+ getAlerts(selectedGroup: TreeGroupData): Subscription {
+ let searchRequest = new SearchRequest();
+ searchRequest.query = this.createQuery(selectedGroup);
+ searchRequest.from = selectedGroup.pagingData.from;
+ searchRequest.size = selectedGroup.pagingData.size;
+ searchRequest.sort = selectedGroup.sortField ? [selectedGroup.sortField] : [];
+
+ return this.searchGroup(selectedGroup, searchRequest);
--- End diff --
Yes, that makes sense to me. @merrimanr
---
[GitHub] metron pull request #768: Metron 1123: Add group by option using faceted sea...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/768#discussion_r144639257
--- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts ---
@@ -0,0 +1,319 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Component, Input, OnChanges, SimpleChanges } from '@angular/core';
+import {Router} from '@angular/router';
+import {Subscription} from 'rxjs/Rx';
+
+import {TableViewComponent} from '../table-view/table-view.component';
+import {SearchResponse} from '../../../model/search-response';
+import {SearchService} from '../../../service/search.service';
+import {TreeGroupData, TreeAlertsSubscription} from './tree-group-data';
+import {GroupResponse} from '../../../model/group-response';
+import {GroupResult} from '../../../model/group-result';
+import {Group} from '../../../model/group';
+import {SortField} from '../../../model/sort-field';
+import {Sort} from '../../../utils/enums';
+import {MetronDialogBox, DialogType} from '../../../shared/metron-dialog-box';
+import {ElasticsearchUtils} from '../../../utils/elasticsearch-utils';
+import {SearchRequest} from '../../../model/search-request';
+
+@Component({
+ selector: 'app-tree-view',
+ templateUrl: './tree-view.component.html',
+ styleUrls: ['./tree-view.component.scss']
+})
+
+export class TreeViewComponent extends TableViewComponent implements OnChanges {
+
+ groupByFields: string[] = [];
+ topGroups: TreeGroupData[] = [];
+ groupResponse: GroupResponse = new GroupResponse();
+ treeGroupSubscriptionMap: {[key: string]: TreeAlertsSubscription } = {};
+
+ constructor(router: Router,
+ searchService: SearchService,
+ metronDialogBox: MetronDialogBox) {
+ super(router, searchService, metronDialogBox);
+ }
+
+ collapseGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level > (level)) {
+ groupArray[i].show = false;
+ groupArray[i].expand = false;
+ } else {
+ break;
+ }
+ }
+ }
+
+ createQuery(selectedGroup: TreeGroupData) {
+ let searchQuery = this.queryBuilder.generateSelect();
+ let groupQery = Object.keys(selectedGroup.groupQueryMap).map(key => {
+ return key.replace(/:/g, '\\:') +
+ ':' +
+ String(selectedGroup.groupQueryMap[key])
+ .replace(/[\*\+\-=~><\"\?^\${}\(\)\:\!\/[\]\\\s]/g, '\\$&') // replace single special characters
+ .replace(/\|\|/g, '\\||') // replace ||
+ .replace(/\&\&/g, '\\&&'); // replace &&
+ }).join(' AND ');
+
+ groupQery += searchQuery === '*' ? '' : (' AND ' + searchQuery);
+ return groupQery;
+ }
+
+ expandGroup(groupArray: TreeGroupData[], level: number, index: number) {
+ for (let i = index + 1; i < groupArray.length; i++) {
+ if (groupArray[i].level === (level + 1)) {
+ groupArray[i].show = true;
+ } else {
+ break;
+ }
+ }
+ }
+
+ getAlerts(selectedGroup: TreeGroupData): Subscription {
+ let searchRequest = new SearchRequest();
+ searchRequest.query = this.createQuery(selectedGroup);
+ searchRequest.from = selectedGroup.pagingData.from;
+ searchRequest.size = selectedGroup.pagingData.size;
+ searchRequest.sort = selectedGroup.sortField ? [selectedGroup.sortField] : [];
+
+ return this.searchGroup(selectedGroup, searchRequest);
--- End diff --
I believe this issue is similar to https://github.com/apache/metron/pull/768#issuecomment-335281376 and should be resolved in the same follow on PR. Do you agree @nickwallen?
---