You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@eagle.apache.org by DadanielZ <gi...@git.apache.org> on 2016/08/04 21:31:12 UTC
[GitHub] incubator-eagle pull request #300: EAGLE-283 Convert fid,uid in MapR's audit...
GitHub user DadanielZ opened a pull request:
https://github.com/apache/incubator-eagle/pull/300
EAGLE-283 Convert fid,uid in MapR's audit log to FIle/folder name, user name
1. Added resolver for user's input of "volume", "file/folder", "Status"
- These resolver provide drop down list for user to select
2. Added NameConvertor to convert user's input to corresponding id and saved it in policy
- volume value will be used to call MapR's REST API to get volume id
- status code will be resolved to status id
- file/folder name will be passed to MapR's command line class to get its id.
3. Modified controller.js
- As soon as user input a value for "volume", "file", "status", it will fetch user's input and call rest service to get corresponding id
- Save ids as a new match criteria in the policy
Eg: after user select "== '/test'" in field "dst", a new match criteria "== 234.567.890" will appear automatically under it, and the policy will end up as:
dst == 'test' or dst == '234.567.890'
Same for volume and status input.
In this way, the policy is human readable and can be used to match the ids in maprfsAuditLog for monitoring.
4. Added maprfs resource resolver to provide drop down list for user
5. Added license for "/eagle-app-example/src/main/java/org/apache/eagle/app/example/ExampleStormConfig.java" to pass rat check.
For user id conversion I haven't come up with a perfect solution, this is because:
- As user id is more OS standard staff, user might need to make OS calls to get the user ids.
- If user use LDAP, it will make it easier as we can query that directly.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/DadanielZ/incubator-eagle develop
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-eagle/pull/300.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #300
----
commit c6d10261e5ba32bf22c1fcee07c8c0ebf6f43a68
Author: Daniel <dz...@binghamton.edu>
Date: 2016-07-05T18:11:12Z
Merge pull request #12 from apache/develop
EAGLE-276 eagle support for mr & spark history job monitoring
commit 92d7a14a529b17608f018fb624039bb9115b9dff
Author: Daniel <dz...@binghamton.edu>
Date: 2016-07-21T05:18:22Z
Merge pull request #13 from apache/develop
update from Eagle develop
commit 2d97d281ec10eb24dca76f64557733d412ce6a19
Author: Daniel <dz...@binghamton.edu>
Date: 2016-07-28T17:56:52Z
Merge pull request #14 from apache/develop
update from eagle-develop
commit 1e2e6a98569d2000f3561c420067e2f926fa2a75
Author: Daniel <dz...@binghamton.edu>
Date: 2016-07-29T19:38:01Z
Merge pull request #15 from apache/develop
update from apache-eagle develop branch
commit 55b6e1d8279be9b72b3958c52fc36bbe72a2ceb0
Author: Daniel <da...@apache.org>
Date: 2016-08-04T05:10:29Z
Merge pull request #16 from apache/develop
update from eagle-develop
commit 7340451cbb0621bfc1eebae1deeed730349ee8a6
Author: Daniel <da...@dataguise.com>
Date: 2016-08-04T21:19:37Z
EAGLE-283: Convert fid,uid in MapR's audit log to FIle/folder name, user name
1. Added resolver for user's input of "volume", "file/folder", "Status"
- These resolver provide drop down list for user to select
2. Added NameConvertor to convert user's input to corresponding id and saved it in policy
- volume value will be used to call MapR's REST API to get volume id
- status code will be resolved to status id
- file/folder name will be passed to MapR's command line class to get its id.
3. Modified controller.js
- As soon as user input a value for "volume", "file", "status", it will fetch user's input and call rest service to get corresponding id
- Save ids as a new match criteria in the policy
Eg: user select "== '/test'" in field "dst", a new match criteria "== 234.567.890" will appear automatically under it, and the policy will end up as:
dst == 'test' or dst == '234.567.890'
Same for volume and status input.
In this way, the policy is human readable and can be used to match the ids in maprfsAuditLog.
4. Added maprfs resource resolver to provide drop down list for user
5. Added license for "/eagle-app-example/src/main/java/org/apache/eagle/app/example/ExampleStormConfig.java" to pass rat check.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle issue #300: EAGLE-283 Convert fid,uid in MapR's audit log to...
Posted by haoch <gi...@git.apache.org>.
Github user haoch commented on the issue:
https://github.com/apache/incubator-eagle/pull/300
@DadanielZ thanks, but I think you don't have to reopen another new pull request but could continue to modify and commit on you current branch as any additional changes will append directly to this PR.
But as a better practice, you are recommended to create an independent feature branch (typically named EAGLE-${JIRA_ID}) from latest apache `develop` branch in your own forked repository for every new pull request instead of using `develop` branch always :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle pull request #300: EAGLE-283 Convert fid,uid in MapR's audit...
Posted by haoch <gi...@git.apache.org>.
Github user haoch commented on a diff in the pull request:
https://github.com/apache/incubator-eagle/pull/300#discussion_r73635073
--- Diff: eagle-security/eagle-security-maprfs-web/pom.xml ---
@@ -64,6 +64,16 @@
<artifactId>eagle-security-common</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>com.mapr.hadoop</groupId>
+ <artifactId>maprfs</artifactId>
+ <version>${maprfs.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ <version>4.5.2</version>
--- End diff --
Please avoid declaring any dependency with explicit version but use property defined in root pom instead. Better way is to define in dependency-management firstly and include without version.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle issue #300: EAGLE-283 Convert fid,uid in MapR's audit log to...
Posted by DadanielZ <gi...@git.apache.org>.
Github user DadanielZ commented on the issue:
https://github.com/apache/incubator-eagle/pull/300
As I mentioned, user id conversion is a little bit tricky, it will be great if someone can give me a hint on user id conversion issue.
Thanks and regards,
Da
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle pull request #300: EAGLE-283 Convert fid,uid in MapR's audit...
Posted by haoch <gi...@git.apache.org>.
Github user haoch commented on a diff in the pull request:
https://github.com/apache/incubator-eagle/pull/300#discussion_r73635506
--- Diff: eagle-webservice/src/main/resources/application.conf ---
@@ -16,13 +16,24 @@
{
eagle {
service {
- storage-type = "hbase"
--- End diff --
Please keep hbase storage as default store type
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle issue #300: EAGLE-283 Convert fid,uid in MapR's audit log to...
Posted by DadanielZ <gi...@git.apache.org>.
Github user DadanielZ commented on the issue:
https://github.com/apache/incubator-eagle/pull/300
By the way, "id converter" will make http request to mapr host.
As mapr use https connection, " javax.net.ssl.SSLHandshakeException Error" will occur if user didn't add the certificate to JVM's trust store.
Will update this on related doc later.
Regards,
Da
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle pull request #300: EAGLE-283 Convert fid,uid in MapR's audit...
Posted by DadanielZ <gi...@git.apache.org>.
Github user DadanielZ commented on a diff in the pull request:
https://github.com/apache/incubator-eagle/pull/300#discussion_r73637287
--- Diff: eagle-webservice/src/main/resources/application.conf ---
@@ -16,13 +16,24 @@
{
eagle {
service {
- storage-type = "hbase"
--- End diff --
thank you for the review\uff0cwill update it and create a new PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle issue #300: EAGLE-283 Convert fid,uid in MapR's audit log to...
Posted by haoch <gi...@git.apache.org>.
Github user haoch commented on the issue:
https://github.com/apache/incubator-eagle/pull/300
@DadanielZ reviewed as above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle issue #300: EAGLE-283 Convert fid,uid in MapR's audit log to...
Posted by DadanielZ <gi...@git.apache.org>.
Github user DadanielZ commented on the issue:
https://github.com/apache/incubator-eagle/pull/300
@haoch thanks for your advice! XD
Working on a feature branch will be a good practice.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle pull request #300: EAGLE-283 Convert fid,uid in MapR's audit...
Posted by DadanielZ <gi...@git.apache.org>.
Github user DadanielZ closed the pull request at:
https://github.com/apache/incubator-eagle/pull/300
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle pull request #300: EAGLE-283 Convert fid,uid in MapR's audit...
Posted by DadanielZ <gi...@git.apache.org>.
Github user DadanielZ closed the pull request at:
https://github.com/apache/incubator-eagle/pull/300
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle pull request #300: EAGLE-283 Convert fid,uid in MapR's audit...
Posted by haoch <gi...@git.apache.org>.
Github user haoch commented on a diff in the pull request:
https://github.com/apache/incubator-eagle/pull/300#discussion_r73635124
--- Diff: eagle-security/eagle-security-maprfs-auditlog/src/main/resources/maprFSAuditLog-init.sh ---
@@ -32,7 +32,7 @@ curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:a
"application":"maprFSAuditLog"
},
"enabled": true,
- "config": "classification.fs.defaultFS=maprfs:///"
+ "config": "classification.fs.defaultFS=hdfs://mapr1.da.dg:7222\nclassification.mapr.webhttps=https://mapr1.da.dg:8443\nclassification.mapr.username=mapr\nclassification.mapr.password=mapr"
--- End diff --
Is `mapr1.da.dg` dataguise sensitive info? :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle pull request #300: EAGLE-283 Convert fid,uid in MapR's audit...
Posted by haoch <gi...@git.apache.org>.
Github user haoch commented on a diff in the pull request:
https://github.com/apache/incubator-eagle/pull/300#discussion_r73635217
--- Diff: eagle-security/eagle-security-maprfs-web/src/main/test/java/MapRNameResolverTest.java ---
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+import org.apache.eagle.service.security.hdfs.rest.MapRNameResolver;
+
+public class MapRNameResolverTest {
+
+ public static void main(String[] args){
--- End diff --
Could you pls. refactor the `main` to a unit test case?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle pull request #300: EAGLE-283 Convert fid,uid in MapR's audit...
Posted by DadanielZ <gi...@git.apache.org>.
GitHub user DadanielZ reopened a pull request:
https://github.com/apache/incubator-eagle/pull/300
EAGLE-283 Convert fid,uid in MapR's audit log to FIle/folder name, user name
1. Added resolver for user's input of "volume", "file/folder", "Status"
- These resolver provide drop down list for user to select
2. Added NameConvertor to convert user's input to corresponding id and saved it in policy
- volume value will be used to call MapR's REST API to get volume id
- status code will be resolved to status id
- file/folder name will be passed to MapR's command line class to get its id.
3. Modified controller.js
- As soon as user input a value for "volume", "file", "status", it will fetch user's input and call rest service to get corresponding id
- Save ids as a new match criteria in the policy
Eg: after user select "== '/test'" in field "dst", a new match criteria "== 234.567.890" will appear automatically under it, and the policy will end up as:
dst == 'test' or dst == '234.567.890'
Same for volume and status input.
In this way, the policy is human readable and can be used to match the ids in maprfsAuditLog for monitoring.
4. Added maprfs resource resolver to provide drop down list for user
5. Added license for "/eagle-app-example/src/main/java/org/apache/eagle/app/example/ExampleStormConfig.java" to pass rat check.
For user id conversion I haven't come up with a perfect solution, this is because:
- As user id is more OS standard staff, user might need to make OS calls to get the user ids.
- If user use LDAP, it will make it easier as we can query that directly.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/DadanielZ/incubator-eagle develop
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-eagle/pull/300.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #300
----
commit c6d10261e5ba32bf22c1fcee07c8c0ebf6f43a68
Author: Daniel <dz...@binghamton.edu>
Date: 2016-07-05T18:11:12Z
Merge pull request #12 from apache/develop
EAGLE-276 eagle support for mr & spark history job monitoring
commit 92d7a14a529b17608f018fb624039bb9115b9dff
Author: Daniel <dz...@binghamton.edu>
Date: 2016-07-21T05:18:22Z
Merge pull request #13 from apache/develop
update from Eagle develop
commit 2d97d281ec10eb24dca76f64557733d412ce6a19
Author: Daniel <dz...@binghamton.edu>
Date: 2016-07-28T17:56:52Z
Merge pull request #14 from apache/develop
update from eagle-develop
commit 1e2e6a98569d2000f3561c420067e2f926fa2a75
Author: Daniel <dz...@binghamton.edu>
Date: 2016-07-29T19:38:01Z
Merge pull request #15 from apache/develop
update from apache-eagle develop branch
commit 55b6e1d8279be9b72b3958c52fc36bbe72a2ceb0
Author: Daniel <da...@apache.org>
Date: 2016-08-04T05:10:29Z
Merge pull request #16 from apache/develop
update from eagle-develop
commit 7340451cbb0621bfc1eebae1deeed730349ee8a6
Author: Daniel <da...@dataguise.com>
Date: 2016-08-04T21:19:37Z
EAGLE-283: Convert fid,uid in MapR's audit log to FIle/folder name, user name
1. Added resolver for user's input of "volume", "file/folder", "Status"
- These resolver provide drop down list for user to select
2. Added NameConvertor to convert user's input to corresponding id and saved it in policy
- volume value will be used to call MapR's REST API to get volume id
- status code will be resolved to status id
- file/folder name will be passed to MapR's command line class to get its id.
3. Modified controller.js
- As soon as user input a value for "volume", "file", "status", it will fetch user's input and call rest service to get corresponding id
- Save ids as a new match criteria in the policy
Eg: user select "== '/test'" in field "dst", a new match criteria "== 234.567.890" will appear automatically under it, and the policy will end up as:
dst == 'test' or dst == '234.567.890'
Same for volume and status input.
In this way, the policy is human readable and can be used to match the ids in maprfsAuditLog.
4. Added maprfs resource resolver to provide drop down list for user
5. Added license for "/eagle-app-example/src/main/java/org/apache/eagle/app/example/ExampleStormConfig.java" to pass rat check.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle pull request #300: EAGLE-283 Convert fid,uid in MapR's audit...
Posted by haoch <gi...@git.apache.org>.
Github user haoch commented on a diff in the pull request:
https://github.com/apache/incubator-eagle/pull/300#discussion_r73635450
--- Diff: eagle-security/eagle-security-maprfs-web/src/main/test/java/org/apache/eagle/service/security/hdfs/rest/HttpRequestTest.java ---
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.eagle.service.security.hdfs.rest;
+
+import org.apache.eagle.service.security.hdfs.resolver.MAPRFSVolumeResolver;
+import org.json.JSONObject;
+
+import java.util.List;
+
+public class HttpRequestTest {
+ public static void main(String args[]) throws Exception {
+ String restUrl = "https://mapr1.da.dg:8443/rest/volume/list";
--- End diff --
- Could you pls. refactor the `main` to a unit test case ?
- Probably sensitive info?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---