You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "curie71 (via GitHub)" <gi...@apache.org> on 2023/03/08 15:58:37 UTC
[GitHub] [hadoop] curie71 opened a new pull request, #5464: HDFS-16944 Add audit log for RouterAdminServer to save privileged operation log seperately.
curie71 opened a new pull request, #5464:
URL: https://github.com/apache/hadoop/pull/5464
HDFS-16944 We found that in other components (like namenode in hdfs or resourcemanager in yarn), debug log and audit log are record seperately, except RouterAdminServer.
We found that in other components (like namenode in hdfs or resourcemanager in yarn), *debug log and audit log are record seperately*, except RouterAdminServer.
There are lots of +simple+ logs to help with debugging for the *developers* who can access to the source code. And there are also audit logs record +privileged operations+ with more +detailed+ information to help *system admins* understand what happened in a real run.
There is an example in yarn:
```java
public static final Log auditLog = LogFactory.getLog(
FSNamesystem.class.getName() + ".audit");
try {
// Safety
userUgi = UserGroupInformation.getCurrentUser();
user = userUgi.getShortUserName();
} catch (IOException ie) {
LOG.warn("Unable to get the current user.", ie); // debug log
RMAuditLogger.logFailure(user, AuditConstants.SUBMIT_APP_REQUEST,
ie.getMessage(), "ClientRMService",
"Exception in submitting application", applicationId, callerContext,
submissionContext.getQueue()); // audit log
throw RPCUtil.getRemoteException(ie);
}
```
So I suggest to add an audit log for *RouterAdminServer* to save privileged operation logs seperately.
The logger' s name may be:
```java
// hadoop security
public static final Logger AUDITLOG =
LoggerFactory.getLogger(
"SecurityLogger." + ServiceAuthorizationManager.class.getName());
// namenode
public static final Log auditLog = LogFactory.getLog(
FSNamesystem.class.getName() + ".audit");
```
I choose className.audit finally and record AUDITLOG instead of LOG for the privileged operations that call permission check function _checkSuperuserPrivilege_.
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
### How was this patch tested?
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-16944 Add audit log for RouterAdminServer to save privileged operation log seperately. [hadoop]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #5464: HDFS-16944 Add audit log for RouterAdminServer to save privileged operation log seperately.
URL: https://github.com/apache/hadoop/pull/5464
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5464: HDFS-16944 Add audit log for RouterAdminServer to save privileged operation log seperately.
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5464:
URL: https://github.com/apache/hadoop/pull/5464#issuecomment-1474184981
Agree to the above suggestion of having a new audit logger class and use it for all operations.
Also Ayush, do you think we should rather have audit logs that indicate the audit and not the result as such? For instance, here it seems we are trying to audit the result instead like "whether nameservice was successfully enabled".
Instead, audit log could have been something like "enable nameservice {ns} is called by client {client/ip etc}". The result of whether or not nameservice was enabled successfully is something to be saved as internal application logs only rather than audit, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-16944 Add audit log for RouterAdminServer to save privileged operation log seperately. [hadoop]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5464:
URL: https://github.com/apache/hadoop/pull/5464#issuecomment-3440030007
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #5464: HDFS-16944 Add audit log for RouterAdminServer to save privileged operation log seperately.
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #5464:
URL: https://github.com/apache/hadoop/pull/5464#issuecomment-1474357461
The only policy around Audit logs that I am aware of, they shouldn't change and the parsers should be able to parse them, So, if we make the format similar to that of the HDFS ones, that should be good but still not a strict ask.
Regarding logging `success`, we do log successful result everytime, it is just the failure which get logs only for ACE. Why only ACE has historical reasons, which we can decide here, if we want to go with that approach in RBF also or not, whatever we choose is good, it is just we can't change it once released.
One main reason for only ACE was: We are always interested In successful cases, since that change the state of the FS, other failure we don't care because they didn't do anything, for RPC load or so use the NamenodeMetrics.
ACE finds special attention as it is bit alarming: that someone who doesn't have access attempted to do some operation, "Some illegal guy in the town trying to do operation X, which he ain't allowed to do"
That is what I know, that is very old stuff, will research sometime more about that....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5464: HDFS-16944 Add audit log for RouterAdminServer to save privileged operation log seperately.
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5464:
URL: https://github.com/apache/hadoop/pull/5464#issuecomment-1474436169
Wow, quite a history behind ACE, thanks a lot for summarizing everything here :)
I was wondering in the past also as to why is it that hdfs audit logs only care for auth success/failure rather than overall operation success/failure but yes it does make sense to audit if authorized access was granted for the given operation, as it's more alarming than other failures. (for other failures, there are tons of logs anyways)
Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5464: HDFS-16944 Add audit log for RouterAdminServer to save privileged operation log seperately.
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5464:
URL: https://github.com/apache/hadoop/pull/5464#issuecomment-1474195335
While hdfs audit logs report whether the given operation had successful authorization (i.e. whether `AccessControlException` was thrown), but perhaps RMAuditLogger seems to have variety of failures being reported in audit log. So maybe we don't follow specific convention of audit as such (i might be wrong), then maybe it's upto us to report success/failure of the given operation in RBF overall?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5464: HDFS-16944 Add audit log for RouterAdminServer to save privileged operation log seperately.
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5464:
URL: https://github.com/apache/hadoop/pull/5464#issuecomment-1461110917
For RBF, if we really want audit logs, they should cover all like add/update mount table entry etc, and not just name service APIs. If we only need auditing for name service RPC, then we can rather name the logger as `NameserviceManager.class.getName() + ".audit"` rather than `RouterAdminServer.class.getName() + ".audit"` correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5464: HDFS-16944 Add audit log for RouterAdminServer to save privileged operation log seperately.
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5464:
URL: https://github.com/apache/hadoop/pull/5464#issuecomment-1460612337
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 25s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 1m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 29s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 16s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5464/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 1m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 24s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 21m 1s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 123m 29s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5464/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d6d53c4ce83d 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 78b6013e1edec591cf0d66d748f46268d8aa43e3 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5464/1/testReport/ |
| Max. process+thread count | 2205 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5464/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org