You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Andy Isaacson (JIRA)" <ji...@apache.org> on 2012/07/27 03:37:37 UTC
[jira] [Created] (HDFS-3733) audit logging must cover WebHdfs
access
Andy Isaacson created HDFS-3733:
-----------------------------------
Summary: audit logging must cover WebHdfs access
Key: HDFS-3733
URL: https://issues.apache.org/jira/browse/HDFS-3733
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.0.0-alpha
Reporter: Andy Isaacson
Assignee: Andy Isaacson
Access via WebHdfs does not result in audit log entries. It should.
{noformat}
% curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
{"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
{noformat}
and observe that no audit log entry is generated.
Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
{noformat}
% curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
...
< HTTP/1.1 307 TEMPORARY_REDIRECT
< Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
...
% curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
...
< HTTP/1.1 200 OK
< Content-Type: application/octet-stream
< Content-Length: 12
< Server: Jetty(6.1.26.cloudera.1)
<
hello world
{noformat}
This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445468#comment-13445468 ]
Andy Isaacson commented on HDFS-3733:
-------------------------------------
bq. I have to logAuditEvent(false under any exception.
This false assumption was the root of my confusion. In fact, if an exception other than ACE occurs, there's no need to logAuditEvent. None of the other callsites do so.
Thanks for bringing this up, Eli. New patch attached.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733-6.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) audit logging must cover WebHdfs
access
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424079#comment-13424079 ]
Andy Isaacson commented on HDFS-3733:
-------------------------------------
bq. WebHDFS access HDFS via NameNodeRpcServer.
Yes, and NNRS calls into FSNamesystem which does generate audit log entries *if* {{isExternalInvocation()}} returns true.
bq. The access paths of WebHDFS should be similar to the ones in RPC.
The codepath is similar, but FSNamesystem only generates audit log events if the current request is coming from an RPC. It tells this by checking {{Server.isRpcInvocation}}.
The attached patch does the following:
{code}
private boolean isExternalInvocation() {
- return Server.isRpcInvocation();
+ return Server.isRpcInvocation() || NameNodeRpcServer.isWebHdfsInvocation();
+ }
{code}
I think this is the approach you're suggesting, no?
bq. Have you checked DistributedFileSystem.getFileStatus(..)? Does it generate audit logs?
Indeed, it does not.
bq. For OPEN, WebHDFS calls getFileInfo(..) and getBlockLocations(..) via NameNodeRpcServer in the namenode side. These two methods seem not generating audit log.
Those get called in the NN (and don't generate audit log events), and then the NN sends the client a 307 redirect to a DN, and then the DN calls getBlockLocations again, which does generate an audit log on successful access. With my patch the NN side generates audit logs for both allowed and disallowed access.
> audit logging must cover WebHdfs access
> ---------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444634#comment-13444634 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3733:
----------------------------------------------
- How about moving isWebHdfsInvocation() and getRemoteIp() from NameNodeRpcServer to NamenodeWebHdfsMethods? These two methods are not RPC related.
- FSNamesystem.getRemoteIp() should be static.
- The following change seems not useful.
{code}
@@ -380,8 +389,11 @@ class NameNodeRpcServer implements NamenodeProtocols {
long length)
throws IOException {
metrics.incrGetBlockLocations();
- return namesystem.getBlockLocations(getClientMachine(),
- src, offset, length);
+ try {
+ return namesystem.getBlockLocations(getClientMachine(),
+ src, offset, length);
+ } finally {
+ }
{code}
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Isaacson updated HDFS-3733:
--------------------------------
Attachment: hdfs-3733-1.txt
Attaching new patch, removing the need for the ThreadLocal curClient by simply using NamenodeWebHdfsMethods#REMOTE_ADDRESS. Various other cleanups, and add test showing that Hftp is also audited.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) audit logging must cover WebHdfs
access
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-3733:
---------------------------------
Status: Patch Available (was: Open)
Marking PA for Andy so that Jenkins runs.
> audit logging must cover WebHdfs access
> ---------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445510#comment-13445510 ]
Eli Collins commented on HDFS-3733:
-----------------------------------
Looks great Andy.
+1 pending jenkins.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733-6.txt, hdfs-3733-7.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) audit logging must cover WebHdfs
access
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423623#comment-13423623 ]
Andy Isaacson commented on HDFS-3733:
-------------------------------------
Additionally, since a denied OPEN does not hit the DN, only allowed OPEN requests generate audit log entries.
> audit logging must cover WebHdfs access
> ---------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445155#comment-13445155 ]
Eli Collins commented on HDFS-3733:
-----------------------------------
Andy, looking good!
- In FSN#getFileInfo why catch UnresolvedLinkException and StandbyException, just AccessControlException is sufficient right?
- Nit, I'd remove the System.out.printlns for debugging in the tests?
- Per jenkins there's a javadoc warning:
{noformat}
[WARNING] /home/eli/src/hadoop2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java:132: warning - Tag @link: reference not found: Server#isRpcInvocation()
{noformat}
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444794#comment-13444794 ]
Hadoop QA commented on HDFS-3733:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12543049/hdfs-3733-4.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestHftpDelegationToken
org.apache.hadoop.hdfs.TestClientReportBadBlock
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3122//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3122//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3122//console
This message is automatically generated.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Isaacson updated HDFS-3733:
--------------------------------
Attachment: hdfs-3733-6.txt
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733-6.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445565#comment-13445565 ]
Hadoop QA commented on HDFS-3733:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12543203/hdfs-3733-7.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestHftpDelegationToken
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3131//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3131//console
This message is automatically generated.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733-6.txt, hdfs-3733-7.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3733:
------------------------------
Target Version/s: 2.2.0-alpha (was: 2.1.0-alpha)
Summary: Audit logs should include WebHDFS access (was: audit logging must cover WebHdfs access)
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) audit logging must cover WebHdfs
access
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Isaacson updated HDFS-3733:
--------------------------------
Attachment: hdfs-3733.txt
Attaching a patch which addresses this issue.
* add getRemoteIp helper to FSNamesystem
* teach FSNamesystem#isExternalInvocation to know about NameNodeRpcServer.isWebHdfsInvocation as well as isRpcInvocation
* add a ThreadLocal to NameNodeRpcServer and use it to implement isWebHdfsInvocation
* add tests for WebHdfs to TestAuditLogs
This patch has the following issues:
* NameNodeRpcServer#CurClient needs to be set in more places; currently it is only set for open().
* as a result the getFileStatus test does not succeed.
I'm posting this work in progress to get feedback on the general approach. If people would like me to take a different tack, I'm happy to hear that.
> audit logging must cover WebHdfs access
> ---------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed HDFS-3733.
-------------------------------
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Fix For: 2.0.2-alpha
>
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733-6.txt, hdfs-3733-7.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443888#comment-13443888 ]
Hadoop QA commented on HDFS-3733:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542898/hdfs-3733-1.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
-1 javac. The patch appears to cause the build to fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3118//console
This message is automatically generated.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445897#comment-13445897 ]
Hudson commented on HDFS-3733:
------------------------------
Integrated in Hadoop-Hdfs-trunk #1151 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1151/])
HDFS-3733. Audit logs should include WebHDFS access. Contributed by Andy Isaacson (Revision 1379278)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379278
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogs.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Fix For: 2.2.0-alpha
>
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733-6.txt, hdfs-3733-7.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444386#comment-13444386 ]
Andy Isaacson commented on HDFS-3733:
-------------------------------------
bq. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
TestFsck is due to audit log changes. TestHftpDelegationToken may be due to the change, it does fail for me but the linkage to audit logs is not obviuos.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) audit logging must cover WebHdfs
access
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425111#comment-13425111 ]
Hadoop QA commented on HDFS-3733:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538118/hdfs-3733.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
org.apache.hadoop.hdfs.TestPersistBlocks
org.apache.hadoop.hdfs.server.namenode.TestFsck
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2919//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2919//console
This message is automatically generated.
> audit logging must cover WebHdfs access
> ---------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) audit logging must cover WebHdfs
access
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423647#comment-13423647 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3733:
----------------------------------------------
Hi Andy,
WebHDFS access HDFS via NameNodeRpcServer. The access paths of WebHDFS should be similar to the ones in RPC. So I think the problems you found are not specific to WebHDFS. Have you checked DistributedFileSystem.getFileStatus(..)? Does it generate audit logs?
For OPEN, WebHDFS calls getFileInfo(..) and getBlockLocations(..) via NameNodeRpcServer in the namenode side. These two methods seem not generating audit log.
Suppose audit log is required. We should change FSNamesystem but not WebHDFS.
> audit logging must cover WebHdfs access
> ---------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Isaacson updated HDFS-3733:
--------------------------------
Attachment: hdfs-3733-3.txt
TestHftpDelegationToken is known flaky. TestFsck broke because Fsck uses getFileInfo under the covers (see NamenodeFsck#fsck) and we now audit log those accesses correctly.
So I've updated TestFsck's verifyAuditLogs to handle this case correctly. New patch attached.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445327#comment-13445327 ]
Hadoop QA commented on HDFS-3733:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12543151/hdfs-3733-6.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
org.apache.hadoop.hdfs.TestHftpDelegationToken
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3127//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3127//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3127//console
This message is automatically generated.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733-6.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443584#comment-13443584 ]
Andy Isaacson commented on HDFS-3733:
-------------------------------------
OK, backing up -- I think my addition of CurClient just duplicates functionality already provided by NamenodeWebHdfsMethods#REMOTE_ADDRESS . So I can drop that new ThreadLocal and just teach NameNodeRpcServer to use REMOTE_ADDRESS appropriately.
Or am I missing something?
bq. getRemoteIp should not just return NamenodeWebHdfsMethods#getRemoteAddress
(I assume you are referring to my newly added {{FSNamesystem#getRemoteIp}}.)
Agreed, FSNamesystem should support all remote methods: RPC, WebHdfs ... and Hftp? The {{FSNamesystem#getRemoteIp}} should handle them all.
The helper {{NameNodeRpcServer#getRemoteIp}} implements the WebHdfs portion of {{FSNamesystem#getRemoteIp}} just as {{Server#getRemoteIp}} implements the RPC portion.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444748#comment-13444748 ]
Andy Isaacson commented on HDFS-3733:
-------------------------------------
bq. How about moving isWebHdfsInvocation() and getRemoteIp() from NameNodeRpcServer to NamenodeWebHdfsMethods? These two methods are not RPC related.
Fair enough, done.
bq. FSNamesystem.getRemoteIp() should be static.
Yep, thanks.
bq. The following change seems not useful.
It made more sense in a previous version of the patch. :) Fixed!
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3733:
-----------------------------------------
Hadoop Flags: Reviewed
+1
Andy, thanks for the update. The new patch looks good.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) audit logging must cover WebHdfs
access
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438182#comment-13438182 ]
Eli Collins commented on HDFS-3733:
-----------------------------------
Overall approach makes sense.
- s/CurClient/curClient/
- Use {@link..} to refer to a.h.ipc.Server#isRpcInvocation
- isWebHdfsInvocation returns true if client != null, and client is intialized with getClientMachine which can return non-null values in the non-webhdfs case, doesn't this imply isWebHdfsInvocation might return true even if the invocation is not for webhdfs?
- getRemoteIp should not just return NamenodeWebHdfsMethods#getRemoteAddress because it returns null in the non-webhdfs case even though Server.getRemoteAddress() has a value, ie we're losing the remote IP for the non-webhdfs case
- Looks like NamenodeWebHdfsMethods#setRemoteAddress is only used by the tests, can we remove it?
- Please add a TestAuditLogs test that covers access via Hftp so we make sure we're not just doing something webhdfs specific
> audit logging must cover WebHdfs access
> ---------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3733:
------------------------------
Resolution: Fixed
Fix Version/s: 2.2.0-alpha
Target Version/s: (was: 2.2.0-alpha)
Status: Resolved (was: Patch Available)
Test failure is unrelated.
I've committed this. Thanks Andy!
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Fix For: 2.2.0-alpha
>
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733-6.txt, hdfs-3733-7.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444586#comment-13444586 ]
Hadoop QA commented on HDFS-3733:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12543005/hdfs-3733-3.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestHftpDelegationToken
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3121//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3121//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3121//console
This message is automatically generated.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444625#comment-13444625 ]
Andy Isaacson commented on HDFS-3733:
-------------------------------------
The findbugs are unrelated and the HftpDelegationToken failure is unrelated. I believe this is ready to go.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Isaacson updated HDFS-3733:
--------------------------------
Attachment: hdfs-3733-4.txt
Attaching latest version of patch.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445259#comment-13445259 ]
Andy Isaacson commented on HDFS-3733:
-------------------------------------
bq. In FSN#getFileInfo why catch UnresolvedLinkException and StandbyException, just AccessControlException is sufficient right?
I have to {{logAuditEvent(false}} under any exception. Todd suggested doing this instead:
{code}
+ } catch (Throwable e) {
if (auditLog.isInfoEnabled() && isExternalInvocation()) {
logAuditEvent(false, UserGroupInformation.getCurrentUser(),
getRemoteIp(),
"getfileinfo", src, null, null);
}
- throw e;
- } catch (StandbyException e) {
- if (auditLog.isInfoEnabled() && isExternalInvocation()) {
- logAuditEvent(false, UserGroupInformation.getCurrentUser(),
- getRemoteIp(),
- "getfileinfo", src, null, null);
- }
- throw e;
+ Throwables.propagateIfPossible(e, AccessControlException.class);
+ Throwables.propagateIfPossible(e, UnresolvedLinkException.class);
+ Throwables.propagateIfPossible(e, StandbyException.class);
+ Throwables.propagateIfPossible(e, IOException.class);
+ throw new RuntimeException("unexpected", e);
{code}
bq. Nit, I'd remove the System.out.printlns for debugging in the tests?
Where's the upside to removing them? It adds a few KB at most to the MBs of test output, and I always end up adding the prinlns when trying to grok failures.
But, whatever. Removed.
bq. javadoc warning
Turns out you have to import anything you want to {{@link}}. Fixed.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733-6.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Isaacson updated HDFS-3733:
--------------------------------
Attachment: hdfs-3733-7.txt
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733-6.txt, hdfs-3733-7.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444356#comment-13444356 ]
Hadoop QA commented on HDFS-3733:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542959/hdfs-3733-2.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestFsck
org.apache.hadoop.hdfs.TestHftpDelegationToken
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3120//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3120//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3120//console
This message is automatically generated.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Isaacson updated HDFS-3733:
--------------------------------
Attachment: hdfs-3733-2.txt
Sync patch with trunk and revert a gratuitous NL-at-EOF change that might be messing up patch application.
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3733) Audit logs should include WebHDFS
access
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445952#comment-13445952 ]
Hudson commented on HDFS-3733:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1182 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1182/])
HDFS-3733. Audit logs should include WebHDFS access. Contributed by Andy Isaacson (Revision 1379278)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379278
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogs.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java
> Audit logs should include WebHDFS access
> ----------------------------------------
>
> Key: HDFS-3733
> URL: https://issues.apache.org/jira/browse/HDFS-3733
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Andy Isaacson
> Assignee: Andy Isaacson
> Fix For: 2.2.0-alpha
>
> Attachments: hdfs-3733-1.txt, hdfs-3733-2.txt, hdfs-3733-3.txt, hdfs-3733-4.txt, hdfs-3733-6.txt, hdfs-3733-7.txt, hdfs-3733.txt
>
>
> Access via WebHdfs does not result in audit log entries. It should.
> {noformat}
> % curl "http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=GETFILESTATUS"
> {"FileStatus":{"accessTime":1343351432395,"blockSize":134217728,"group":"supergroup","length":12,"modificationTime":1342808158399,"owner":"adi","pathSuffix":"","permission":"644","replication":1,"type":"FILE"}}
> {noformat}
> and observe that no audit log entry is generated.
> Interestingly, OPEN requests do not generate audit log entries when the NN generates the redirect, but do generate audit log entries when the second phase against the DN is executed.
> {noformat}
> % curl -v 'http://nn1:50070/webhdfs/v1/user/adi/hello.txt?op=OPEN'
> ...
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Location: http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020&offset=0
> ...
> % curl -v 'http://dn01:50075/webhdfs/v1/user/adi/hello.txt?op=OPEN&namenoderpcaddress=nn1:8020'
> ...
> < HTTP/1.1 200 OK
> < Content-Type: application/octet-stream
> < Content-Length: 12
> < Server: Jetty(6.1.26.cloudera.1)
> <
> hello world
> {noformat}
> This happens because {{DatanodeWebHdfsMethods#get}} uses {{DFSClient#open}} thereby triggering the existing {{logAuditEvent}} code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira