You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Daryn Sharp (Created) (JIRA)" <ji...@apache.org> on 2012/01/10 18:07:39 UTC
[jira] [Created] (HADOOP-7967) Need generalized multi-token
filesystem support
Need generalized multi-token filesystem support
-----------------------------------------------
Key: HADOOP-7967
URL: https://issues.apache.org/jira/browse/HADOOP-7967
Project: Hadoop Common
Issue Type: Bug
Components: fs, security
Affects Versions: 0.24.0, 0.23.1
Reporter: Daryn Sharp
Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-7967:
---------------------------------
Attachment: hadoop7967-javadoc.patch
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch, hadoop7967-javadoc.patch, hadoop7967-javadoc.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Attachment: HADOOP-7967.newapi.3.patch
# Added the provided javadocs patch
# Completely removed {{getDelegationTokens}} methods, even for 23. Might as well find bugs in the upper stack components now than later...
# As suggested, {{addDelegationTokens(renewer,null)}} semantically behaves like the removed {{getDelegationTokens(renewer)}}
# Create {{TestFileSystemTokens}} in common and move most of the applicable token fetching tests into it from mapreduce. Tests are now more comprehensive with greater coverage.
# Remove GETDELEGATIONTOKENS from httpfs since it was removed from webhdfs
# Downgraded visibility of {{FileSystem#getDelegationToken}} to protected
#* caused a lot of test updates
#* exposed multi-token bug in fetchdt
#* had to update webhdfs token renewal
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch, hadoop7967-javadoc.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Attachment: HADOOP-7967.newapi.patch
Implements the new api within {{FileSystem}} classes and {{TokenCache}}.
The change is already so large that I'd like to separate out {{FileContext}} and {{AbstractFileSystem}} changes into a separate jira. There is no code reuse between fs and fc/afs so it's a completely different set of changes. The fc is not used by MR for token acquisition so this patch will stand on its own.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Attachment: HADOOP-7967.newapi.5.patch
Add my new test file I dropped in the last patch, and fix conflict in DelegationTokenFetcher where someone else removed the unnecessary imports that I removed too.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.5.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Attachment: HADOOP-7967.newapi.4.patch
Includes Sanjay's diff.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated HADOOP-7967:
----------------------------------
Priority: Critical (was: Major)
Target Version/s: 2.0.0-alpha, 0.23.3, 3.0.0 (was: 0.23.3, 2.0.0-alpha, 3.0.0)
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420853#comment-13420853 ]
Sanjay Radia commented on HADOOP-7967:
--------------------------------------
@daryn .. I'm not we should tie whether a fs has a token to whether it has children. ..
Only as a default impl. The methods can be overrided when needed. In your patch, ViewFileSystem returns null for getCanonicalName and getDelegationToken. This will be true for very many file systems that are built using other file systems (but one can override).
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202789#comment-13202789 ]
Sanjay Radia commented on HADOOP-7967:
--------------------------------------
Look at my comment in MAPREDUCE-3825.
There IS support for multi-token file system (hence we changed from getDelegationToken() to getDelegationTokenS().
I think this is more of an issue of efficiency (see my example in MAPREDUCE-3825).
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.24.0, 0.23.1
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13432130#comment-13432130 ]
Sanjay Radia commented on HADOOP-7967:
--------------------------------------
+1 - i have some changes in the delta patch I am attaching.
* Some minor improvements to javadoc and added comments to some tests (in my delta patch)
* Given that Hadoop 1 did not have audience annotations we can't make getDelegationToken protected. I have fixed that in my delta patch. Note the changes you made in tests to call addDelegationTokens instead of getDelegationToken are correct and should remain.
* File a Jira to make getDelegationTokens protected - lets see if community feels this can be done at some stage.
* File a Jira to make the corresponding changes to FileContext/AbstractFileSystem as we discussed.
Thanks for adding more tests and for refactoring some of the test internals.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch, hadoop7967-javadoc.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425318#comment-13425318 ]
Sanjay Radia commented on HADOOP-7967:
--------------------------------------
Webhdfs - okay with your patch - getDelegationToken is good enough; removing getDelegationTokens, which wasn't in Hadoop 1, is correct.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch, hadoop7967-javadoc.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Attachment: HADOOP-7967-4.patch
Fix the warning in a test case.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.24.0, 0.23.1
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202844#comment-13202844 ]
Daryn Sharp commented on HADOOP-7967:
-------------------------------------
We should continue the discussion on MAPREDUCE-3825. There is no dispute that multi-token support exists. However, it's broken in a number of ways, and is incompatible with a {{TokenCache}} that assumes a 1 to 1 service/token mapping.
Regarding efficiency, leaking duplicate tokens everywhere increases the memory load on the namenode, fills the edit logs, increases the txn burden, increases startup time, increases GC time, leaked tokens aren't cancelled so they live until they expire, and it increases the load on the secret manager.
It may be debated if duplicate tokens are a bug or performance, but fixing the duplicate tokens is only incidentally fixed when fixing the bugs such as the broken viewfs mount table.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.24.0, 0.23.1
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-7967:
---------------------------------
Attachment: (was: hadoop7967-javadoc.patch)
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch, hadoop7967-javadoc.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435994#comment-13435994 ]
Daryn Sharp commented on HADOOP-7967:
-------------------------------------
Test failures aren't related.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.5.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Target Version/s: 2.0.0-alpha, 0.23.3, 3.0.0 (was: 0.23.3, 2.0.0-alpha, 3.0.0)
Status: Patch Available (was: Open)
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270515#comment-13270515 ]
Daryn Sharp commented on HADOOP-7967:
-------------------------------------
@Sanjay: I'd like to make a suggestion to accelerate fixing these token bugs. The mutually agreed upon solution in MAPREDUCE-3825 is more extensive and breaks api compatibility, so would it be reasonable to commit this backward-compatible change, with an immediate followup jira for the new apis?
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418608#comment-13418608 ]
Sanjay Radia commented on HADOOP-7967:
--------------------------------------
Some early comments:
A suggestion: center the code around "FileSystem#getChildFileSystems()"
* FileSystems which embed others such as ViewFS will only need to implement getChildFileSystems()
* the default impl of getCanaonicalName will return null if getChildFileSystems() returns null
* the defaul impl of getDelegationToken will return null if getChildFileSystems() returns null
* In collectDelegationTokens use "if (getChildFileSystems()==null)" rather than "if (getCanaonicalName() == null)
{code}
collectDelegationTokens(....) {
....
children = getChildFileSystems();
if (children == null) { // get token directly from FS
... your code ...
} else { // get the tokens from the children
... your code
}
}
{code}
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-7967:
---------------------------------
Attachment: hadoop7967-deltas.patch
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420860#comment-13420860 ]
Sanjay Radia commented on HADOOP-7967:
--------------------------------------
@sanjay the defaul impl of getDelegationToken will return null if getChildFileSystems() returns null ...
Sorry I misspoke, the default impl for getDelegationToken should always be null (as is today).
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425276#comment-13425276 ]
Hadoop QA commented on HADOOP-7967:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538430/hadoop7967-javadoc.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1234//console
This message is automatically generated.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch, hadoop7967-javadoc.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated HADOOP-7967:
----------------------------------------
Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.2)
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Attachment: HADOOP-7967-compat.patch
Resubmitting patch since the build didn't run.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-7967:
---------------------------------
Attachment: hadoop7967-javadoc.patch
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch, hadoop7967-javadoc.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Attachment: (was: HADOOP-7967-compat.patch)
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435475#comment-13435475 ]
Hadoop QA commented on HADOOP-7967:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12541059/HADOOP-7967.newapi.4.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1309//console
This message is automatically generated.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291143#comment-13291143 ]
Sanjay Radia commented on HADOOP-7967:
--------------------------------------
@Daryn ... so would it be reasonable to commit this backward-compatible change, with an immediate followup jira for the new apis?..
Can you please describe what the patch does and if it is proper subset of the the originally agreed solution and what will remain for the follow up jira?
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Status: Open (was: Patch Available)
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.24.0, 0.23.1
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290396#comment-13290396 ]
Daryn Sharp commented on HADOOP-7967:
-------------------------------------
TestViewFsTrash is notorious for failing. I can't reproduce the failure in TestDatanodeBlockScanner, and these changes have nothing to do with the DN, so it must be a sporadic failure.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Attachment: HADOOP-7967-3.patch
Add a lot more tests to strenuously test token retrieval.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.24.0, 0.23.1
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398740#comment-13398740 ]
Sanjay Radia commented on HADOOP-7967:
--------------------------------------
Daryn, rather than do this in two or three steps, let us do a single patch to address this problem completely. I have summarized our discussion in MAPREDUCE-3825 in this [comment.|https://issues.apache.org/jira/browse/MAPREDUCE-3825?focusedCommentId=13398726&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13398726]
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp reassigned HADOOP-7967:
-----------------------------------
Assignee: Daryn Sharp
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.24.0, 0.23.1
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436349#comment-13436349 ]
Sanjay Radia commented on HADOOP-7967:
--------------------------------------
+1. please commit it.
Thanks for working with me through several design alternatives
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.5.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-7967:
---------------------------------
Target Version/s: 2.0.0-alpha, 0.23.3, 3.0.0 (was: 0.23.3, 2.0.0-alpha, 3.0.0)
Status: Open (was: Patch Available)
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch, hadoop7967-javadoc.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290288#comment-13290288 ]
Hadoop QA commented on HADOOP-7967:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12531113/HADOOP-7967-compat.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.fs.viewfs.TestViewFsTrash
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1088//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1088//console
This message is automatically generated.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201829#comment-13201829 ]
Hadoop QA commented on HADOOP-7967:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513522/HADOOP-7967-2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/567//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/567//console
This message is automatically generated.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.24.0, 0.23.1
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13432482#comment-13432482 ]
Sanjay Radia commented on HADOOP-7967:
--------------------------------------
Clarification:
>Given that Hadoop 1 did not have audience annotations we can't make getDelegationToken protected
What I meant was that in Hadoop1, getDelegationToken was public without any annotations; hence we would be breaking compatibility if we made it protected.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Attachment: HADOOP-7967.patch
This is an untested patch that presents an alternate approach. Let's discuss if this approach should be pursued or if there are other clean approaches.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.24.0, 0.23.1
> Reporter: Daryn Sharp
> Attachments: HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202548#comment-13202548 ]
Hadoop QA commented on HADOOP-7967:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513632/HADOOP-7967-3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 1053 javac compiler warnings (more than the trunk's current 1052 warnings).
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/569//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/569//console
This message is automatically generated.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.24.0, 0.23.1
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436772#comment-13436772 ]
Hudson commented on HADOOP-7967:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2624 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2624/])
HADOOP-7967. Need generalized multi-token filesystem support (daryn) (Revision 1374271)
Result = FAILURE
daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374271
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemTestHelper.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemTokens.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSKerberosAuthenticator.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSKerberosAuthenticationHandler.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSKerberosAuthenticationHandler.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSWithKerberos.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemAtHdfsRoot.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemHdfs.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationTokenForProxyUser.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.5.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202572#comment-13202572 ]
Hadoop QA commented on HADOOP-7967:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513643/HADOOP-7967-4.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/571//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/571//console
This message is automatically generated.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.24.0, 0.23.1
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Target Version/s: 2.0.0-alpha, 0.23.3, 3.0.0 (was: 0.23.3, 2.0.0-alpha, 3.0.0)
Fix Version/s: 2.2.0-alpha
3.0.0
2.1.0-alpha
0.23.3
Had to omit changes on 2.1 and 0.23.3 for the following non-existent files:
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSWithKerberos.java
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSKerberosAuthenticationHandler.java
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSKerberosAuthenticationHandler.java
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSKerberosAuthenticator.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemAtHdfsRoot.java
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Fix For: 0.23.3, 2.1.0-alpha, 3.0.0, 2.2.0-alpha
>
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.5.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Target Version/s: 0.24.0, 0.23.1 (was: 0.23.1, 0.24.0)
Status: Patch Available (was: Open)
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.24.0, 0.23.1
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422528#comment-13422528 ]
Daryn Sharp commented on HADOOP-7967:
-------------------------------------
* getDelegationToken should have audience of HDFS and MapReduce. Why the deprecate with a comment?
It was already deprecated, and is in fact no longer used by MR. The goal is to eventually reduce the visibility to prevent external calls. Hence why I removed MR from the audience.
* getDelegationTokens - remove from viewFileSystem test
Ok, assuming we don't keep it around a little longer (see next point).
* getDelegationTokens - remove this method - only 0.23 needs to keep it for transition
We discussed that option before, but I think we may want to reconsider retaining it as deprecated for at least one more alpha release. {{getDelegationToken}} is supposedly limited audience to mapreduce and hdfs. Unfortunately I found documentation, such as secure impersonation, that tells people to use it! When {{getDelegationToken}} was deprecated in 1.x, the javadocs referred people to {{getDelegationToken(S)}}. Abruptly removing {{getDelegationToken(S)}} w/o notice might break other projects.
* collectDelegationTokens - add javadoc - esp the parameter returnAllTokens
It's a private method, but ok.
* collectDelegationTokens - Shouldn't the code be (or are you saying that a fs can have its own token + those of its children)
Yes, I don't think an intrinsic token vs child tokens are mutually exclusive. I'd like to make the api flexible to allow a fs to have its own token and/or child tokens.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437312#comment-13437312 ]
Hudson commented on HADOOP-7967:
--------------------------------
Integrated in Hadoop-Hdfs-trunk #1138 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1138/])
HADOOP-7967. Need generalized multi-token filesystem support (daryn) (Revision 1374271)
Result = FAILURE
daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374271
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemTestHelper.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemTokens.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSKerberosAuthenticator.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSKerberosAuthenticationHandler.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSKerberosAuthenticationHandler.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSWithKerberos.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemAtHdfsRoot.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemHdfs.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationTokenForProxyUser.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.5.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291215#comment-13291215 ]
Daryn Sharp commented on HADOOP-7967:
-------------------------------------
Certainly. I suggested in a May 8 comment to commit this minimal patch (essentially the original patch minus the bits that have since gone in on separate jiras), with the more extensive changes implemented on a followup jira. The high level summary of the patch for this jira:
# I fixed the existing apis, with no semantic changes, in order to maintain backwards-compat
# A filesystem's service always matches the token's service to prevent dups since tokens have always been fetched if a filesystem's service is not already in the creds
# Ability to query a fs for its set of leaf filesystems
# {{FileSystem}} obtains only missing tokens for a fs, instead of getting all tokens and ignoring the dups
# {{ViewFileSystem}} won't get tokens for identical mountpoints
# Implicit side-effects include being able to better stack multi-token and/or filtered filesystems
The remaining followup work is the (incompatible) api changes, auto-population of the given credentials, and new methods like getting direct child filesystems. This patch does not conflict or complicate the larger goal since it only fixes the existing apis.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422388#comment-13422388 ]
Sanjay Radia commented on HADOOP-7967:
--------------------------------------
* I will email you some javadoc improvements
* getDelegationToken should have audience of HDFS and MapReduce
** Why the deprecate with a comment?
* getDelegationTokens
** remove from viewFileSystem test
** remove this method - only 0.23 needs to keep it for transition
* collectDelegationTokens
** add javadoc - esp the parameter returnAllTokens
** Typo in comment - "if (serviceName != null) { // fs has no token" - you mean fs has tokens
** Shouldn't the code be (or are you saying that a fs can have its own token + those of its children)
{code}
if (serviceName != null) { // fs has a token of its own
...
} else { // fs has no token of its own, get tokens from its children
..
}
{code}
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438663#comment-13438663 ]
Hudson commented on HADOOP-7967:
--------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #350 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/350/])
svn merge -c 1374346 FIXES: HADOOP-7967. Need generalized multi-token filesystem support (daryn) (Revision 1375063)
Result = SUCCESS
daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375063
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemTestHelper.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemTokens.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenRenewer.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemHdfs.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationTokenForProxyUser.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Fix For: 0.23.3, 2.1.0-alpha, 3.0.0, 2.2.0-alpha
>
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.5.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437334#comment-13437334 ]
Hudson commented on HADOOP-7967:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk #1170 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1170/])
HADOOP-7967. Need generalized multi-token filesystem support (daryn) (Revision 1374271)
Result = FAILURE
daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374271
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemTestHelper.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemTokens.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSKerberosAuthenticator.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSKerberosAuthenticationHandler.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSKerberosAuthenticationHandler.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSWithKerberos.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemAtHdfsRoot.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemHdfs.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationTokenForProxyUser.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.5.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Resolution: Fixed
Target Version/s: 2.0.0-alpha, 0.23.3, 3.0.0 (was: 0.23.3, 2.0.0-alpha, 3.0.0)
Status: Resolved (was: Patch Available)
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Fix For: 0.23.3, 2.1.0-alpha, 3.0.0, 2.2.0-alpha
>
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.5.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Attachment: HADOOP-7967-compat.patch
Suggested patch that fixes many token acquisition bugs (viewfs, dup tokens, inability to stack filtered filesystems, etc) w/o changing existing public apis, their semantics, or introducing incompatibilities. This is intended only as a short term fix until the apis can be overhauled as described in a linked jira.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417352#comment-13417352 ]
Sanjay Radia commented on HADOOP-7967:
--------------------------------------
@daryn I'd like to separate out FileContext and AbstractFileSystem changes into a separate jira
That sounds good since there is not code share as you state.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428537#comment-13428537 ]
Hadoop QA commented on HADOOP-7967:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12539093/HADOOP-7967.newapi.3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 16 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.hdfs.TestDFSClientRetries
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1249//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1249//console
This message is automatically generated.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch, hadoop7967-javadoc.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425265#comment-13425265 ]
Sanjay Radia commented on HADOOP-7967:
--------------------------------------
* Noticed that in DelegateToFileSystem#getDelegationTokens() you simply want the tokens and pass in an empty credentials to addDTs - shall we have addDelegationTokens deal with the null credentials parameter to indicate that it should simply return the tokens rather than add. I am okay either way.
* getDelegationToken
** undeprecate getDelegationToken - since getDelegationTokens is being removed (the deprecation was added when we added getDelegationTokens)
** Does it work to make it protected?
* Since the audience for getDelegationTokens was HDFS & MapReduce originally, it is safe to remove getDelegationTokens right away from trunk and target this jira for Hadoop 2. Hadoop 0.23 can remove it later. Such changes are allowed in Alphas especially since the audience was limitedPrivate.
* Some of the new tests in TestViewSystemDelegationTokenSupport is using the old deprecated getDelegationTokens - it should use the new addDelegationTokens.
* WebHdfs rest API - not sure what is needed here - noticed you removed GETDELEGATIONTOKENS. I need to think this one through.
* TestTokenCache
** Can't you take advantage of the default impl of addDelegationTokens and hence not need getTokens?
** add javadoc to createFileSystemForService - change name to createFileSystemForServiceName
* File Jira for fixing Viewfs and AbstractFileSystem
* I have attached patch that improves some of the javadocs - you can apply it on top of yours.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch, hadoop7967-javadoc.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Attachment: HADOOP-7967.newapi.2.patch
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Attachment: HADOOP-7967-2.patch
Took a slightly different approach to avoid changing semi-private apis. Added {{getFileSystems}} that returns a unique list of filesystems. This prevents the {{TokenCache}} from trying to use the service of a filesystem (like viewfs) as a key into its hash, when the filesystem is never going to return a token with that service. Whereas the flattened {{FileSystem}} list allows the {{TokenCache}} to safely "assume" the service of the token that will be returned. I'll try to further cleanup the {{TokenCache}} later.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.24.0, 0.23.1
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420882#comment-13420882 ]
Sanjay Radia commented on HADOOP-7967:
--------------------------------------
I think you are right: (getCanonicalName() != null) => FS has token
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed HADOOP-7967.
---------------------------------
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Fix For: 0.23.3, 2.0.2-alpha
>
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.5.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436752#comment-13436752 ]
Hudson commented on HADOOP-7967:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #2594 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2594/])
HADOOP-7967. Need generalized multi-token filesystem support (daryn) (Revision 1374271)
Result = SUCCESS
daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374271
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemTestHelper.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemTokens.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSKerberosAuthenticator.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSKerberosAuthenticationHandler.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSKerberosAuthenticationHandler.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSWithKerberos.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemAtHdfsRoot.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemHdfs.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationTokenForProxyUser.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.5.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HADOOP-7967:
--------------------------------
Target Version/s: 2.0.0-alpha, 0.23.3, 3.0.0 (was: 0.23.3, 2.0.0-alpha, 3.0.0)
Status: Patch Available (was: Open)
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420808#comment-13420808 ]
Daryn Sharp commented on HADOOP-7967:
-------------------------------------
Would you please explain further? I'm not we should tie whether a fs has a token to whether it has children. That assumption may be true for viewfs, but isn't a requirement.
bq. FileSystems which embed others such as ViewFS will only need to implement getChildFileSystems()
I think the proposals run counter to that goal. The methods {{getCanonicalService}} and {{getDelegation}} are the simple atoms used by the implementation of {{addDelegationTokens}} that an fs either does or does not implement.
bq. the default impl of getCanaonicalName will return null if getChildFileSystems() returns null
What would be the behavior when {{getChildFileSystems}} _does not_ return null? Note that viewfs has to override it to always return null instead of the uri's authority (mount table in this case). If the default is changed to always return null, which I can do, it becomes another incompatible change...
bq. the defaul impl of getDelegationToken will return null if getChildFileSystems() returns null
What would be the behavior when {{getChildFileSystems}} _does not_ return null? The default is/was return null and the fs overrides only if it has a token.
The logic is currently: see if the fs itself has a token, yes, get it. Irrespective, see if the fs has children, yes, repeat the process for each child.
{code}
collectDelegationTokens(....) {
....
if (getCanonicalService() != null) { // get a token for myself
getDelegationToken(...);
}
foreach childFs : getChildFileSystems() { // ask each of my children for a token
childFs.collectDelegationTokens(...);
}
}
{code}
It has to be recursive to allow for arbitrarily stacked filesystems. Only allowing the top-level fs to have multiple children will not allow mergefs to work.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435636#comment-13435636 ]
Hadoop QA commented on HADOOP-7967:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12541137/HADOOP-7967.newapi.5.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 16 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.ha.TestZKFailoverController
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1310//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1310//console
This message is automatically generated.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.5.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436749#comment-13436749 ]
Hudson commented on HADOOP-7967:
--------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2658 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2658/])
HADOOP-7967. Need generalized multi-token filesystem support (daryn) (Revision 1374271)
Result = SUCCESS
daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374271
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemTestHelper.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemTokens.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSKerberosAuthenticator.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSKerberosAuthenticationHandler.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSKerberosAuthenticationHandler.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSWithKerberos.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemAtHdfsRoot.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemHdfs.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationTokenForProxyUser.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.4.patch, HADOOP-7967.newapi.5.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416574#comment-13416574 ]
Hadoop QA commented on HADOOP-7967:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536854/HADOOP-7967.newapi.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified test files.
-1 javac. The applied patch generated 2074 javac compiler warnings (more than the trunk's current 2067 warnings).
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.fs.TestFilterFileSystem
org.apache.hadoop.ha.TestZKFailoverController
org.apache.hadoop.io.file.tfile.TestTFileByteArrays
org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
org.apache.hadoop.hdfs.TestPersistBlocks
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1199//testReport/
Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1199//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1199//console
This message is automatically generated.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7967) Need generalized multi-token
filesystem support
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433684#comment-13433684 ]
Daryn Sharp commented on HADOOP-7967:
-------------------------------------
The dilemma over breaking compatibility by reducing the visibility of {{getDelegationToken}} is interesting. Code that attempts to use this method is incompatible with {{ViewFileSystem}}. Viewfs can't return tokens and then tasks will unexpectedly fail due to the missing tokens...
I'm willing to revert the visibility, although this is maybe one of the few cases where making things break is the right thing to do?
> Need generalized multi-token filesystem support
> -----------------------------------------------
>
> Key: HADOOP-7967
> URL: https://issues.apache.org/jira/browse/HADOOP-7967
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, security
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967-compat.patch, hadoop7967-deltas.patch, hadoop7967-javadoc.patch, HADOOP-7967.newapi.2.patch, HADOOP-7967.newapi.3.patch, HADOOP-7967.newapi.patch, HADOOP-7967.patch
>
>
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic. The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently {{FileSystem}} will leak tokens even when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down into the {{FileSystem}} level. The {{TokenCache}} should be ignorant and simply request tokens from each {{FileSystem}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira