You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Kan Zhang (JIRA)" <ji...@apache.org> on 2009/06/16 06:26:07 UTC
[jira] Created: (HADOOP-6048) Need to handle access token
expiration when re-establishing the pipeline for dfs write
Need to handle access token expiration when re-establishing the pipeline for dfs write
--------------------------------------------------------------------------------------
Key: HADOOP-6048
URL: https://issues.apache.org/jira/browse/HADOOP-6048
Project: Hadoop Core
Issue Type: Bug
Components: dfs, security
Reporter: Kan Zhang
Assignee: Kan Zhang
Original access token may have expired when re-establishing the pipeline within processDatanodeError().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6048) Need to handle access token
expiration when re-establishing the pipeline for dfs write
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-6048:
------------------------------
Attachment: 6048-18.patch
> Need to handle access token expiration when re-establishing the pipeline for dfs write
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-6048
> URL: https://issues.apache.org/jira/browse/HADOOP-6048
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, security
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 6048-09.patch, 6048-11.patch, 6048-12.patch, 6048-14.patch, 6048-15.patch, 6048-16.patch, 6048-17.patch, 6048-18.patch
>
>
> Original access token may have expired when re-establishing the pipeline within processDatanodeError().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6048) Need to handle access token
expiration when re-establishing the pipeline for dfs write
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720494#action_12720494 ]
Kan Zhang commented on HADOOP-6048:
-----------------------------------
To be clear to the reader, the last patch removed null checking on the return value from recoverBlock() since it always returns a LocatedBlock object or throw an exception.
> Need to handle access token expiration when re-establishing the pipeline for dfs write
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-6048
> URL: https://issues.apache.org/jira/browse/HADOOP-6048
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, security
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 6048-09.patch, 6048-11.patch, 6048-12.patch, 6048-14.patch
>
>
> Original access token may have expired when re-establishing the pipeline within processDatanodeError().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6048) Need to handle access token
expiration when re-establishing the pipeline for dfs write
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-6048:
------------------------------
Status: Open (was: Patch Available)
> Need to handle access token expiration when re-establishing the pipeline for dfs write
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-6048
> URL: https://issues.apache.org/jira/browse/HADOOP-6048
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, security
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 6048-09.patch, 6048-11.patch, 6048-12.patch, 6048-14.patch, 6048-15.patch, 6048-16.patch
>
>
> Original access token may have expired when re-establishing the pipeline within processDatanodeError().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6048) Need to handle access token
expiration when re-establishing the pipeline for dfs write
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721606#action_12721606 ]
Kan Zhang commented on HADOOP-6048:
-----------------------------------
attached a new patch, which bumps up the ClientDatanodeProtocol version to signal the fact that now recoverBlock() never returns null and always returns a newly generated access token as part of the return value. Also, reverted the changes made in previous patches that changed the way how lastException is set by sync() and createBlockOutputStream().
> Need to handle access token expiration when re-establishing the pipeline for dfs write
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-6048
> URL: https://issues.apache.org/jira/browse/HADOOP-6048
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, security
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 6048-09.patch, 6048-11.patch, 6048-12.patch, 6048-14.patch, 6048-15.patch, 6048-16.patch, 6048-17.patch, 6048-18.patch
>
>
> Original access token may have expired when re-establishing the pipeline within processDatanodeError().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6048) Need to handle access token
expiration when re-establishing the pipeline for dfs write
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-6048:
------------------------------
Status: Patch Available (was: Open)
> Need to handle access token expiration when re-establishing the pipeline for dfs write
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-6048
> URL: https://issues.apache.org/jira/browse/HADOOP-6048
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, security
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 6048-09.patch, 6048-11.patch, 6048-12.patch, 6048-14.patch, 6048-15.patch, 6048-16.patch
>
>
> Original access token may have expired when re-establishing the pipeline within processDatanodeError().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6048) Need to handle access token
expiration when re-establishing the pipeline for dfs write
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-6048:
------------------------------
Attachment: 6048-16.patch
attached a new patch. removed the testing for InvalidAccessTokenException in processDatanodeError() since it can never happen. processDatanodeError() is called in 2 places. 1) when an error happens while writing. since the pipeline has already been set up in this case, the error can't be access token error. 2) when initializing for append, in which case, there is no previous error yet.
> Need to handle access token expiration when re-establishing the pipeline for dfs write
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-6048
> URL: https://issues.apache.org/jira/browse/HADOOP-6048
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, security
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 6048-09.patch, 6048-11.patch, 6048-12.patch, 6048-14.patch, 6048-15.patch, 6048-16.patch
>
>
> Original access token may have expired when re-establishing the pipeline within processDatanodeError().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6048) Need to handle access token
expiration when re-establishing the pipeline for dfs write
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721724#action_12721724 ]
Hadoop QA commented on HADOOP-6048:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12411172/6048-18.patch
against trunk revision 786278.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/533/console
This message is automatically generated.
> Need to handle access token expiration when re-establishing the pipeline for dfs write
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-6048
> URL: https://issues.apache.org/jira/browse/HADOOP-6048
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, security
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 6048-09.patch, 6048-11.patch, 6048-12.patch, 6048-14.patch, 6048-15.patch, 6048-16.patch, 6048-17.patch, 6048-18.patch
>
>
> Original access token may have expired when re-establishing the pipeline within processDatanodeError().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6048) Need to handle access token
expiration when re-establishing the pipeline for dfs write
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-6048:
------------------------------
Attachment: 6048-15.patch
one cosmetic change.
> Need to handle access token expiration when re-establishing the pipeline for dfs write
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-6048
> URL: https://issues.apache.org/jira/browse/HADOOP-6048
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, security
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 6048-09.patch, 6048-11.patch, 6048-12.patch, 6048-14.patch, 6048-15.patch
>
>
> Original access token may have expired when re-establishing the pipeline within processDatanodeError().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6048) Need to handle access token
expiration when re-establishing the pipeline for dfs write
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-6048:
------------------------------
Attachment: 6048-17.patch
> Need to handle access token expiration when re-establishing the pipeline for dfs write
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-6048
> URL: https://issues.apache.org/jira/browse/HADOOP-6048
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, security
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 6048-09.patch, 6048-11.patch, 6048-12.patch, 6048-14.patch, 6048-15.patch, 6048-16.patch, 6048-17.patch
>
>
> Original access token may have expired when re-establishing the pipeline within processDatanodeError().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6048) Need to handle access token
expiration when re-establishing the pipeline for dfs write
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-6048:
------------------------------
Attachment: 6048-12.patch
attaching a new patch, which relies on block recovery to always return a newly generated access token, so that there is no need for adding a new method to ClientProtocol to get access token.
> Need to handle access token expiration when re-establishing the pipeline for dfs write
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-6048
> URL: https://issues.apache.org/jira/browse/HADOOP-6048
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, security
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 6048-09.patch, 6048-11.patch, 6048-12.patch
>
>
> Original access token may have expired when re-establishing the pipeline within processDatanodeError().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6048) Need to handle access token
expiration when re-establishing the pipeline for dfs write
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-6048:
------------------------------
Status: Patch Available (was: Open)
> Need to handle access token expiration when re-establishing the pipeline for dfs write
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-6048
> URL: https://issues.apache.org/jira/browse/HADOOP-6048
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, security
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 6048-09.patch
>
>
> Original access token may have expired when re-establishing the pipeline within processDatanodeError().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6048) Need to handle access token
expiration when re-establishing the pipeline for dfs write
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-6048:
------------------------------
Attachment: 6048-14.patch
minor updates
> Need to handle access token expiration when re-establishing the pipeline for dfs write
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-6048
> URL: https://issues.apache.org/jira/browse/HADOOP-6048
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, security
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 6048-09.patch, 6048-11.patch, 6048-12.patch, 6048-14.patch
>
>
> Original access token may have expired when re-establishing the pipeline within processDatanodeError().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6048) Need to handle access token
expiration when re-establishing the pipeline for dfs write
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-6048:
------------------------------
Attachment: 6048-11.patch
updated ClientProtocol version.
> Need to handle access token expiration when re-establishing the pipeline for dfs write
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-6048
> URL: https://issues.apache.org/jira/browse/HADOOP-6048
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, security
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 6048-09.patch, 6048-11.patch
>
>
> Original access token may have expired when re-establishing the pipeline within processDatanodeError().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6048) Need to handle access token
expiration when re-establishing the pipeline for dfs write
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-6048:
------------------------------
Attachment: 6048-09.patch
this patch does the following.
1. when the primary datanode recovers a block, it sends back a newly generated access token along a new generation stamp if any.
2. add a namenode method so that a DFSClient can call to get a new access token when access token becomes expired during datanode error recovery.
3. To differentiate access token error from other errors, datanode now needs to send back a status code together with firstBadLink during pipeline setup for dfs write (only for DFSClients, not for other datanodes).
> Need to handle access token expiration when re-establishing the pipeline for dfs write
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-6048
> URL: https://issues.apache.org/jira/browse/HADOOP-6048
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs, security
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: 6048-09.patch
>
>
> Original access token may have expired when re-establishing the pipeline within processDatanodeError().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.