You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Xiao Kang (JIRA)" <ji...@apache.org> on 2010/05/17 12:18:43 UTC
[jira] Created: (HADOOP-6768) RPC client can response more
efficiently when sendParam() got IOException
RPC client can response more efficiently when sendParam() got IOException
-------------------------------------------------------------------------
Key: HADOOP-6768
URL: https://issues.apache.org/jira/browse/HADOOP-6768
Project: Hadoop Common
Issue Type: Improvement
Components: ipc
Affects Versions: 0.22.0
Reporter: Xiao Kang
Fix For: 0.22.0
Under current RPC client implementation, when Client.Connection.sendParam() encounters IOException it just marks the exception and wait receiveResponse() thread to handle the exception. However, receiveResponse() may block ipc.ping.interval on socket read().
This means that RPC client may wait ipc.ping.interval(typically 1 miniute) when exception in sendParam().
It maybe not so reasonable in some situation. For example using "hadoop fs -put " a small file, DFSClient.close() will interupt renewLease() thread and it may wait 1 minute.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6768) RPC client can response more
efficiently when sendParam() got IOException
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875423#action_12875423 ]
Todd Lipcon commented on HADOOP-6768:
-------------------------------------
Hi Xiao,
For your test case, I think it would be much simpler to just use a Writable which throws IOE, rather than mocking out all of Socket, etc. If you look at TestIPC in trunk, there was a case added by HADOOP-6723 that fixed a similar bug for exceptions in readFields, but we didn't test writeFields. It should be easy to extend that test case and share the code there.
Also please take a look at HADOOP-6762. which solves the specific problem of InterruptedException but not the more general problem of IOE or RTE. Would appreciate your thoughts on that issue.
> RPC client can response more efficiently when sendParam() got IOException
> -------------------------------------------------------------------------
>
> Key: HADOOP-6768
> URL: https://issues.apache.org/jira/browse/HADOOP-6768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
> Affects Versions: 0.22.0
> Reporter: Xiao Kang
> Fix For: 0.22.0
>
> Attachments: HADOOP-6768.patch
>
>
> Under current RPC client implementation, when Client.Connection.sendParam() encounters IOException it just marks the exception and wait receiveResponse() thread to handle the exception. However, receiveResponse() may block ipc.ping.interval on socket read().
> This means that RPC client may wait ipc.ping.interval(typically 1 miniute) when exception in sendParam().
> It maybe not so reasonable in some situation. For example using "hadoop fs -put " a small file, DFSClient.close() will interupt renewLease() thread and it may wait 1 minute.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6768) RPC client can response more
efficiently when sendParam() got IOException
Posted by "Xiao Kang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Kang updated HADOOP-6768:
------------------------------
Attachment: HADOOP-6768.patch
patch attached.
A. change org.apache.hadoop.ipc.Client
1. add call.setException(e) when sendParam() got IOException to set call.done to avoid current call wait receiveResponse() thread.
2. interrupt receiveResponse() thread to cleanup other calls on this connection
B. test case for it
> RPC client can response more efficiently when sendParam() got IOException
> -------------------------------------------------------------------------
>
> Key: HADOOP-6768
> URL: https://issues.apache.org/jira/browse/HADOOP-6768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
> Affects Versions: 0.22.0
> Reporter: Xiao Kang
> Fix For: 0.22.0
>
> Attachments: HADOOP-6768.patch
>
>
> Under current RPC client implementation, when Client.Connection.sendParam() encounters IOException it just marks the exception and wait receiveResponse() thread to handle the exception. However, receiveResponse() may block ipc.ping.interval on socket read().
> This means that RPC client may wait ipc.ping.interval(typically 1 miniute) when exception in sendParam().
> It maybe not so reasonable in some situation. For example using "hadoop fs -put " a small file, DFSClient.close() will interupt renewLease() thread and it may wait 1 minute.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6768) RPC client can response more
efficiently when sendParam() got IOException
Posted by "Xiao Kang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874572#action_12874572 ]
Xiao Kang commented on HADOOP-6768:
-----------------------------------
Another case found for this issuse.
In mapreduce task execution, each task will report progress to TaskTracker through Ubilical RPC. When the task completes, it will interrupt and join the reporter thread, this may cause Exception for sendParam() of report progress RPC and wait ipc.ping.interval(typically 1 miniute).
> RPC client can response more efficiently when sendParam() got IOException
> -------------------------------------------------------------------------
>
> Key: HADOOP-6768
> URL: https://issues.apache.org/jira/browse/HADOOP-6768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
> Affects Versions: 0.22.0
> Reporter: Xiao Kang
> Fix For: 0.22.0
>
> Attachments: HADOOP-6768.patch
>
>
> Under current RPC client implementation, when Client.Connection.sendParam() encounters IOException it just marks the exception and wait receiveResponse() thread to handle the exception. However, receiveResponse() may block ipc.ping.interval on socket read().
> This means that RPC client may wait ipc.ping.interval(typically 1 miniute) when exception in sendParam().
> It maybe not so reasonable in some situation. For example using "hadoop fs -put " a small file, DFSClient.close() will interupt renewLease() thread and it may wait 1 minute.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6768) RPC client can response more
efficiently when sendParam() got IOException
Posted by "Lin Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868499#action_12868499 ]
Lin Yang commented on HADOOP-6768:
----------------------------------
又看见师兄提bug了...
--
杨林
http://blog.csdn.net/idning
> RPC client can response more efficiently when sendParam() got IOException
> -------------------------------------------------------------------------
>
> Key: HADOOP-6768
> URL: https://issues.apache.org/jira/browse/HADOOP-6768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
> Affects Versions: 0.22.0
> Reporter: Xiao Kang
> Fix For: 0.22.0
>
>
> Under current RPC client implementation, when Client.Connection.sendParam() encounters IOException it just marks the exception and wait receiveResponse() thread to handle the exception. However, receiveResponse() may block ipc.ping.interval on socket read().
> This means that RPC client may wait ipc.ping.interval(typically 1 miniute) when exception in sendParam().
> It maybe not so reasonable in some situation. For example using "hadoop fs -put " a small file, DFSClient.close() will interupt renewLease() thread and it may wait 1 minute.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.