You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Doug Cutting (JIRA)" <ji...@apache.org> on 2007/09/05 20:37:34 UTC
[jira] Created: (HADOOP-1841) IPC server should write repsonses
asynchronously
IPC server should write repsonses asynchronously
------------------------------------------------
Key: HADOOP-1841
URL: https://issues.apache.org/jira/browse/HADOOP-1841
Project: Hadoop
Issue Type: Improvement
Components: ipc
Reporter: Doug Cutting
Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548022 ]
dhruba borthakur commented on HADOOP-1841:
------------------------------------------
Mukubd: Thanks for running the performance numbers.
Doug: This patch (by itself) will not improve performance. It is mostly a bug fix. Koji's earlier comment describes how slow clients caused a denial-of-service of the namenode on a real cluster. Mukund's performance runs show that performance is not impacted. However, I plan on attempting a follow-on patch to NameNode that will move the sync-to-edits-log from occuring in the RPC Handler thread, thus possibly improving scalability. Do you think that this patch is good to go?
Owen: would you like to review the code again?
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Attachment: transactionLogSync7.patch
The close() call cannot lose data because a flush and sync should have been called earlier. However, I added an exception to close() to detect this case if it ever happens. I also removed extra imports from NameNodeMetrics.java.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: Enis Soztutar
> Attachments: transactionLogSync7.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546858 ]
Mukund Madhugiri commented on HADOOP-1841:
------------------------------------------
I ran nnbench benchmark on 500 nodes and here is the data:
trunk:
- createWrite: 4982
- openRead: 44523
- rename: 4371
- delete: 3757
trunk + patch:
- createWrite: 5005
- openRead: 46584
- rename: 4400
- delete: 3946
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC-2.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Attachment: asyncRPC-9.patch
Inserted a LOG messages as suggested by Doug.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC-8.patch, asyncRPC-9.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Attachment: asyncRPC-4.patch
This patch is in response to Doug's comments that we need to show improvement in performance in the face of slow clients.
This patch has a unit test that creates a server with one handler thread. One thread makes an RPC and stops processing the response. Another thread then issues another RPC and it completes successfully even though the first thread has not yet consumed the RPC response. This test passes successfully with this patch whereas it fails with trunk.
Please let me know is it addresses your concerns. If so, then the only remaining thing to make this patch committable is to demonstrate that it does not degrade performance for sort runs.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Attachment: (was: transactionLogSync7.patch)
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: Enis Soztutar
> Attachments: asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548019 ]
Mukund Madhugiri commented on HADOOP-1841:
------------------------------------------
I ran sort benchmark on 500 nodes and here is the data:
trunk:
* sort: 1.776 hr
* random writer: 0.459 hr
* sort validation: 0.341 hr
trunk + patch:
* sort: 1.776 hr
* random writer: 0.423 hr
* sort validation: 0.450 hr
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Enis Soztutar updated HADOOP-1841:
----------------------------------
Attachment: asyncRPC.patch
I am attaching a slightly modified version, in which Call#response is changed from byte[] to ByteBuffer.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: Enis Soztutar
> Attachments: asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Status: Patch Available (was: Open)
Fixed findbugs warnings. I would request Owen to review this patch.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Attachment: asyncRPC-6.patch
Fixed findbugs warnings.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548689 ]
Hadoop QA commented on HADOOP-1841:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12371003/asyncRPC-5.patch
against trunk revision r601232.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs -1. The patch appears to introduce 2 new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1272/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1272/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1272/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1272/console
This message is automatically generated.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12549308 ]
dhruba borthakur commented on HADOOP-1841:
------------------------------------------
Can either Doug or Owen please please review this patch? Thanks.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12543968 ]
Doug Cutting commented on HADOOP-1841:
--------------------------------------
Does it improve performance or reliability?
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC-2.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Status: Open (was: Patch Available)
Findbugs warnings.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528413 ]
Raghu Angadi commented on HADOOP-1841:
--------------------------------------
I agree w.r.t performance. But main purpose of this is not to increase the performance. In fact, the normal case is more data is written in reply than read (except for blocReport() RPCs), in that sense, doing async write in single thread could actually be the bottleneck (but rarely is, we have never seen reading in single thread being a bottleneck). Of course we could write with multiple threads, or all the IPC handlers could be involved in write.
The main issue this proposes to fix is this : The actual RPC call itself is internal to Server, but writing depends on how fast client is receiving as well. We don't want excternal factors like slow or down clients to influence how fast server handles RPCs (from other clients). We should certainly test with artificially slow clients and should be able to show the difference.
A few more things :
- we should keep track of amount of pending reply buffer to a client and not read from it if there is more than a few KB.
- also we should have limit on over all memory in pending write buffers.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: Enis Soztutar
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Attachment: asyncRPC.patch
Hi Enis, I took the liberty of writing some code to do asynchronous RPC responses from the server. This patch has the following:
1. There is *one* additional Responder thread, whose job is to write the RPC responses back to the RPC clients. It does non-blocking IO on the socket. Because it does non-blocking IO, only one Responder thread is good enough.
2. The RPC handler threads queues the responses of the RPCs into buffers associated with the response-channel. These responses are flushed to the client by the Responder thread described above.
3. A test case that creates 50 threads and issues concurrent RPCs. Each of these RPCs transfers 400K of data from client to server and back from server to client.
This is very preliminary code but I would like to enhance the test to measure whether this change is making any difference to performance.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: Enis Soztutar
> Attachments: asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12543990 ]
Doug Cutting commented on HADOOP-1841:
--------------------------------------
I think we need to both show that it doesn't slow things in the normal case, and that it improves performance in this particular case (clients slow to consume output). Do you have a realistic benchmark for that case?
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC-2.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Attachment: asyncRPC-2.patch
Merged patch with latest trunk. This patch has been lying idle fro a while. I would really like somebody to review this uploaded patch and comment on the JIRA.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC-2.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC-8.patch, asyncRPC-9.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Attachment: asyncRPC-8.patch
All debug log messages are inside If LOG.debguEnabled()
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC-8.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12547337 ]
Koji Noguchi commented on HADOOP-1841:
--------------------------------------
Just wanted to note that we had couple of occasions when one user
1) Started many dfsclients on a single node
2) Load on that client node became too high that its rpc read speed slowed down significantly
3) Namenode became unresponsive
Rebooting that client node brought back the namenode.
I'm hoping this patch would solve such cases.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC-2.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12549542 ]
Doug Cutting commented on HADOOP-1841:
--------------------------------------
- re-organizing imports injects noise into the patch
- any reason we can't make the listener thread also the responder, using a single selector?
- catching OutOfMemoryError seems suspect, but i guess we already do it...
- it appears to me there are ways processResponse could exit with an exception without error being set to true. Might it be safer to assume error=true, close the connection in a 'finally' clause, and only set error=false for normal returns?
- 'status' would better be named 'more'
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551068 ]
Doug Cutting commented on HADOOP-1841:
--------------------------------------
Looks good. Should we log the exception caught after the call to doAsyncWrite()?
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC-8.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551160 ]
Hadoop QA commented on HADOOP-1841:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12371560/asyncRPC-9.patch
against trunk revision r603652.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1331/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1331/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1331/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1331/console
This message is automatically generated.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC-8.patch, asyncRPC-9.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Attachment: asyncRPC-7.patch
Incorporated Doug's review comments.
1. Cleaned up imports to reduce patch noise
2. I kept the Listener and the Responder as separate threads. These two functions are inherently parallel. The Responder needs to take locks on the response queue and putting this functionality in the Listener might interrupt the consumption rate of incoming RPCs.
3. Fixed processResponse to be inside a try-finally clause.
4. Renamed variable "status" as "more".
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551140 ]
Doug Cutting commented on HADOOP-1841:
--------------------------------------
+1 This looks good to me.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC-8.patch, asyncRPC-9.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550224 ]
Raghu Angadi commented on HADOOP-1841:
--------------------------------------
Dhruba,
Could you move couple of debug statements into '{{if (LOG.debugEnabled())}' It looked like they are executed for every RPC.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12544003 ]
dhruba borthakur commented on HADOOP-1841:
------------------------------------------
I have a test case that triggers this new portion of code. I can extend it to gather test-timings.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC-2.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533764 ]
Raghu Angadi commented on HADOOP-1841:
--------------------------------------
> This is very preliminary code but I would like to enhance the test to measure whether this change is making any difference to performance.
Why do you think there will be any improvement in performance? This is really not performance improvement in normal case.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Fix Version/s: 0.16.0
Status: Patch Available (was: Open)
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Status: Open (was: Patch Available)
Canceling patch to incorporate Raghu's comments.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550268 ]
Hadoop QA commented on HADOOP-1841:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12371392/asyncRPC-8.patch
against trunk revision r603084.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1311/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1311/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1311/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1311/console
This message is automatically generated.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC-8.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528323 ]
Enis Soztutar commented on HADOOP-1841:
---------------------------------------
I started to work on this issue, but before any code, i would like to discuss a few things.
Current ipc/rpc handling is done as a 1 dispatcher N worker scenario. The Listener thread asynchronously listens to the socket for incoming connections, accepts them(creating Connection instances), or reads the data and puts in the queue for the handler threads for later processing.
The handler threads on the other hand, listens to the call queue, removes the first element and process the Call. It first makes the actual call to the method(through {{Server#call}} ), gets the return value and writes back the results to output stream(blocking).
As Doug suggested the performance might be improved if the results would have been written asynchronously. But i am not sure that it will affect the performance, if any, a lot. The costly operation in handler threads is expected to be the method calls. How do others think about this? We should start to think about developing benchmark tests. It would be really cool if we could improved the performance of ipc calls say %10. I am thinking about using jmeter for this, however maybe we need to develop our own benchmarking framework to test all the components of Hadoop. Any thoughts ?
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: Enis Soztutar
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Status: Patch Available (was: Open)
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC-8.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12543973 ]
dhruba borthakur commented on HADOOP-1841:
------------------------------------------
It improves performance in the face of clients that are very slow to consume output. I am yet to run performance benchmarks (e.g. sort) to prove that it does not decrease performance in the normal case.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC-2.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Enis Soztutar reassigned HADOOP-1841:
-------------------------------------
Assignee: dhruba borthakur (was: Enis Soztutar)
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548004 ]
Doug Cutting commented on HADOOP-1841:
--------------------------------------
> Please let me know is it addresses your concerns.
The unit test sounds good. Do we have any particular field uses where we think this will improve performance?
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548799 ]
Hadoop QA commented on HADOOP-1841:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12371062/asyncRPC-6.patch
against trunk revision r601383.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1273/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1273/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1273/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1273/console
This message is automatically generated.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Status: Patch Available (was: Open)
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Status: Patch Available (was: Open)
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC-8.patch, asyncRPC-9.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550777 ]
dhruba borthakur commented on HADOOP-1841:
------------------------------------------
I have incorporated most of Doug's comments. Doug: if you would care to take another look at this one, please let me know. Thanks.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC-8.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12547335 ]
dhruba borthakur commented on HADOOP-1841:
------------------------------------------
As expected, this patch does not decrease the performance of the namenode. But I agree with owen that we should determine that it does not affect sort performance too.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC-2.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12547222 ]
Owen O'Malley commented on HADOOP-1841:
---------------------------------------
I think on RPC patches, it is a very good idea to run a sort on 500 nodes to make sure that the system remains stable.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC-2.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Attachment: asyncRPC-5.patch
Merged with latest trunk.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533804 ]
dhruba borthakur commented on HADOOP-1841:
------------------------------------------
Please let me explain why i think this is a necessary pre-requisite for increasing namenode transaction performance.
The namenode has 40 threads serving client requests. A majority of these threads are waiting for a transaction to complete, i.e. a disk sync. The disk-sync times are pretty high on loaded systems. If we can avoid doing the edit-log sync from the IPC handler thread, instead do it from a separate Sycner thread, we can free up the IPC handler thread to process new client requests. To do this, we need the ability to be able to respond to a RPC from a thread that is separate from the IPC handler thread.
Enis: Thanks for looking over the patch I submitted and making enhancements to it.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Attachments: asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Status: Open (was: Patch Available)
Cancelling patch to incorporate Doug's comments,
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Enis Soztutar reassigned HADOOP-1841:
-------------------------------------
Assignee: Enis Soztutar
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: Enis Soztutar
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12532597 ]
Enis Soztutar commented on HADOOP-1841:
---------------------------------------
Dhruba, i believe this patch belongs to HADOOP-1942 *smile*.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: Enis Soztutar
> Attachments: transactionLogSync7.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1841:
-------------------------------------
Status: Open (was: Patch Available)
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC-8.patch, asyncRPC-9.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550230 ]
Hadoop QA commented on HADOOP-1841:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12371382/asyncRPC-7.patch
against trunk revision r603000.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1309/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1309/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1309/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1309/console
This message is automatically generated.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551480 ]
Hudson commented on HADOOP-1841:
--------------------------------
Integrated in Hadoop-Nightly #331 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/331/])
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: asyncRPC-2.patch, asyncRPC-4.patch, asyncRPC-5.patch, asyncRPC-6.patch, asyncRPC-7.patch, asyncRPC-8.patch, asyncRPC-9.patch, asyncRPC.patch, asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1841) IPC server should write repsonses
asynchronously
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533755 ]
Enis Soztutar commented on HADOOP-1841:
---------------------------------------
Hi Dhruba,
I had written a similar architecture having a pseudo-code something like below, but could not made it pass the tests.
{code}
OutputHandler#run() {
while(running) {
selector.select();
//write data
//register pending calls to selector
}
}
Handler#run() {
//handle the call
//prepare ByteBuffer
//add call to the response queue
selector.wakeup()
}
{code}
now looking at your patch, the reason mine did not work is that i've used SelectionKeys and ResponseQueues per Call, rather that Connections.
Anyway i'm +1 for this after we justify its necessity.
> IPC server should write repsonses asynchronously
> ------------------------------------------------
>
> Key: HADOOP-1841
> URL: https://issues.apache.org/jira/browse/HADOOP-1841
> Project: Hadoop
> Issue Type: Improvement
> Components: ipc
> Reporter: Doug Cutting
> Assignee: Enis Soztutar
> Attachments: asyncRPC.patch
>
>
> Hadoop's IPC Server currently writes responses from request handler threads using blocking writes. Performance and scalability might be improved if responses were written asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.