You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2010/04/19 07:54:49 UTC

[jira] Created: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

The RPC server Listener thread is a scalability bottleneck
----------------------------------------------------------

                 Key: HADOOP-6713
                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
             Project: Hadoop Common
          Issue Type: Improvement
          Components: ipc
    Affects Versions: 0.20.2, 0.20.1, 0.20.0
            Reporter: dhruba borthakur
            Assignee: Dmytro Molkov


The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861108#action_12861108 ] 

Hairong Kuang commented on HADOOP-6713:
---------------------------------------

This is a great idea! Separating "accept" from "read" should also greatly reduce the Connection reset errors observed at the client when NameNode is busy. Dhruba asked me to review this patch. So here are a few comments:

1. Please remove the System.out.println or change it to be a log statement;
2. Listener#run() should remove doRead() else branch;
3. Now that accept is done is a separate thread, doAccept() should accept as many as possible (not limit to up to 10 as in the trunk). Another option is to use a blocking accept channel.
4. Optional: the synchronization between listener thread & read thread is very interesting. It took me a while to figure out that it works. But it seems to me that the code is hard to understand and maintain. Another option is that each reader thread maintains a queue of pending registration channels. After choosing a reader, a listener thread simply adds an accepted channel into its pending queue and then wakes up the reader thread. Each reader thread  registers all the pending channels before select().

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Bharath Mundlapudi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-6713:
---------------------------------------

    Attachment:     (was: HADOOP-6713-rel20.patch.1)

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6713-rel20.2.patch, HADOOP-6713-rel20.patch, HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dmytro Molkov updated HADOOP-6713:
----------------------------------

    Status: Patch Available  (was: Open)

Resubmitting for Hudson

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Bharath Mundlapudi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-6713:
---------------------------------------

    Attachment: HADOOP-6713-rel20.2.patch

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6713-rel20.2.patch, HADOOP-6713-rel20.patch, HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dmytro Molkov updated HADOOP-6713:
----------------------------------

    Status: Open  (was: Patch Available)

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861466#action_12861466 ] 

Hairong Kuang commented on HADOOP-6713:
---------------------------------------

+1

Note that this patch assigns read channels to readers in a round-robin fashion. Ideally it would be nicer if all readers share a set of selected keys and ready-to-read channels are dynamically assigned to each reader. This would enforce better balance among readers. But I think we could have this improvement at a later time.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859599#action_12859599 ] 

Hadoop QA commented on HADOOP-6713:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442483/HADOOP-6713.patch
  against trunk revision 936463.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/49/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/49/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/49/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/49/console

This message is automatically generated.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859935#action_12859935 ] 

Dmytro Molkov commented on HADOOP-6713:
---------------------------------------

We were running performance test on our test cluster.
The test itself is creating a tree of directories with files on the leafs in a depths first search fashion: there is a root and we create N directories in the root directory for the test, each mapper then starts in one of those directories and creates its own subtree with files on the leafs.

Then there is a read job that for each mapper does ls on the directory, chooses random element in ls, if it is a directory then repeat if it is a file then do read on the file. The files are 4K in size so the read time is small and we are mostly hitting the namenode with this job.

We were running the branch that had this fix and it also had read write locks for namenode instead of synchronized sections.

The version without fixes could only get namenode to use 175% cpu. With fixes in place we were using 750% cpu for read only load (when the second job was running on its own and 550% for read-write load when two jobs were running in parallel. 

In the read-write mode the ration of reads to writes was 8:1 (800 read clients vs 100 write clients).

We are not putting the read-write locks in production in this iteration, seems we feel like we need to do more testing on it. As soon as I have some results for the branch with this fix only I will post my findings here.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861130#action_12861130 ] 

Hadoop QA commented on HADOOP-6713:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442902/HADOOP-6713.2.patch
  against trunk revision 938136.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/52/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/52/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/52/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/52/console

This message is automatically generated.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Bharath Mundlapudi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-6713:
---------------------------------------

    Attachment: HADOOP-6713-rel20.patch.1

Attached the patch addressing the comments.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6713-rel20.patch, HADOOP-6713-rel20.patch.1, HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885605#action_12885605 ] 

Suresh Srinivas commented on HADOOP-6713:
-----------------------------------------

+1 for the patch.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6713-rel20.2.patch, HADOOP-6713-rel20.3.patch, HADOOP-6713-rel20.patch, HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dmytro Molkov updated HADOOP-6713:
----------------------------------

    Attachment: HADOOP-6713.patch

Please have a look. We did the same change on hadoop-0.20 and ran it for a day under heavy write and read load on the test cluster.
The unittest included just reruns the TestRCP with multiple number of reader threads

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dmytro Molkov updated HADOOP-6713:
----------------------------------

               Status: Patch Available  (was: Open)
    Affects Version/s: 0.21.0
                           (was: 0.20.0)
                           (was: 0.20.1)
                           (was: 0.20.2)

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859905#action_12859905 ] 

Koji Noguchi commented on HADOOP-6713:
--------------------------------------

Nice. Any performance numbers ?

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858390#action_12858390 ] 

dhruba borthakur commented on HADOOP-6713:
------------------------------------------

My proposal is that the Server.Listener.run() thread only executes doAccept(). Also there are a bunch of open Selector objects (instead of just one Selector object in the current code).  Each of these Selector objects are handled by its own thread.

The Server.Listener.run() invoke doAccept() that inserts a SocketChannel into a randomly selected Selector object. The thread that is monitoring this Selector object will then invoke doRead() and process the RPC.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Bharath Mundlapudi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-6713:
---------------------------------------

    Attachment: HADOOP-6713-rel20.patch

Attached the patch for Y20S branch.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6713-rel20.patch, HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859848#action_12859848 ] 

dhruba borthakur commented on HADOOP-6713:
------------------------------------------

+1 code looks good.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858392#action_12858392 ] 

dhruba borthakur commented on HADOOP-6713:
------------------------------------------

Hi Christian, I agree with you. We have around 40K clients hitting the namenode and the NN  ( inspite of running NN on a newest and greatest Nehalem) has one CPU completely maxed out by the Server.Listener thread.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6713:
----------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.22.0
       Resolution: Fixed

I've just committed this. Thanks, Dmytro!

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dmytro Molkov updated HADOOP-6713:
----------------------------------

    Attachment: HADOOP-6713.2.patch

modifying the patch to address Hairong's comments:

1. Removed the System.out
2. Removed doRead from listener#run
3. Changed the code to accept all the connections in the queue

4. I wrote a comment on startAdd method to explain the synchronization a little. The reason I went with this decision is to keep the semantics of doAccept. After the iteration with certain channel that channel is registered with the selector and is ready to be read. Otherwise it possibly could affect other parts of the code, when the connection is accepted (doAccept iteration for the channel is over) but connectionList is not modified yet. And the connection is only created when the channel is registered with selector.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Bharath Mundlapudi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-6713:
---------------------------------------

    Attachment: HADOOP-6713-rel20.3.patch

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6713-rel20.2.patch, HADOOP-6713-rel20.3.patch, HADOOP-6713-rel20.patch, HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858391#action_12858391 ] 

Christian Kunz commented on HADOOP-6713:
----------------------------------------

That would be a great improvement for our clusters as well. Probably for any cluster with a lot of clients not necessarily being map-reduce applications.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck

Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884817#action_12884817 ] 

Suresh Srinivas commented on HADOOP-6713:
-----------------------------------------

Comments for Y20 version of the patch.
# Please remove YAHOO-CHANGES.txt from the patch
# Remove blank changes that is diff between two patches: lines 51, 73, 138, 171, 203, 220. There could be others.
# line 283 exceeds 80 chars


> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6713-rel20.patch, HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.