You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by "Alexey Polbitsyn (JIRA)" <ji...@apache.org> on 2011/07/02 18:44:21 UTC
[jira] [Created] (SSHD-136) Reduce consumption of
ClientInputStreamPump threads and NioProcessor threads.
Reduce consumption of ClientInputStreamPump threads and NioProcessor threads.
------------------------------------------------------------------------------
Key: SSHD-136
URL: https://issues.apache.org/jira/browse/SSHD-136
Project: MINA SSHD
Issue Type: Improvement
Affects Versions: 0.5.0
Reporter: Alexey Polbitsyn
When using the library to work with 1000 concurrent ssh connections, it takes 1000 ClientInputStreamPump threads and it can. 1 thread for 1 channel. It would be nice to be able to configure this behavior, for example use 1 ClientInputStreamPump for all channels of sshclient or session.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SSHD-136) Reduce consumption of
ClientInputStreamPump threads and NioProcessor threads.
Posted by "Guillaume Nodet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SSHD-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462685#comment-13462685 ]
Guillaume Nodet commented on SSHD-136:
--------------------------------------
I wonder if we could somehow leverage nio channels with the Pipe class and a selector instead of spawning threads.
We could maybe keep both APIs or a simple wrapper to ease the work for the user using blocking io.
> Reduce consumption of ClientInputStreamPump threads and NioProcessor threads.
> ------------------------------------------------------------------------------
>
> Key: SSHD-136
> URL: https://issues.apache.org/jira/browse/SSHD-136
> Project: MINA SSHD
> Issue Type: Improvement
> Affects Versions: 0.5.0
> Reporter: Alexey Polbitsyn
> Attachments: src-diff.zip, src.zip, SSHD-136.patch
>
>
> When using the library to work with 1000 concurrent ssh connections, it takes 1000 ClientInputStreamPump threads. 1 thread for 1 channel. It would be nice to be able to configure this behavior, for example use 1 ClientInputStreamPump for all channels of sshclient or session.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (SSHD-136) Reduce consumption of
ClientInputStreamPump threads and NioProcessor threads.
Posted by "Alexey Polbitsyn (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SSHD-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059086#comment-13059086 ]
Alexey Polbitsyn edited comment on SSHD-136 at 7/2/11 4:50 PM:
---------------------------------------------------------------
possible solution:
src-diff - diff files
src - sources
the ability to customize count of NioProcessor threads before start SshClient:
SshClient.setNioProcessorCount
the ability to customize count of ClientInputStreamPump threads.
1) 1 ClientInputStreamPump on channel
2) 1 client ClientInputStreamPump on session
3) 1 client ClientInputStreamPump on SshClient
4) turn off automatically pumping
public enum PumpingMethod
{
SELF,/* class will pump streams/stream */
PARENT,/* delegating pumping to parent code(session for channel,client for session,external code for client)
OFF /* torn off auto pumping */
}
add public API to channel, session and client:
public boolean pump(); // try to pump input stream for channel, pump all channels with PumpingMethod=PARENT for session, pump all sessions with PumpingMethod=PARENT for client. Return true if at least one stream was pumped.
public void setPumpingMethod(PumpingMethod pumpingMethod) - set pumping method
public PumpingMethod getPumpingMethod()
add implementation for SshClient pumping:
use one thread for pumping all sessions with PumpingMethod=PARENT and accordingly all channels with PumpingMethod=PARENT
was (Author: polbitsyn):
src-diff - diff files
src - sources
> Reduce consumption of ClientInputStreamPump threads and NioProcessor threads.
> ------------------------------------------------------------------------------
>
> Key: SSHD-136
> URL: https://issues.apache.org/jira/browse/SSHD-136
> Project: MINA SSHD
> Issue Type: Improvement
> Affects Versions: 0.5.0
> Reporter: Alexey Polbitsyn
> Attachments: src-diff.zip, src.zip
>
>
> When using the library to work with 1000 concurrent ssh connections, it takes 1000 ClientInputStreamPump threads and it can. 1 thread for 1 channel. It would be nice to be able to configure this behavior, for example use 1 ClientInputStreamPump for all channels of sshclient or session.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SSHD-136) Reduce consumption of
ClientInputStreamPump threads and NioProcessor threads.
Posted by "Guillaume Nodet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SSHD-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094516#comment-13094516 ]
Guillaume Nodet commented on SSHD-136:
--------------------------------------
I wonder if there is any need to have the pump method configurable. Why not simply moving everything to the session instead ?
> Reduce consumption of ClientInputStreamPump threads and NioProcessor threads.
> ------------------------------------------------------------------------------
>
> Key: SSHD-136
> URL: https://issues.apache.org/jira/browse/SSHD-136
> Project: MINA SSHD
> Issue Type: Improvement
> Affects Versions: 0.5.0
> Reporter: Alexey Polbitsyn
> Attachments: SSHD-136.patch, src-diff.zip, src.zip
>
>
> When using the library to work with 1000 concurrent ssh connections, it takes 1000 ClientInputStreamPump threads. 1 thread for 1 channel. It would be nice to be able to configure this behavior, for example use 1 ClientInputStreamPump for all channels of sshclient or session.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SSHD-136) Reduce consumption of
ClientInputStreamPump threads and NioProcessor threads.
Posted by "Guillaume Nodet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SSHD-136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Guillaume Nodet updated SSHD-136:
---------------------------------
Attachment: SSHD-136.patch
Reworked patch with a single file.
> Reduce consumption of ClientInputStreamPump threads and NioProcessor threads.
> ------------------------------------------------------------------------------
>
> Key: SSHD-136
> URL: https://issues.apache.org/jira/browse/SSHD-136
> Project: MINA SSHD
> Issue Type: Improvement
> Affects Versions: 0.5.0
> Reporter: Alexey Polbitsyn
> Attachments: SSHD-136.patch, src-diff.zip, src.zip
>
>
> When using the library to work with 1000 concurrent ssh connections, it takes 1000 ClientInputStreamPump threads. 1 thread for 1 channel. It would be nice to be able to configure this behavior, for example use 1 ClientInputStreamPump for all channels of sshclient or session.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SSHD-136) Reduce consumption of
ClientInputStreamPump threads and NioProcessor threads.
Posted by "Alexey Polbitsyn (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SSHD-136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Polbitsyn updated SSHD-136:
----------------------------------
Description: When using the library to work with 1000 concurrent ssh connections, it takes 1000 ClientInputStreamPump threads. 1 thread for 1 channel. It would be nice to be able to configure this behavior, for example use 1 ClientInputStreamPump for all channels of sshclient or session. (was: When using the library to work with 1000 concurrent ssh connections, it takes 1000 ClientInputStreamPump threads and it can. 1 thread for 1 channel. It would be nice to be able to configure this behavior, for example use 1 ClientInputStreamPump for all channels of sshclient or session.)
> Reduce consumption of ClientInputStreamPump threads and NioProcessor threads.
> ------------------------------------------------------------------------------
>
> Key: SSHD-136
> URL: https://issues.apache.org/jira/browse/SSHD-136
> Project: MINA SSHD
> Issue Type: Improvement
> Affects Versions: 0.5.0
> Reporter: Alexey Polbitsyn
> Attachments: src-diff.zip, src.zip
>
>
> When using the library to work with 1000 concurrent ssh connections, it takes 1000 ClientInputStreamPump threads. 1 thread for 1 channel. It would be nice to be able to configure this behavior, for example use 1 ClientInputStreamPump for all channels of sshclient or session.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SSHD-136) Reduce consumption of
ClientInputStreamPump threads and NioProcessor threads.
Posted by "Guillaume Nodet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SSHD-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280016#comment-13280016 ]
Guillaume Nodet commented on SSHD-136:
--------------------------------------
One real problem is that such a solution leads to requirements on the InputStream / OutputStreams used: the InputStream#available() method has to report accurate data, but more importantantly, there's no way to know when the stream is finished or not.
{code}
public boolean pump() {
try {
if (!closeFuture.isClosed()) {
int len = Math.min(in.available(), remoteWindow.getPacketSize());
if (len > 0 && remoteWindow.consumeIfAvailable(len)) {
Buffer buffer = session.createBuffer(SshConstants.Message.SSH_MSG_CHANNEL_DATA, 0);
buffer.putInt(recipient);
buffer.putInt(len);
int wpos = buffer.wpos(); // keep buffer position for data write
buffer.wpos(wpos + remoteWindow.getPacketSize()); // Make room
securedRead(in, buffer.array(), wpos, len); // read data into buffer
log.debug("Send SSH_MSG_CHANNEL_DATA on channel {}", id);
session.writePacket(buffer);
return true;
}
}
} catch (Exception e) {
if (!closing) {
log.info("Caught exception", e);
close(false);
}
}
return false;
}
{code}
The above code has the mentioned problem, because eof() is not sent anymore.
I think for the proposed logic to work, we need to use enhanced interfaces instead of InputStream / OutputStream (making sure we have a way to detect real eof).
> Reduce consumption of ClientInputStreamPump threads and NioProcessor threads.
> ------------------------------------------------------------------------------
>
> Key: SSHD-136
> URL: https://issues.apache.org/jira/browse/SSHD-136
> Project: MINA SSHD
> Issue Type: Improvement
> Affects Versions: 0.5.0
> Reporter: Alexey Polbitsyn
> Attachments: SSHD-136.patch, src-diff.zip, src.zip
>
>
> When using the library to work with 1000 concurrent ssh connections, it takes 1000 ClientInputStreamPump threads. 1 thread for 1 channel. It would be nice to be able to configure this behavior, for example use 1 ClientInputStreamPump for all channels of sshclient or session.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SSHD-136) Reduce consumption of
ClientInputStreamPump threads and NioProcessor threads.
Posted by "Alexey Polbitsyn (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SSHD-136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Polbitsyn updated SSHD-136:
----------------------------------
Attachment: src.zip
src-diff.zip
src-diff - diff files
src - sources
> Reduce consumption of ClientInputStreamPump threads and NioProcessor threads.
> ------------------------------------------------------------------------------
>
> Key: SSHD-136
> URL: https://issues.apache.org/jira/browse/SSHD-136
> Project: MINA SSHD
> Issue Type: Improvement
> Affects Versions: 0.5.0
> Reporter: Alexey Polbitsyn
> Attachments: src-diff.zip, src.zip
>
>
> When using the library to work with 1000 concurrent ssh connections, it takes 1000 ClientInputStreamPump threads and it can. 1 thread for 1 channel. It would be nice to be able to configure this behavior, for example use 1 ClientInputStreamPump for all channels of sshclient or session.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira