You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@synapse.apache.org by "Andreas Veithen (JIRA)" <ji...@apache.org> on 2008/08/23 18:29:44 UTC
[jira] Created: (SYNAPSE-434) Concurrency level provided by the
mail and VFS transports listeners is insufficient
Concurrency level provided by the mail and VFS transports listeners is insufficient
-----------------------------------------------------------------------------------
Key: SYNAPSE-434
URL: https://issues.apache.org/jira/browse/SYNAPSE-434
Project: Synapse
Issue Type: Bug
Components: Transports
Affects Versions: 1.2
Reporter: Andreas Veithen
Priority: Minor
Scenario:
* Two services are configured to use the (mail or VFS) transport.
* Two messages are sent two each of these services.
In this case, one would expect that the transport listener will be able to process at least 2 (better 4) messages concurrently. However the listener currently only processes one message at a time, i.e. the listener behaves as if it was single-threaded.
There are two reasons for this:
1. AbstractPollingTransportListener contains the following code:
workerPool.execute(new Runnable() {
public void run() {
synchronized (pollLock) {
pollInProgress = true;
try {
onPoll();
} finally {
pollInProgress = false;
}
}
}
});
Since pollLock is a member of AbstractPollingTransportListener, there can only be one poll in progress at a time, even if several services are configured to use the transport (in which case nothing would prevent the transport from executing the polls for the different services in parallel).
2. Both MailTransportListener and VFSTransportListener process incoming messages sequentially. Here again, the transport could process several messages in parallel during the execution of the onPoll method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
[jira] Commented: (SYNAPSE-434) Concurrency level provided by the
mail and VFS transports listeners is insufficient
Posted by "Andreas Veithen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SYNAPSE-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688011#action_12688011 ]
Andreas Veithen commented on SYNAPSE-434:
-----------------------------------------
Of course we need to avoid to process the same message twice. This can be achieved if me make sure that for a given service, a new poll is only scheduled when the current poll is completed. For the moment the mail and VFS transports work as follows:
1. Start poll (open mail folder and read message list; read directory).
2. Sequentially process each message.
3. Schedule the next poll.
The idea is to change that (as an option) to:
1. Start poll
2. Process the messages in parallel (with a configurable level of concurrency)
3. Wait until all messages have been processed and schedule the next poll.
Note that this solution probably doesn't provide enough concurrency for what is requested in the linked issue WSCOMMONS-454.
> Concurrency level provided by the mail and VFS transports listeners is insufficient
> -----------------------------------------------------------------------------------
>
> Key: SYNAPSE-434
> URL: https://issues.apache.org/jira/browse/SYNAPSE-434
> Project: Synapse
> Issue Type: Bug
> Components: Transports
> Affects Versions: 1.2
> Reporter: Andreas Veithen
> Assignee: Andreas Veithen
> Priority: Minor
>
> Scenario:
> * Two services are configured to use the (mail or VFS) transport.
> * Two messages are sent two each of these services.
> In this case, one would expect that the transport listener will be able to process at least 2 (better 4) messages concurrently. However the listener currently only processes one message at a time, i.e. the listener behaves as if it was single-threaded.
> There are two reasons for this:
> 1. AbstractPollingTransportListener contains the following code:
> workerPool.execute(new Runnable() {
> public void run() {
> synchronized (pollLock) {
> pollInProgress = true;
> try {
> onPoll();
> } finally {
> pollInProgress = false;
> }
> }
> }
> });
> Since pollLock is a member of AbstractPollingTransportListener, there can only be one poll in progress at a time, even if several services are configured to use the transport (in which case nothing would prevent the transport from executing the polls for the different services in parallel).
> 2. Both MailTransportListener and VFSTransportListener process incoming messages sequentially. Here again, the transport could process several messages in parallel during the execution of the onPoll method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
[jira] Assigned: (SYNAPSE-434) Concurrency level provided by the
mail and VFS transports listeners is insufficient
Posted by "Andreas Veithen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SYNAPSE-434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andreas Veithen reassigned SYNAPSE-434:
---------------------------------------
Assignee: Andreas Veithen
> Concurrency level provided by the mail and VFS transports listeners is insufficient
> -----------------------------------------------------------------------------------
>
> Key: SYNAPSE-434
> URL: https://issues.apache.org/jira/browse/SYNAPSE-434
> Project: Synapse
> Issue Type: Bug
> Components: Transports
> Affects Versions: 1.2
> Reporter: Andreas Veithen
> Assignee: Andreas Veithen
> Priority: Minor
>
> Scenario:
> * Two services are configured to use the (mail or VFS) transport.
> * Two messages are sent two each of these services.
> In this case, one would expect that the transport listener will be able to process at least 2 (better 4) messages concurrently. However the listener currently only processes one message at a time, i.e. the listener behaves as if it was single-threaded.
> There are two reasons for this:
> 1. AbstractPollingTransportListener contains the following code:
> workerPool.execute(new Runnable() {
> public void run() {
> synchronized (pollLock) {
> pollInProgress = true;
> try {
> onPoll();
> } finally {
> pollInProgress = false;
> }
> }
> }
> });
> Since pollLock is a member of AbstractPollingTransportListener, there can only be one poll in progress at a time, even if several services are configured to use the transport (in which case nothing would prevent the transport from executing the polls for the different services in parallel).
> 2. Both MailTransportListener and VFSTransportListener process incoming messages sequentially. Here again, the transport could process several messages in parallel during the execution of the onPoll method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
[jira] Commented: (SYNAPSE-434) Concurrency level provided by the
mail and VFS transports listeners is insufficient
Posted by "Asankha C. Perera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SYNAPSE-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12685418#action_12685418 ]
Asankha C. Perera commented on SYNAPSE-434:
-------------------------------------------
The reason I did this was to actually prevent concurrent processing by Mail, VFS etc., where we do not want to say process the same file/email by two threads concurrently.. Do you have a proposed solution to prevent that?
> Concurrency level provided by the mail and VFS transports listeners is insufficient
> -----------------------------------------------------------------------------------
>
> Key: SYNAPSE-434
> URL: https://issues.apache.org/jira/browse/SYNAPSE-434
> Project: Synapse
> Issue Type: Bug
> Components: Transports
> Affects Versions: 1.2
> Reporter: Andreas Veithen
> Assignee: Andreas Veithen
> Priority: Minor
>
> Scenario:
> * Two services are configured to use the (mail or VFS) transport.
> * Two messages are sent two each of these services.
> In this case, one would expect that the transport listener will be able to process at least 2 (better 4) messages concurrently. However the listener currently only processes one message at a time, i.e. the listener behaves as if it was single-threaded.
> There are two reasons for this:
> 1. AbstractPollingTransportListener contains the following code:
> workerPool.execute(new Runnable() {
> public void run() {
> synchronized (pollLock) {
> pollInProgress = true;
> try {
> onPoll();
> } finally {
> pollInProgress = false;
> }
> }
> }
> });
> Since pollLock is a member of AbstractPollingTransportListener, there can only be one poll in progress at a time, even if several services are configured to use the transport (in which case nothing would prevent the transport from executing the polls for the different services in parallel).
> 2. Both MailTransportListener and VFSTransportListener process incoming messages sequentially. Here again, the transport could process several messages in parallel during the execution of the onPoll method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
[jira] Commented: (SYNAPSE-434) Concurrency level provided by the
mail and VFS transports listeners is insufficient
Posted by "Asankha C. Perera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SYNAPSE-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689880#action_12689880 ]
Asankha C. Perera commented on SYNAPSE-434:
-------------------------------------------
I've fixed WSCOMMONS-454 for the Mail transport. We could use the same concept and make sure that one file is processed only once (i.e. using the file path) for VFS. Of course, one would need to pin VFS services to an Active/Passive configuration to ensure that multiple nodes on a cluster would not conflict
> Concurrency level provided by the mail and VFS transports listeners is insufficient
> -----------------------------------------------------------------------------------
>
> Key: SYNAPSE-434
> URL: https://issues.apache.org/jira/browse/SYNAPSE-434
> Project: Synapse
> Issue Type: Bug
> Components: Transports
> Affects Versions: 1.2
> Reporter: Andreas Veithen
> Assignee: Andreas Veithen
> Priority: Minor
>
> Scenario:
> * Two services are configured to use the (mail or VFS) transport.
> * Two messages are sent two each of these services.
> In this case, one would expect that the transport listener will be able to process at least 2 (better 4) messages concurrently. However the listener currently only processes one message at a time, i.e. the listener behaves as if it was single-threaded.
> There are two reasons for this:
> 1. AbstractPollingTransportListener contains the following code:
> workerPool.execute(new Runnable() {
> public void run() {
> synchronized (pollLock) {
> pollInProgress = true;
> try {
> onPoll();
> } finally {
> pollInProgress = false;
> }
> }
> }
> });
> Since pollLock is a member of AbstractPollingTransportListener, there can only be one poll in progress at a time, even if several services are configured to use the transport (in which case nothing would prevent the transport from executing the polls for the different services in parallel).
> 2. Both MailTransportListener and VFSTransportListener process incoming messages sequentially. Here again, the transport could process several messages in parallel during the execution of the onPoll method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
[jira] Commented: (SYNAPSE-434) Concurrency level provided by the
mail and VFS transports listeners is insufficient
Posted by "Amila Chinthaka Suriarachchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SYNAPSE-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688103#action_12688103 ]
Amila Chinthaka Suriarachchi commented on SYNAPSE-434:
------------------------------------------------------
I have also seen this same issue and it is a serious problem for RM synchronize invocations.
In the earlier Axis2 SMTP transport it also polls the SMTP server using one thread. But after creating the message context it starts a new thread to send it through the Axis Engine.
synchronized (receiver) {
receiver.connect();
Message[] msgs = receiver.receiveMessages();
if ((msgs != null) && (msgs.length > 0)) {
log.info(msgs.length + " Message(s) Found");
for (int i = 0; i < msgs.length; i++) {
MimeMessage msg = (MimeMessage) msgs[i];
try {
MessageContext mc = createMessageContextToMailWorker(msg);
msg.setFlag(Flags.Flag.DELETED, true);
if(mc==null){
continue;
}
MailWorker worker = new MailWorker(configurationContext,mc);
this.configurationContext.getThreadPool().execute(worker);
} catch (Exception e) {
log.error("Error in SimpleMailListener - processing mail", e);
} finally {
// delete mail in any case
}
}
}
receiver.disconnect();
}
Can't we do the same thing here as well.
thanks,
Amila.
> Concurrency level provided by the mail and VFS transports listeners is insufficient
> -----------------------------------------------------------------------------------
>
> Key: SYNAPSE-434
> URL: https://issues.apache.org/jira/browse/SYNAPSE-434
> Project: Synapse
> Issue Type: Bug
> Components: Transports
> Affects Versions: 1.2
> Reporter: Andreas Veithen
> Assignee: Andreas Veithen
> Priority: Minor
>
> Scenario:
> * Two services are configured to use the (mail or VFS) transport.
> * Two messages are sent two each of these services.
> In this case, one would expect that the transport listener will be able to process at least 2 (better 4) messages concurrently. However the listener currently only processes one message at a time, i.e. the listener behaves as if it was single-threaded.
> There are two reasons for this:
> 1. AbstractPollingTransportListener contains the following code:
> workerPool.execute(new Runnable() {
> public void run() {
> synchronized (pollLock) {
> pollInProgress = true;
> try {
> onPoll();
> } finally {
> pollInProgress = false;
> }
> }
> }
> });
> Since pollLock is a member of AbstractPollingTransportListener, there can only be one poll in progress at a time, even if several services are configured to use the transport (in which case nothing would prevent the transport from executing the polls for the different services in parallel).
> 2. Both MailTransportListener and VFSTransportListener process incoming messages sequentially. Here again, the transport could process several messages in parallel during the execution of the onPoll method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org