You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@oodt.apache.org by "Chris A. Mattmann (JIRA)" <ji...@apache.org> on 2011/06/19 01:56:48 UTC
[jira] [Created] (OODT-203) Create a Non-Blocking threaded
implementation of the Workflow Engine
Create a Non-Blocking threaded implementation of the Workflow Engine
--------------------------------------------------------------------
Key: OODT-203
URL: https://issues.apache.org/jira/browse/OODT-203
Project: OODT
Issue Type: Bug
Components: workflow manager
Environment: from JPL's last internal WM JIRA release
Reporter: David Woollard
Fix For: 0.4
The current implementation of the threaded workflow engine uses a thread from its thread pool for each of the workflows it creates. This can cause a problem if a significant number of workflows submitted block waiting on preconditions to be satisfied. Each of these paused workflows hangs on to the thread allocated from the thread pool, so workflows that could be run are blocked by workflows waiting on preconditions if the thread pool is fully allocated.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OODT-203) Create a Non-Blocking threaded
implementation of the Workflow Engine
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris A. Mattmann updated OODT-203:
-----------------------------------
Issue Type: Sub-task (was: Bug)
Parent: OODT-215
> Create a Non-Blocking threaded implementation of the Workflow Engine
> --------------------------------------------------------------------
>
> Key: OODT-203
> URL: https://issues.apache.org/jira/browse/OODT-203
> Project: OODT
> Issue Type: Sub-task
> Components: workflow manager
> Environment: from JPL's last internal WM JIRA release
> Reporter: David Woollard
> Fix For: 0.4
>
>
> The current implementation of the threaded workflow engine uses a thread from its thread pool for each of the workflows it creates. This can cause a problem if a significant number of workflows submitted block waiting on preconditions to be satisfied. Each of these paused workflows hangs on to the thread allocated from the thread pool, so workflows that could be run are blocked by workflows waiting on preconditions if the thread pool is fully allocated.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (OODT-203) Create a Non-Blocking threaded
implementation of the Workflow Engine
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris A. Mattmann resolved OODT-203.
------------------------------------
Resolution: Fixed
- this is the current PriorityQueueBasedWorkflowEngine.
> Create a Non-Blocking threaded implementation of the Workflow Engine
> --------------------------------------------------------------------
>
> Key: OODT-203
> URL: https://issues.apache.org/jira/browse/OODT-203
> Project: OODT
> Issue Type: Sub-task
> Components: workflow manager
> Environment: from JPL's last internal WM JIRA release
> Reporter: David Woollard
> Assignee: Chris A. Mattmann
> Fix For: 0.5
>
>
> The current implementation of the threaded workflow engine uses a thread from its thread pool for each of the workflows it creates. This can cause a problem if a significant number of workflows submitted block waiting on preconditions to be satisfied. Each of these paused workflows hangs on to the thread allocated from the thread pool, so workflows that could be run are blocked by workflows waiting on preconditions if the thread pool is fully allocated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OODT-203) Create a Non-Blocking threaded
implementation of the Workflow Engine
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051621#comment-13051621 ]
Chris A. Mattmann commented on OODT-203:
----------------------------------------
Since OODT-202 was filed, I would recommend the following things in order to close this issue:
1. reproducible unit tests that demonstrate that OODT-202 isn't happening anymore. These can be modeled after the single JVM unit tests of higher level file manager services. These should create a sequence of 4 or so workflows akin to PEATE example provided by [~bfoster].
2. as [~woollard] mentioned, this was an initial version – ideally we would probably make some methods protected and turn the core WorkflowEngine into an AbstractBaseWorkflowEngine that provides functionality that both ThreadPoolWorkflowEngine and NonBlockingThreadPoolWorkflowEngine can inherit from.
> Create a Non-Blocking threaded implementation of the Workflow Engine
> --------------------------------------------------------------------
>
> Key: OODT-203
> URL: https://issues.apache.org/jira/browse/OODT-203
> Project: OODT
> Issue Type: Bug
> Components: workflow manager
> Environment: from JPL's last internal WM JIRA release
> Reporter: David Woollard
> Fix For: 0.4
>
>
> The current implementation of the threaded workflow engine uses a thread from its thread pool for each of the workflows it creates. This can cause a problem if a significant number of workflows submitted block waiting on preconditions to be satisfied. Each of these paused workflows hangs on to the thread allocated from the thread pool, so workflows that could be run are blocked by workflows waiting on preconditions if the thread pool is fully allocated.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (OODT-203) Create a Non-Blocking
threaded implementation of the Workflow Engine
Posted by "Sheryl John (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123401#comment-13123401 ]
Sheryl John edited comment on OODT-203 at 10/8/11 8:05 PM:
-----------------------------------------------------------
Hi,
I have a few questions regarding this issue and OODT-202:
1.Has the NonBlockingThreadPoolWorkflowEngine implementation been updated after the initial one?
I went through the code from the checked out version around r1156658 of OODT 0.4 and it looks just as it is described by David's comment.
2. In the above comment, which higher level file manager services are you referring for modeling the unit test?
3. By the PEATE example, do you mean the case described in OODT-202?
4. How will AbstractBaseWorkflowEngine be different from the current core WorkflowEngine? Are you thinking of adding some new functionality to the existing ones?
was (Author: sherylj):
Hi,
I have a few of questions here:
1.Has the NonBlockingThreadPoolWorkflowEngine implementation been updated after the initial one?
I went through the code from the checked out version around r1156658 of OODT 0.4 and it looks just as it is described by David's comment.
2. In the above comment, which higher level file manager services are you referring for modeling the unit test?
3. By the PEATE example, do you mean the case described in OODT-202?
4. How will AbstractBaseWorkflowEngine be different from the current core WorkflowEngine? Are you thinking of adding some new functionality to the existing ones?
> Create a Non-Blocking threaded implementation of the Workflow Engine
> --------------------------------------------------------------------
>
> Key: OODT-203
> URL: https://issues.apache.org/jira/browse/OODT-203
> Project: OODT
> Issue Type: Sub-task
> Components: workflow manager
> Environment: from JPL's last internal WM JIRA release
> Reporter: David Woollard
> Assignee: Chris A. Mattmann
> Fix For: 0.4
>
>
> The current implementation of the threaded workflow engine uses a thread from its thread pool for each of the workflows it creates. This can cause a problem if a significant number of workflows submitted block waiting on preconditions to be satisfied. Each of these paused workflows hangs on to the thread allocated from the thread pool, so workflows that could be run are blocked by workflows waiting on preconditions if the thread pool is fully allocated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OODT-203) Create a Non-Blocking threaded
implementation of the Workflow Engine
Posted by "Sheryl John (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124722#comment-13124722 ]
Sheryl John commented on OODT-203:
----------------------------------
Thank you for your answers, Chris.
So for the workflow manager unit tests, we should test a XmlRpcWorkflowManager client to run some workflows with predefined tasks,conditions,metadata etc right?
Before I start, I wanted to confirm if I should define some test workflows/tasks files like the test.txt and met files that were used to test the ingestion in the XmlRpcFileManager unit tests.
> Create a Non-Blocking threaded implementation of the Workflow Engine
> --------------------------------------------------------------------
>
> Key: OODT-203
> URL: https://issues.apache.org/jira/browse/OODT-203
> Project: OODT
> Issue Type: Sub-task
> Components: workflow manager
> Environment: from JPL's last internal WM JIRA release
> Reporter: David Woollard
> Assignee: Chris A. Mattmann
> Fix For: 0.4
>
>
> The current implementation of the threaded workflow engine uses a thread from its thread pool for each of the workflows it creates. This can cause a problem if a significant number of workflows submitted block waiting on preconditions to be satisfied. Each of these paused workflows hangs on to the thread allocated from the thread pool, so workflows that could be run are blocked by workflows waiting on preconditions if the thread pool is fully allocated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OODT-203) Create a Non-Blocking threaded
implementation of the Workflow Engine
Posted by "Sheryl John (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123401#comment-13123401 ]
Sheryl John commented on OODT-203:
----------------------------------
Hi,
I have a few of questions here:
1.Has the NonBlockingThreadPoolWorkflowEngine implementation been updated after the initial one?
I went through the code from the checked out version around r1156658 of OODT 0.4 and it looks just as it is described by David's comment.
2. In the above comment, which higher level file manager services are you referring for modeling the unit test?
3. By the PEATE example, do you mean the case described in OODT-202?
4. How will AbstractBaseWorkflowEngine be different from the current core WorkflowEngine? Are you thinking of adding some new functionality to the existing ones?
> Create a Non-Blocking threaded implementation of the Workflow Engine
> --------------------------------------------------------------------
>
> Key: OODT-203
> URL: https://issues.apache.org/jira/browse/OODT-203
> Project: OODT
> Issue Type: Sub-task
> Components: workflow manager
> Environment: from JPL's last internal WM JIRA release
> Reporter: David Woollard
> Assignee: Chris A. Mattmann
> Fix For: 0.4
>
>
> The current implementation of the threaded workflow engine uses a thread from its thread pool for each of the workflows it creates. This can cause a problem if a significant number of workflows submitted block waiting on preconditions to be satisfied. Each of these paused workflows hangs on to the thread allocated from the thread pool, so workflows that could be run are blocked by workflows waiting on preconditions if the thread pool is fully allocated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OODT-203) Create a Non-Blocking threaded
implementation of the Workflow Engine
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051620#comment-13051620 ]
Chris A. Mattmann commented on OODT-203:
----------------------------------------
Original comment from [~woollard]:
{quote}
Initial implementation of NonBlocking ThreadPoolWorkflowEngine delivered in revision 5541. This Engine uses a vector of queued workflow instances to hold workflows with unsatisfied preconditions. A shepard thread is introduced that iterates over this vector, checking for satisfied preconditions and, as workflows with satisfied precoditions are identified, the engine uses a thread from the thread pool to monitor the execution of the workflow instance. When a workflow consists of multiple tasks, the current task pointer is incremented to the next task in the workflow and the worklfow is requeued in the vector until the next task's preconditions are satisfied.
{quote}
> Create a Non-Blocking threaded implementation of the Workflow Engine
> --------------------------------------------------------------------
>
> Key: OODT-203
> URL: https://issues.apache.org/jira/browse/OODT-203
> Project: OODT
> Issue Type: Bug
> Components: workflow manager
> Environment: from JPL's last internal WM JIRA release
> Reporter: David Woollard
> Fix For: 0.4
>
>
> The current implementation of the threaded workflow engine uses a thread from its thread pool for each of the workflows it creates. This can cause a problem if a significant number of workflows submitted block waiting on preconditions to be satisfied. Each of these paused workflows hangs on to the thread allocated from the thread pool, so workflows that could be run are blocked by workflows waiting on preconditions if the thread pool is fully allocated.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (OODT-203) Create a Non-Blocking threaded
implementation of the Workflow Engine
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris A. Mattmann reassigned OODT-203:
--------------------------------------
Assignee: Chris A. Mattmann
> Create a Non-Blocking threaded implementation of the Workflow Engine
> --------------------------------------------------------------------
>
> Key: OODT-203
> URL: https://issues.apache.org/jira/browse/OODT-203
> Project: OODT
> Issue Type: Sub-task
> Components: workflow manager
> Environment: from JPL's last internal WM JIRA release
> Reporter: David Woollard
> Assignee: Chris A. Mattmann
> Fix For: 0.4
>
>
> The current implementation of the threaded workflow engine uses a thread from its thread pool for each of the workflows it creates. This can cause a problem if a significant number of workflows submitted block waiting on preconditions to be satisfied. Each of these paused workflows hangs on to the thread allocated from the thread pool, so workflows that could be run are blocked by workflows waiting on preconditions if the thread pool is fully allocated.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OODT-203) Create a Non-Blocking threaded
implementation of the Workflow Engine
Posted by "Chris A. Mattmann (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123718#comment-13123718 ]
Chris A. Mattmann commented on OODT-203:
----------------------------------------
Hi Sheryl,
Thanks for your questions. Comments below:
bq. Has the NonBlockingThreadPoolWorkflowEngine implementation been updated after the initial one?
I went through the code from the checked out version around r1156658 of OODT 0.4 and it looks just as it is described by David's comment.
There was a NonBlockingThreadPoolWorkflowEngine. In a way you could call it a predecessor to the Wengine branch. As I started to integrate Wengine in OODT-215 into trunk, I decided to get rid of the existing old implementation, and bring Wengine down in new. The stub I'm about to attach in OODT-310 that contains the PrioritizedQueueBasedWorkflowEngine is the functional replacement.
bq. In the above comment, which higher level file manager services are you referring for modeling the unit test?
I mean the tests in filemgr that actually instantiate an XmlRpcFileManager service, and then ingest files into it. Similarly, we should have some tests in the workflow manager that stand up an XmlRpcWorkflowManager service, and then run workflows and check the expected results. Those would be great!
bq. By the PEATE example, do you mean the case described in OODT-202?
Yep. More specifically, PEATE is a Product Evaluation and Testbed Element (PEATE) project part of the NASA/DOD/NOAA NPOESS Preparatory Project. This code and effort originally came out of the NPP Sounder PEATE, which resides at JPL, and their use cases.
bq. How will AbstractBaseWorkflowEngine be different from the current core WorkflowEngine? Are you thinking of adding some new functionality to the existing ones?
In the old design there was thinking that we could aggregate a bunch of common functionality in an AbstractWorkflowEngine, that the subclasses (like ThreadPoolWorkflowEngine) inherited from. I'm not sure we're headed down that direction anymore though.
> Create a Non-Blocking threaded implementation of the Workflow Engine
> --------------------------------------------------------------------
>
> Key: OODT-203
> URL: https://issues.apache.org/jira/browse/OODT-203
> Project: OODT
> Issue Type: Sub-task
> Components: workflow manager
> Environment: from JPL's last internal WM JIRA release
> Reporter: David Woollard
> Assignee: Chris A. Mattmann
> Fix For: 0.4
>
>
> The current implementation of the threaded workflow engine uses a thread from its thread pool for each of the workflows it creates. This can cause a problem if a significant number of workflows submitted block waiting on preconditions to be satisfied. Each of these paused workflows hangs on to the thread allocated from the thread pool, so workflows that could be run are blocked by workflows waiting on preconditions if the thread pool is fully allocated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (OODT-203) Create a Non-Blocking
threaded implementation of the Workflow Engine
Posted by "Chris A. Mattmann (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123718#comment-13123718 ]
Chris A. Mattmann edited comment on OODT-203 at 10/9/11 4:29 PM:
-----------------------------------------------------------------
Hi Sheryl,
Thanks for your questions. Comments below:
bq. Has the NonBlockingThreadPoolWorkflowEngine implementation been updated after the initial one? I went through the code from the checked out version around r1156658 of OODT 0.4 and it looks just as it is described by David's comment.
There was a NonBlockingThreadPoolWorkflowEngine. In a way you could call it a predecessor to the Wengine branch. As I started to integrate Wengine in OODT-215 into trunk, I decided to get rid of the existing old implementation, and bring Wengine down in new. The stub I'm about to attach in OODT-310 that contains the PrioritizedQueueBasedWorkflowEngine is the functional replacement.
bq. In the above comment, which higher level file manager services are you referring for modeling the unit test?
I mean the tests in filemgr that actually instantiate an XmlRpcFileManager service, and then ingest files into it. Similarly, we should have some tests in the workflow manager that stand up an XmlRpcWorkflowManager service, and then run workflows and check the expected results. Those would be great!
bq. By the PEATE example, do you mean the case described in OODT-202?
Yep. More specifically, PEATE is a Product Evaluation and Testbed Element (PEATE) project part of the NASA/DOD/NOAA NPOESS Preparatory Project. This code and effort originally came out of the NPP Sounder PEATE, which resides at JPL, and their use cases.
bq. How will AbstractBaseWorkflowEngine be different from the current core WorkflowEngine? Are you thinking of adding some new functionality to the existing ones?
In the old design there was thinking that we could aggregate a bunch of common functionality in an AbstractWorkflowEngine, that the subclasses (like ThreadPoolWorkflowEngine) inherited from. I'm not sure we're headed down that direction anymore though.
was (Author: chrismattmann):
Hi Sheryl,
Thanks for your questions. Comments below:
bq. Has the NonBlockingThreadPoolWorkflowEngine implementation been updated after the initial one?
I went through the code from the checked out version around r1156658 of OODT 0.4 and it looks just as it is described by David's comment.
There was a NonBlockingThreadPoolWorkflowEngine. In a way you could call it a predecessor to the Wengine branch. As I started to integrate Wengine in OODT-215 into trunk, I decided to get rid of the existing old implementation, and bring Wengine down in new. The stub I'm about to attach in OODT-310 that contains the PrioritizedQueueBasedWorkflowEngine is the functional replacement.
bq. In the above comment, which higher level file manager services are you referring for modeling the unit test?
I mean the tests in filemgr that actually instantiate an XmlRpcFileManager service, and then ingest files into it. Similarly, we should have some tests in the workflow manager that stand up an XmlRpcWorkflowManager service, and then run workflows and check the expected results. Those would be great!
bq. By the PEATE example, do you mean the case described in OODT-202?
Yep. More specifically, PEATE is a Product Evaluation and Testbed Element (PEATE) project part of the NASA/DOD/NOAA NPOESS Preparatory Project. This code and effort originally came out of the NPP Sounder PEATE, which resides at JPL, and their use cases.
bq. How will AbstractBaseWorkflowEngine be different from the current core WorkflowEngine? Are you thinking of adding some new functionality to the existing ones?
In the old design there was thinking that we could aggregate a bunch of common functionality in an AbstractWorkflowEngine, that the subclasses (like ThreadPoolWorkflowEngine) inherited from. I'm not sure we're headed down that direction anymore though.
> Create a Non-Blocking threaded implementation of the Workflow Engine
> --------------------------------------------------------------------
>
> Key: OODT-203
> URL: https://issues.apache.org/jira/browse/OODT-203
> Project: OODT
> Issue Type: Sub-task
> Components: workflow manager
> Environment: from JPL's last internal WM JIRA release
> Reporter: David Woollard
> Assignee: Chris A. Mattmann
> Fix For: 0.4
>
>
> The current implementation of the threaded workflow engine uses a thread from its thread pool for each of the workflows it creates. This can cause a problem if a significant number of workflows submitted block waiting on preconditions to be satisfied. Each of these paused workflows hangs on to the thread allocated from the thread pool, so workflows that could be run are blocked by workflows waiting on preconditions if the thread pool is fully allocated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira