Re: Workflow & Turbine design questions

[James Strachan <ja...@yahoo.co.uk>]

Hi Erik

I totally agree. I think a Scheduler which just sends JMS messages when its
time to execute tasks is a great way to solve the problem. Then fault
tolerance and load balancing can be achieved by having server farms to
execute the tasks. (e.g. you could use Messenger with a cluster of Tomcat
servers to execute the tasks which would take care of load balancing & fault
tolerance for you).

The JMS provider will handle load balancing, acknowledgement, recovery,
ensuring messages are delivered only once etc. If need be database-centric
tasks could be implemented with XA 2 phase commit with the JMS provider. If
2 phase commit is required then it could be used by the scheduler as well,
updating the schedule database and sending the JMS message in the same
transaction.

I guess for systems with less transactional requirements then HTTP, servlets
or email could be used as alternatives to JMS.

James
----- Original Message -----
From: "Erik Worth" <ew...@altoweb.com>
To: "Jakarta Commons Developers List" <co...@jakarta.apache.org>
Sent: Friday, November 30, 2001 7:26 PM
Subject: RE: Workflow & Turbine design questions


> Hi Gang,
> The tricky thing with building a scheduler suitable for workflow is the
need
> to schedule tasks in the future (i.e. do this task 10 days from now at
2:35
> PM).  When scheduling future tasks, one needs to persist the schedule.
This
> type of service is also difficult to support in a clustered or fault
> tolerant environment because it is relatively difficult to keep multiple
> instances of the same scheduler in sync so that when the main one (you
only
> want one of these scheduling tasks at a time or you get duplicate work
> scheduled) goes down, the other one automatically picks up where the other
> left off.
>
> I am not trying to dampen anyone's enthusiasm with regards to building one
> of these for the open source community, but it would be nice if such an
> endeavor was taken up, it would address these types of issues.  I built a
> CORBA-based implementation of such a scheduler a number of years back that
> handled the persistence case, but when we considered the level of effort
for
> a fault tolerant version, we decided to live with the single point of
> failure.
>
> One aspect of the approach I took worked out rather well.  When scheduling
a
> task, I passed a command object along with the schedule time and recipient
> (and periodicity if any).  If this were implemented using JMS as the
> transport, one could pass a serializable command object (an object
> implementing an interface that specifies an execute method), a JMS
> destination (as the recipient), the schedule time, and the periodicity
using
> an JMS ObjectMessage payload.  The scheduler (in this case a stand-alone
> daemon type application) keeps a time ordered list of tasks and when a
> task's time comes due, sends the command object to the designated
> destination and removes the task if is not a periodic task.  If Message
> Driven Beans are listening on the destination queues, the threading is
> already taken care of and the listener simply invokes the execute method
on
> the command object (simple JMS listeners can also execute tasks, but they
> need to manage their own thread pool).  The execute method is implemented
to
> perform a task or invoke on something that performs the task.
>
> Using this model, a client is able to schedule some other service to
perform
> a task at a specific time in the future (or periodically starting at some
> time in the future).  It decouples the schedule maintenance from the task
> execution, and it has a hope of being made fault tolerant.
>
> Food for thought :-)
> -- Erik.
>
>
> -----Original Message-----
> From: James Strachan [mailto:james_strachan@yahoo.co.uk]
> Sent: Friday, November 30, 2001 2:42 AM
> To: Jakarta Commons Developers List
> Subject: Re: Workflow & Turbine design questions
>
>
> +1.
>
> I've wanted a scheduler component for some time - up until now everything
> I've seen has been integrated into larger frameworks or has been payware.
> Maybe a combination of a general ThreadPool that asynchronous tasks can be
> dispatched to (probably using the Runnable interface to represent the
tasks)
> with a Scheduler capable of dispatching tasks at fixed or repeated
> intervals.
>
> The Timer and TimerTask in JDK1.3 are a pretty good start for a Scheduler
if
> used with ThreadPools for the heavy-lifting. Though Timer/TimerTask is
only
> available on 1.3. Its a real shame we're not allowed to redistribute
classes
> from the newer JDKs so we can use them on older JDKs like 1.2. (Ditto for
> other stuff in 1.4).
>
> James
> ----- Original Message -----
> From: "Craig R. McClanahan" <cr...@apache.org>
> To: "Jakarta Commons Developers List" <co...@jakarta.apache.org>;
> <br...@mac.com>
> Sent: Friday, November 30, 2001 2:32 AM
> Subject: RE: Workflow & Turbine design questions
>
>
> > One way to approach these issues (and would even fit into the Commons
> > charter :-) would be to conceive of a Commons package to do background
job
> > scheduling and execution (what legacy OSs called "batch job queues")
> > within a JVM.  Executing a particular workflow script would be only one
of
> > the interesting things you could do -- ideally the definition of a "job"
> > would be something like "call method X on object Y" so it could be fully
> > generalized.
> >
> > The "jobs" package could be configured with limits on how many
background
> > threads it could run, support ways to schedule jobs repeatedly, and all
> > the other stuff you can do with "cron" type utilities, if we wanted.  It
> > could be embedded in any sort of long-running JVM (such as in a webapp
on
> > a servlet container), as well as stand-alone background jobs.
> >
> > This is a bigger task than just something that can run Workflow scripts
in
> > background threads (which isn't terribly difficult to hard code in your
> > application by starting a new thread and executing the Activity there),
> > but it would be a generally useful package.  What do you think?
> >
> > Craig
> >
> >
> > On Thu, 29 Nov 2001, Brett Gullan wrote:
> >
> > > Date: Thu, 29 Nov 2001 09:24:55 +0800
> > > From: Brett Gullan <br...@mac.com>
> > > Reply-To: Jakarta Commons Developers List
> <co...@jakarta.apache.org>,
> > >      brettgullan@mac.com
> > > To: 'Jakarta Commons Developers List' <co...@jakarta.apache.org>
> > > Subject: RE: Workflow & Turbine design questions
> > >
> > > Almost...
> > >
> > > > #1. User starts workflow process by performing action.
> > > Let's say User A starts workflow process, by for instance, copying a
> > > file from his/her desktop to a directory on a local/remote server
(this
> > > can/will be achieved by a variety of methods/protocols).
> > >
> > > > #2. Workflow performs tasks that may take time.
> > > Yes
> > >
> > > > #3. User should not be held up waiting for a response from the
server
> > > by
> > > background workflow tasks.
> > > Yes. However, User A may never even use the Web UI. User B on the
other
> > > hand may be watching the queue/progress monitor for regular updates.
> > >
> > > The web UI is intended to be a monitoring and configuration tool for
the
> > > underlying workflow engine. For the most part, general users would
> > > submit files (by desktop drag-n-drop, FTP, web-form, etc...) then use
> > > the web UI to monitor progress or change the queue priority (if
> > > authorised).
> > > Administrative users would have access to web forms that configure the
> > > workflow processing activity steps and the 'hot' folders that feed the
> > > processing queue.
> > > The application is intended to support deployment on a 'headless'
server
> > > where all interaction is via a web UI.
> > > I hope that makes more sense?
> > > Regards,
> > > Brett
> > > --
> > > Brett Gullan
> > > brettgullan@mac.com
> > >
> > >
> > >
> > > --
> > > To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> > > For additional commands, e-mail:
> <ma...@jakarta.apache.org>
> > >
> > >
> >
> >
> > --
> > To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> > For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
>
> _________________________________________________________
> Do You Yahoo!?
> Get your free @yahoo.com address at http://mail.yahoo.com
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
> --
> To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
> For additional commands, e-mail:
<ma...@jakarta.apache.org>


_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com


--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>