You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Brett Gullan <br...@mac.com> on 2001/11/28 04:11:07 UTC
Workflow & Turbine design questions
I am working on a project to develop a Turbine-based workflow engine for
file processing, using the Commons Workflow package and I would be
interested in any feedback or suggestions as to how best to implement
this.
The main focus of the project is to automate file processing for
print/publishing workflow. Users move/copy files into predefined
"hotfolders" which trigger a series of workflow steps such as converting
a file from PostScript to PDF, compressing images or transferring a file
or directory to a remote server (via FTP, HTTP, WebDAV, etc...).
Currently I have created a WorkflowService, modelled after the Turbine
SchedulerService that retrieves items from a queue and initiates the
processing sequence. A scheduled job periodically retrieves the list of
hotfolders (from storage; they are implemented as OM/Peers), checks for
any new files and submits them to the workflow queue, along with a
reference to an Activity definition that will be used to process the
files.
My intention is that as each entry is retrieved from the queue the
associated Activity definition file is 'digested' and a processing
thread started to perform the workflow steps.
I also need to ensure there is a mechanism for maintaining
state/progress information for each workflow step -- both to report back
to the client UI and for 'recovery' should a process be interrupted.
Workflow steps may be quite expensive and/or time consuming -- for
instance, PDF conversion of large publishing files typically takes 1-5
minutes, thus processing a directory of files could take significantly
longer. During this period a user needs to see progress information such
as a progress bar or estimated time to completion.
I am unclear at this stage as to how the workflow Context and Scope fit
in. I figure the processing thread itself should probably be a Context
implementation and maybe the Scope implementation is the conduit to the
UI? A Velocity Screen/Action could request a list of processing
threads/Contexts from the WorkflowService and insert each Scope
implementation into the Velocity Context? The Scope would probably also
need to be backed by an OM/Peer to persist state information.
I would appreciate any suggestions or alternative design ideas.
Thanks,
Brett
--
Brett Gullan
brettgullan@mac.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Workflow & Turbine design questions
Posted by James Strachan <ja...@yahoo.co.uk>.
From: "James Strachan" <ja...@yahoo.co.uk>
> Its a real shame we're not allowed to redistribute classes (*)
> from the newer JDKs so we can use them on older JDKs like 1.2. (Ditto for
> other stuff in 1.4).
* - even redistributing them in binary form would be nice....
James
_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Workflow & Turbine design questions
Posted by James Strachan <ja...@yahoo.co.uk>.
+1.
I've wanted a scheduler component for some time - up until now everything
I've seen has been integrated into larger frameworks or has been payware.
Maybe a combination of a general ThreadPool that asynchronous tasks can be
dispatched to (probably using the Runnable interface to represent the tasks)
with a Scheduler capable of dispatching tasks at fixed or repeated
intervals.
The Timer and TimerTask in JDK1.3 are a pretty good start for a Scheduler if
used with ThreadPools for the heavy-lifting. Though Timer/TimerTask is only
available on 1.3. Its a real shame we're not allowed to redistribute classes
from the newer JDKs so we can use them on older JDKs like 1.2. (Ditto for
other stuff in 1.4).
James
----- Original Message -----
From: "Craig R. McClanahan" <cr...@apache.org>
To: "Jakarta Commons Developers List" <co...@jakarta.apache.org>;
<br...@mac.com>
Sent: Friday, November 30, 2001 2:32 AM
Subject: RE: Workflow & Turbine design questions
> One way to approach these issues (and would even fit into the Commons
> charter :-) would be to conceive of a Commons package to do background job
> scheduling and execution (what legacy OSs called "batch job queues")
> within a JVM. Executing a particular workflow script would be only one of
> the interesting things you could do -- ideally the definition of a "job"
> would be something like "call method X on object Y" so it could be fully
> generalized.
>
> The "jobs" package could be configured with limits on how many background
> threads it could run, support ways to schedule jobs repeatedly, and all
> the other stuff you can do with "cron" type utilities, if we wanted. It
> could be embedded in any sort of long-running JVM (such as in a webapp on
> a servlet container), as well as stand-alone background jobs.
>
> This is a bigger task than just something that can run Workflow scripts in
> background threads (which isn't terribly difficult to hard code in your
> application by starting a new thread and executing the Activity there),
> but it would be a generally useful package. What do you think?
>
> Craig
>
>
> On Thu, 29 Nov 2001, Brett Gullan wrote:
>
> > Date: Thu, 29 Nov 2001 09:24:55 +0800
> > From: Brett Gullan <br...@mac.com>
> > Reply-To: Jakarta Commons Developers List
<co...@jakarta.apache.org>,
> > brettgullan@mac.com
> > To: 'Jakarta Commons Developers List' <co...@jakarta.apache.org>
> > Subject: RE: Workflow & Turbine design questions
> >
> > Almost...
> >
> > > #1. User starts workflow process by performing action.
> > Let's say User A starts workflow process, by for instance, copying a
> > file from his/her desktop to a directory on a local/remote server (this
> > can/will be achieved by a variety of methods/protocols).
> >
> > > #2. Workflow performs tasks that may take time.
> > Yes
> >
> > > #3. User should not be held up waiting for a response from the server
> > by
> > background workflow tasks.
> > Yes. However, User A may never even use the Web UI. User B on the other
> > hand may be watching the queue/progress monitor for regular updates.
> >
> > The web UI is intended to be a monitoring and configuration tool for the
> > underlying workflow engine. For the most part, general users would
> > submit files (by desktop drag-n-drop, FTP, web-form, etc...) then use
> > the web UI to monitor progress or change the queue priority (if
> > authorised).
> > Administrative users would have access to web forms that configure the
> > workflow processing activity steps and the 'hot' folders that feed the
> > processing queue.
> > The application is intended to support deployment on a 'headless' server
> > where all interaction is via a web UI.
> > I hope that makes more sense?
> > Regards,
> > Brett
> > --
> > Brett Gullan
> > brettgullan@mac.com
> >
> >
> >
> > --
> > To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
> > For additional commands, e-mail:
<ma...@jakarta.apache.org>
> >
> >
>
>
> --
> To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
> For additional commands, e-mail:
<ma...@jakarta.apache.org>
_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Workflow & Turbine design questions
Posted by "Geir Magnusson Jr." <ge...@optonline.net>.
On 11/29/01 9:32 PM, "Craig R. McClanahan" <cr...@apache.org> wrote:
> One way to approach these issues (and would even fit into the Commons
> charter :-) would be to conceive of a Commons package to do background job
> scheduling and execution (what legacy OSs called "batch job queues")
> within a JVM. Executing a particular workflow script would be only one of
> the interesting things you could do -- ideally the definition of a "job"
> would be something like "call method X on object Y" so it could be fully
> generalized.
>
> The "jobs" package could be configured with limits on how many background
> threads it could run, support ways to schedule jobs repeatedly, and all
> the other stuff you can do with "cron" type utilities, if we wanted. It
> could be embedded in any sort of long-running JVM (such as in a webapp on
> a servlet container), as well as stand-alone background jobs.
>
> This is a bigger task than just something that can run Workflow scripts in
> background threads (which isn't terribly difficult to hard code in your
> application by starting a new thread and executing the Activity there),
> but it would be a generally useful package. What do you think?
Certainly useful, but I would think that it must by lying around Jakarta
already. I assume that since you suggested it, it's not in Struts. What
about Turbine or Avalon?
geir
--
Geir Magnusson Jr. geirm@optonline.net
System and Software Consulting
"They that can give up essential liberty to obtain a little temporary safety
deserve neither liberty nor safety." - Benjamin Franklin
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Workflow & Turbine design questions
Posted by James Strachan <ja...@yahoo.co.uk>.
From: "Warren Janssens" <wh...@yahoo.ca>
> One idea would be to have something like javabeans or
> jmx events and listeners.
> Of course it would be nice
> if you could register the event senders and the
> listeners with a configuration file instead of in
> code. The messaging loop would be a nice avalon
> component (or block?).
The Messenger project can use used to do exactly this with JMS. An XML
configuration file can contain JMS subscriptions which when a JMS messsage
arrives can then fire MessageListeners or dispatch to Servlets or JSP.
Its nothing to do with JMX though - maybe Craig's Modeler project might help
do JMX events & listeners
> A file system monitor would be
> just another event sender.
A file system monitor would be a great idea too - I could use one of those.
Should be fairly easy to add a new Task to a ThreadPool / Scheduler, which
would watch for files that have changed since a certain timestamp.
James
_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Workflow & Turbine design questions
Posted by Warren Janssens <wh...@yahoo.ca>.
One idea would be to have something like javabeans or
jmx events and listeners. Of course it would be nice
if you could register the event senders and the
listeners with a configuration file instead of in
code. The messaging loop would be a nice avalon
component (or block?). A file system monitor would be
just another event sender.
--- "Craig R. McClanahan" <cr...@apache.org> wrote:
> One way to approach these issues (and would even fit
> into the Commons
> charter :-) would be to conceive of a Commons
> package to do background job
> scheduling and execution (what legacy OSs called
> "batch job queues")
> within a JVM. Executing a particular workflow
> script would be only one of
> the interesting things you could do -- ideally the
> definition of a "job"
> would be something like "call method X on object Y"
> so it could be fully
> generalized.
>
> The "jobs" package could be configured with limits
> on how many background
> threads it could run, support ways to schedule jobs
> repeatedly, and all
> the other stuff you can do with "cron" type
> utilities, if we wanted. It
> could be embedded in any sort of long-running JVM
> (such as in a webapp on
> a servlet container), as well as stand-alone
> background jobs.
>
> This is a bigger task than just something that can
> run Workflow scripts in
> background threads (which isn't terribly difficult
> to hard code in your
> application by starting a new thread and executing
> the Activity there),
> but it would be a generally useful package. What do
> you think?
>
> Craig
>
>
> On Thu, 29 Nov 2001, Brett Gullan wrote:
>
> > Date: Thu, 29 Nov 2001 09:24:55 +0800
> > From: Brett Gullan <br...@mac.com>
> > Reply-To: Jakarta Commons Developers List
> <co...@jakarta.apache.org>,
> > brettgullan@mac.com
> > To: 'Jakarta Commons Developers List'
> <co...@jakarta.apache.org>
> > Subject: RE: Workflow & Turbine design questions
> >
> > Almost...
> >
> > > #1. User starts workflow process by performing
> action.
> > Let's say User A starts workflow process, by for
> instance, copying a
> > file from his/her desktop to a directory on a
> local/remote server (this
> > can/will be achieved by a variety of
> methods/protocols).
> >
> > > #2. Workflow performs tasks that may take time.
> > Yes
> >
> > > #3. User should not be held up waiting for a
> response from the server
> > by
> > background workflow tasks.
> > Yes. However, User A may never even use the Web
> UI. User B on the other
> > hand may be watching the queue/progress monitor
> for regular updates.
> >
> > The web UI is intended to be a monitoring and
> configuration tool for the
> > underlying workflow engine. For the most part,
> general users would
> > submit files (by desktop drag-n-drop, FTP,
> web-form, etc...) then use
> > the web UI to monitor progress or change the queue
> priority (if
> > authorised).
> > Administrative users would have access to web
> forms that configure the
> > workflow processing activity steps and the 'hot'
> folders that feed the
> > processing queue.
> > The application is intended to support deployment
> on a 'headless' server
> > where all interaction is via a web UI.
> > I hope that makes more sense?
> > Regards,
> > Brett
> > --
> > Brett Gullan
> > brettgullan@mac.com
> >
> >
> >
> > --
> > To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> > For additional commands, e-mail:
> <ma...@jakarta.apache.org>
> >
> >
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
_______________________________________________________
Build your own website in minutes and for free at http://ca.geocities.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Workflow & Turbine design questions
Posted by Paulo Gaspar <pa...@krankikom.de>.
> From: Jon Stevens [mailto:jon@latchkey.com]
> Sent: Saturday, December 01, 2001 10:01 PM
>
> on 12/1/01 8:22 AM, "Paulo Gaspar" <pa...@krankikom.de> wrote:
>
> > There is a CRON like scheduler at jakarta-avalon-cornerstone package:
> > org.apache.avalon.cornerstone.services.scheduler
>
> There is a cron scheduler in the jakarta-turbine-fulcrum package:
> org.apache.fulcrum.services.scheduler
Another scheduler available.
=:o)
> > P.S.: Being a bit more aware of Avalon could save you a lot of work.
>
> P.S.: Being a bit more aware of Turbine could save you a lot of work
> (especially given that Turbine had a scheduler service before Avalon did).
Of course. I am using some code from Turbine too. However, "before" is not
a compelling argument since:
older != better
> > There is a lot of interesting common use code there.
>
> I agree. Turbine does have a lot of interesting common use code.
>
> > I just hope they will follow the trend to cut on that kind of
> > dependencies as they did with the LogKit.
>
> Through the use of an interface which will prevent you from fully
> using all
> the features of a logging package (lots of people have complained
> about that before...personally, I'm for it).
Yes, they have a common interface and adapters for Log4J, LogKit and
the JDK1.4 thing.
I have been using it and I personally like it. IMHO the interface
allows me to use all of the features I need for the logging work.
Keeping the logger configuration/initialization separated I can still
use most of the sophisticated features each Logger has to offer just
by changing the configuration bit.
Have fun,
Paulo Gaspar
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Workflow & Turbine design questions
Posted by Jon Stevens <jo...@latchkey.com>.
on 12/1/01 8:22 AM, "Paulo Gaspar" <pa...@krankikom.de> wrote:
> There is a CRON like scheduler at jakarta-avalon-cornerstone package:
> org.apache.avalon.cornerstone.services.scheduler
There is a cron scheduler in the jakarta-turbine-fulcrum package:
org.apache.fulcrum.services.scheduler
> P.S.: Being a bit more aware of Avalon could save you a lot of work.
P.S.: Being a bit more aware of Turbine could save you a lot of work
(especially given that Turbine had a scheduler service before Avalon did).
> There is a lot of interesting common use code there.
I agree. Turbine does have a lot of interesting common use code.
> I just hope they will follow the trend to cut on that kind of
> dependencies as they did with the LogKit.
Through the use of an interface which will prevent you from fully using all
the features of a logging package (lots of people have complained about that
before...personally, I'm for it).
-jon
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Workflow & Turbine design questions
Posted by Paulo Gaspar <pa...@krankikom.de>.
There is a CRON like scheduler at jakarta-avalon-cornerstone package:
org.apache.avalon.cornerstone.services.scheduler
It even includes a CRON service and it was quite easy to make it
independent from Avalon and adapt it to something like what you are
describing - including having it executing a Runnable or some similar
interface.
My implementation executes jobs using a thread pool as the one you
can find at the scratchpad of jakarta-avalon-excalibur at
\src\scratchpad\org\apache\avalon\excalibur\thread\impl
(so, package org.apache.avalon.excalibur.thread.impl).
I do not use Avalon's pool. I use an evolution of the Commons pool
instead, and it was not hard to adapt.
I made the all thing work in a weekend and I was still learning a lot
about multithreading in the process.
Another very interesting mechanism is the asynchronous notifications
in the threading project of commons-sandbox... but I am sure you know
that one. =:o) This could be used for the already suggested JMS
alternative. Its notification mechanism queue could be a nice message
sender.
Now, my implementation, although still a bit simplistic, works. Maybe
I can contribute it if someone wants to pick it but I am not in the
mood to go trough the politics of it. I only go trough the politics
of things that I can not implement this fast on my own - which is a
simple time efficiency oriented criteria.
So, I can post it and let you decide what to do with it IF you have
any interest on it.
I am adding the "IF you have interest" clause instead of just posting
it because it still involves the work of cleaning possible dependencies
to my framework and making sure it still works afterwards and this will
still take me some hours.
If you do not have interest on my code, you can always use the above
mentioned packages as I did to achieve the same thing. At least it is
a path that worked for me.
Have fun,
Paulo Gaspar
P.S.: Being a bit more aware of Avalon could save you a lot of work.
There is a lot of interesting common use code there.
I also think that create too many dependencies between some of their
common stuff and their framework structure, but it is still worth the
trouble.
I just hope they will follow the trend to cut on that kind of
dependencies as they did with the LogKit.
> -----Original Message-----
> From: Craig R. McClanahan
> Sent: Friday, November 30, 2001 3:33 AM
>
> One way to approach these issues (and would even fit into the Commons
> charter :-) would be to conceive of a Commons package to do background job
> scheduling and execution (what legacy OSs called "batch job queues")
> within a JVM. Executing a particular workflow script would be only one of
> the interesting things you could do -- ideally the definition of a "job"
> would be something like "call method X on object Y" so it could be fully
> generalized.
>
> The "jobs" package could be configured with limits on how many background
> threads it could run, support ways to schedule jobs repeatedly, and all
> the other stuff you can do with "cron" type utilities, if we wanted. It
> could be embedded in any sort of long-running JVM (such as in a webapp on
> a servlet container), as well as stand-alone background jobs.
>
> This is a bigger task than just something that can run Workflow scripts in
> background threads (which isn't terribly difficult to hard code in your
> application by starting a new thread and executing the Activity there),
> but it would be a generally useful package. What do you think?
>
> Craig
>
>
> On Thu, 29 Nov 2001, Brett Gullan wrote:
>
> > Date: Thu, 29 Nov 2001 09:24:55 +0800
> > From: Brett Gullan <br...@mac.com>
> > Reply-To: Jakarta Commons Developers List
> <co...@jakarta.apache.org>,
> > brettgullan@mac.com
> > To: 'Jakarta Commons Developers List' <co...@jakarta.apache.org>
> > Subject: RE: Workflow & Turbine design questions
> >
> > Almost...
> >
> > > #1. User starts workflow process by performing action.
> > Let's say User A starts workflow process, by for instance, copying a
> > file from his/her desktop to a directory on a local/remote server (this
> > can/will be achieved by a variety of methods/protocols).
> >
> > > #2. Workflow performs tasks that may take time.
> > Yes
> >
> > > #3. User should not be held up waiting for a response from the server
> > by
> > background workflow tasks.
> > Yes. However, User A may never even use the Web UI. User B on the other
> > hand may be watching the queue/progress monitor for regular updates.
> >
> > The web UI is intended to be a monitoring and configuration tool for the
> > underlying workflow engine. For the most part, general users would
> > submit files (by desktop drag-n-drop, FTP, web-form, etc...) then use
> > the web UI to monitor progress or change the queue priority (if
> > authorised).
> > Administrative users would have access to web forms that configure the
> > workflow processing activity steps and the 'hot' folders that feed the
> > processing queue.
> > The application is intended to support deployment on a 'headless' server
> > where all interaction is via a web UI.
> > I hope that makes more sense?
> > Regards,
> > Brett
> > --
> > Brett Gullan
> > brettgullan@mac.com
> >
> >
> >
> > --
> > To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
> For additional commands, e-mail:
<ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Workflow & Turbine design questions
Posted by Scott Sanders <sa...@stonecobra.com>.
I would say that this would be an excellent addition to Avalon, if it is
not already there, or could be built in commons. I think running this
as a JMX MBean would be great for the likes of JBoss. Maybe JBoss
already has one?
Scott Sanders
-----Original Message-----
From: craigmcc@localhost [mailto:craigmcc@localhost] On Behalf Of Craig
R. McClanahan
Sent: Thursday, November 29, 2001 6:33 PM
To: Jakarta Commons Developers List; brettgullan@mac.com
Subject: RE: Workflow & Turbine design questions
One way to approach these issues (and would even fit into the Commons
charter :-) would be to conceive of a Commons package to do background
job
scheduling and execution (what legacy OSs called "batch job queues")
within a JVM. Executing a particular workflow script would be only one
of
the interesting things you could do -- ideally the definition of a "job"
would be something like "call method X on object Y" so it could be fully
generalized.
The "jobs" package could be configured with limits on how many
background
threads it could run, support ways to schedule jobs repeatedly, and all
the other stuff you can do with "cron" type utilities, if we wanted. It
could be embedded in any sort of long-running JVM (such as in a webapp
on
a servlet container), as well as stand-alone background jobs.
This is a bigger task than just something that can run Workflow scripts
in
background threads (which isn't terribly difficult to hard code in your
application by starting a new thread and executing the Activity there),
but it would be a generally useful package. What do you think?
Craig
On Thu, 29 Nov 2001, Brett Gullan wrote:
> Date: Thu, 29 Nov 2001 09:24:55 +0800
> From: Brett Gullan <br...@mac.com>
> Reply-To: Jakarta Commons Developers List
<co...@jakarta.apache.org>,
> brettgullan@mac.com
> To: 'Jakarta Commons Developers List' <co...@jakarta.apache.org>
> Subject: RE: Workflow & Turbine design questions
>
> Almost...
>
> > #1. User starts workflow process by performing action.
> Let's say User A starts workflow process, by for instance, copying a
> file from his/her desktop to a directory on a local/remote server
(this
> can/will be achieved by a variety of methods/protocols).
>
> > #2. Workflow performs tasks that may take time.
> Yes
>
> > #3. User should not be held up waiting for a response from the
server
> by
> background workflow tasks.
> Yes. However, User A may never even use the Web UI. User B on the
other
> hand may be watching the queue/progress monitor for regular updates.
>
> The web UI is intended to be a monitoring and configuration tool for
the
> underlying workflow engine. For the most part, general users would
> submit files (by desktop drag-n-drop, FTP, web-form, etc...) then use
> the web UI to monitor progress or change the queue priority (if
> authorised).
> Administrative users would have access to web forms that configure the
> workflow processing activity steps and the 'hot' folders that feed the
> processing queue.
> The application is intended to support deployment on a 'headless'
server
> where all interaction is via a web UI.
> I hope that makes more sense?
> Regards,
> Brett
> --
> Brett Gullan
> brettgullan@mac.com
>
>
>
> --
> To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
> For additional commands, e-mail:
<ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Workflow & Turbine design questions
Posted by "Craig R. McClanahan" <cr...@apache.org>.
One way to approach these issues (and would even fit into the Commons
charter :-) would be to conceive of a Commons package to do background job
scheduling and execution (what legacy OSs called "batch job queues")
within a JVM. Executing a particular workflow script would be only one of
the interesting things you could do -- ideally the definition of a "job"
would be something like "call method X on object Y" so it could be fully
generalized.
The "jobs" package could be configured with limits on how many background
threads it could run, support ways to schedule jobs repeatedly, and all
the other stuff you can do with "cron" type utilities, if we wanted. It
could be embedded in any sort of long-running JVM (such as in a webapp on
a servlet container), as well as stand-alone background jobs.
This is a bigger task than just something that can run Workflow scripts in
background threads (which isn't terribly difficult to hard code in your
application by starting a new thread and executing the Activity there),
but it would be a generally useful package. What do you think?
Craig
On Thu, 29 Nov 2001, Brett Gullan wrote:
> Date: Thu, 29 Nov 2001 09:24:55 +0800
> From: Brett Gullan <br...@mac.com>
> Reply-To: Jakarta Commons Developers List <co...@jakarta.apache.org>,
> brettgullan@mac.com
> To: 'Jakarta Commons Developers List' <co...@jakarta.apache.org>
> Subject: RE: Workflow & Turbine design questions
>
> Almost...
>
> > #1. User starts workflow process by performing action.
> Let's say User A starts workflow process, by for instance, copying a
> file from his/her desktop to a directory on a local/remote server (this
> can/will be achieved by a variety of methods/protocols).
>
> > #2. Workflow performs tasks that may take time.
> Yes
>
> > #3. User should not be held up waiting for a response from the server
> by
> background workflow tasks.
> Yes. However, User A may never even use the Web UI. User B on the other
> hand may be watching the queue/progress monitor for regular updates.
>
> The web UI is intended to be a monitoring and configuration tool for the
> underlying workflow engine. For the most part, general users would
> submit files (by desktop drag-n-drop, FTP, web-form, etc...) then use
> the web UI to monitor progress or change the queue priority (if
> authorised).
> Administrative users would have access to web forms that configure the
> workflow processing activity steps and the 'hot' folders that feed the
> processing queue.
> The application is intended to support deployment on a 'headless' server
> where all interaction is via a web UI.
> I hope that makes more sense?
> Regards,
> Brett
> --
> Brett Gullan
> brettgullan@mac.com
>
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Workflow & Turbine design questions
Posted by Brett Gullan <br...@mac.com>.
Almost...
> #1. User starts workflow process by performing action.
Let's say User A starts workflow process, by for instance, copying a
file from his/her desktop to a directory on a local/remote server (this
can/will be achieved by a variety of methods/protocols).
> #2. Workflow performs tasks that may take time.
Yes
> #3. User should not be held up waiting for a response from the server
by
background workflow tasks.
Yes. However, User A may never even use the Web UI. User B on the other
hand may be watching the queue/progress monitor for regular updates.
The web UI is intended to be a monitoring and configuration tool for the
underlying workflow engine. For the most part, general users would
submit files (by desktop drag-n-drop, FTP, web-form, etc...) then use
the web UI to monitor progress or change the queue priority (if
authorised).
Administrative users would have access to web forms that configure the
workflow processing activity steps and the 'hot' folders that feed the
processing queue.
The application is intended to support deployment on a 'headless' server
where all interaction is via a web UI.
I hope that makes more sense?
Regards,
Brett
--
Brett Gullan
brettgullan@mac.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Workflow & Turbine design questions
Posted by Jon Stevens <jo...@latchkey.com>.
on 11/28/01 4:57 PM, "Brett Gullan" <br...@mac.com> wrote:
> Thanks Jon,
>
> Would you have any ideas as to how the processing thread might be
> integrated with the Workflow Context and Scope interfaces?
> I need to allow for the fact that in most cases, the web UI would be
> reporting on the activity of the server-based processing. This
> processing would not typically be invoked from the web session; rather
> it is the result of another user or process moving a file into a 'hot'
> directory.
Let me see if I hear you correctly:
#1. User starts workflow process by performing action.
#2. Workflow performs tasks that may take time.
#3. User should not be held up waiting for a response from the server by
background workflow tasks.
So, what the user does is spawn a background thread that executes the
workflow.
The workflow then updates a database table with regards to its status as
things process through the workflow.
User can then query the database table to find out how far along the
workflow process is.
> P.S: If I get this working, I'd be happy to contribute it!
Great!
-jon
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Workflow & Turbine design questions
Posted by Brett Gullan <br...@mac.com>.
Thanks Jon,
Would you have any ideas as to how the processing thread might be
integrated with the Workflow Context and Scope interfaces?
I need to allow for the fact that in most cases, the web UI would be
reporting on the activity of the server-based processing. This
processing would not typically be invoked from the web session; rather
it is the result of another user or process moving a file into a 'hot'
directory.
Regards,
Brett
P.S: If I get this working, I'd be happy to contribute it!
--
Brett Gullan
brettgullan@mac.com
-----Original Message-----
From: Jon Stevens [mailto:jon@latchkey.com]
Sent: Thursday, 29 November 2001 3:33 AM
To: Jakarta Commons Developers List
Subject: Re: Workflow & Turbine design questions
Woah! I hope you decide to contribute your service back to the Turbine
group. :-) At some point soon, I'm going to need something very similar
for
Scarab...
Brain dump:
That said, one way that you can do things in a web environment is to
spawn a
thread to do the processing and not make the user wait. Since web
environments are very disconnected (ie: you don't know if the user just
closed their browser window)...
Then, when the thread is finished, an email is sent to the user to let
them
know that things have finished. The process can also do things like
update a
database table which can be read each time the user is accesses a page
in
the site. That table would be able to maintain the state of what is
going on
with regards to the processing and relay that to the user as they browse
the
site.
I hope that helps a bit...
-jon
on 11/27/01 7:11 PM, "Brett Gullan" <br...@mac.com> wrote:
>
> I am working on a project to develop a Turbine-based workflow engine
for
> file processing, using the Commons Workflow package and I would be
> interested in any feedback or suggestions as to how best to implement
> this.
> The main focus of the project is to automate file processing for
> print/publishing workflow. Users move/copy files into predefined
> "hotfolders" which trigger a series of workflow steps such as
converting
> a file from PostScript to PDF, compressing images or transferring a
file
> or directory to a remote server (via FTP, HTTP, WebDAV, etc...).
> Currently I have created a WorkflowService, modelled after the Turbine
> SchedulerService that retrieves items from a queue and initiates the
> processing sequence. A scheduled job periodically retrieves the list
of
> hotfolders (from storage; they are implemented as OM/Peers), checks
for
> any new files and submits them to the workflow queue, along with a
> reference to an Activity definition that will be used to process the
> files.
> My intention is that as each entry is retrieved from the queue the
> associated Activity definition file is 'digested' and a processing
> thread started to perform the workflow steps.
> I also need to ensure there is a mechanism for maintaining
> state/progress information for each workflow step -- both to report
back
> to the client UI and for 'recovery' should a process be interrupted.
> Workflow steps may be quite expensive and/or time consuming -- for
> instance, PDF conversion of large publishing files typically takes 1-5
> minutes, thus processing a directory of files could take significantly
> longer. During this period a user needs to see progress information
such
> as a progress bar or estimated time to completion.
> I am unclear at this stage as to how the workflow Context and Scope
fit
> in. I figure the processing thread itself should probably be a Context
> implementation and maybe the Scope implementation is the conduit to
the
> UI? A Velocity Screen/Action could request a list of processing
> threads/Contexts from the WorkflowService and insert each Scope
> implementation into the Velocity Context? The Scope would probably
also
> need to be backed by an OM/Peer to persist state information.
> I would appreciate any suggestions or alternative design ideas.
> Thanks,
> Brett
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Workflow & Turbine design questions
Posted by Jon Stevens <jo...@latchkey.com>.
Woah! I hope you decide to contribute your service back to the Turbine
group. :-) At some point soon, I'm going to need something very similar for
Scarab...
Brain dump:
That said, one way that you can do things in a web environment is to spawn a
thread to do the processing and not make the user wait. Since web
environments are very disconnected (ie: you don't know if the user just
closed their browser window)...
Then, when the thread is finished, an email is sent to the user to let them
know that things have finished. The process can also do things like update a
database table which can be read each time the user is accesses a page in
the site. That table would be able to maintain the state of what is going on
with regards to the processing and relay that to the user as they browse the
site.
I hope that helps a bit...
-jon
on 11/27/01 7:11 PM, "Brett Gullan" <br...@mac.com> wrote:
>
> I am working on a project to develop a Turbine-based workflow engine for
> file processing, using the Commons Workflow package and I would be
> interested in any feedback or suggestions as to how best to implement
> this.
> The main focus of the project is to automate file processing for
> print/publishing workflow. Users move/copy files into predefined
> "hotfolders" which trigger a series of workflow steps such as converting
> a file from PostScript to PDF, compressing images or transferring a file
> or directory to a remote server (via FTP, HTTP, WebDAV, etc...).
> Currently I have created a WorkflowService, modelled after the Turbine
> SchedulerService that retrieves items from a queue and initiates the
> processing sequence. A scheduled job periodically retrieves the list of
> hotfolders (from storage; they are implemented as OM/Peers), checks for
> any new files and submits them to the workflow queue, along with a
> reference to an Activity definition that will be used to process the
> files.
> My intention is that as each entry is retrieved from the queue the
> associated Activity definition file is 'digested' and a processing
> thread started to perform the workflow steps.
> I also need to ensure there is a mechanism for maintaining
> state/progress information for each workflow step -- both to report back
> to the client UI and for 'recovery' should a process be interrupted.
> Workflow steps may be quite expensive and/or time consuming -- for
> instance, PDF conversion of large publishing files typically takes 1-5
> minutes, thus processing a directory of files could take significantly
> longer. During this period a user needs to see progress information such
> as a progress bar or estimated time to completion.
> I am unclear at this stage as to how the workflow Context and Scope fit
> in. I figure the processing thread itself should probably be a Context
> implementation and maybe the Scope implementation is the conduit to the
> UI? A Velocity Screen/Action could request a list of processing
> threads/Contexts from the WorkflowService and insert each Scope
> implementation into the Velocity Context? The Scope would probably also
> need to be backed by an OM/Peer to persist state information.
> I would appreciate any suggestions or alternative design ideas.
> Thanks,
> Brett
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>