You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Michael Starch <st...@umich.edu> on 2014/08/06 17:29:47 UTC
Multiple Processing Paradigms at Once
All,
I am working on upgrading OODT to allow it to process streaming data,
alongside traditional non-streaming jobs. This means that some jobs need
to be run by the resource manager, and other jobs need to be submitted to
the stream-processing. Therefore, processing needs to be forked or
multiplexed at some point in the life-cycle.
There are two places where this can be done: workflow manager runners, and
the resource manager. Currently, I am working on building workflow
runners, and doing the job-multiplexing there because this cuts out one
superfluous step for the streaming jobs (namely going to the resource
manager before being routed).
Are there any comments on this approach or does this approach make sense?
-Michael Starch
Re: Multiple Processing Paradigms at Once
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Thanks for the pointer and article reference, Bruce!
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Bruce Barkstrom <br...@gmail.com>
Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Date: Wednesday, August 6, 2014 8:52 AM
To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Subject: Re: Multiple Processing Paradigms at Once
>I don't have a deep enough insight into the OODT architecture
>to know for sure how that would work. From an architectural
>point of view, it would be useful to have UML synchronization
>diagrams that show the threads of computation and the message
>passing that you need to undertake for this work. It's a familiar
>problem in the embedded systems world (avionics, multi-computer
>automobile control systems with 80 computers per car). A good
>reference on this kind of design work is Bruce Douglass' "Real-Time
>UML". It might also be useful to find a copy of Klein, et al., 1993:
>"A Practicioner's Handbook for Real-Time Analysis: Guide to
>Rate Monotonic Analysis for Real-Time Systems", Kluwer.
>It isn't clear whether you want to do the design for the hard
>real-time kind of system (where the system fails if messages
>aren't received within a set priority) or less stringent kinds of
>systems. Exception handling gets to be a really "interesting"
>part of these kinds of designs.
>
>Bruce B.
>
>
>On Wed, Aug 6, 2014 at 11:29 AM, Michael Starch <st...@umich.edu>
>wrote:
>
>> All,
>>
>> I am working on upgrading OODT to allow it to process streaming data,
>> alongside traditional non-streaming jobs. This means that some jobs
>>need
>> to be run by the resource manager, and other jobs need to be submitted
>>to
>> the stream-processing. Therefore, processing needs to be forked or
>> multiplexed at some point in the life-cycle.
>>
>> There are two places where this can be done: workflow manager runners,
>>and
>> the resource manager. Currently, I am working on building workflow
>> runners, and doing the job-multiplexing there because this cuts out one
>> superfluous step for the streaming jobs (namely going to the resource
>> manager before being routed).
>>
>> Are there any comments on this approach or does this approach make
>>sense?
>>
>> -Michael Starch
>>
Re: Multiple Processing Paradigms at Once
Posted by Bruce Barkstrom <br...@gmail.com>.
I don't have a deep enough insight into the OODT architecture
to know for sure how that would work. From an architectural
point of view, it would be useful to have UML synchronization
diagrams that show the threads of computation and the message
passing that you need to undertake for this work. It's a familiar
problem in the embedded systems world (avionics, multi-computer
automobile control systems with 80 computers per car). A good
reference on this kind of design work is Bruce Douglass' "Real-Time
UML". It might also be useful to find a copy of Klein, et al., 1993:
"A Practicioner's Handbook for Real-Time Analysis: Guide to
Rate Monotonic Analysis for Real-Time Systems", Kluwer.
It isn't clear whether you want to do the design for the hard
real-time kind of system (where the system fails if messages
aren't received within a set priority) or less stringent kinds of
systems. Exception handling gets to be a really "interesting"
part of these kinds of designs.
Bruce B.
On Wed, Aug 6, 2014 at 11:29 AM, Michael Starch <st...@umich.edu> wrote:
> All,
>
> I am working on upgrading OODT to allow it to process streaming data,
> alongside traditional non-streaming jobs. This means that some jobs need
> to be run by the resource manager, and other jobs need to be submitted to
> the stream-processing. Therefore, processing needs to be forked or
> multiplexed at some point in the life-cycle.
>
> There are two places where this can be done: workflow manager runners, and
> the resource manager. Currently, I am working on building workflow
> runners, and doing the job-multiplexing there because this cuts out one
> superfluous step for the streaming jobs (namely going to the resource
> manager before being routed).
>
> Are there any comments on this approach or does this approach make sense?
>
> -Michael Starch
>
Re: Multiple Processing Paradigms at Once
Posted by Michael Starch <st...@umich.edu>.
Chris,
This makes sense to me. Do you see any problems with a BatchMgr
implementation that submits to children BatchMgr implementations? This
would allow for users to mux to any BatchMgrs that they choose.
-Michael
On Wed, Aug 6, 2014 at 9:35 AM, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Gotcha, thanks Mike, great job and glad you are using the trunk.
> I think it may make sense to do this as a BatchMgr implementation
> rather than a Runner.
>
> The big reason is that the whole intent of obfuscating the where/how
> something is run is architecturally supposed to be the job of the
> Resource Manager. In Wengine this was blurred a bit, and the Runner
> framework is great that was developed there, but I'd like to get to
> pushing this functionality into the Resource Manager. IOW, to me
> it makes sense to update the Resource Manager to talk to Mesos, not
> the Workflow Manager (which shouldn't care). Does that make sense?
>
> Cheers,
> Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Michael Starch <st...@umich.edu>
> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Date: Wednesday, August 6, 2014 9:22 AM
> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Subject: Re: Multiple Processing Paradigms at Once
>
> >Chris,
> >
> >I did find the Runners in the trunk, and used them for the prototype piece
> >to submit to Spark. Specifically, "
> >
> http://svn.apache.org/repos/asf/oodt/branches/wengine-branch/wengine/src/m
> >ain/java/org/apache/oodt/cas/workflow/engine/runner/MappedMultiRunner.java
> >has yet to be ported over to the trunk, and I am wondering if there is a
> >better solution than to using a Runner to submit to multiple Runners.
> >
> >
> >-Michael
> >
> >
> >On Wed, Aug 6, 2014 at 9:08 AM, Mattmann, Chris A (3980) <
> >chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> >> See OODT-215 and OODT-491 the functionality is not absent it is present
> >> and those issues are the current status
> >>
> >> Sent from my iPhone
> >>
> >> > On Aug 6, 2014, at 8:53 AM, "Michael Starch" <st...@umich.edu>
> >>wrote:
> >> >
> >> > Chris,
> >> >
> >> > I believe I am working on the truck of OODT. The WEngine branch
> >>already
> >> > does what I need it to, however; the absence of this functionality in
> >> trunk
> >> > led me to wonder if this approach is undesired...hence this question.
> >> >
> >> > -Michael
> >> >
> >> >
> >> > On Wed, Aug 6, 2014 at 8:50 AM, Chris Mattmann
> >><chris.mattmann@gmail.com
> >> >
> >> > wrote:
> >> >
> >> >> Hi Mike,
> >> >>
> >> >> I think you are using the wengine branch of Apache OODT.
> >> >> That is unmaintained. I would sincerely urge you to get
> >> >> this working in trunk, that's where the developers are working
> >> >> right now.
> >> >>
> >> >> Cheers,
> >> >> Chris
> >> >>
> >> >> P.S. Let me think more about the below I have some ideas there.
> >> >>
> >> >> ------------------------
> >> >> Chris Mattmann
> >> >> chris.mattmann@gmail.com
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> -----Original Message-----
> >> >> From: Michael Starch <st...@umich.edu>
> >> >> Reply-To: <de...@oodt.apache.org>
> >> >> Date: Wednesday, August 6, 2014 8:29 AM
> >> >> To: <de...@oodt.apache.org>
> >> >> Subject: Multiple Processing Paradigms at Once
> >> >>
> >> >>> All,
> >> >>>
> >> >>> I am working on upgrading OODT to allow it to process streaming
> >>data,
> >> >>> alongside traditional non-streaming jobs. This means that some jobs
> >> need
> >> >>> to be run by the resource manager, and other jobs need to be
> >>submitted
> >> to
> >> >>> the stream-processing. Therefore, processing needs to be forked or
> >> >>> multiplexed at some point in the life-cycle.
> >> >>>
> >> >>> There are two places where this can be done: workflow manager
> >>runners,
> >> and
> >> >>> the resource manager. Currently, I am working on building workflow
> >> >>> runners, and doing the job-multiplexing there because this cuts out
> >>one
> >> >>> superfluous step for the streaming jobs (namely going to the
> >>resource
> >> >>> manager before being routed).
> >> >>>
> >> >>> Are there any comments on this approach or does this approach make
> >> sense?
> >> >>>
> >> >>> -Michael Starch
> >> >>
> >> >>
> >> >>
> >>
>
>
Re: Multiple Processing Paradigms at Once
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Gotcha, thanks Mike, great job and glad you are using the trunk.
I think it may make sense to do this as a BatchMgr implementation
rather than a Runner.
The big reason is that the whole intent of obfuscating the where/how
something is run is architecturally supposed to be the job of the
Resource Manager. In Wengine this was blurred a bit, and the Runner
framework is great that was developed there, but I'd like to get to
pushing this functionality into the Resource Manager. IOW, to me
it makes sense to update the Resource Manager to talk to Mesos, not
the Workflow Manager (which shouldn't care). Does that make sense?
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Michael Starch <st...@umich.edu>
Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Date: Wednesday, August 6, 2014 9:22 AM
To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Subject: Re: Multiple Processing Paradigms at Once
>Chris,
>
>I did find the Runners in the trunk, and used them for the prototype piece
>to submit to Spark. Specifically, "
>http://svn.apache.org/repos/asf/oodt/branches/wengine-branch/wengine/src/m
>ain/java/org/apache/oodt/cas/workflow/engine/runner/MappedMultiRunner.java
>has yet to be ported over to the trunk, and I am wondering if there is a
>better solution than to using a Runner to submit to multiple Runners.
>
>
>-Michael
>
>
>On Wed, Aug 6, 2014 at 9:08 AM, Mattmann, Chris A (3980) <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> See OODT-215 and OODT-491 the functionality is not absent it is present
>> and those issues are the current status
>>
>> Sent from my iPhone
>>
>> > On Aug 6, 2014, at 8:53 AM, "Michael Starch" <st...@umich.edu>
>>wrote:
>> >
>> > Chris,
>> >
>> > I believe I am working on the truck of OODT. The WEngine branch
>>already
>> > does what I need it to, however; the absence of this functionality in
>> trunk
>> > led me to wonder if this approach is undesired...hence this question.
>> >
>> > -Michael
>> >
>> >
>> > On Wed, Aug 6, 2014 at 8:50 AM, Chris Mattmann
>><chris.mattmann@gmail.com
>> >
>> > wrote:
>> >
>> >> Hi Mike,
>> >>
>> >> I think you are using the wengine branch of Apache OODT.
>> >> That is unmaintained. I would sincerely urge you to get
>> >> this working in trunk, that's where the developers are working
>> >> right now.
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> P.S. Let me think more about the below I have some ideas there.
>> >>
>> >> ------------------------
>> >> Chris Mattmann
>> >> chris.mattmann@gmail.com
>> >>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Michael Starch <st...@umich.edu>
>> >> Reply-To: <de...@oodt.apache.org>
>> >> Date: Wednesday, August 6, 2014 8:29 AM
>> >> To: <de...@oodt.apache.org>
>> >> Subject: Multiple Processing Paradigms at Once
>> >>
>> >>> All,
>> >>>
>> >>> I am working on upgrading OODT to allow it to process streaming
>>data,
>> >>> alongside traditional non-streaming jobs. This means that some jobs
>> need
>> >>> to be run by the resource manager, and other jobs need to be
>>submitted
>> to
>> >>> the stream-processing. Therefore, processing needs to be forked or
>> >>> multiplexed at some point in the life-cycle.
>> >>>
>> >>> There are two places where this can be done: workflow manager
>>runners,
>> and
>> >>> the resource manager. Currently, I am working on building workflow
>> >>> runners, and doing the job-multiplexing there because this cuts out
>>one
>> >>> superfluous step for the streaming jobs (namely going to the
>>resource
>> >>> manager before being routed).
>> >>>
>> >>> Are there any comments on this approach or does this approach make
>> sense?
>> >>>
>> >>> -Michael Starch
>> >>
>> >>
>> >>
>>
Re: Multiple Processing Paradigms at Once
Posted by Michael Starch <st...@umich.edu>.
Chris,
I did find the Runners in the trunk, and used them for the prototype piece
to submit to Spark. Specifically, "
http://svn.apache.org/repos/asf/oodt/branches/wengine-branch/wengine/src/main/java/org/apache/oodt/cas/workflow/engine/runner/MappedMultiRunner.java
has yet to be ported over to the trunk, and I am wondering if there is a
better solution than to using a Runner to submit to multiple Runners.
-Michael
On Wed, Aug 6, 2014 at 9:08 AM, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> See OODT-215 and OODT-491 the functionality is not absent it is present
> and those issues are the current status
>
> Sent from my iPhone
>
> > On Aug 6, 2014, at 8:53 AM, "Michael Starch" <st...@umich.edu> wrote:
> >
> > Chris,
> >
> > I believe I am working on the truck of OODT. The WEngine branch already
> > does what I need it to, however; the absence of this functionality in
> trunk
> > led me to wonder if this approach is undesired...hence this question.
> >
> > -Michael
> >
> >
> > On Wed, Aug 6, 2014 at 8:50 AM, Chris Mattmann <chris.mattmann@gmail.com
> >
> > wrote:
> >
> >> Hi Mike,
> >>
> >> I think you are using the wengine branch of Apache OODT.
> >> That is unmaintained. I would sincerely urge you to get
> >> this working in trunk, that's where the developers are working
> >> right now.
> >>
> >> Cheers,
> >> Chris
> >>
> >> P.S. Let me think more about the below I have some ideas there.
> >>
> >> ------------------------
> >> Chris Mattmann
> >> chris.mattmann@gmail.com
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Michael Starch <st...@umich.edu>
> >> Reply-To: <de...@oodt.apache.org>
> >> Date: Wednesday, August 6, 2014 8:29 AM
> >> To: <de...@oodt.apache.org>
> >> Subject: Multiple Processing Paradigms at Once
> >>
> >>> All,
> >>>
> >>> I am working on upgrading OODT to allow it to process streaming data,
> >>> alongside traditional non-streaming jobs. This means that some jobs
> need
> >>> to be run by the resource manager, and other jobs need to be submitted
> to
> >>> the stream-processing. Therefore, processing needs to be forked or
> >>> multiplexed at some point in the life-cycle.
> >>>
> >>> There are two places where this can be done: workflow manager runners,
> and
> >>> the resource manager. Currently, I am working on building workflow
> >>> runners, and doing the job-multiplexing there because this cuts out one
> >>> superfluous step for the streaming jobs (namely going to the resource
> >>> manager before being routed).
> >>>
> >>> Are there any comments on this approach or does this approach make
> sense?
> >>>
> >>> -Michael Starch
> >>
> >>
> >>
>
Re: Multiple Processing Paradigms at Once
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
See OODT-215 and OODT-491 the functionality is not absent it is present and those issues are the current status
Sent from my iPhone
> On Aug 6, 2014, at 8:53 AM, "Michael Starch" <st...@umich.edu> wrote:
>
> Chris,
>
> I believe I am working on the truck of OODT. The WEngine branch already
> does what I need it to, however; the absence of this functionality in trunk
> led me to wonder if this approach is undesired...hence this question.
>
> -Michael
>
>
> On Wed, Aug 6, 2014 at 8:50 AM, Chris Mattmann <ch...@gmail.com>
> wrote:
>
>> Hi Mike,
>>
>> I think you are using the wengine branch of Apache OODT.
>> That is unmaintained. I would sincerely urge you to get
>> this working in trunk, that's where the developers are working
>> right now.
>>
>> Cheers,
>> Chris
>>
>> P.S. Let me think more about the below I have some ideas there.
>>
>> ------------------------
>> Chris Mattmann
>> chris.mattmann@gmail.com
>>
>>
>>
>>
>> -----Original Message-----
>> From: Michael Starch <st...@umich.edu>
>> Reply-To: <de...@oodt.apache.org>
>> Date: Wednesday, August 6, 2014 8:29 AM
>> To: <de...@oodt.apache.org>
>> Subject: Multiple Processing Paradigms at Once
>>
>>> All,
>>>
>>> I am working on upgrading OODT to allow it to process streaming data,
>>> alongside traditional non-streaming jobs. This means that some jobs need
>>> to be run by the resource manager, and other jobs need to be submitted to
>>> the stream-processing. Therefore, processing needs to be forked or
>>> multiplexed at some point in the life-cycle.
>>>
>>> There are two places where this can be done: workflow manager runners, and
>>> the resource manager. Currently, I am working on building workflow
>>> runners, and doing the job-multiplexing there because this cuts out one
>>> superfluous step for the streaming jobs (namely going to the resource
>>> manager before being routed).
>>>
>>> Are there any comments on this approach or does this approach make sense?
>>>
>>> -Michael Starch
>>
>>
>>
Re: Multiple Processing Paradigms at Once
Posted by Michael Starch <st...@umich.edu>.
Chris,
I believe I am working on the truck of OODT. The WEngine branch already
does what I need it to, however; the absence of this functionality in trunk
led me to wonder if this approach is undesired...hence this question.
-Michael
On Wed, Aug 6, 2014 at 8:50 AM, Chris Mattmann <ch...@gmail.com>
wrote:
> Hi Mike,
>
> I think you are using the wengine branch of Apache OODT.
> That is unmaintained. I would sincerely urge you to get
> this working in trunk, that's where the developers are working
> right now.
>
> Cheers,
> Chris
>
> P.S. Let me think more about the below I have some ideas there.
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Michael Starch <st...@umich.edu>
> Reply-To: <de...@oodt.apache.org>
> Date: Wednesday, August 6, 2014 8:29 AM
> To: <de...@oodt.apache.org>
> Subject: Multiple Processing Paradigms at Once
>
> >All,
> >
> >I am working on upgrading OODT to allow it to process streaming data,
> >alongside traditional non-streaming jobs. This means that some jobs need
> >to be run by the resource manager, and other jobs need to be submitted to
> >the stream-processing. Therefore, processing needs to be forked or
> >multiplexed at some point in the life-cycle.
> >
> >There are two places where this can be done: workflow manager runners, and
> >the resource manager. Currently, I am working on building workflow
> >runners, and doing the job-multiplexing there because this cuts out one
> >superfluous step for the streaming jobs (namely going to the resource
> >manager before being routed).
> >
> >Are there any comments on this approach or does this approach make sense?
> >
> >-Michael Starch
>
>
>
Re: Multiple Processing Paradigms at Once
Posted by Chris Mattmann <ch...@gmail.com>.
Hi Mike,
I think you are using the wengine branch of Apache OODT.
That is unmaintained. I would sincerely urge you to get
this working in trunk, that's where the developers are working
right now.
Cheers,
Chris
P.S. Let me think more about the below I have some ideas there.
------------------------
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: Michael Starch <st...@umich.edu>
Reply-To: <de...@oodt.apache.org>
Date: Wednesday, August 6, 2014 8:29 AM
To: <de...@oodt.apache.org>
Subject: Multiple Processing Paradigms at Once
>All,
>
>I am working on upgrading OODT to allow it to process streaming data,
>alongside traditional non-streaming jobs. This means that some jobs need
>to be run by the resource manager, and other jobs need to be submitted to
>the stream-processing. Therefore, processing needs to be forked or
>multiplexed at some point in the life-cycle.
>
>There are two places where this can be done: workflow manager runners, and
>the resource manager. Currently, I am working on building workflow
>runners, and doing the job-multiplexing there because this cuts out one
>superfluous step for the streaming jobs (namely going to the resource
>manager before being routed).
>
>Are there any comments on this approach or does this approach make sense?
>
>-Michael Starch