Gump3 inter-component orchestration/communications

[Adam Jack <aj...@mric.coop>]

Why are CvsUpdater, SvnUpdater pre-process plug-ins, not 'process' visitors?
Clearly (as of now) it make little difference, I'm just trying to
understand.

I'm not sure I'm comfortable with the three pre-process/process/post-process
walks. Is this a Gump2 hold over that we want in Gump3? I'm not wanting us
to design a generic mechanism, just solve our problem, but is the approach
the right one? Why are these three walks better than (say) putting
Updates/Builders/Databasers(DynaGumper) in sequence inside one mutlicast
plugin?

One side effect of the current approach is that (during a long slow build)
we'll only update the database at the end of the run, after all modules have
been updated and all projects have been built. This was something we found
unpleasant w/ Gump2, 'cos folks want more instant gratification. We ought
update a DB and (say) notify of a failure (e.g. e-mail or RSS) ASAP [i.e. at
t he time of the processing], to give communities the most opportunity to
solve things before the next Gump run.

I like how Gump3 is simple, and fires events out to plugins (letting them
decide if they wish to process them, not centralizing this logic) however I
think that'll put a higher burden onto inter-component orchestration and
communications. I think using the model object as a "message board" by
writing attributes is the right Pythonic way to solve the latter. It is the
former that worries me.

I don't have a solution here, I'm just asking questions...

regards,

Adam


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org

Re: Gump3 inter-component orchestration/communications

["Adam R. B. Jack" <aj...@apache.org>]

> > I'm not wanting us
> > to design a generic mechanism, just solve our problem, but is the
approach
> > the right one? Why are these three walks better than (say) putting
> > Updates/Builders/Databasers(DynaGumper) in sequence inside one mutlicast
> > plugin?
>
> Do you think we could do without multiple stages completely?

I think it just comes down to order. The "walker logic" I've preferred most
is the one that iterates through projects, visiting modules (and
repositiories and workspaces, etc.) on demand (for those projects) and
visiting each plug-in in sequence. A sequence of (1) start time stamp (2)
build (3) end time stamp (4) DB update (5) notify -- seems fine w/o need for
pre or post processing. So, I guess (as of my view right now) "yes, maybe".

That said, I've not thought throguh some of the more "intelligent"
behaviours we'd like, such as aggregations (so we don't spam folks w/
e-mails or RSS/Atom feeds, etc.)

> > One side effect of the current approach is that (during a long slow
build)
> > we'll only update the database at the end of the run, after all modules
have
> > been updated and all projects have been built. This was something we
found
> > unpleasant w/ Gump2, 'cos folks want more instant gratification.
>
> Really? I've always found it really annoying to want to debug something
and
> then seeing that a build is "in progress" meaning part of that tree is
from
> the run before and part of it is new. Very confusing...

Yup, agreed, but that is something I hope we can fix with the Gump3
separation of run from presentation, via DB. Given that we'll uniquely
identify DB updates per run, they ought be able to be made any time, and the
presentation still show "last complete run" w/o confusion.

    ----

This all said, I'm in no hurry to redesign things and remove the three
phase, I was just wondering. Time will tell.

BTW: One last thought .... property bloat leading to memory bloat. I've
found with Python (for Gump2) that object/property bloat can really clog
things up. Sure, a lot of it was the DOM tree (and those are being pruned)
but I do think we'll hit this later, with full runs. I'm not saying we ought
think about this now (I just have scars from months of fighting performance,
so can't forget) but we might want some sort of Reaper plug-in that keeps us
lean. That'd need to run after all plug-ins for a project. Again, time will
tell...

regards


Adam


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org

Re: Gump3 inter-component orchestration/communications

[Leo Simons <ma...@leosimons.com>]

On 11-04-2005 01:19, "Adam  Jack" <aj...@mric.coop> wrote:
> Why are CvsUpdater, SvnUpdater pre-process plug-ins, not 'process' visitors?
> Clearly (as of now) it make little difference, I'm just trying to
> understand.

I don't know. It's a good question!

> I'm not sure I'm comfortable with the three pre-process/process/post-process
> walks. Is this a Gump2 hold over that we want in Gump3?

Well, what I tried to do so far is just make it easier to change these
things. Right now, making CvsUpdate part of "process" instead of
"pre-process" is something like a three-line change to the config.py file.
Beyond that, I don't have good answers to many of these kinds of questions,
other than "its a hunch".

> I'm not wanting us
> to design a generic mechanism, just solve our problem, but is the approach
> the right one? Why are these three walks better than (say) putting
> Updates/Builders/Databasers(DynaGumper) in sequence inside one mutlicast
> plugin?

Do you think we could do without multiple stages completely?

I don't remember exactly why I put in 3 stages, but I think we discussed it
a little before. Maybe indeed simply because gump2 has 3 stages, I didn't
see much wrong with it, so I kept it that way. If it ain't broke don't fix
it.

Here's some test driven design for you: we don't have a test case that's
going to fail if we make it into one stage only, so we probably only need
one :-D

> One side effect of the current approach is that (during a long slow build)
> we'll only update the database at the end of the run, after all modules have
> been updated and all projects have been built. This was something we found
> unpleasant w/ Gump2, 'cos folks want more instant gratification.

Really? I've always found it really annoying to want to debug something and
then seeing that a build is "in progress" meaning part of that tree is from
the run before and part of it is new. Very confusing...

> We ought
> update a DB and (say) notify of a failure (e.g. e-mail or RSS) ASAP [i.e. at
> t he time of the processing], to give communities the most opportunity to
> solve things before the next Gump run.

Like before, I agree with you here, but it makes a lot of stuff more
complex. For example, its difficult to report on all the side effects of a
failure if those side effects haven't actually occurred yet. You shouldn't
tell project A that project B has failed because it failed without providing
a link detailing the failure of project B showing how B depends on A.

Anyway, you can have a database plugin at the end of the pre-process stage
that puts info into the database about the results of pre-processing, then
another database plugin at the end of the processing stage about the results
of the processing, then another database plugin at the end of the
post-processing stage that handles all the "linking" bits as in the example
above. And you can move plugins from one stage to another, or "soft-merge"
them using a multicaster or something similar.

Anyway, maybe that's why I put in three distinct stages. You can define lots
of simple rules such as

"after pre-processing is done, projects that don't have the nobuild property
set are guaranteed to be current"

"post-processing plugins can access all model information and all context
information added by all pre-processing and processing plugins"

"plugins in different stages should interact exclusively through the model"

"post-processing plugins should not interact with each other"

"post-processing plugins should not modify the model"

"pre-processing nor processing plugins should query the database"

Which are tied to a particular stage and are more important conceptually
than implementation-wise. That makes plugin writing easier. Of course, those
rules need to be defined, and we need to figure out which ones make sense.
Not even talking about testing them or gently guiding plugin writers to use
them. See how complex a real simple plugin model already is?

> I like how Gump3 is simple, and fires events out to plugins (letting them
> decide if they wish to process them, not centralizing this logic) however I
> think that'll put a higher burden onto inter-component orchestration and
> communications. I think using the model object as a "message board"
                                                      ^^^
That's a cool way to put it. Lot better than referring to things like
"inter-process communication" or other ugly stuff ;)

> by
> writing attributes is the right Pythonic way to solve the latter. It is the
> former that worries me.

Here's a hunch: provided plugins are small and most model-independent
functionality ends up in isolated helper classes/methods, I believe we don't
need much "inter-component orchestration" at all. Or in TDD speak: gimme a
test case that shows we do :-D

> I don't have a solution here, I'm just asking questions...

And they're probably all the right questions! I don't have the answers
either. However, I'm confident that the current setup is flexible enough to
experiment with different answers to these questions. We can just try them
out, having requirements and plugins drop out of our use cases and unit
tests.

It's a real fun way of doing this imo: always just go ahead and make change
x, *provided* it results in

A) simpler code
   - and/or -
B) fewer failing unit tests

And the reverse, if someone disagrees with change x they should either
provide even simpler code or a failing unit test.


Phew, long e-mail! You make me think hard... :-D

Cheers,

Leo



---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org