[RT] Gump service (was: brutus)

[Michael Davey <Mi...@coderage.org>]

Adam R. B. Jack wrote:

>
>4) Figuring out a way (some service?) to have a continuous Gump. The "gump
>every three hours" line in crontab is a waste of two hours when builds take
>4 hours. I'd like some sort of 'permanent restart' loop.
>  
>
I keep thinking about this (especially as one of my private gumps 
usually takes 25 hours to run).  The absolute simplest way would be to 
simpy have a bash script that calls gumpy.py in an infinate while loop.  
However, I think this would be a very good time to refactor gump into a 
workflow.

Server component to look at the projects and work out which ones Gump 
needs to try to build next.  To start off, the algorithm stays as it is 
now.  In future we can look to detect if there have been CVS commits 
since last build and so on.  Server component puts "n" work pieces into 
a pool.

Service/Client component takes one project out of the pool, performs cvs 
update and attempts to build (again using the current algorithm).  Puts 
the results into a second pool.

View component takes the results from the second pool and produces the 
web site.  Alternatively, the second pool could be replaced with a 
filestore and the view generates the pages on the fly from the 
filestore.  The view component is forrest with a little glue logic.

The nice thing about this is that it opens up many more possibilities 
for the future.  For instance, it would enable distributed gump at the 
client component.

-- 
Michael


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org

Re: [RT] Gump service

[Michael Davey <Mi...@coderage.org>]

Adam R. B. Jack wrote:

[snip]

>>Server component to look at the projects and work out which ones Gump
>>needs to try to build next.  To start off, the algorithm stays as it is
>>now.  In future we can look to detect if there have been CVS commits
>>since last build and so on.  Server component puts "n" work pieces into
>>a pool.
>>    
>>

[snip]

>>Service/Client component takes one project out of the pool, performs cvs
>>update and attempts to build (again using the current algorithm).  Puts
>>the results into a second pool.
>>    
>>
>
>I've been thinking about this, but for threads. I recently split Gump into
>pieces with some for of listener & event/request pattern, so we could
>dispatch to unnamed parties. I reworked a 'runner' so we could have various
>types, my goal is one that take 'next available module or project' from a
>list, so we can have multiple concurrent threads working it.
>
>  
>
>>View component takes the results from the second pool and produces the
>>web site.  Alternatively, the second pool could be replaced with a
>>filestore and the view generates the pages on the fly from the
>>filestore.  The view component is forrest with a little glue logic.
>>    
>>
>
>I agree, something (lower priority?) could be building a WWW site at the
>same time that real biulds are going on.
>
>I keep thinking we could serialize the context (the build information) to a
>file, but I like having lots of information & haven't found time to
>serialzie it all.
>
>  
>
>>The nice thing about this is that it opens up many more possibilities
>>for the future.  For instance, it would enable distributed gump at the
>>client component.
>>    
>>
>
>Yup, very nice.
>
>My personal goals are for cascading gumps, then multi-threaded (to make use
>of the two CPUs on Brutus), and one day, distributed.
>
>Thanks for the thoughts. Now, how do we make the incremental progress to get
>there?
>  
>
I'll respond as no one else has yet.  I am very new to Python - I can 
play about with simple things like checking of executables but pretty 
much anything else is beyond me, so I am unlikely to be able to help 
with coding for this. That said...

You said that you have recently split Gump up into pieces using an 
event/listener pattern.  I infer that the listener works across the 
equivalent of my proposed server/client interface but you don't have an 
equivalent between client and view.  The steps would seem to be:

1a.  Find a way of describing a piece of work to the client (a 'type' as 
you put it - a project or module).  For the moment, I think we can 
assume that server and client have a consistent view of the metadata, so 
just the module or project name should suffice for the moment.

1b.  Rework the runner to work over some form of sockets.  The client 
will request a piece of work, the server will describe the piece of work 
as per [1a] above and mark it as checked out to the client (simply 
remove the work piece from its list in the initial implementation).  
When the server runs out of work pieces, it starts again from the beginning.

2a.  Find a way of describing the results of a build.  We need to decide 
whether only the log files are to be supplied; log files and location of 
the build on the client (assuming it is public access); or log files 
plus the actual build.

2b. The client will contact the view and supply the results of the build 
as per [2a] above and the view will acknowledge receipt.

2c.  Rework the view to understand how to process the results from the 
client.

The actual sockets layer could be implemented using ftp, http, svn or 
our own protocol. 

Later we can add a feedback loop from view to server to let the server 
know that the client has supplied the results and make the mechanism 
more bulletproof by timing out a client if it doesn't supply the results 
in a timely manner (so the work piece can be supplied to a different 
client).  We could also send the work out to multiple clients to ensure 
they agree upon the results.

-- 
Michael


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org

Re: [RT] Gump service (was: brutus)

["Adam R. B. Jack" <aj...@trysybase.com>]

Where to start? Ah yes, loved this mail...

> I keep thinking about this (especially as one of my private gumps
> usually takes 25 hours to run).

My first thougt here is cascading Gumps, the main ones store jars in a
public repository, and downstream Gumps (typically private ones) just
use/download the latest Gumped jars. For personal Gump I think this would be
a huge saving.

> The absolute simplest way would be to
> simpy have a bash script that calls gumpy.py in an infinate while loop.

I'd like one that does N 'optiomized' (not yet coded) runs that skipped
things that had not changed, then one official (build it all anyway) run.
Seems easy to code.

> However, I think this would be a very good time to refactor gump into a
> workflow.

Yup.

> Server component to look at the projects and work out which ones Gump
> needs to try to build next.  To start off, the algorithm stays as it is
> now.  In future we can look to detect if there have been CVS commits
> since last build and so on.  Server component puts "n" work pieces into
> a pool.

I thought we had this 'any changes' covered, but found we did not (quite).

When we do the update from CVS|SVN we attempt to detect if changes have
occured.
We mark modules (and underlying projects) with isUpdated() and could simply
build those. [It isn't that simple, but enough for now.]

For CVS the "-q -n update" (and checking for any output) ought do it ('cos
the output ought list changes, since last run). I added that to the CVS
update part. [I don't like scanning for output to detect changes, but I am
no CVS expert, and I do like that this is 'fail safe' (we don't look for
changes since a given time, we look for sinice last update.) I am open to
suggestions on how to do this.

For SVN, this is not an option (unless one parses the output) since it likes
to tell one the current id, even if in sync. I know I tried the 'svn
status --show-updates' but I don't recall if it was enough to tell me
changes, or if I still needed to parse the output.

So ... I wrote a way to check if the 'sync' command found any difference.
Unfortunately I didn't write it unidirectional (it said 'sure there are N
diferences, that build dir you created, etc. etc.) I could attempt to only
report difference from the remote server's copy (adds/dels, not vice verse).

I basically ran out of time on the problem. I would like to revisit it,
because I think there are two good uses for knowing no changes, with your's
here being the most important.

> Service/Client component takes one project out of the pool, performs cvs
> update and attempts to build (again using the current algorithm).  Puts
> the results into a second pool.

I've been thinking about this, but for threads. I recently split Gump into
pieces with some for of listener & event/request pattern, so we could
dispatch to unnamed parties. I reworked a 'runner' so we could have various
types, my goal is one that take 'next available module or project' from a
list, so we can have multiple concurrent threads working it.

> View component takes the results from the second pool and produces the
> web site.  Alternatively, the second pool could be replaced with a
> filestore and the view generates the pages on the fly from the
> filestore.  The view component is forrest with a little glue logic.

I agree, something (lower priority?) could be building a WWW site at the
same time that real biulds are going on.

I keep thinking we could serialize the context (the build information) to a
file, but I like having lots of information & haven't found time to
serialzie it all.

> The nice thing about this is that it opens up many more possibilities
> for the future.  For instance, it would enable distributed gump at the
> client component.

Yup, very nice.

My personal goals are for cascading gumps, then multi-threaded (to make use
of the two CPUs on Brutus), and one day, distributed.

Thanks for the thoughts. Now, how do we make the incremental progress to get
there?

regards

Adam


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org