You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@gump.apache.org by "Adam R. B. Jack" <aj...@trysybase.com> on 2004/06/17 18:50:54 UTC

Gump work-in-progress (CleanUp branch)

I've been struggling with Gump performance issues for months now. That is
'clogs up' (on large workspaces) has continually hampered usage & (frankly)
my progress .

I've turned over every rock that comes to my mind, but I keep failing. I've
been getting pretty frustrated with Python (I find it somewhat immature in
inspection/tuning) -- I've been very frustrated with my abilities (I likely
don't have the experience with Python to figure things out) -- and I've been
ultra frustrated with environmental issues. Since I can only cause the
slowdown on large projects (small ones go fast) it is those monsters I have
to get inside, and I need to do that remotely.

All this has basically hampered my progress, stopped me moving on with new
features/functions ('cos I clearly didn't have a true grasp of the basics.)

I started to add a bunch of __del__ (destructor) overrides, so I could
explicitly break circular links. I believe Python was failing to garbage
collect because it needed a hand with these. I can't prove that, but I felt
it. So, I started the 'Clean-Up' branch to work on cleaning up memory,
trying to avoid clog up. That is how I started down this path, which is now
quite a lot more than CleanUp...

When I found that the SAX to Python object code was slowed by something
minor I added (for no clear reason) I just lost it! I was building on sand
(at least in my knowledge). So (not that DOM is pretty/thin) but I had to
move to it, to simplify things & get control. Since we streamed SAX into
memory, I saw no harm. [I don't think our problem is memory usage, so much
as memory management.] For once, I felt good about ripping out clever code
and adding pedestrian code ... I just needed controlled.

Along the way I decided that moving to Python 2.3 (the latest/greatest bug
fixed release per Python.org) was smart. If there were GC implementation
fixes, that is where I'd find them. Also, it allowed me to use the boolean
type & get away from 0 & 1. It felt a lot cleaner to be on Python 2.3. [I
added code to allow usage of /usr/bin/python2.3 interpreter, in case (like
on gump.try) on has RHEL.]

I started to feel 'bold and brave'. I started work on calling Depot to
download jars (kinda like we download code via CVS/SVN). I'll wiki more of
this, and I'll mail here. I have some final things to resolve, but I am
finally getting there. This is huge (for me) this is what I want to be
working on...

At the same time the Forrest folk divided to do some major (copyless) work,
and they removed the ability to pass work/site location parameters to the
batch file. Gump used these. Since I've not managed to install Forrest into
TomCat (when I once tried it failed, and the Forrest guys were too busy w/
"copyless" to fix it), I started to feel the same pain as others. I finally
decided I could cope no longer with the *effectively mandatory* Forrest
dependency for Gump. The inability for Gump to generate pages w/o Forrest
was too much. As it now stands Gump's page generator can now generate XHTML
or XDOCS. The XHTML isn't pretty, but with a little CSS it is perfectly
adequate [and we get colours again, as it looks w/ might w/ Forrest also].
Basically XDOCS and XHTML are similar enough that I can if/else
sufficiently. Take a look at:

    http://gump.try.sybase.com/buildLog.html

At the same time I decided to add the 'view as you go' ability Sam had been
looking for. I need to optimise it (I think the buildLog.html|buildLog.xml
page generation is fat) but it works. As of now, the site gets updated as it
goes.

I've added/changed other stuff, but minor things (in comparison). I do like
how branching allows me to make bold moves; I think the nightly/continual
builds were causing me too much stress & slowing down my progress. I know
I've broken a bunch of things along the way (the DOM change was non-trivial
& I have bugs w/ 'merge' & duplicates and such) but I'll work through those.

My goal is to install this (on Gump.try, then probably on Brutus, possibly
w/ JDK1.4 -- but to move to JDK 1.5 when stable) -- and work out the issues.
There are issues, but I feel I am seeing light at the end of the tunnel. I
won't attempt any merge back into the main branch until I've resolved them
all/worked through them w/ folks here.

regards,

Adam
--
Experience the Unwired Enterprise:
http://www.sybase.com/unwiredenterprise
Try Sybase: http://www.try.sybase.com


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org


merging CleanUp branch back into CVS HEAD

Posted by "Adam R. B. Jack" <aj...@trysybase.com>.
Are folks brave enough for me to merge the CleanUp branch back into CVS
HEAD? It is a major re-work (see some details below, so not guaranteed to be
bug-free ;-) but it has performed ok on this server:

    http://gump.try.sybase.com/buildLog.html

I think it'd be smart to turn off nagging for a few days, just to verify
nothing truly offensive occurs immediately.

Thoughts?

regards,

Adam
----- Original Message ----- 
From: "Adam R. B. Jack" <aj...@trysybase.com>
To: <ge...@gump.apache.org>
Sent: Thursday, June 17, 2004 10:50 AM
Subject: Gump work-in-progress (CleanUp branch)


> I've been struggling with Gump performance issues for months now. That is
> 'clogs up' (on large workspaces) has continually hampered usage &
(frankly)
> my progress .
>
> I've turned over every rock that comes to my mind, but I keep failing.
I've
> been getting pretty frustrated with Python (I find it somewhat immature in
> inspection/tuning) -- I've been very frustrated with my abilities (I
likely
> don't have the experience with Python to figure things out) -- and I've
been
> ultra frustrated with environmental issues. Since I can only cause the
> slowdown on large projects (small ones go fast) it is those monsters I
have
> to get inside, and I need to do that remotely.
>
> All this has basically hampered my progress, stopped me moving on with new
> features/functions ('cos I clearly didn't have a true grasp of the
basics.)
>
> I started to add a bunch of __del__ (destructor) overrides, so I could
> explicitly break circular links. I believe Python was failing to garbage
> collect because it needed a hand with these. I can't prove that, but I
felt
> it. So, I started the 'Clean-Up' branch to work on cleaning up memory,
> trying to avoid clog up. That is how I started down this path, which is
now
> quite a lot more than CleanUp...
>
> When I found that the SAX to Python object code was slowed by something
> minor I added (for no clear reason) I just lost it! I was building on sand
> (at least in my knowledge). So (not that DOM is pretty/thin) but I had to
> move to it, to simplify things & get control. Since we streamed SAX into
> memory, I saw no harm. [I don't think our problem is memory usage, so much
> as memory management.] For once, I felt good about ripping out clever code
> and adding pedestrian code ... I just needed controlled.
>
> Along the way I decided that moving to Python 2.3 (the latest/greatest bug
> fixed release per Python.org) was smart. If there were GC implementation
> fixes, that is where I'd find them. Also, it allowed me to use the boolean
> type & get away from 0 & 1. It felt a lot cleaner to be on Python 2.3. [I
> added code to allow usage of /usr/bin/python2.3 interpreter, in case (like
> on gump.try) on has RHEL.]
>
> I started to feel 'bold and brave'. I started work on calling Depot to
> download jars (kinda like we download code via CVS/SVN). I'll wiki more of
> this, and I'll mail here. I have some final things to resolve, but I am
> finally getting there. This is huge (for me) this is what I want to be
> working on...
>
> At the same time the Forrest folk divided to do some major (copyless)
work,
> and they removed the ability to pass work/site location parameters to the
> batch file. Gump used these. Since I've not managed to install Forrest
into
> TomCat (when I once tried it failed, and the Forrest guys were too busy w/
> "copyless" to fix it), I started to feel the same pain as others. I
finally
> decided I could cope no longer with the *effectively mandatory* Forrest
> dependency for Gump. The inability for Gump to generate pages w/o Forrest
> was too much. As it now stands Gump's page generator can now generate
XHTML
> or XDOCS. The XHTML isn't pretty, but with a little CSS it is perfectly
> adequate [and we get colours again, as it looks w/ might w/ Forrest also].
> Basically XDOCS and XHTML are similar enough that I can if/else
> sufficiently. Take a look at:
>
>     http://gump.try.sybase.com/buildLog.html
>
> At the same time I decided to add the 'view as you go' ability Sam had
been
> looking for. I need to optimise it (I think the buildLog.html|buildLog.xml
> page generation is fat) but it works. As of now, the site gets updated as
it
> goes.
>
> I've added/changed other stuff, but minor things (in comparison). I do
like
> how branching allows me to make bold moves; I think the nightly/continual
> builds were causing me too much stress & slowing down my progress. I know
> I've broken a bunch of things along the way (the DOM change was
non-trivial
> & I have bugs w/ 'merge' & duplicates and such) but I'll work through
those.
>
> My goal is to install this (on Gump.try, then probably on Brutus, possibly
> w/ JDK1.4 -- but to move to JDK 1.5 when stable) -- and work out the
issues.
> There are issues, but I feel I am seeing light at the end of the tunnel. I
> won't attempt any merge back into the main branch until I've resolved them
> all/worked through them w/ folks here.
>
> regards,
>
> Adam
> --
> Experience the Unwired Enterprise:
> http://www.sybase.com/unwiredenterprise
> Try Sybase: http://www.try.sybase.com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
> For additional commands, e-mail: general-help@gump.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org


[Forrest Gump] Re: Gump work-in-progress (CleanUp branch)

Posted by "Adam R. B. Jack" <aj...@trysybase.com>.
> Adam R. B. Jack wrote:

> > At the same time the Forrest folk divided to do some major (copyless)
work,
> > and they removed the ability to pass work/site location parameters to
the
> > batch file. Gump used these.
>
> They will be back somehow, don't worry :-)

No worries. As you said, we probably ought not default to using the batch
mechanism (it is a waste of CPU/resource), and persuade them to use the
dynamic webapplication instead. I've gone one further and remove the
temptation, by removing the ability. ;-)

I won't use it, so don't worry (from my part) about adding it again.

> > Since I've not managed to install Forrest into
> > TomCat (when I once tried it failed, and the Forrest guys were too busy
w/
> > "copyless" to fix it), I started to feel the same pain as others. I
finally
> > decided I could cope no longer with the *effectively mandatory* Forrest
> > dependency for Gump. The inability for Gump to generate pages w/o
Forrest
> > was too much. As it now stands Gump's page generator can now generate
XHTML
> > or XDOCS. The XHTML isn't pretty, but with a little CSS it is perfectly
> > adequate [and we get colours again, as it looks w/ might w/ Forrest
also].
>
> Forrest can use html instead of xdocs to generate the site. This way we
> could have a single generation system.

Again, no worries. We have a single system, it just has a few if/else for
XDOCS verses XHTML. I still want to use Forrest on the public Gump sites,
these XDOCS are just for folks less inclined. It was (when I got ballsy) low
hanging fruit, and ought reduce the on ramp to personal Gumps...

regards,

Adam


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org


Re: Gump work-in-progress (CleanUp branch)

Posted by Nicola Ken Barozzi <ni...@apache.org>.
Adam R. B. Jack wrote:

...
> At the same time the Forrest folk divided to do some major (copyless) work,
> and they removed the ability to pass work/site location parameters to the
> batch file. Gump used these. 

They will be back somehow, don't worry :-)

> Since I've not managed to install Forrest into
> TomCat (when I once tried it failed, and the Forrest guys were too busy w/
> "copyless" to fix it), I started to feel the same pain as others. I finally
> decided I could cope no longer with the *effectively mandatory* Forrest
> dependency for Gump. The inability for Gump to generate pages w/o Forrest
> was too much. As it now stands Gump's page generator can now generate XHTML
> or XDOCS. The XHTML isn't pretty, but with a little CSS it is perfectly
> adequate [and we get colours again, as it looks w/ might w/ Forrest also].

Forrest can use html instead of xdocs to generate the site. This way we 
could have a single generation system.

-- 
Nicola Ken Barozzi                   nicolaken@apache.org
             - verba volant, scripta manent -
    (discussions get forgotten, just code remains)
---------------------------------------------------------------------


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org


Re: Gump work-in-progress (CleanUp branch)

Posted by "Adam R. B. Jack" <aj...@trysybase.com>.
> I've been struggling with Gump performance issues for months now. That is
> 'clogs up' (on large workspaces) has continually hampered usage &
(frankly)
> my progress .

One thing that I did find when I was doing performance analysis/tuning of
the DOM-like (create a tree of objects then serialize) code was that this
code was one huge major total and complete *leak*. It really ought just add
a simple attribute to the class the leverages it (i.e. owner) and it ought
clean up (when __del__ is called remove that, breaking circular links) -- 
**but it doesn't**. I watch memory grow with it in, and not without it. I
don't get Python!

I removed use of this & the leak goes away.


http://cvs.apache.org/viewcvs.cgi/gump/python/gump/utils/owner.py?rev=1.4.4.2&view=markup

This needs to be made prettier, but see it runs in 2 something hours, which
isn't bad for a virtual machines:

    http://gump.try.sybase.com/buildLog.html

regards

Adam


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org