You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@gump.apache.org by Antoine Lévy-Lambert <an...@antbuild.com> on 2004/02/24 00:25:58 UTC
various gump remarks/problems
1) running gumpy under cygwin :
--------------------------------
is actually difficult, maybe not something to pursue.
The reason : gumpy starts ant with classpaths and various properties
pointing to files/or directories
in cygwin format. ant is in Java and does not know anything about
cygwin, so it does not work
2) running gumpy under DOS, with cygwin utilities
--------------------------------------------------
I have tried this one.
It requires to hack tools.py and forrest.py, so that the following
commands :
cp
sync
cat
get passed file or directory arguments reconverted back to cygwin format
with the cygpath utility.
This works OK.
*As Adam wrote already, the best (and real) solution is to replace all
these system calls by pure python code.*
I was wondering whether cat is used in tools.py for something else than
checking for the existence & readability of a file.
Is not this a 2 liner in python ?
cp and sync could probably also be replaced by pure python code
this page gives idea of functions which could be used.
http://www.python.org/doc/current/lib/module-shutil.html
3) problem with Documenter
------------------------------
When I tried my run yesterday, after I had fixed my problems with the
shell utilities, my ant builds starting being successful
(also in the first place because I added the *name* attribute in my
workspace defnition.)
But gumpy was seeming to hang in Documenter, and to consume a lot of CPU.
I am not sure, whether it is the fault of gumpy or of forrest.
Does one need to set ANT_OPTS to a high memory value in local-env-py ....
Cheers,
Antoine
Re: various gump remarks/problems
Posted by "Adam R. B. Jack" <aj...@trysybase.com>.
> I think all we want is to sync two directories (we can forget about the
> r from rsync).
Agreed.
> I think we better have our own, but we can use outside inspiration.
Works for me.
> I am willing to write it.
Works best of all.
> Sync is simple enough (to me) so that we write our own solution. I think
> it is just a recursive copy (possibly skipping files which have same
> datetime and size
> at source and destination), then a deletion of files which exist at
> destination and do not exist at source. I am willing to write the stuff.
If you do write it this way, a few thoughts:
1) I use 'cp' (even when rsync is available) to copy the forrest template
into a work tree, and then copy a local (site) forrest template on top [if
available]. We can't rsync the second because we'd loose the main template.
Your sync w/o the following deletions would maek a good copy.
2) I suspect that os.walk would be a beautiful thing here, but we are trying
(for a short while longer only, I hope) keep to Python 2.2 & not force 2.3.
3) Anything in dircmp useful?
4) Right now gump/utils/sync.py would seem appropriate. An interface that
logged debug/info/warn/error, and raise issues when critical would be fine
for me to wire into Gumpy. I don't think a file by file output is needed, or
anything other than 'it failed to do as you ask', but if so I can help you
w/ annotated objects (notes) and such.
5) A unit test (home grown gump/test/pyunit.py) would be awesome. Perhaps
extend gump/test/tools.py which now conttains the (not yet removed)
listDir/catFile AsWork and ToFileHolder (the replacements.)
> I think we can start with a very basic implementation, ignoring symlinks
> and file permissions issues. I understand we are mostly syncing CVS
> trees which I don't think contain symlinks, and where all the files
> are read write. If needs be we can always make it more complicated.
CVS and/or SVN (and that one template dir of dirs/plain files, taken from
Gump CVS tree) so yes -- no need for complexity, I feel.
> If we port all these, then we do not need cygwin any more. What we do
> need is a cvs client, people setting up Gump instances
> on M$ can choose between a Win32 or a cygwin cvs client, both can work.
CVS/SVN(/Ruper)(/Maven) are all fine by me to have external for the
foreseable. The other thing we launch, pgrep, is a pain but I don't know a
pure Python way to detect child processes in order to kill them (in a
timeout).
> put the
> content back and run forrest by hand and it looks like it is working.
Awesome. Forrest might be a tad heavyweight for our needs, but I really want
to persist with it and keep Gumpy simple. Writing more than xdocs would
likely lead to 'getting pretty' (read browser woes), and all sorts of
unnecessary time wasting stuff. That, and I think forest sites look snappy &
enjoy standing on their shoulders.
regards,
Adam
Re: various gump remarks/problems
Posted by Davanum Srinivas <di...@yahoo.com>.
rsync.py - http://www.vdesmedt.com/~vds2212/rsync.html
--- Antoine_L�vy-Lambert <an...@antbuild.com> wrote:
> Adam R. B. Jack wrote:
>
> >
> >Could you take on a challenge that I've not been able to, yet? Rsync ... it
> >is a major pain, 'cos it has bugs [not deleting what it ought] & 'cos it is
> >not pure Python. There are Python choices (thanks Nicola for the original
> >reference) but we'd need to weigh these verse writting our own:
> >
> >
> >
> I think all we want is to sync two directories (we can forget about the
> r from rsync).
> I think we better have our own, but we can use outside inspiration.
> I am willing to write it.
>
> >http://nagoya.apache.org/eyebrowse/ReadMsg?listName=gump@jakarta.apache.org&msgNo=3026
> >
> >http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/231501
> >
> >http://sourceforge.net/projects/dirssync/
> >
> >http://www.homepages.lu/pu/dfp.html
> >
> >I don't know the OSS world well enough to know if we can use GPL code,
> >bundle it, whatever.
> >
> No, we should leave GPL code alone, because the GPL license is viral.
>
> >I don't know the Python world to know if we ought
> >'expect' packages installed, or (like we did w/ logging) we ought dump them
> >in our CVS. Are you able to look into this, and select a solution? If you
> >get one, I'd work with you to get it into Gumpy.
> >
> >
> Sync is simple enough (to me) so that we write our own solution. I think
> it is just a recursive copy (possibly skipping files which have same
> datetime and size
> at source and destination), then a deletion of files which exist at
> destination and do not exist at source. I am willing to write the stuff.
>
> I think we can start with a very basic implementation, ignoring symlinks
> and file permissions issues. I understand we are mostly syncing CVS
> trees which I don't think contain symlinks, and where all the files
> are read write. If needs be we can always make it more complicated.
>
> >BTW: If we port all these, would we not needs cgywin? I'm hoping not.
> >
> >
> >
> If we port all these, then we do not need cygwin any more. What we do
> need is a cvs client, people setting up Gump instances
> on M$ can choose between a Win32 or a cygwin cvs client, both can work.
>
> >>>BTW: How long does it 'hang' in forrest? How big is your workspace? [If
> >>>
> >>>
> >you
> >
> >
> >>>don't have forrest on the path it ought use text...]
> >>>
> >>>
> >>>
> >>I have seen messages on the screen running Forrest Documenter and
> >>running Text Documenter, I do not remember in which order.
> >>It took ages, and in the end my CPU was all eaten, I could not start
> >>another program.
> >>
> >>
> >
> >
> >
> I tried to hack the code in forrest.py where the seed command is called,
> but hacked it badly.
> I have just tried to run forrest by hand, and it was complaining about
> the skinconf.xml missing a <toc> element,
> then about other files missing.
> Now I moved temporarily my content, then run forrest seed, put the
> content back and run forrest by hand and it looks like it is working.
> Cheers,
> Antoine
>
> >The Forrest Documenter does (as one page inside itself) run the Text
> >Documenter [which we need to update to be nicer] so it is running Forrest.
> >Gumpy needs to be smarter about documentation choices, but for now forrest
> >ought be ok. I can't think why it chewed your CPU. Can you go to the forrest
> >directory under your work area & run forrest by hand? Alternatively, what
> >did the forrest.txt file show? [I forget if Gumpy sets --debug on forrest or
> >not.]
> >
> >regards,
> >
> >Adam
> >
> >
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: gump-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: gump-help@jakarta.apache.org
>
=====
Davanum Srinivas - http://webservices.apache.org/~dims/
Re: various gump remarks/problems
Posted by Antoine Lévy-Lambert <an...@antbuild.com>.
Adam R. B. Jack wrote:
>
>Could you take on a challenge that I've not been able to, yet? Rsync ... it
>is a major pain, 'cos it has bugs [not deleting what it ought] & 'cos it is
>not pure Python. There are Python choices (thanks Nicola for the original
>reference) but we'd need to weigh these verse writting our own:
>
>
>
I think all we want is to sync two directories (we can forget about the
r from rsync).
I think we better have our own, but we can use outside inspiration.
I am willing to write it.
>http://nagoya.apache.org/eyebrowse/ReadMsg?listName=gump@jakarta.apache.org&msgNo=3026
>
>http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/231501
>
>http://sourceforge.net/projects/dirssync/
>
>http://www.homepages.lu/pu/dfp.html
>
>I don't know the OSS world well enough to know if we can use GPL code,
>bundle it, whatever.
>
No, we should leave GPL code alone, because the GPL license is viral.
>I don't know the Python world to know if we ought
>'expect' packages installed, or (like we did w/ logging) we ought dump them
>in our CVS. Are you able to look into this, and select a solution? If you
>get one, I'd work with you to get it into Gumpy.
>
>
Sync is simple enough (to me) so that we write our own solution. I think
it is just a recursive copy (possibly skipping files which have same
datetime and size
at source and destination), then a deletion of files which exist at
destination and do not exist at source. I am willing to write the stuff.
I think we can start with a very basic implementation, ignoring symlinks
and file permissions issues. I understand we are mostly syncing CVS
trees which I don't think contain symlinks, and where all the files
are read write. If needs be we can always make it more complicated.
>BTW: If we port all these, would we not needs cgywin? I'm hoping not.
>
>
>
If we port all these, then we do not need cygwin any more. What we do
need is a cvs client, people setting up Gump instances
on M$ can choose between a Win32 or a cygwin cvs client, both can work.
>>>BTW: How long does it 'hang' in forrest? How big is your workspace? [If
>>>
>>>
>you
>
>
>>>don't have forrest on the path it ought use text...]
>>>
>>>
>>>
>>I have seen messages on the screen running Forrest Documenter and
>>running Text Documenter, I do not remember in which order.
>>It took ages, and in the end my CPU was all eaten, I could not start
>>another program.
>>
>>
>
>
>
I tried to hack the code in forrest.py where the seed command is called,
but hacked it badly.
I have just tried to run forrest by hand, and it was complaining about
the skinconf.xml missing a <toc> element,
then about other files missing.
Now I moved temporarily my content, then run forrest seed, put the
content back and run forrest by hand and it looks like it is working.
Cheers,
Antoine
>The Forrest Documenter does (as one page inside itself) run the Text
>Documenter [which we need to update to be nicer] so it is running Forrest.
>Gumpy needs to be smarter about documentation choices, but for now forrest
>ought be ok. I can't think why it chewed your CPU. Can you go to the forrest
>directory under your work area & run forrest by hand? Alternatively, what
>did the forrest.txt file show? [I forget if Gumpy sets --debug on forrest or
>not.]
>
>regards,
>
>Adam
>
>
>
>
Re: various gump remarks/problems
Posted by "Adam R. B. Jack" <aj...@trysybase.com>.
> No problem, gumpy is an important tool, and I would like to get it to
> work for me.
Awesome, thanks.
> I am willing to work with you to replace the invocations of ls, cat, cp
> and sync with Python code. (in tools.py and forrest.py)
As you may have seen from check-ins I working on ls and cat.
Could you take on a challenge that I've not been able to, yet? Rsync ... it
is a major pain, 'cos it has bugs [not deleting what it ought] & 'cos it is
not pure Python. There are Python choices (thanks Nicola for the original
reference) but we'd need to weigh these verse writting our own:
http://nagoya.apache.org/eyebrowse/ReadMsg?listName=gump@jakarta.apache.org&msgNo=3026
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/231501
http://sourceforge.net/projects/dirssync/
http://www.homepages.lu/pu/dfp.html
I don't know the OSS world well enough to know if we can use GPL code,
bundle it, whatever. I don't know the Python world to know if we ought
'expect' packages installed, or (like we did w/ logging) we ought dump them
in our CVS. Are you able to look into this, and select a solution? If you
get one, I'd work with you to get it into Gumpy.
BTW: If we port all these, would we not needs cgywin? I'm hoping not.
> >BTW: How long does it 'hang' in forrest? How big is your workspace? [If
you
> >don't have forrest on the path it ought use text...]
> >
> I have seen messages on the screen running Forrest Documenter and
> running Text Documenter, I do not remember in which order.
> It took ages, and in the end my CPU was all eaten, I could not start
> another program.
The Forrest Documenter does (as one page inside itself) run the Text
Documenter [which we need to update to be nicer] so it is running Forrest.
Gumpy needs to be smarter about documentation choices, but for now forrest
ought be ok. I can't think why it chewed your CPU. Can you go to the forrest
directory under your work area & run forrest by hand? Alternatively, what
did the forrest.txt file show? [I forget if Gumpy sets --debug on forrest or
not.]
regards,
Adam
Re: various gump remarks/problems
Posted by Antoine Lévy-Lambert <an...@antbuild.com>.
Adam R. B. Jack wrote:
>First, awesome, thanks for the detailed feedback.
>
>
>
No problem, gumpy is an important tool, and I would like to get it to
work for me.
>Third, I'm game to work with anybody in order to fix these things. I'd
>prefer to work w/ somebody else coding (about time I help folks learn how),
>but I'd work w/ folks who JIRA'ed issues also.
>
>
I am willing to work with you to replace the invocations of ls, cat, cp
and sync with Python code. (in tools.py and forrest.py)
>BTW: How long does it 'hang' in forrest? How big is your workspace? [If you
>don't have forrest on the path it ought use text...]
>
>
>
I have seen messages on the screen running Forrest Documenter and
running Text Documenter, I do not remember in which order.
It took ages, and in the end my CPU was all eaten, I could not start
another program.
>regards,
>
>Adam
>
>
Cheers,
Antoine
Re: various gump remarks/problems
Posted by "Adam R. B. Jack" <aj...@trysybase.com>.
I wrote:
> I used 'cat' 'cos I wasn't thinking ...
Upon revisiting, there is a little more to it than that. Various objects
(W/S, modules, projects) are 'Workable', i.e. can have work applied to
them -- and that work is typically a build, or an update, i.e. a forked
command with an optional output file. I coded the xdoc writer to display all
work using separate HTML pages. Basically, I re-used that mechanism & got
kinda carried away with it.
http://lsd.student.utwente.nl/gump/ws-axis/ws-axis-test.html#Project-level+Work
I think I need to write 'Fileable' (or something named nicer) that allows a
list of files to be associated with such an object. I'll take a stab as soon
as I can.
regards
Adam
Re: various gump remarks/problems
Posted by "Adam R. B. Jack" <aj...@trysybase.com>.
First, awesome, thanks for the detailed feedback.
Second, I've only ever tinkered w/ Gumpy w/ MKS on DOS, never used cygwin,
so never noticed many of these. I've also only got Linux boxes the 'nice'
side of my slow modem, so never spent much time fixing things for M$. I used
'cat' 'cos I wasn't thinking ...
Third, I'm game to work with anybody in order to fix these things. I'd
prefer to work w/ somebody else coding (about time I help folks learn how),
but I'd work w/ folks who JIRA'ed issues also.
BTW: How long does it 'hang' in forrest? How big is your workspace? [If you
don't have forrest on the path it ought use text...]
regards,
Adam
----- Original Message -----
From: "Antoine Lévy-Lambert" <an...@antbuild.com>
To: "jakarta-gump list" <gu...@jakarta.apache.org>
Sent: Monday, February 23, 2004 4:25 PM
Subject: various gump remarks/problems
> 1) running gumpy under cygwin :
> --------------------------------
> is actually difficult, maybe not something to pursue.
> The reason : gumpy starts ant with classpaths and various properties
> pointing to files/or directories
> in cygwin format. ant is in Java and does not know anything about
> cygwin, so it does not work
>
> 2) running gumpy under DOS, with cygwin utilities
> --------------------------------------------------
> I have tried this one.
>
> It requires to hack tools.py and forrest.py, so that the following
> commands :
> cp
> sync
> cat
> get passed file or directory arguments reconverted back to cygwin format
> with the cygpath utility.
>
> This works OK.
>
> *As Adam wrote already, the best (and real) solution is to replace all
> these system calls by pure python code.*
>
> I was wondering whether cat is used in tools.py for something else than
> checking for the existence & readability of a file.
>
> Is not this a 2 liner in python ?
>
> cp and sync could probably also be replaced by pure python code
>
> this page gives idea of functions which could be used.
> http://www.python.org/doc/current/lib/module-shutil.html
>
>
> 3) problem with Documenter
> ------------------------------
> When I tried my run yesterday, after I had fixed my problems with the
> shell utilities, my ant builds starting being successful
> (also in the first place because I added the *name* attribute in my
> workspace defnition.)
>
> But gumpy was seeming to hang in Documenter, and to consume a lot of CPU.
> I am not sure, whether it is the fault of gumpy or of forrest.
>
> Does one need to set ANT_OPTS to a high memory value in local-env-py ....
>
> Cheers,
>
>
> Antoine
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: gump-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: gump-help@jakarta.apache.org
>
Re: various gump remarks/problems
Posted by Antoine Lévy-Lambert <an...@antbuild.com>.
Adam R. B. Jack wrote:
>Could you explain to me what cygwin format is?
>
>regards
>
>Adam
>
>
cygwin has a mount table
here is how it looks on my PC :
C:\Programme\cygwin\usr\X11R6\lib\X11\fonts on /usr/X11R6/lib/X11/fonts
type system (binmode)
C:\Programme\cygwin\bin on /usr/bin type system (binmode)
C:\Programme\cygwin\lib on /usr/lib type system (binmode)
C:\Programme\cygwin on / type system (binmode)
C:\opt on /opt type system (binmode)
C:\var on /var type system (binmode)
c: on /cygdrive/c type user (binmode,noumount)
my javac seen from cygwin is under /cygdrive/c/j2sdk1.4.1_02/bin/javac
my cvs.exe (the one from cygwin) is under /usr/bin/cvs.exe
(C:\Programme\cygwin\bin\cvs.exe for Windows)
there is a utility called cygpath which can convert paths :
$ cygpath --windows "/cygdrive/c/j2sdk1.4.1_02/bin"
c:\j2sdk1.4.1_02\bin
$ cygpath --unix "c:\j2sdk1.4.1_02\bin"
/cygdrive/c/j2sdk1.4.1_02/bin
Apart from that, shell scripting under cygwin is like shell scripting
under UNIX.
When starting a java program, all command line arguments need to be
expressed in Windows terms, because Java is not cygwin aware.
Antoine
Re: various gump remarks/problems
Posted by "Adam R. B. Jack" <aj...@trysybase.com>.
Could you explain to me what cygwin format is?
regards
Adam
----- Original Message -----
From: "Antoine Lévy-Lambert" <an...@antbuild.com>
To: "jakarta-gump list" <gu...@jakarta.apache.org>
Sent: Monday, February 23, 2004 4:25 PM
Subject: various gump remarks/problems
> 1) running gumpy under cygwin :
> --------------------------------
> is actually difficult, maybe not something to pursue.
> The reason : gumpy starts ant with classpaths and various properties
> pointing to files/or directories
> in cygwin format. ant is in Java and does not know anything about
> cygwin, so it does not work
>
> 2) running gumpy under DOS, with cygwin utilities
> --------------------------------------------------
> I have tried this one.
>
> It requires to hack tools.py and forrest.py, so that the following
> commands :
> cp
> sync
> cat
> get passed file or directory arguments reconverted back to cygwin format
> with the cygpath utility.
>
> This works OK.
>
> *As Adam wrote already, the best (and real) solution is to replace all
> these system calls by pure python code.*
>
> I was wondering whether cat is used in tools.py for something else than
> checking for the existence & readability of a file.
>
> Is not this a 2 liner in python ?
>
> cp and sync could probably also be replaced by pure python code
>
> this page gives idea of functions which could be used.
> http://www.python.org/doc/current/lib/module-shutil.html
>
>
> 3) problem with Documenter
> ------------------------------
> When I tried my run yesterday, after I had fixed my problems with the
> shell utilities, my ant builds starting being successful
> (also in the first place because I added the *name* attribute in my
> workspace defnition.)
>
> But gumpy was seeming to hang in Documenter, and to consume a lot of CPU.
> I am not sure, whether it is the fault of gumpy or of forrest.
>
> Does one need to set ANT_OPTS to a high memory value in local-env-py ....
>
> Cheers,
>
>
> Antoine
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: gump-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: gump-help@jakarta.apache.org
>