You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@gump.apache.org by Stefan Bodewig <bo...@apache.org> on 2005/07/11 17:16:38 UTC
Re: Runtime.exec & Ant & Gump3
On Mon, 11 Jul 2005, Leo Simons <ma...@leosimons.com> wrote:
> I suspect that using any JVM for which Ant's Execute takes a
> different approach (ie not using Runtime.exec) makes the problem
> "go away".
Hmm, have you run your tests with the equivalent of <exec>'s
vmlauncher="false"? I.e. force Execute to use the ShellLauncher
even if it is on JDK > 1.2?
> My hunch is that we'd have to implement a work around in Ant...
I can probably get away with yet another magic property if I know
how to cure the problem ;-)
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org
Re: Runtime.exec & Ant & Gump3
Posted by Leo Simons <ma...@leosimons.com>.
Stefano Mazzocchi wrote:
> do you think it's time to throw a python task force at it?
what would you suggest?
> I'm sure the python community would be quite pleased to help us making
> the entire apache software (especially all the java stuff) with a python
> script.
I think so too [1]. I hope the python community comes to ApacheCon :-)
LSD
[1] - http://jroller.com/page/lsd/20050530#hey_you_python_expert_apache
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org
Re: Runtime.exec & Ant & Gump3
Posted by Stefano Mazzocchi <st...@apache.org>.
Leo Simons wrote:
> Adam R. B. Jack wrote:
>
>>>No. Gump2 does not set the process group id. Gump3 calls setpgrp (man
>>>setpgrp :-)). Note there isn't a whole lot of software out there that
>>>make setpgrp calls...
>>
>>Yes Leo, it does.
>
>
> NooOOOhOOooohhoh! :-)
>
> WTF. Crap. So now I really have no clue where the problem is. We may
> have a python bug then.
>
>
>>Recall I implemented a scheme right before you got input
>>on how to do it (we even discussed if my approach would work, or not. :-)
>>The code is messy (for historical reasons, and 'cos it was tryign to do
>>sometihng on M$) but it is here.
>
>
> And, more importantly, it works. Remember it took the python people a
> POpen, POpen2, POpen3, POpen4, POpen5 to get to subprocess.POpen, which
> as it turns out isn't all that perfect yet either.
>
> I just really really want to know *why* it works (or not).
>
>
>>https://svn.apache.org/repos/asf/gump/trunk/python/gump/util/process/launcher.py
>>
>>BTW: A month or so ago I notice Gump2 hanging, when I ran it from command
>>line, and it seemed to improve if I passed it "< /dev/null".
>
>
> Would you be able to figure out something more specific than "A month ago"?
>
>
>>I kinda figured
>>something had changed in Ant, or in one of the Ant scripts, that was
>>blocking on input. That said, I never tracked anything down
>
>
> I think that might be worth it.
>
>
>>& I don't know
>>if it is still happening.
>
>
> I suspect the important bit here is that gump2 does this:
>
> # Allow redirect
> cmd += ' >>' + str(outputFile) + ' 2>&1'
>
> # Run the command
> systemReturn = os.system(cmd)
>
> cuz otherwise most of the end results are the same, eg gump2 also does
>
> (childPID, waitcode) = os.waitpid(forkPID,0)
>
> and the like. However, subprocess.py uses os.execvp() or os.execvpe()
> and not os.system. I'll have to take a look at the source code of
> posix.system() (which powers os.system). There's definitely some file
> descriptor magic to fix, somewhere.
do you think it's time to throw a python task force at it?
I'm sure the python community would be quite pleased to help us making
the entire apache software (especially all the java stuff) with a python
script.
--
Stefano, ego-tickle master ;-)
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org
Re: Runtime.exec & Ant & Gump3
Posted by Leo Simons <ma...@leosimons.com>.
Adam R. B. Jack wrote:
>>No. Gump2 does not set the process group id. Gump3 calls setpgrp (man
>>setpgrp :-)). Note there isn't a whole lot of software out there that
>>make setpgrp calls...
>
> Yes Leo, it does.
NooOOOhOOooohhoh! :-)
WTF. Crap. So now I really have no clue where the problem is. We may
have a python bug then.
> Recall I implemented a scheme right before you got input
> on how to do it (we even discussed if my approach would work, or not. :-)
> The code is messy (for historical reasons, and 'cos it was tryign to do
> sometihng on M$) but it is here.
And, more importantly, it works. Remember it took the python people a
POpen, POpen2, POpen3, POpen4, POpen5 to get to subprocess.POpen, which
as it turns out isn't all that perfect yet either.
I just really really want to know *why* it works (or not).
> https://svn.apache.org/repos/asf/gump/trunk/python/gump/util/process/launcher.py
>
> BTW: A month or so ago I notice Gump2 hanging, when I ran it from command
> line, and it seemed to improve if I passed it "< /dev/null".
Would you be able to figure out something more specific than "A month ago"?
> I kinda figured
> something had changed in Ant, or in one of the Ant scripts, that was
> blocking on input. That said, I never tracked anything down
I think that might be worth it.
> & I don't know
> if it is still happening.
I suspect the important bit here is that gump2 does this:
# Allow redirect
cmd += ' >>' + str(outputFile) + ' 2>&1'
# Run the command
systemReturn = os.system(cmd)
cuz otherwise most of the end results are the same, eg gump2 also does
(childPID, waitcode) = os.waitpid(forkPID,0)
and the like. However, subprocess.py uses os.execvp() or os.execvpe()
and not os.system. I'll have to take a look at the source code of
posix.system() (which powers os.system). There's definitely some file
descriptor magic to fix, somewhere.
cheers,
LSD
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org
Re: Runtime.exec & Ant & Gump3
Posted by "Adam R. B. Jack" <aj...@apache.org>.
> No. Gump2 does not set the process group id. Gump3 calls setpgrp (man
> setpgrp :-)). Note there isn't a whole lot of software out there that
> make setpgrp calls...
Yes Leo, it does. Recall I implemented a scheme right before you got input
on how to do it (we even discussed if my approach would work, or not. :-)
The code is messy (for historical reasons, and 'cos it was tryign to do
sometihng on M$) but it is here.
https://svn.apache.org/repos/asf/gump/trunk/python/gump/util/process/launcher.py
BTW: A month or so ago I notice Gump2 hanging, when I ran it from command
line, and it seemed to improve if I passed it "< /dev/null". I kinda figured
something had changed in Ant, or in one of the Ant scripts, that was
blocking on input. That said, I never tracked anything down & I don't know
if it is still happening.
regards
Adam
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org
Re: Runtime.exec & Ant & Gump3
Posted by Leo Simons <ma...@leosimons.com>.
Stefan Bodewig wrote:
> does the same happen in Gump 2? So far I haven't seen a trace of
> stalled processes.
No. Gump2 does not set the process group id. Gump3 calls setpgrp (man
setpgrp :-)). Note there isn't a whole lot of software out there that
make setpgrp calls...
> You said you had some smallish testcase, can we get access to that?
Well, sure, but that one doesn't have this problem. Might be because of
GC never kicking in or something similar.
> It looks as if I was too busy to spend considerable time on it right
> now, so I'll try to get ant-dev involved. Besides there are a few
> people on that list that are smarter than me 8-)
Kewl.
LSD
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org
Re: Runtime.exec & Ant & Gump3
Posted by Stefan Bodewig <bo...@apache.org>.
Leo,
does the same happen in Gump 2? So far I haven't seen a trace of
stalled processes.
You said you had some smallish testcase, can we get access to that?
It looks as if I was too busy to spend considerable time on it right
now, so I'll try to get ant-dev involved. Besides there are a few
people on that list that are smarter than me 8-)
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org
Re: Runtime.exec & Ant & Gump3
Posted by Leo Simons <ma...@leosimons.com>.
Stefan Bodewig wrote:
>>I have now. It doesn't help. But it does give a new bit of info --
>>the /bin/sh process created by the ShellLauncher stalls as well,
>
> Likely some thread waiting for the process to read or the process
> waiting for Ant to write something. There have been some process
> deadlocks due to streams getting read too early or written too late.
Yeah its something I thought about. I've tried manually sending bits and
bytes into some of the unix pipes that get created but without much
understanding of all the custom pipe stuff its pretty much impossible to
understand.
> You are certainly using Ant's CVS HEAD, aren't you?
Yep. Have also tried using 1.6.2 with the same result.
>>I've verified that the same problem occurs when running the xalan
>>build through gump on vmgump, so if you want to have a go at
>>figuring it out, that's all set to go :-)
>
> I'll try to free up some time, thanks.
Kewl!
LSD
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org
Re: Runtime.exec & Ant & Gump3
Posted by Stefan Bodewig <bo...@apache.org>.
On Mon, 11 Jul 2005, Leo Simons <ma...@leosimons.com> wrote:
> Stefan Bodewig wrote:
>> On Mon, 11 Jul 2005, Leo Simons <ma...@leosimons.com> wrote:
>>
>>>I suspect that using any JVM for which Ant's Execute takes a
>>>different approach (ie not using Runtime.exec) makes the problem
>>>"go away".
>>
>> Hmm, have you run your tests with the equivalent of <exec>'s
>> vmlauncher="false"? I.e. force Execute to use the ShellLauncher
>> even if it is on JDK > 1.2?
>
> I have now. It doesn't help. But it does give a new bit of info --
> the /bin/sh process created by the ShellLauncher stalls as well,
Likely some thread waiting for the process to read or the process
waiting for Ant to write something. There have been some process
deadlocks due to streams getting read too early or written too late.
You are certainly using Ant's CVS HEAD, aren't you?
> I've verified that the same problem occurs when running the xalan
> build through gump on vmgump, so if you want to have a go at
> figuring it out, that's all set to go :-)
I'll try to free up some time, thanks.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org
Re: Runtime.exec & Ant & Gump3
Posted by Leo Simons <ma...@leosimons.com>.
Stefan Bodewig wrote:
> On Mon, 11 Jul 2005, Leo Simons <ma...@leosimons.com> wrote:
>
>>I suspect that using any JVM for which Ant's Execute takes a
>>different approach (ie not using Runtime.exec) makes the problem
>>"go away".
>
> Hmm, have you run your tests with the equivalent of <exec>'s
> vmlauncher="false"? I.e. force Execute to use the ShellLauncher
> even if it is on JDK > 1.2?
I have now. It doesn't help. But it does give a new bit of info -- the
/bin/sh process created by the ShellLauncher stalls as well, before ever
firing up a new java process. I.e. we never get beyond
return Runtime.getRuntime().exec(cmd, env);
on line 736 of Execute.java.
>>My hunch is that we'd have to implement a work around in Ant...
>
> I can probably get away with yet another magic property if I know
> how to cure the problem ;-)
It'd be nice to know :-)
I've verified that the same problem occurs when running the xalan build
through gump on vmgump, so if you want to have a go at figuring it out,
that's all set to go :-)
cheers,
Leo
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org