You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ant.apache.org by Jeff Adamson <jw...@us.ibm.com> on 2016/10/11 15:51:00 UTC
ant wrapper script testing
I have been doing some work on trying to correct errors in the ant shell
script with how special characters and whitespace and handled in arguments.
I have a scratch branch in github. I do forced pushes as I clean it up and
try new things, so beware forking.
https://github.com/jwadamson/ant-1/tree/shell-detection
https://github.com/jwadamson/ant-1/blob/shell-detection/src/script/ant
I do trials by running a script test.sh:
#!/bin/sh
./bin/ant --execdebug \
"-Dfoo=dollar\$_backtick\`_single'_double\"\"_trailingbackslash\\" \
"-Dbar=trailingnewline
" \
"-Dnewline=
" \
"-Ddoublespace= " \
"-Dx=y" \
"-f" "test.xml"
where test.xml is simply
<project default="run">
<target name="run">
<echo message="hello world"/>
<echo message="foo=${foo}_"/>
<echo message="bar=${bar}_"/>
<echo message="newline=${newline}_"/>
<echo message="doublespace=${doublespace}_"/>
<echo message="end"/>
</target>
</project>
The trailing _ characters are so that echo task does not trim the output
for trailing whitespace scenarios.
The only solution I have found to date that avoids all behavior quirks such
as echo, sed, awk, and whitespace handling is a bashism/ksh/zsh-ism. So
that does not work for ubuntu's /bin/sh. I have not figured out a universal
correct script for dash (ubuntu) yet.
It may be worth detecting those shells to leverage that when we can.
I have run the script with bash, ksh, zsh, and dash under Ubuntu and Mac.
Csh and tcsh do not seem to work (do we expect them to?)
I do not have an HP-UX or Solaris setup yet (though there are vbox images
for Solaris).
I may not have much time for more testing today, but would appreciate any
advise:
1) from people with Solaris and HP-UX and AIX
2) thoughts on introducing shell detection and bashisms into the script
3) how to do a clean string substitution or sed/awk based replacement that
doesn't munge newline and trailing spaces
4) anything else that appears fishy in the test cases
Regards,
Jeff Adamson
Re: ant wrapper script testing
Posted by Jeff Adamson <jw...@us.ibm.com>.
A little more cleanup on my part but looking good. Take what you need as I
presume we do not want this merged in directly due to comments, alternative
impls, and testing scripts.
https://github.com/jwadamson/ant-1/blob/shell-detection/src/script/ant
Should I make a new pull request and/or squash this down first?
Is it ready to test by our Solaris/AIX/HPUX users or not until we get it
merged into the mainline repo?
Regards,
Jeff Adamson
From: Jeff Adamson/Cleveland/IBM@IBMUS
To: "Ant Developers List" <de...@ant.apache.org>
Date: 10/13/2016 09:54 AM
Subject: Re: ant wrapper script testing
Things I found out/rediscovered during experiments:
1) `echo` bash built-in modifies the input (escape interpretation, trailing
newline) and not all versions have options to suppress those behaviors.
Using printf instead to get a "clean" echo.
2) `sed` on mac does not support group-0 reference, created an extra
capture group and reference that instead
3) "${var/?/Y/Z}" is a perfect substring replacement for our needs. Is
supported by bash, zsh, and ksh93 but is not supported by the dash (ubuntu)
or ksh88 (older AIX?) shells.
4) `awk` and `sed` will both consume trailing newlines and/or whitespace
from the input. Resolved by padding input value with non-whitespace and
strip result afterwards.
5) http://pubs.opengroup.org/onlinepubs/9699919799/ section 2.6.2 of Shell
volume documents ${x%w} ${x#w} expansions
The basic code that seems to work for my on ubuntu+dash+gnu_sed and mac
+bash+bsd_sed (which is the default for those platforms):
esc_arg="X${arg}X"
esc_arg="$(printf '%s' "$esc_arg" | sed -e 's@\([$"\\`]\)@\\\1@g')"
esc_arg="${esc_arg#X}" # posix remove single leading X if present
esc_arg="${esc_arg%X}" # posix remove single trailing X if present
http://pubs.opengroup.org/onlinepubs/9699919799/
quoted_arg="\"$esc_arg\""
ant_exec_args="$ant_exec_args $quoted_arg"
This pattern passes a checkbashisms script cleanly.
Concerns:
1) is awk or sed a better choice for cross platform consistency
2) did we still need the trailing newline for sed now that the value is
padded? I did not see that as necessary in my testing on either mac or
ubuntu once padded and had removed it.
The branch name is a bit of a misnomer since I removed the actuall shell
detection, but wanted to leave that commit in the history locally in case
we have reason to adopt that sort of detection.
The trade off is doing ugly'ish zsh/ksh93/bash detection for the benefit of
keeping everything internal and not having the child tools do it.
I have more comments inline
https://github.com/jwadamson/ant-1/blob/shell-detection/src/script/ant as
well as an awk pattern alternative to sed ( commit
c591ade4aa7c19ef7e640ed9759fd95e8c4582d8)
Regards,
Jeff Adamson
Stefan Bodewig ---10/12/2016 05:57:39 AM---On 2016-10-11, Jeff Adamson
wrote: > I have been doing some work on trying to correct errors in the
From: Stefan Bodewig <bo...@apache.org>
To: dev@ant.apache.org
Date: 10/12/2016 05:57 AM
Subject: Re: ant wrapper script testing
On 2016-10-11, Jeff Adamson wrote:
> I have been doing some work on trying to correct errors in the ant shell
> script with how special characters and whitespace and handled in
arguments.
Many thanks for that.
> The only solution I have found to date that avoids all behavior quirks
such
> as echo, sed, awk, and whitespace handling is a bashism/ksh/zsh-ism. So
> that does not work for ubuntu's /bin/sh. I have not figured out a
universal
> correct script for dash (ubuntu) yet.
> It may be worth detecting those shells to leverage that when we can.
understood. If we are able to get it to work without relying on sed
which seems to be way more platform specific than I had thought, then
I'd prefer such a solution.
> I have run the script with bash, ksh, zsh, and dash under Ubuntu and Mac.
> Csh and tcsh do not seem to work (do we expect them to?)
No, we don't expect the csh family of shells to be able to run /bin/sh
scripts.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
RE: ant wrapper script testing
Posted by Martin Gainty <mg...@hotmail.com>.
MG>please see quick comment below
Subject: Re: ant wrapper script testing
To: dev@ant.apache.org
From: jwadamson@us.ibm.com
Date: Thu, 13 Oct 2016 09:53:19 -0400
Things I found out/rediscovered during experiments:
`echo` bash built-in modifies the input (escape interpretation, trailing newline) and not all versions have options to suppress those behaviors. Using printf instead to get a "clean" echo.
`sed` on mac does not support group-0 reference, created an extra capture group and reference that instead
"${var/?/Y/Z}" is a perfect substring replacement for our needs. Is supported by bash, zsh, and ksh93 but is not supported by the dash (ubuntu) or ksh88 (older AIX?) shells.
`awk` and `sed` will both consume trailing newlines and/or whitespace from the input. Resolved by padding input value with non-whitespace and strip result afterwards.
http://pubs.opengroup.org/onlinepubs/9699919799/ section 2.6.2 of Shell volume documents ${x%w} ${x#w} expansions
The basic code that seems to work for my on ubuntu+dash+gnu_sed and mac+bash+bsd_sed (which is the default for those platforms):
esc_arg="X${arg}X"
esc_arg="$(printf '%s' "$esc_arg" | sed -e 's@\([$"\\`]\)@\\\1@g')"
esc_arg="${esc_arg#X}" # posix remove single leading X if present
esc_arg="${esc_arg%X}" # posix remove single trailing X if present http://pubs.opengroup.org/onlinepubs/9699919799/
quoted_arg="\"$esc_arg\""
ant_exec_args="$ant_exec_args $quoted_arg"
This pattern passes a checkbashisms script cleanly.
Concerns:
is awk or sed a better choice for cross platform consistency did we still need the trailing newline for sed now that the value is padded? I did not see that as necessary in my testing on either mac or ubuntu once padded and had removed it.
MG>I *hope* sed utility would read the environment and thus pad 0X0D or 0X0D0x0A depending on implementing OS
The branch name is a bit of a misnomer since I removed the actuall shell detection, but wanted to leave that commit in the history locally in case we have reason to adopt that sort of detection.
The trade off is doing ugly'ish zsh/ksh93/bash detection for the benefit of keeping everything internal and not having the child tools do it.
I have more comments inline https://github.com/jwadamson/ant-1/blob/shell-detection/src/script/ant as well as an awk pattern alternative to sed ( commit c591ade4aa7c19ef7e640ed9759fd95e8c4582d8)
MG>speaking of ant can you build above features into http://ant-contrib.sourceforge.net/tasks/tasks/propertyregex.html
MG>there are many of us who would like the above features incorporated a Xplatform solution invoked from ant
Regards,
Jeff Adamson
MG>Many Thanks Jeff (and Stefan)
Stefan Bodewig ---10/12/2016 05:57:39 AM---On 2016-10-11, Jeff Adamson wrote: > I have been doing some work on trying to correct errors in the
From: Stefan Bodewig <bo...@apache.org>
To: dev@ant.apache.org
Date: 10/12/2016 05:57 AM
Subject: Re: ant wrapper script testing
On 2016-10-11, Jeff Adamson wrote:
> I have been doing some work on trying to correct errors in the ant shell
> script with how special characters and whitespace and handled in arguments.
Many thanks for that.
> The only solution I have found to date that avoids all behavior quirks such
> as echo, sed, awk, and whitespace handling is a bashism/ksh/zsh-ism. So
> that does not work for ubuntu's /bin/sh. I have not figured out a universal
> correct script for dash (ubuntu) yet.
> It may be worth detecting those shells to leverage that when we can.
understood. If we are able to get it to work without relying on sed
which seems to be way more platform specific than I had thought, then
I'd prefer such a solution.
> I have run the script with bash, ksh, zsh, and dash under Ubuntu and Mac.
> Csh and tcsh do not seem to work (do we expect them to?)
No, we don't expect the csh family of shells to be able to run /bin/sh
scripts.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: ant wrapper script testing
Posted by Jeff Adamson <jw...@us.ibm.com>.
Things I found out/rediscovered during experiments:
`echo` bash built-in modifies the input (escape interpretation, trailing
newline) and not all versions have options to suppress those behaviors.
Using printf instead to get a "clean" echo.
`sed` on mac does not support group-0 reference, created an extra
capture group and reference that instead
"${var/?/Y/Z}" is a perfect substring replacement for our needs. Is
supported by bash, zsh, and ksh93 but is not supported by the dash
(ubuntu) or ksh88 (older AIX?) shells.
`awk` and `sed` will both consume trailing newlines and/or whitespace
from the input. Resolved by padding input value with non-whitespace and
strip result afterwards.
http://pubs.opengroup.org/onlinepubs/9699919799/ section 2.6.2 of Shell
volume documents ${x%w} ${x#w} expansions
The basic code that seems to work for my on ubuntu+dash+gnu_sed and mac
+bash+bsd_sed (which is the default for those platforms):
esc_arg="X${arg}X"
esc_arg="$(printf '%s' "$esc_arg" | sed -e 's@\([$"\\`]\)@\\\1@g')"
esc_arg="${esc_arg#X}" # posix remove single leading X if present
esc_arg="${esc_arg%X}" # posix remove single trailing X if present
http://pubs.opengroup.org/onlinepubs/9699919799/
quoted_arg="\"$esc_arg\""
ant_exec_args="$ant_exec_args $quoted_arg"
This pattern passes a checkbashisms script cleanly.
Concerns:
is awk or sed a better choice for cross platform consistency
did we still need the trailing newline for sed now that the value is
padded? I did not see that as necessary in my testing on either mac or
ubuntu once padded and had removed it.
The branch name is a bit of a misnomer since I removed the actuall shell
detection, but wanted to leave that commit in the history locally in case
we have reason to adopt that sort of detection.
The trade off is doing ugly'ish zsh/ksh93/bash detection for the benefit of
keeping everything internal and not having the child tools do it.
I have more comments inline
https://github.com/jwadamson/ant-1/blob/shell-detection/src/script/ant as
well as an awk pattern alternative to sed ( commit
c591ade4aa7c19ef7e640ed9759fd95e8c4582d8)
Regards,
Jeff Adamson
From: Stefan Bodewig <bo...@apache.org>
To: dev@ant.apache.org
Date: 10/12/2016 05:57 AM
Subject: Re: ant wrapper script testing
On 2016-10-11, Jeff Adamson wrote:
> I have been doing some work on trying to correct errors in the ant shell
> script with how special characters and whitespace and handled in
arguments.
Many thanks for that.
> The only solution I have found to date that avoids all behavior quirks
such
> as echo, sed, awk, and whitespace handling is a bashism/ksh/zsh-ism. So
> that does not work for ubuntu's /bin/sh. I have not figured out a
universal
> correct script for dash (ubuntu) yet.
> It may be worth detecting those shells to leverage that when we can.
understood. If we are able to get it to work without relying on sed
which seems to be way more platform specific than I had thought, then
I'd prefer such a solution.
> I have run the script with bash, ksh, zsh, and dash under Ubuntu and Mac.
> Csh and tcsh do not seem to work (do we expect them to?)
No, we don't expect the csh family of shells to be able to run /bin/sh
scripts.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: ant wrapper script testing
Posted by Stefan Bodewig <bo...@apache.org>.
On 2016-10-11, Jeff Adamson wrote:
> I have been doing some work on trying to correct errors in the ant shell
> script with how special characters and whitespace and handled in arguments.
Many thanks for that.
> The only solution I have found to date that avoids all behavior quirks such
> as echo, sed, awk, and whitespace handling is a bashism/ksh/zsh-ism. So
> that does not work for ubuntu's /bin/sh. I have not figured out a universal
> correct script for dash (ubuntu) yet.
> It may be worth detecting those shells to leverage that when we can.
understood. If we are able to get it to work without relying on sed
which seems to be way more platform specific than I had thought, then
I'd prefer such a solution.
> I have run the script with bash, ksh, zsh, and dash under Ubuntu and Mac.
> Csh and tcsh do not seem to work (do we expect them to?)
No, we don't expect the csh family of shells to be able to run /bin/sh
scripts.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org