You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@groovy.apache.org by Dimitar Vassilev <di...@gmail.com> on 2020/07/02 13:32:50 UTC
Re: Pointers needed for complex groovy shell variable escapes
Hi all,
I will need some help with the following extended code around curl and
the authentication. We switched to authenticated setup on the app
side. It seems now I cannot get past the authorization process as I
probably need more escapes.
I would be grateful for any pointers on how to improve the
authenticated curl calls by escapes or by using withCredentials and
escapes.
environment {
CREDS_USR = credentials('common-creds')
SEARCH_CSV_URL = "https://foo.com/search.csv"
VIEW = "baz"
}
script {
sh '''#!/usr/bin/env bash
TTL_PRJ="$(curl ${CURL_OPTS} --user
${CREDS_USR}:${CREDS_PSW} ${SEARCH_CSV_URL} | awk -F , 'NR > 1 {print
$2}'| wc -l)"
FIN_PRJ="$(curl ${CURL_OPTS} --user
${CREDS_USR}:${CREDS_PSW} ${SEARCH_CSV_URL} | awk -F , 'NR > 1 {print
$2}'| grep Finished | wc -l)"
SUCC_RATE="$(awk -v succ_prj="${FIN_PRJ}" -v
tl_prj="${TTL_PRJ}" 'BEGIN { print int(( succ_prj / tl_prj )*100) }')"
mail -s "build stats attached for "\${VIEW}""
foo@foo.com <<EOF
================Printing build stats:==================
Total projects for "\${VIEW}" is: ${TTL_PRJ}
Finished projects for "\${VIEW}" is: ${FIN_PRJ}
Success rate in percentage is: ${SUCC_RATE}
EOF
'''
}
}
}
}
}
На вт, 24.03.2020 г. в 17:43 Dimitar Vassilev
<di...@gmail.com> написа:
>
> Thanks Paul.
> After switching to ''' and moving the EOF marker to the very beginning of the line ( yep I know I break the indent), but I don't like warnings like
> script.sh: line 16: warning: here-document at line 10 delimited by end-of-file (wanted `EOF')
> Next I will try using csv and https parser to do the stuff better.
> Thank you and the community!
> Regards,
> Dimitar
>
> На вт, 24.03.2020 г. в 15:43 Paul King <pa...@asert.com.au> написа:
>>
>> $ in Groovy double quoted strings indicates the start of a placeholder, so you are expecting Groovy to supply the values for things like ${CURL_OPTS}.
>> I suspect you want the shell environment to replace those. So you can either use the triple single quoted string (exactly what it is intended for) or you can escape each $, e.g. \$.
>> The error is for expressions like $2 which isn't a valid Groovy variable name so not suitable for replacement.
>>
>> On Tue, Mar 24, 2020 at 11:27 PM Dimitar Vassilev <di...@gmail.com> wrote:
>>>
>>> Greetings all,
>>> Hope you are well and safe. I will need some pointers how to properly escape the following piece of code into double quoted multi line shell step
>>> ```
>>> #!/usr/bin/env bash
>>> CURR_BL="foo_200227_203942"
>>> HUB_URL="https://baz-baz.com:8045"
>>> SEARCH_CSV_URL="${HUB_URL}/project_search.csv?query="${CURR_BL}"&scope=all"
>>> CURL_OPTS="-s"
>>> TTL_PRJ="$(curl "${CURL_OPTS}" ${SEARCH_CSV_URL} | awk -F , '{print $2}'| wc -l)"
>>> FIN_PRJ="$(curl "${CURL_OPTS}" ${SEARCH_CSV_URL} | awk -F , '{print $2}'| grep Finished | wc -l)"
>>> SUCC_RATE="$(awk -v succ_prj="${FIN_PRJ}" -v tl_prj="${TTL_PRJ}" 'BEGIN { print int(( succ_prj / tl_prj )*100) }')"
>>> LOG_PLACE="/foo/baz"
>>> APPL_NAME="foobar"
>>> CURR_VIEW="foo_200227_203942_seafood_int"
>>>
>>> cd "${LOG_PLACE}/${APPL_NAME}/${CURR_VIEW}"/;
>>> cat "${LOG_PLACE}/${APPL_NAME}/${CURR_VIEW}"/.build_[0-9]*/*.err >> "${CURR_VIEW}_err.log"
>>> mail -a "${CURR_VIEW}_err.log" -s "Code quality build stats and error logs attached for ${CURR_VIEW}" me@me.em << EOF
>>> ================Printing Code quality build stats:==================
>>> Total projects for ${CURR_BL} is: ${TTL_PRJ}
>>> Finished projects for ${CURR_BL} is: ${FIN_PRJ}
>>> Success rate in percentage is: ${SUCC_RATE}
>>> EOF
>>> ```
>>> When running the above script as Jenkins shell step with """ I get the error
>>> illegal string body character after dollar sign;
>>> solution: either escape a literal dollar sign "\$5" or bracket the value expression "${5}" @ line 13, column 86.
>>> earch.csv?query="${CURR_BL}"&scope=a
>>>
>>> Sample code with the error trigger is included
>>> ```
>>> pipeline {
>>> agent {label 'baz'}
>>> stages {
>>> stage('Test') {
>>> steps {
>>> script{
>>> sh"""#!/usr/bin/env bash
>>> CURR_BL="foo_200227_203942"
>>> HUB_URL="https://baz-baz.com:8045"
>>> SEARCH_CSV_URL="${HUB_URL}/project_search.csv?query="${CURR_BL}"&scope=all"
>>> CURL_OPTS="-s"
>>> TTL_PRJ="$(curl "${CURL_OPTS}" ${SEARCH_CSV_URL} | awk -F , '{print $2}'| wc -l)"
>>> FIN_PRJ="$(curl "${CURL_OPTS}" ${SEARCH_CSV_URL} | awk -F , '{print $2}'| grep Finished | wc -l)"
>>> SUCC_RATE="$(awk -v succ_prj="${FIN_PRJ}" -v tl_prj="${TTL_PRJ}" 'BEGIN { print int(( succ_prj / tl_prj )*100) }')"
>>> LOG_PLACE="/foo/baz"
>>> APPL_NAME="foobar"
>>> CURR_VIEW="foo_200227_203942_seafood_int"
>>> cd "${LOG_PLACE}/${APPL_NAME}/${CURR_VIEW}"/;
>>> cat "${LOG_PLACE}/${APPL_NAME}/${CURR_VIEW}"/.build_[0-9]*/*.err >> "${CURR_VIEW}_err.log"
>>> mail -a "${CURR_VIEW}_err.log" -s "Code quality build stats and error logs attached for ${CURR_VIEW}" me@me.em << EOF
>>> ================Printing Code quality build stats:==================
>>> Total projects for ${CURR_BL} is: ${TTL_PRJ}
>>> Finished projects for ${CURR_BL} is: ${FIN_PRJ}
>>> Success rate in percentage is: ${SUCC_RATE}
>>> EOF
>>> """
>>> }
>>> }
>>> }
>>> }
>>> }
>>> ```
>>> So it seems I can't make groovy enough groovy using """. With ''' it works fine. However I prefer to educate myself if possible. Thanks for making this happen.
Re: Pointers needed for complex groovy shell variable escapes
Posted by Dimitar Vassilev <di...@gmail.com>.
Long story short
when using intermediate variables from text files, ensure 2 things
a) echo -n $VAR > file name
b) properly encode the variable - see
https://stackoverflow.com/questions/296536/how-to-urlencode-data-for-curl-command
and other examples - application-related.
In the end I did go with \\\"$var\\\".
Thanks!
На чт, 2.07.2020 г. в 16:32 Dimitar Vassilev
<di...@gmail.com> написа:
>
> Hi all,
> I will need some help with the following extended code around curl and
> the authentication. We switched to authenticated setup on the app
> side. It seems now I cannot get past the authorization process as I
> probably need more escapes.
> I would be grateful for any pointers on how to improve the
> authenticated curl calls by escapes or by using withCredentials and
> escapes.
>
> environment {
> CREDS_USR = credentials('common-creds')
> SEARCH_CSV_URL = "https://foo.com/search.csv"
> VIEW = "baz"
> }
> script {
> sh '''#!/usr/bin/env bash
> TTL_PRJ="$(curl ${CURL_OPTS} --user
> ${CREDS_USR}:${CREDS_PSW} ${SEARCH_CSV_URL} | awk -F , 'NR > 1 {print
> $2}'| wc -l)"
> FIN_PRJ="$(curl ${CURL_OPTS} --user
> ${CREDS_USR}:${CREDS_PSW} ${SEARCH_CSV_URL} | awk -F , 'NR > 1 {print
> $2}'| grep Finished | wc -l)"
> SUCC_RATE="$(awk -v succ_prj="${FIN_PRJ}" -v
> tl_prj="${TTL_PRJ}" 'BEGIN { print int(( succ_prj / tl_prj )*100) }')"
> mail -s "build stats attached for "\${VIEW}""
> foo@foo.com <<EOF
> ================Printing build stats:==================
> Total projects for "\${VIEW}" is: ${TTL_PRJ}
> Finished projects for "\${VIEW}" is: ${FIN_PRJ}
> Success rate in percentage is: ${SUCC_RATE}
> EOF
> '''
>
> }
> }
> }
> }
> }
>
> На вт, 24.03.2020 г. в 17:43 Dimitar Vassilev
> <di...@gmail.com> написа:
> >
> > Thanks Paul.
> > After switching to ''' and moving the EOF marker to the very beginning of the line ( yep I know I break the indent), but I don't like warnings like
> > script.sh: line 16: warning: here-document at line 10 delimited by end-of-file (wanted `EOF')
> > Next I will try using csv and https parser to do the stuff better.
> > Thank you and the community!
> > Regards,
> > Dimitar
> >
> > На вт, 24.03.2020 г. в 15:43 Paul King <pa...@asert.com.au> написа:
> >>
> >> $ in Groovy double quoted strings indicates the start of a placeholder, so you are expecting Groovy to supply the values for things like ${CURL_OPTS}.
> >> I suspect you want the shell environment to replace those. So you can either use the triple single quoted string (exactly what it is intended for) or you can escape each $, e.g. \$.
> >> The error is for expressions like $2 which isn't a valid Groovy variable name so not suitable for replacement.
> >>
> >> On Tue, Mar 24, 2020 at 11:27 PM Dimitar Vassilev <di...@gmail.com> wrote:
> >>>
> >>> Greetings all,
> >>> Hope you are well and safe. I will need some pointers how to properly escape the following piece of code into double quoted multi line shell step
> >>> ```
> >>> #!/usr/bin/env bash
> >>> CURR_BL="foo_200227_203942"
> >>> HUB_URL="https://baz-baz.com:8045"
> >>> SEARCH_CSV_URL="${HUB_URL}/project_search.csv?query="${CURR_BL}"&scope=all"
> >>> CURL_OPTS="-s"
> >>> TTL_PRJ="$(curl "${CURL_OPTS}" ${SEARCH_CSV_URL} | awk -F , '{print $2}'| wc -l)"
> >>> FIN_PRJ="$(curl "${CURL_OPTS}" ${SEARCH_CSV_URL} | awk -F , '{print $2}'| grep Finished | wc -l)"
> >>> SUCC_RATE="$(awk -v succ_prj="${FIN_PRJ}" -v tl_prj="${TTL_PRJ}" 'BEGIN { print int(( succ_prj / tl_prj )*100) }')"
> >>> LOG_PLACE="/foo/baz"
> >>> APPL_NAME="foobar"
> >>> CURR_VIEW="foo_200227_203942_seafood_int"
> >>>
> >>> cd "${LOG_PLACE}/${APPL_NAME}/${CURR_VIEW}"/;
> >>> cat "${LOG_PLACE}/${APPL_NAME}/${CURR_VIEW}"/.build_[0-9]*/*.err >> "${CURR_VIEW}_err.log"
> >>> mail -a "${CURR_VIEW}_err.log" -s "Code quality build stats and error logs attached for ${CURR_VIEW}" me@me.em << EOF
> >>> ================Printing Code quality build stats:==================
> >>> Total projects for ${CURR_BL} is: ${TTL_PRJ}
> >>> Finished projects for ${CURR_BL} is: ${FIN_PRJ}
> >>> Success rate in percentage is: ${SUCC_RATE}
> >>> EOF
> >>> ```
> >>> When running the above script as Jenkins shell step with """ I get the error
> >>> illegal string body character after dollar sign;
> >>> solution: either escape a literal dollar sign "\$5" or bracket the value expression "${5}" @ line 13, column 86.
> >>> earch.csv?query="${CURR_BL}"&scope=a
> >>>
> >>> Sample code with the error trigger is included
> >>> ```
> >>> pipeline {
> >>> agent {label 'baz'}
> >>> stages {
> >>> stage('Test') {
> >>> steps {
> >>> script{
> >>> sh"""#!/usr/bin/env bash
> >>> CURR_BL="foo_200227_203942"
> >>> HUB_URL="https://baz-baz.com:8045"
> >>> SEARCH_CSV_URL="${HUB_URL}/project_search.csv?query="${CURR_BL}"&scope=all"
> >>> CURL_OPTS="-s"
> >>> TTL_PRJ="$(curl "${CURL_OPTS}" ${SEARCH_CSV_URL} | awk -F , '{print $2}'| wc -l)"
> >>> FIN_PRJ="$(curl "${CURL_OPTS}" ${SEARCH_CSV_URL} | awk -F , '{print $2}'| grep Finished | wc -l)"
> >>> SUCC_RATE="$(awk -v succ_prj="${FIN_PRJ}" -v tl_prj="${TTL_PRJ}" 'BEGIN { print int(( succ_prj / tl_prj )*100) }')"
> >>> LOG_PLACE="/foo/baz"
> >>> APPL_NAME="foobar"
> >>> CURR_VIEW="foo_200227_203942_seafood_int"
> >>> cd "${LOG_PLACE}/${APPL_NAME}/${CURR_VIEW}"/;
> >>> cat "${LOG_PLACE}/${APPL_NAME}/${CURR_VIEW}"/.build_[0-9]*/*.err >> "${CURR_VIEW}_err.log"
> >>> mail -a "${CURR_VIEW}_err.log" -s "Code quality build stats and error logs attached for ${CURR_VIEW}" me@me.em << EOF
> >>> ================Printing Code quality build stats:==================
> >>> Total projects for ${CURR_BL} is: ${TTL_PRJ}
> >>> Finished projects for ${CURR_BL} is: ${FIN_PRJ}
> >>> Success rate in percentage is: ${SUCC_RATE}
> >>> EOF
> >>> """
> >>> }
> >>> }
> >>> }
> >>> }
> >>> }
> >>> ```
> >>> So it seems I can't make groovy enough groovy using """. With ''' it works fine. However I prefer to educate myself if possible. Thanks for making this happen.