You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "zentol (via GitHub)" <gi...@apache.org> on 2023/08/11 12:07:12 UTC
[GitHub] [flink] zentol opened a new pull request, #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
zentol opened a new pull request, #23195:
URL: https://github.com/apache/flink/pull/23195
This PR allows devs to run this script locally, just like that. Lower-level scripts are also easier to use.
The commits are quite granular and all have a commit message explaining the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "zentol (via GitHub)" <gi...@apache.org>.
zentol commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1297297491
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -17,31 +17,58 @@
# limitations under the License.
#
+usage() {
+ echo "Usage: $0 <maven-build-output>"
+ echo " <maven-build-output> A file containing the output of the Maven build."
+ echo ""
+ echo "mvnw clean package > <maven-build-output>"
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of ShadeOptionalChecker."
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
+ exit 0
+ ;;
+ esac
+done
+
+if [[ "$#" != "1" ]]; then
+ usage
+ exit 1
+fi
+
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/dependency_tree_optional.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
+$MVN dependency:tree -B -T1 > "${dependency_plugin_output}"
Review Comment:
yes, this could easily get lost int he commit history since it's not set on a separate line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "zentol (via GitHub)" <gi...@apache.org>.
zentol commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1294773109
##########
tools/ci/compile.sh:
##########
@@ -18,37 +18,48 @@
################################################################################
#
-# This file contains tooling for compiling Flink
+# This script compiles Flink and runs all QA checks apart from tests.
+#
+# This script should not contain any CI-specific logic; put these into compile_ci.sh instead.
+#
+# Usage: [MVN=/path/to/maven] tools/ci/compile.sh [additional maven args]
+# - Use the MVN environment variable to point the script to another maven installation.
+# - Any script argument is forwarded to the Flink maven build. Use it to skip/modify parts of the build process.
+#
+# Tips:
+# - '-Pskip-webui-build' skips the WebUI build.
+# - '-Dfast' skips Maven QA checks.
+# - '-Dmaven.clean.skip' skips recompilation of classes.
+# Example: tools/ci/compile.sh -Dmaven.clean.skip -Dfast -Pskip-webui-build, use -Dmaven.clean.skip to avoid recompiling classes.
+#
+# Warnings:
+# - Skipping modules via '-pl [!]<module>' is not recommended because checks may assume/require a full build.
#
-HERE="`dirname \"$0\"`" # relative
-HERE="`( cd \"$HERE\" && pwd )`" # absolutized and normalized
-if [ -z "$HERE" ] ; then
- exit 1 # fail
-fi
-CI_DIR="$HERE/../ci"
+CI_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
MVN_CLEAN_COMPILE_OUT="/tmp/clean_compile.out"
+MVN=${MVN:-./mvnw}
Review Comment:
If you call the low-level scripts, then mvnw is used by default. Since the commit targets the low level scripts this should be fine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "zentol (via GitHub)" <gi...@apache.org>.
zentol commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1294815629
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -19,22 +19,16 @@
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/optional_dep.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
-
-run_mvn dependency:tree -B > "${dependency_plugin_output}"
+$MVN dependency:tree -B > "${dependency_plugin_output}"
cat "${dependency_plugin_output}"
-cd "${CI_DIR}/flink-ci-tools/" || exit
-
-run_mvn exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args=\""${MVN_CLEAN_COMPILE_OUT}" "${dependency_plugin_output}"\"
+$MVN -pl tools/ci/flink-ci-tools exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args="${MVN_CLEAN_COMPILE_OUT} ${dependency_plugin_output}"
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ]; then
Review Comment:
> It's kind of annoying to scroll through the stdout logs
I think that's more of a symptom to be solved by sorting the output by severity (important stuff at the end). You shouldn't need to scroll at all to find the thing that broke CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "zentol (via GitHub)" <gi...@apache.org>.
zentol commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1296932433
##########
tools/ci/license_check.sh:
##########
@@ -17,6 +17,34 @@
# limitations under the License.
################################################################################
+usage() {
+ if [[ "$#" != "2" ]]; then
+ echo "Usage: $0 <maven-build-output> <deployed-artifacts-folder>"
+ echo " <maven-build-output> A file containing the output of the Maven build."
+ echo " <deployed-artifacts-folder> A directory containing a Maven repository into which the Flink artifacts were deployed."
+ echo ""
+ echo "Example preparation:"
+ echo " mvnw clean deploy -DaltDeploymentRepository=validation_repository::default::file:<deployed-artifacts-folder> > <maven-build-output>"
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of LicenseChecker."
+ fi
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
Review Comment:
doesnt work when used with brackets
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1297097220
##########
tools/ci/license_check.sh:
##########
@@ -17,6 +17,34 @@
# limitations under the License.
################################################################################
+usage() {
+ if [[ "$#" != "2" ]]; then
+ echo "Usage: $0 <maven-build-output> <deployed-artifacts-folder>"
+ echo " <maven-build-output> A file containing the output of the Maven build."
+ echo " <deployed-artifacts-folder> A directory containing a Maven repository into which the Flink artifacts were deployed."
+ echo ""
+ echo "Example preparation:"
+ echo " mvnw clean deploy -DaltDeploymentRepository=validation_repository::default::file:<deployed-artifacts-folder> > <maven-build-output>"
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of LicenseChecker."
+ fi
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
Review Comment:
:disappointed: true, I mixed that one up :facepalm: whatever :shrug:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1296832114
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -17,24 +17,51 @@
# limitations under the License.
#
+usage() {
+ echo "Usage: $0 <maven-build-output>"
+ echo " <maven-build-output> A file containing the output of the Maven build."
+ echo ""
+ echo "mvnw clean package > <maven-build-output>"
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of ShadeOptionalChecker."
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
+ exit 0
+ ;;
+ esac
+done
+
+if [[ "$#" != "1" ]]; then
+ usage
+ exit 1
+fi
+
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/optional_dep.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
+$MVN dependency:tree -B > "${dependency_plugin_output}"
Review Comment:
I did another pass over the code and identified one other location where we should add the `-T1` parameter. I commented on it below in a separate review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "zentol (via GitHub)" <gi...@apache.org>.
zentol commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1294777301
##########
tools/ci/compile_ci.sh:
##########
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+#
+# This script is the CI entrypoint for compiling Flink and running QA checks that don't require tests.
+#
Review Comment:
IMO we shouldn't put additional roadblocks into the way if you want to run the CI version locally. It will work (I guess).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1296771740
##########
tools/ci/verify_scala_suffixes.sh:
##########
@@ -37,20 +37,33 @@
#
# The script uses 'mvn dependency:tree -Dincludes=org.scala-lang' to list Scala
# dependent modules.
-CI_DIR=$1
-FLINK_ROOT=$2
+
+
+usage() {
+ echo "Usage: $0 "
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of ScalaSuffixChecker."
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
+ exit 0
+ ;;
+ esac
+done
+
+MVN=${MVN:-./mvnw}
echo "--- Flink Scala Dependency Analyzer ---"
echo "Analyzing modules for Scala dependencies using 'mvn dependency:tree'."
echo "If you haven't built the project, please do so first by running \"mvn clean install -DskipTests\""
-source "${CI_DIR}/maven-utils.sh"
+dependency_plugin_output=/tmp/dep.txt
-cd "$FLINK_ROOT" || exit
-
-dependency_plugin_output=${CI_DIR}/dep.txt
-
-run_mvn dependency:tree -Dincludes=org.scala-lang,:*_2.1*:: ${MAVEN_ARGUMENTS} >> "${dependency_plugin_output}"
+$MVN dependency:tree -Dincludes=org.scala-lang,:*_2.1*:: ${MAVEN_ARGUMENTS} > "${dependency_plugin_output}"
Review Comment:
```suggestion
$MVN -T1 dependency:tree -Dincludes=org.scala-lang,:*_2.1*:: ${MAVEN_ARGUMENTS} > "${dependency_plugin_output}"
```
##########
tools/ci/verify_scala_suffixes.sh:
##########
@@ -37,20 +37,33 @@
#
# The script uses 'mvn dependency:tree -Dincludes=org.scala-lang' to list Scala
# dependent modules.
-CI_DIR=$1
-FLINK_ROOT=$2
+
+
+usage() {
+ echo "Usage: $0 "
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of ScalaSuffixChecker."
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
+ exit 0
+ ;;
+ esac
+done
+
+MVN=${MVN:-./mvnw}
echo "--- Flink Scala Dependency Analyzer ---"
echo "Analyzing modules for Scala dependencies using 'mvn dependency:tree'."
echo "If you haven't built the project, please do so first by running \"mvn clean install -DskipTests\""
-source "${CI_DIR}/maven-utils.sh"
+dependency_plugin_output=/tmp/dep.txt
Review Comment:
```suggestion
dependency_plugin_output=/tmp/dependency-tree-including-scala.txt
```
nit: We might want to give it a more meaningful name to differentiate it from the other dependency:tree temporary file. ...even if it's only a temporary file. That makes things easier to debug issues.
##########
tools/ci/verify_scala_suffixes.sh:
##########
@@ -37,20 +37,33 @@
#
# The script uses 'mvn dependency:tree -Dincludes=org.scala-lang' to list Scala
# dependent modules.
-CI_DIR=$1
-FLINK_ROOT=$2
+
+
+usage() {
+ echo "Usage: $0 "
Review Comment:
```suggestion
echo "Usage: $0"
```
nit: just to avoid someone coming up with a hotfix PR
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -17,24 +17,51 @@
# limitations under the License.
#
+usage() {
+ echo "Usage: $0 <maven-build-output>"
+ echo " <maven-build-output> A file containing the output of the Maven build."
+ echo ""
+ echo "mvnw clean package > <maven-build-output>"
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of ShadeOptionalChecker."
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
+ exit 0
+ ;;
+ esac
+done
+
+if [[ "$#" != "1" ]]; then
+ usage
+ exit 1
+fi
+
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/optional_dep.txt
Review Comment:
```suggestion
dependency_plugin_output=/tmp/dependency-tree.txt
```
nit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1297104015
##########
tools/ci/compile.sh:
##########
@@ -69,34 +80,38 @@ fi
echo "============ Checking Javadocs ============"
+javadoc_output=/tmp/javadoc.out
+
# use the same invocation as .github/workflows/docs.sh
-run_mvn javadoc:aggregate -DadditionalJOption='-Xdoclint:none' \
+$MVN javadoc:aggregate -DadditionalJOption='-Xdoclint:none' \
Review Comment:
```suggestion
$MVN javadoc:aggregate -T1 -DadditionalJOption='-Xdoclint:none' \
```
according to your statement "every call (apart from mvn exec has to run with a parallelism of 1.". But tbh, I don't see value here because we're not parsing the output. In this sense, it would be up for the user to decide. :shrug:
##########
tools/ci/compile.sh:
##########
@@ -69,34 +80,38 @@ fi
echo "============ Checking Javadocs ============"
+javadoc_output=/tmp/javadoc.out
+
# use the same invocation as .github/workflows/docs.sh
-run_mvn javadoc:aggregate -DadditionalJOption='-Xdoclint:none' \
+$MVN javadoc:aggregate -DadditionalJOption='-Xdoclint:none' \
-Dmaven.javadoc.failOnError=false -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -Drat.skip=true \
- -Dheader=someTestHeader > javadoc.out
+ -Dheader=someTestHeader > ${javadoc_output}
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ] ; then
echo "ERROR in Javadocs. Printing full output:"
- cat javadoc.out ; rm javadoc.out
+ cat ${javadoc_output}
exit $EXIT_CODE
fi
echo "============ Checking Scaladocs ============"
-run_mvn scala:doc -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -pl flink-scala 2> scaladoc.out
+scaladoc_output=/tmp/scaladoc.out
+
+$MVN scala:doc -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -pl flink-scala 2> ${scaladoc_output}
Review Comment:
```suggestion
$MVN scala:doc -T1 -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -pl flink-scala 2> ${scaladoc_output}
```
according to your statement "every call (apart from mvn exec has to run with a parallelism of 1.". But here as well: We're not parsing the output. So it would be up to the user how to deal with it.
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -17,31 +17,58 @@
# limitations under the License.
#
+usage() {
+ echo "Usage: $0 <maven-build-output>"
+ echo " <maven-build-output> A file containing the output of the Maven build."
+ echo ""
+ echo "mvnw clean package > <maven-build-output>"
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of ShadeOptionalChecker."
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
+ exit 0
+ ;;
+ esac
+done
+
+if [[ "$#" != "1" ]]; then
+ usage
+ exit 1
+fi
+
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/dependency_tree_optional.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
+$MVN dependency:tree -B -T1 > "${dependency_plugin_output}"
Review Comment:
```suggestion
# dependency:tree needs to run without multi-threading support to ensure that the output stays parseable (the lines appear in a sequential order)
$MVN dependency:tree -B -T1 > "${dependency_plugin_output}"
```
Should we add a comment to ensure that there is no desire to "optimize" the maven call in the future?
##########
tools/ci/verify_scala_suffixes.sh:
##########
@@ -37,41 +37,52 @@
#
# The script uses 'mvn dependency:tree -Dincludes=org.scala-lang' to list Scala
# dependent modules.
-CI_DIR=$1
-FLINK_ROOT=$2
+
+
+usage() {
+ echo "Usage: $0"
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of ScalaSuffixChecker."
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
+ exit 0
+ ;;
+ esac
+done
+
+MVN=${MVN:-./mvnw}
echo "--- Flink Scala Dependency Analyzer ---"
echo "Analyzing modules for Scala dependencies using 'mvn dependency:tree'."
echo "If you haven't built the project, please do so first by running \"mvn clean install -DskipTests\""
-source "${CI_DIR}/maven-utils.sh"
-
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/dependency_tree_scala.txt
-dependency_plugin_output=${CI_DIR}/dep.txt
-
-run_mvn dependency:tree -Dincludes=org.scala-lang,:*_2.1*:: ${MAVEN_ARGUMENTS} >> "${dependency_plugin_output}"
+$MVN dependency:tree -Dincludes=org.scala-lang,:*_2.1*:: ${MAVEN_ARGUMENTS} -T1 > "${dependency_plugin_output}"
Review Comment:
```suggestion
# dependency:tree needs to run without multi-threading support to ensure that the output stays parseable (the lines appear in a sequential order)
$MVN dependency:tree -Dincludes=org.scala-lang,:*_2.1*:: ${MAVEN_ARGUMENTS} -T1 > "${dependency_plugin_output}"
```
Should we add a comment to ensure that there is no desire to "optimize" the maven call in the future?
##########
tools/ci/compile.sh:
##########
@@ -18,37 +18,48 @@
################################################################################
#
-# This file contains tooling for compiling Flink
+# This script compiles Flink and runs all QA checks apart from tests.
+#
+# This script should not contain any CI-specific logic; put these into compile_ci.sh instead.
+#
+# Usage: [MVN=/path/to/maven] tools/ci/compile.sh [additional maven args]
+# - Use the MVN environment variable to point the script to another maven installation.
+# - Any script argument is forwarded to the Flink maven build. Use it to skip/modify parts of the build process.
+#
+# Tips:
+# - '-Pskip-webui-build' skips the WebUI build.
+# - '-Dfast' skips Maven QA checks.
+# - '-Dmaven.clean.skip' skips recompilation of classes.
+# Example: tools/ci/compile.sh -Dmaven.clean.skip -Dfast -Pskip-webui-build, use -Dmaven.clean.skip to avoid recompiling classes.
+#
+# Warnings:
+# - Skipping modules via '-pl [!]<module>' is not recommended because checks may assume/require a full build.
#
-HERE="`dirname \"$0\"`" # relative
-HERE="`( cd \"$HERE\" && pwd )`" # absolutized and normalized
-if [ -z "$HERE" ] ; then
- exit 1 # fail
-fi
-CI_DIR="$HERE/../ci"
+CI_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
MVN_CLEAN_COMPILE_OUT="/tmp/clean_compile.out"
+MVN=${MVN:-./mvnw}
# Deploy into this directory, to run license checks on all jars staged for deployment.
# This helps us ensure that ALL artifacts we deploy to maven central adhere to our license conditions.
MVN_VALIDATION_DIR="/tmp/flink-validation-deployment"
+rm -rf ${MVN_VALIDATION_DIR}
# source required ci scripts
source "${CI_DIR}/stage.sh"
source "${CI_DIR}/shade.sh"
-source "${CI_DIR}/maven-utils.sh"
echo "Maven version:"
-run_mvn -version
+$MVN -version
echo "=============================================================================="
echo "Compiling Flink"
echo "=============================================================================="
EXIT_CODE=0
-run_mvn clean deploy -DaltDeploymentRepository=validation_repository::default::file:$MVN_VALIDATION_DIR -Dflink.convergence.phase=install -Pcheck-convergence \
- -Dmaven.javadoc.skip=true -U -DskipTests | tee $MVN_CLEAN_COMPILE_OUT
+$MVN clean deploy -DaltDeploymentRepository=validation_repository::default::file:$MVN_VALIDATION_DIR -Dflink.convergence.phase=install -Pcheck-convergence \
Review Comment:
```suggestion
# deploy needs to run without multi-threading support to ensure that the output stays parseable (the lines appear in a sequential order)
$MVN clean deploy -DaltDeploymentRepository=validation_repository::default::file:$MVN_VALIDATION_DIR -Dflink.convergence.phase=install -Pcheck-convergence \
```
just as a proposal here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1294597504
##########
tools/ci/compile.sh:
##########
@@ -18,37 +18,48 @@
################################################################################
#
-# This file contains tooling for compiling Flink
+# This script compiles Flink and runs all QA checks apart from tests.
+#
+# This script should not contain any CI-specific logic; put these into compile_ci.sh instead.
+#
+# Usage: [MVN=/path/to/maven] tools/ci/compile.sh [additional maven args]
+# - Use the MVN environment variable to point the script to another maven installation.
+# - Any script argument is forwarded to the Flink maven build. Use it to skip/modify parts of the build process.
+#
+# Tips:
+# - '-Pskip-webui-build' skips the WebUI build.
+# - '-Dfast' skips Maven QA checks.
+# - '-Dmaven.clean.skip' skips recompilation of classes.
+# Example: tools/ci/compile.sh -Dmaven.clean.skip -Dfast -Pskip-webui-build, use -Dmaven.clean.skip to avoid recompiling classes.
+#
+# Warnings:
+# - Skipping modules via '-pl [!]<module>' is not recommended because checks may assume/require a full build.
#
-HERE="`dirname \"$0\"`" # relative
-HERE="`( cd \"$HERE\" && pwd )`" # absolutized and normalized
-if [ -z "$HERE" ] ; then
- exit 1 # fail
-fi
-CI_DIR="$HERE/../ci"
+CI_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
MVN_CLEAN_COMPILE_OUT="/tmp/clean_compile.out"
+MVN=${MVN:-./mvnw}
Review Comment:
> \- use mvnw by default
fyi: The log message of ceb63ffa is a bit confusing. You claim that the Maven Wrapper is used by default in that commit. But this is only set up in d9fe1649
##########
tools/ci/maven-utils.sh:
##########
@@ -21,12 +21,10 @@ function run_mvn {
MVN_CMD="${M2_HOME}/bin/mvn"
fi
- ARGS=$@
- INVOCATION="$MVN_CMD $MVN_GLOBAL_OPTIONS $ARGS"
if [[ "$MVN_RUN_VERBOSE" != "false" ]]; then
- echo "Invoking mvn with '$INVOCATION'"
+ echo "Invoking mvn with '$MVN_GLOBAL_OPTIONS ${@}'"
Review Comment:
From the commit message:
> This comes with a slight maintenance cost to maven-utils.sh#run_mvn, but given that we'll likely throw this out in 1.19 anyway (because there are better ways to implement this on later maven versions), we can bear that cost.
What are the costs? The quotes around `$@`? :thinking: Is there a Jira issue for the "1.19 work"?
##########
tools/ci/compile_ci.sh:
##########
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+#
+# This script is the CI entrypoint for compiling Flink and running QA checks that don't require tests.
+#
Review Comment:
```suggestion
if [[ ! -z "${AZURE_LOCATION}" ]]; then
echo 'No "$AZURE_LOCATION" environmental found. This script is the CI entrypoint for compiling Flink and running QA checks that don't require tests. Please use ./compile.sh for local setups.'
exit 1
fi
```
##########
tools/ci/compile.sh:
##########
@@ -69,34 +80,38 @@ fi
echo "============ Checking Javadocs ============"
+javadoc_output=/tmp/javadoc.out
+
# use the same invocation as .github/workflows/docs.sh
-run_mvn javadoc:aggregate -DadditionalJOption='-Xdoclint:none' \
+$MVN javadoc:aggregate -DadditionalJOption='-Xdoclint:none' \
-Dmaven.javadoc.failOnError=false -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -Drat.skip=true \
- -Dheader=someTestHeader > javadoc.out
+ -Dheader=someTestHeader > ${javadoc_output}
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ] ; then
echo "ERROR in Javadocs. Printing full output:"
- cat javadoc.out ; rm javadoc.out
+ cat ${javadoc_output}
exit $EXIT_CODE
fi
echo "============ Checking Scaladocs ============"
-run_mvn scala:doc -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -pl flink-scala 2> scaladoc.out
+scaladoc_output=/tmp/scaladoc.out
+
+$MVN scala:doc -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -pl flink-scala 2> ${scaladoc_output}
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ] ; then
echo "ERROR in Scaladocs. Printing full output:"
- cat scaladoc.out ; rm scaladoc.out
+ cat ${scaladoc_output}
exit $EXIT_CODE
fi
echo "============ Checking bundled dependencies marked as optional ============"
-${CI_DIR}/verify_bundled_optional.sh $MVN_CLEAN_COMPILE_OUT "$CI_DIR" "$(pwd)" || exit $?
+MVN=$MVN ${CI_DIR}/verify_bundled_optional.sh $MVN_CLEAN_COMPILE_OUT || exit $?
echo "============ Checking scala suffixes ============"
-${CI_DIR}/verify_scala_suffixes.sh "$CI_DIR" "$(pwd)" || exit $?
+MVN=$MVN ${CI_DIR}/verify_scala_suffixes.sh || exit $?
Review Comment:
Same as above: Obsolete variable initialization `MVN`, AFAICS
##########
tools/ci/compile.sh:
##########
@@ -112,7 +127,7 @@ echo "============ Run license check ============"
find $MVN_VALIDATION_DIR
# We use a different Scala version with Java 17
if [[ ${PROFILE} != *"jdk17"* ]]; then
- ${CI_DIR}/license_check.sh $MVN_CLEAN_COMPILE_OUT $CI_DIR $(pwd) $MVN_VALIDATION_DIR || exit $?
+ MVN=$MVN ${CI_DIR}/license_check.sh $MVN_CLEAN_COMPILE_OUT $MVN_VALIDATION_DIR || exit $?
Review Comment:
Same as above: Obsolete variable initialization MVN, AFAICS
##########
tools/ci/compile.sh:
##########
@@ -69,34 +80,38 @@ fi
echo "============ Checking Javadocs ============"
+javadoc_output=/tmp/javadoc.out
+
# use the same invocation as .github/workflows/docs.sh
-run_mvn javadoc:aggregate -DadditionalJOption='-Xdoclint:none' \
+$MVN javadoc:aggregate -DadditionalJOption='-Xdoclint:none' \
-Dmaven.javadoc.failOnError=false -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -Drat.skip=true \
- -Dheader=someTestHeader > javadoc.out
+ -Dheader=someTestHeader > ${javadoc_output}
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ] ; then
echo "ERROR in Javadocs. Printing full output:"
- cat javadoc.out ; rm javadoc.out
+ cat ${javadoc_output}
exit $EXIT_CODE
fi
echo "============ Checking Scaladocs ============"
-run_mvn scala:doc -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -pl flink-scala 2> scaladoc.out
+scaladoc_output=/tmp/scaladoc.out
+
+$MVN scala:doc -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -pl flink-scala 2> ${scaladoc_output}
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ] ; then
echo "ERROR in Scaladocs. Printing full output:"
- cat scaladoc.out ; rm scaladoc.out
+ cat ${scaladoc_output}
exit $EXIT_CODE
fi
echo "============ Checking bundled dependencies marked as optional ============"
-${CI_DIR}/verify_bundled_optional.sh $MVN_CLEAN_COMPILE_OUT "$CI_DIR" "$(pwd)" || exit $?
+MVN=$MVN ${CI_DIR}/verify_bundled_optional.sh $MVN_CLEAN_COMPILE_OUT || exit $?
Review Comment:
Isn't `MVN_$MVN` kind of obsolete? Or is it meant for documentation purposes? :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "zentol (via GitHub)" <gi...@apache.org>.
zentol commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1294775040
##########
tools/ci/compile.sh:
##########
@@ -69,34 +80,38 @@ fi
echo "============ Checking Javadocs ============"
+javadoc_output=/tmp/javadoc.out
+
# use the same invocation as .github/workflows/docs.sh
-run_mvn javadoc:aggregate -DadditionalJOption='-Xdoclint:none' \
+$MVN javadoc:aggregate -DadditionalJOption='-Xdoclint:none' \
-Dmaven.javadoc.failOnError=false -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -Drat.skip=true \
- -Dheader=someTestHeader > javadoc.out
+ -Dheader=someTestHeader > ${javadoc_output}
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ] ; then
echo "ERROR in Javadocs. Printing full output:"
- cat javadoc.out ; rm javadoc.out
+ cat ${javadoc_output}
exit $EXIT_CODE
fi
echo "============ Checking Scaladocs ============"
-run_mvn scala:doc -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -pl flink-scala 2> scaladoc.out
+scaladoc_output=/tmp/scaladoc.out
+
+$MVN scala:doc -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -pl flink-scala 2> ${scaladoc_output}
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ] ; then
echo "ERROR in Scaladocs. Printing full output:"
- cat scaladoc.out ; rm scaladoc.out
+ cat ${scaladoc_output}
exit $EXIT_CODE
fi
echo "============ Checking bundled dependencies marked as optional ============"
-${CI_DIR}/verify_bundled_optional.sh $MVN_CLEAN_COMPILE_OUT "$CI_DIR" "$(pwd)" || exit $?
+MVN=$MVN ${CI_DIR}/verify_bundled_optional.sh $MVN_CLEAN_COMPILE_OUT || exit $?
Review Comment:
I wasn't sure so; it being bash and all that. Can't hurt to be explicit/consistent about it I guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "zentol (via GitHub)" <gi...@apache.org>.
zentol commented on PR #23195:
URL: https://github.com/apache/flink/pull/23195#issuecomment-1679154000
> Additionally, I would like to have a separate section (maybe ## Continuous Integration?) next to ## Building Apache Flink from Source in Flink's README.md. WDYT?
I think that we need to come up with a clear place for developer documentation. I didn't think of the README.md at all personally, and was wondering whether it should be added to the flinkDev/building.md docs, but they are also woefully unclear as to what their audience is (users vs dev), and so decided against it.
Realistically you ain't gonna lock at "build from source docs" when trying to replicate CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1294822852
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -19,22 +19,16 @@
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/optional_dep.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
-
-run_mvn dependency:tree -B > "${dependency_plugin_output}"
+$MVN dependency:tree -B > "${dependency_plugin_output}"
cat "${dependency_plugin_output}"
-cd "${CI_DIR}/flink-ci-tools/" || exit
-
-run_mvn exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args=\""${MVN_CLEAN_COMPILE_OUT}" "${dependency_plugin_output}"\"
+$MVN -pl tools/ci/flink-ci-tools exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args="${MVN_CLEAN_COMPILE_OUT} ${dependency_plugin_output}"
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ]; then
Review Comment:
For me, I'm confronted with the following output when running the `tools/ci/compile.sh` locally:
```
[INFO] --- exec-maven-plugin:3.1.0:java (default-cli) @ flink-ci-tools ---
16:18:36,976 DEBUG org.apache.flink.tools.ci.optional.ShadeOptionalChecker [] - Checking module 'flink-parent'.
16:18:36,980 DEBUG org.apache.flink.tools.ci.optional.ShadeOptionalChecker [] - Module is not bundling any dependencies.
16:18:36,980 INFO org.apache.flink.tools.ci.optional.ShadeOptionalChecker [] - OK: flink-parent
16:18:36,980 DEBUG org.apache.flink.tools.ci.optional.ShadeOptionalChecker [] - Checking module 'flink-dstl'.
[WARNING]
java.lang.IllegalStateException: Module flink-dstl listed by shade-plugin, but not dependency-plugin.
at org.apache.flink.tools.ci.optional.ShadeOptionalChecker.checkOptionalFlags (ShadeOptionalChecker.java:134)
at org.apache.flink.tools.ci.optional.ShadeOptionalChecker.main (ShadeOptionalChecker.java:85)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:279)
at java.lang.Thread.run (Thread.java:750)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.029 s (Wall Clock)
[INFO] Finished at: 2023-08-15T16:18:36+02:00
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "skip-webui-build" could not be activated because it does not exist.
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.1.0:java (default-cli) on project flink-ci-tools: An exception occurred while executing the Java class. Module flink-dstl listed by shade-plugin, but not dependency-plugin. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
==============================================================================
Optional Check failed. See previous output for details.
==============================================================================
```
If I want to verify manually where the error comes from, I have to go through the logs. AFAIU, this output is actually not the usual script output but an unexpected error, huh? That would be a reason to go with your argument... :thinking: I'm gonna investigate it tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1296070723
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -17,24 +17,51 @@
# limitations under the License.
#
+usage() {
+ echo "Usage: $0 <maven-build-output>"
+ echo " <maven-build-output> A file containing the output of the Maven build."
+ echo ""
+ echo "mvnw clean package > <maven-build-output>"
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of ShadeOptionalChecker."
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
+ exit 0
+ ;;
+ esac
+done
+
+if [[ "$#" != "1" ]]; then
+ usage
+ exit 1
+fi
+
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/optional_dep.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
+$MVN dependency:tree -B > "${dependency_plugin_output}"
Review Comment:
```suggestion
$MVN dependency:tree -T1 -B > "${dependency_plugin_output}"
```
That caused the error on my side. I use a local `.mvn/maven.config` to set default parameters (`-DskipTests -Dfast -Pskip-webui-build -T1C` and a custom settings xml that points to a separate local repository). The multi-threading is the problem here. `dependency:tree` can run in multiple threads. But that will screw up the output of course which causes the dependency tree parsing to malfunction :facepalm:
##########
tools/ci/license_check.sh:
##########
@@ -17,6 +17,34 @@
# limitations under the License.
################################################################################
+usage() {
+ if [[ "$#" != "2" ]]; then
Review Comment:
I guess, the if condition is not necessary here
##########
tools/ci/license_check.sh:
##########
@@ -17,6 +17,34 @@
# limitations under the License.
################################################################################
+usage() {
+ if [[ "$#" != "2" ]]; then
+ echo "Usage: $0 <maven-build-output> <deployed-artifacts-folder>"
+ echo " <maven-build-output> A file containing the output of the Maven build."
+ echo " <deployed-artifacts-folder> A directory containing a Maven repository into which the Flink artifacts were deployed."
+ echo ""
+ echo "Example preparation:"
+ echo " mvnw clean deploy -DaltDeploymentRepository=validation_repository::default::file:<deployed-artifacts-folder> > <maven-build-output>"
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of LicenseChecker."
+ fi
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
Review Comment:
nit: Even though it's possible to do it like that I think we should call the method with brackets (i.e. `usage()`) to improve the readability of the code.
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -17,24 +17,51 @@
# limitations under the License.
#
+usage() {
+ echo "Usage: $0 <maven-build-output>"
+ echo " <maven-build-output> A file containing the output of the Maven build."
+ echo ""
+ echo "mvnw clean package > <maven-build-output>"
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of ShadeOptionalChecker."
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
+ exit 0
+ ;;
+ esac
+done
+
+if [[ "$#" != "1" ]]; then
+ usage
+ exit 1
+fi
+
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/optional_dep.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
+$MVN dependency:tree -B > "${dependency_plugin_output}"
Review Comment:
I'm wondering whether we should apply this in every mvn command for which we analyze the output. To be on the safe side :thinking:
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -17,24 +17,51 @@
# limitations under the License.
#
+usage() {
+ echo "Usage: $0 <maven-build-output>"
+ echo " <maven-build-output> A file containing the output of the Maven build."
+ echo ""
+ echo "mvnw clean package > <maven-build-output>"
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of ShadeOptionalChecker."
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
+ exit 0
+ ;;
+ esac
+done
+
+if [[ "$#" != "1" ]]; then
+ usage
+ exit 1
+fi
+
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/optional_dep.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
+$MVN dependency:tree -B > "${dependency_plugin_output}"
+EXIT_CODE=$?
-run_mvn dependency:tree -B > "${dependency_plugin_output}"
+if [ $EXIT_CODE != 0 ]; then
+ cat ${dependency_plugin_output}
+ echo "=============================================================================="
+ echo "Optional Check failed. The dependency tree could not be determined. See previous output for details."
+ echo "=============================================================================="
+ exit 1
Review Comment:
that could be also applied to the unchange code below.
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -17,24 +17,51 @@
# limitations under the License.
#
+usage() {
+ echo "Usage: $0 <maven-build-output>"
+ echo " <maven-build-output> A file containing the output of the Maven build."
+ echo ""
+ echo "mvnw clean package > <maven-build-output>"
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of ShadeOptionalChecker."
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
+ exit 0
+ ;;
+ esac
+done
+
+if [[ "$#" != "1" ]]; then
+ usage
+ exit 1
+fi
+
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/optional_dep.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
+$MVN dependency:tree -B > "${dependency_plugin_output}"
+EXIT_CODE=$?
-run_mvn dependency:tree -B > "${dependency_plugin_output}"
+if [ $EXIT_CODE != 0 ]; then
+ cat ${dependency_plugin_output}
+ echo "=============================================================================="
+ echo "Optional Check failed. The dependency tree could not be determined. See previous output for details."
+ echo "=============================================================================="
+ exit 1
Review Comment:
```suggestion
exit $EXIT_CODE
```
nit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol merged pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "zentol (via GitHub)" <gi...@apache.org>.
zentol merged PR #23195:
URL: https://github.com/apache/flink/pull/23195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1294643648
##########
tools/ci/license_check.sh:
##########
@@ -18,15 +18,11 @@
################################################################################
MVN_CLEAN_COMPILE_OUT=$1
Review Comment:
The same would apply to `tools/ci/verify_bundled_optional.sh` and `tools/ci/verify_scala_suffixes.sh`
##########
tools/ci/license_check.sh:
##########
@@ -18,15 +18,11 @@
################################################################################
MVN_CLEAN_COMPILE_OUT=$1
Review Comment:
```suggestion
if [[ "$#" != "2" ]]; then
echo "Usage: $0 <maven-clean-compile-output> <deployed-artifacts-folder>"
echo " <maven-clean-compile-output> A file containing the Maven run's log output."
echo " <deployed-artifacts-folder> A mirror folder that contains the deployed Flink artifacts."
echo ""
echo "The environment variable MVN is used to specify the Maven binaries."
echo "See further details in the JavaDoc of LicenseChecker.".
exit 1
fi
MVN_CLEAN_COMPILE_OUT=$1
```
can we add a usage info in case a user wants to call a script individually? It serves as some kind of documentation for this specific script. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "zentol (via GitHub)" <gi...@apache.org>.
zentol commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1294770709
##########
tools/ci/maven-utils.sh:
##########
@@ -21,12 +21,10 @@ function run_mvn {
MVN_CMD="${M2_HOME}/bin/mvn"
fi
- ARGS=$@
- INVOCATION="$MVN_CMD $MVN_GLOBAL_OPTIONS $ARGS"
if [[ "$MVN_RUN_VERBOSE" != "false" ]]; then
- echo "Invoking mvn with '$INVOCATION'"
+ echo "Invoking mvn with '$MVN_GLOBAL_OPTIONS ${@}'"
Review Comment:
> What are the costs?
The cost is `$MVN_GLOBAL_OPTIONS ${@}` being duplicated in the code.
> Is there a Jira issue for the "1.19 work"?
There is https://issues.apache.org/jira/browse/FLINK-32181, although it lacks a subtasks for this specific item.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #23195:
URL: https://github.com/apache/flink/pull/23195#issuecomment-1674651885
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "70c0149c6e1453a09972b071e2afdda0243d945a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70c0149c6e1453a09972b071e2afdda0243d945a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70c0149c6e1453a09972b071e2afdda0243d945a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "zentol (via GitHub)" <gi...@apache.org>.
zentol commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1295738061
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -19,22 +19,16 @@
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/optional_dep.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
-
-run_mvn dependency:tree -B > "${dependency_plugin_output}"
+$MVN dependency:tree -B > "${dependency_plugin_output}"
cat "${dependency_plugin_output}"
-cd "${CI_DIR}/flink-ci-tools/" || exit
-
-run_mvn exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args=\""${MVN_CLEAN_COMPILE_OUT}" "${dependency_plugin_output}"\"
+$MVN -pl tools/ci/flink-ci-tools exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args="${MVN_CLEAN_COMPILE_OUT} ${dependency_plugin_output}"
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ]; then
Review Comment:
note that you can also do `tools/ci/compile.sh | tee output.txt`. The scripts doesn't necessarily have to take care of that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "zentol (via GitHub)" <gi...@apache.org>.
zentol commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1296947756
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -17,24 +17,51 @@
# limitations under the License.
#
+usage() {
+ echo "Usage: $0 <maven-build-output>"
+ echo " <maven-build-output> A file containing the output of the Maven build."
+ echo ""
+ echo "mvnw clean package > <maven-build-output>"
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of ShadeOptionalChecker."
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
+ exit 0
+ ;;
+ esac
+done
+
+if [[ "$#" != "1" ]]; then
+ usage
+ exit 1
+fi
+
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/optional_dep.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
+$MVN dependency:tree -B > "${dependency_plugin_output}"
Review Comment:
_every_ call (apart from `mvn exec` has to run with a parallelism of 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1294668273
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -19,22 +19,16 @@
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/optional_dep.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
-
-run_mvn dependency:tree -B > "${dependency_plugin_output}"
+$MVN dependency:tree -B > "${dependency_plugin_output}"
cat "${dependency_plugin_output}"
-cd "${CI_DIR}/flink-ci-tools/" || exit
-
-run_mvn exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args=\""${MVN_CLEAN_COMPILE_OUT}" "${dependency_plugin_output}"\"
+$MVN -pl tools/ci/flink-ci-tools exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args="${MVN_CLEAN_COMPILE_OUT} ${dependency_plugin_output}"
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ]; then
Review Comment:
I'm wondering whether we should pipe the console output of `ShadeOptionalChecker` through `tee` and have the output available in a file. We could add the file path to the echo statement below (rather than referring to the "previous output for details". It's kind of annoying to scroll through the stdout logs
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -19,22 +19,16 @@
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/optional_dep.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
-
-run_mvn dependency:tree -B > "${dependency_plugin_output}"
+$MVN dependency:tree -B > "${dependency_plugin_output}"
cat "${dependency_plugin_output}"
-cd "${CI_DIR}/flink-ci-tools/" || exit
-
-run_mvn exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args=\""${MVN_CLEAN_COMPILE_OUT}" "${dependency_plugin_output}"\"
+$MVN -pl tools/ci/flink-ci-tools exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args="${MVN_CLEAN_COMPILE_OUT} ${dependency_plugin_output}"
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ]; then
Review Comment:
That might apply to the license checker and the scala suffix checker as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1294670637
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -19,22 +19,16 @@
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/optional_dep.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
-
-run_mvn dependency:tree -B > "${dependency_plugin_output}"
+$MVN dependency:tree -B > "${dependency_plugin_output}"
cat "${dependency_plugin_output}"
-cd "${CI_DIR}/flink-ci-tools/" || exit
-
-run_mvn exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args=\""${MVN_CLEAN_COMPILE_OUT}" "${dependency_plugin_output}"\"
+$MVN -pl tools/ci/flink-ci-tools exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args="${MVN_CLEAN_COMPILE_OUT} ${dependency_plugin_output}"
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ]; then
Review Comment:
We could also print the input parameters of the script to allow a copy&paste to trigger a rerun.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1296771740
##########
tools/ci/verify_scala_suffixes.sh:
##########
@@ -37,20 +37,33 @@
#
# The script uses 'mvn dependency:tree -Dincludes=org.scala-lang' to list Scala
# dependent modules.
-CI_DIR=$1
-FLINK_ROOT=$2
+
+
+usage() {
+ echo "Usage: $0 "
+ echo ""
+ echo "The environment variable MVN is used to specify the Maven binaries; defaults to 'mvnw'."
+ echo "See further details in the JavaDoc of ScalaSuffixChecker."
+}
+
+while getopts 'h' o; do
+ case "${o}" in
+ h)
+ usage
+ exit 0
+ ;;
+ esac
+done
+
+MVN=${MVN:-./mvnw}
echo "--- Flink Scala Dependency Analyzer ---"
echo "Analyzing modules for Scala dependencies using 'mvn dependency:tree'."
echo "If you haven't built the project, please do so first by running \"mvn clean install -DskipTests\""
-source "${CI_DIR}/maven-utils.sh"
+dependency_plugin_output=/tmp/dep.txt
-cd "$FLINK_ROOT" || exit
-
-dependency_plugin_output=${CI_DIR}/dep.txt
-
-run_mvn dependency:tree -Dincludes=org.scala-lang,:*_2.1*:: ${MAVEN_ARGUMENTS} >> "${dependency_plugin_output}"
+$MVN dependency:tree -Dincludes=org.scala-lang,:*_2.1*:: ${MAVEN_ARGUMENTS} > "${dependency_plugin_output}"
Review Comment:
```suggestion
$MVN -T1 dependency:tree -Dincludes=org.scala-lang,:*_2.1*:: ${MAVEN_ARGUMENTS} > "${dependency_plugin_output}"
```
Here we have to add `-T1` as well to make sure that the output is actually parseable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "zentol (via GitHub)" <gi...@apache.org>.
zentol commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1297137606
##########
tools/ci/compile.sh:
##########
@@ -69,34 +80,38 @@ fi
echo "============ Checking Javadocs ============"
+javadoc_output=/tmp/javadoc.out
+
# use the same invocation as .github/workflows/docs.sh
-run_mvn javadoc:aggregate -DadditionalJOption='-Xdoclint:none' \
+$MVN javadoc:aggregate -DadditionalJOption='-Xdoclint:none' \
Review Comment:
javadoc can run in parallel.
##########
tools/ci/compile.sh:
##########
@@ -69,34 +80,38 @@ fi
echo "============ Checking Javadocs ============"
+javadoc_output=/tmp/javadoc.out
+
# use the same invocation as .github/workflows/docs.sh
-run_mvn javadoc:aggregate -DadditionalJOption='-Xdoclint:none' \
+$MVN javadoc:aggregate -DadditionalJOption='-Xdoclint:none' \
-Dmaven.javadoc.failOnError=false -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -Drat.skip=true \
- -Dheader=someTestHeader > javadoc.out
+ -Dheader=someTestHeader > ${javadoc_output}
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ] ; then
echo "ERROR in Javadocs. Printing full output:"
- cat javadoc.out ; rm javadoc.out
+ cat ${javadoc_output}
exit $EXIT_CODE
fi
echo "============ Checking Scaladocs ============"
-run_mvn scala:doc -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -pl flink-scala 2> scaladoc.out
+scaladoc_output=/tmp/scaladoc.out
+
+$MVN scala:doc -Dcheckstyle.skip=true -Denforcer.skip=true -Dspotless.skip=true -pl flink-scala 2> ${scaladoc_output}
Review Comment:
scaladoc can run in parallel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "zentol (via GitHub)" <gi...@apache.org>.
zentol commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1295729827
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -19,22 +19,16 @@
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/optional_dep.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
-
-run_mvn dependency:tree -B > "${dependency_plugin_output}"
+$MVN dependency:tree -B > "${dependency_plugin_output}"
cat "${dependency_plugin_output}"
-cd "${CI_DIR}/flink-ci-tools/" || exit
-
-run_mvn exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args=\""${MVN_CLEAN_COMPILE_OUT}" "${dependency_plugin_output}"\"
+$MVN -pl tools/ci/flink-ci-tools exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args="${MVN_CLEAN_COMPILE_OUT} ${dependency_plugin_output}"
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ]; then
Review Comment:
I ran it locally and it worked fine.
IIRC we don't check that the dependency-plugin call fails; in which case we pipe garbage into the checker which typically gives you this error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a diff in pull request #23195: [FLINK-32834] Ease local use of tools/ci/compile.sh
Posted by "zentol (via GitHub)" <gi...@apache.org>.
zentol commented on code in PR #23195:
URL: https://github.com/apache/flink/pull/23195#discussion_r1295742223
##########
tools/ci/verify_bundled_optional.sh:
##########
@@ -19,22 +19,16 @@
## Checks that all bundled dependencies are marked as optional in the poms
MVN_CLEAN_COMPILE_OUT=$1
-CI_DIR=$2
-FLINK_ROOT=$3
-source "${CI_DIR}/maven-utils.sh"
+MVN=${MVN:-./mvnw}
-cd "$FLINK_ROOT" || exit
+dependency_plugin_output=/tmp/optional_dep.txt
-dependency_plugin_output=${CI_DIR}/optional_dep.txt
-
-run_mvn dependency:tree -B > "${dependency_plugin_output}"
+$MVN dependency:tree -B > "${dependency_plugin_output}"
cat "${dependency_plugin_output}"
-cd "${CI_DIR}/flink-ci-tools/" || exit
-
-run_mvn exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args=\""${MVN_CLEAN_COMPILE_OUT}" "${dependency_plugin_output}"\"
+$MVN -pl tools/ci/flink-ci-tools exec:java -Dexec.mainClass=org.apache.flink.tools.ci.optional.ShadeOptionalChecker -Dexec.args="${MVN_CLEAN_COMPILE_OUT} ${dependency_plugin_output}"
EXIT_CODE=$?
if [ $EXIT_CODE != 0 ]; then
Review Comment:
> IIRC we don't check that the dependency-plugin call fails; in which case we pipe garbage into the checker which typically gives you this error.
added a check to prevent this from silently happening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org