You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by GitBox <gi...@apache.org> on 2022/10/31 12:53:19 UTC
[GitHub] [flink-connector-shared-utils] zentol opened a new pull request, #1: Add first version of release scripts
zentol opened a new pull request, #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1
This PR adds the first version of shared release utilities for connectors.
The expectation is that the `release_utils` branch will be mounted as a git submodule into each connector repo.
See the README for the purpose of each script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] zentol commented on a diff in pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1#discussion_r1011501540
##########
stage_jars.sh:
##########
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
+
+source "${SCRIPT_DIR}/_init.sh"
+source "${SCRIPT_DIR}/_utils.sh"
+
+###########################
+
+check_variable_set FLINK_MINOR_VERSION
+
+###########################
+
+function deploy_staging_jars {
+ cd "${SOURCE_DIR}"
+ mkdir -p "${RELEASE_DIR}"
+
+ project_version=$(get_pom_version)
+ if [[ ${project_version} =~ -SNAPSHOT$ ]]; then
+ echo "Jars should not be created for SNAPSHOT versions. Use 'update_branch_version.sh' first."
+ exit 1
+ fi
+ version=$(project_version)-${FLINK_MINOR_VERSION}
+
+ echo "Deploying jars v${version} to repository.apache.org"
+ echo "To revert this step, login to 'https://repository.apache.org' -> 'Staging repositories' -> Select repository -> 'Drop'"
+
+ clone_dir=$(create_pristine_source "${SOURCE_DIR}" "${RELEASE_DIR}")
+ cd "${clone_dir}"
+ set_pom_version "${version}"
+
+ options="-Prelease,docs-and-source -DskipTests -DretryFailedDeploymentCount=10"
+ ${MVN} clean deploy ${options}
Review Comment:
This should set the `flink.version` property. May require the user to provide a full Flink version from which we extract the minor version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] MartijnVisser commented on pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1#issuecomment-1299988788
> Thanks @zentol and @MartijnVisser for driving this, but I saw `dev` mail list received all PR updates information from this repo, is there some settings incorrect?
Yes, we need to add an `.asf.yaml` file to this repo. Will do in a moment :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] leonardBang commented on pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
leonardBang commented on PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1#issuecomment-1303154579
Happy to see you start working @zentol , Could you take a look my comments here? https://github.com/apache/flink/pull/21227 sorry to reply under this issue but I've tried ping you on slack but no response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] zentol commented on a diff in pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1#discussion_r1011301544
##########
publish_git_tag.sh:
##########
@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
+
+source ${SCRIPT_DIR}/_init.sh
+source ${SCRIPT_DIR}/_utils.sh
+
+###########################
+
+RC_NUM=${RC_NUM:-none}
+
+###########################
+
+function create_release_tag {
+ cd "${SOURCE_DIR}"
+
+ version=$(get_pom_version)
+
+ tag=v${version}
+ if [ "$RC_NUM" != "none" ]; then
+ tag=${tag}-rc${RC_NUM}
+ fi
Review Comment:
We could do that, yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] zentol commented on pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
zentol commented on PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1#issuecomment-1297059496
Note that we could just do a review round, and then try them out for the ES connector release before merging them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] zentol commented on a diff in pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1#discussion_r1011501540
##########
stage_jars.sh:
##########
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
+
+source "${SCRIPT_DIR}/_init.sh"
+source "${SCRIPT_DIR}/_utils.sh"
+
+###########################
+
+check_variable_set FLINK_MINOR_VERSION
+
+###########################
+
+function deploy_staging_jars {
+ cd "${SOURCE_DIR}"
+ mkdir -p "${RELEASE_DIR}"
+
+ project_version=$(get_pom_version)
+ if [[ ${project_version} =~ -SNAPSHOT$ ]]; then
+ echo "Jars should not be created for SNAPSHOT versions. Use 'update_branch_version.sh' first."
+ exit 1
+ fi
+ version=$(project_version)-${FLINK_MINOR_VERSION}
+
+ echo "Deploying jars v${version} to repository.apache.org"
+ echo "To revert this step, login to 'https://repository.apache.org' -> 'Staging repositories' -> Select repository -> 'Drop'"
+
+ clone_dir=$(create_pristine_source "${SOURCE_DIR}" "${RELEASE_DIR}")
+ cd "${clone_dir}"
+ set_pom_version "${version}"
+
+ options="-Prelease,docs-and-source -DskipTests -DretryFailedDeploymentCount=10"
+ ${MVN} clean deploy ${options}
Review Comment:
This should set the `flink.version` property so we actually compile against the correct Flink version.
May require the user to provide a full Flink version from which we extract the minor version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] zentol commented on a diff in pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1#discussion_r1011296837
##########
check_environment.sh:
##########
@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
+
+source "${SCRIPT_DIR}/_init.sh"
+
+EXIT_CODE=0
+
+function check_program_available {
+ if program=$(command -v ${1}); then
+ printf "\t%-10s%s\n" "${1}" "using ${program}"
+ else
+ printf "\t%-10s%s\n" "${1}" "is not available."
+ EXIT_CODE=1
+ fi
+}
+
+echo "Checking program availability:"
+check_program_available git
+check_program_available tar
+check_program_available rsync
+check_program_available gpg
+check_program_available perl
+check_program_available sed
+check_program_available svn
+check_program_available ${MVN}
Review Comment:
This script doesn't neither enforces a maven version nor uses fancy features that would implicitly require a newer maven version.
This is all left to the connector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] zentol commented on a diff in pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1#discussion_r1016492375
##########
README.md:
##########
@@ -1 +1,79 @@
-This repository contains utilities for [Apache Flink](https://flink.apache.org/) connectors.
\ No newline at end of file
+This is a collection of release utils for [Apache Flink](https://flink.apache.org/) connectors.
+
+# Integration
+
+The scripts assume that they are integrated into a connector repo as a submodule into the connector repo
+under `tools/releasing/shared`.
+
+# Usage
+
+Some scripts rely on environment variables to be set.
+These are checked at the start of each script.
+Any instance of `${some_variable}` in this document refers to an environment variable that is used by the respective
+script.
+
+## check_environment.sh
+
+Runs some pre-release checks for the current environment, for example that all required programs are available.
+This should be run once at the start of the release process.
+
+## publish_snapshot_branch.sh
+
+Creates (and pushes!) a new snapshot branch for the current commit.
+The branch name is automatically determined from the version in the pom.
+This script should be called when work on a new major/minor version of the connector has started.
+
+## update_branch_version.sh
+
+Updates the version in the poms of the current branch to `${NEW_VERSION}`.
+
+## stage_source_release.sh
+
+Creates a source release from the current branch and pushes it via `svn`
+to [dist.apache.org](https://dist.apache.org/repops/dist/dev/flink).
+The version is automatically determined from the version in the pom.
+The created `svn` directory will contain a `-rc${RC_NUM}` suffix.
+
+## stage_jars.sh
+
+Creates the jars from the current branch and deploys them to [repository.apache.org](https://repository.apache.org).
+The version will be suffixed with `-${FLINK_MINOR_VERSION}` to indicate the supported Flink version.
+If a particular version of a connector supports multiple Flink versions then this script should be called multiple
+times.
+
+## publish_git_tag.sh
+
+Creates a release tag for the current branch and pushes it to GitHub.
+The tag will be suffixed with `-rc${RC_NUM}`, if `${RC_NUM}` was set.
+This script should only be used _after_ the `-SNAPSHOT` version suffix was removd via `update_branch_version.sh`.
+
+## update_japicmp_configuration.sh
+
+Sets the japicmp reference version in the pom of the current branch to `${NEW_VERSION}`, enables compatibility checks
+for `@PublicEvolving` when used on snapshot branches an clears the list of exclusions.
+This should be called after a release on the associated snapshot branch. If it was a minor release it should
+additionally be called on the `main` branch.
+
+# Common workflow
+
+1. run `publish_snapshot_branch.sh`
+2. do some development work on the created snapshot branch
+3. checkout a specific commit to create a release from
+4. run `check_environment.sh`
+5. run `update_branch_version.sh`
+6. run `stage_source_release.sh`
+7. run `stage_jars.sh` (once for each supported Flink version)
+8. run `publish_git_tag.sh` (with `RC_NUM`)
+9. vote on release
+10. finalize release or cancel and go back to step 2
+11. run `publish_git_tag.sh` (without `RC_NUM`)
+12. run `update_japicmp_configuration.sh` (on snapshot branch, and maybe `main`)
+
+# Script naming conventions
+
+| Prefix | Meaning |
+|---------|------------------------------------------------------------------------|
+| check | Verifies conditions without making any changes. |
+| update | Applies modifications locally to the current branch. |
+| stage | Publishes an artifact to an intermediate location for voting purposes. |
+| publish | Releases an artifact to a user-facing location. |
Review Comment:
```suggestion
| release | Publishes an artifact to a user-facing location. |
```
It's a bit of a mess with "publishing" having 2 different meanings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] zentol commented on a diff in pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1#discussion_r1011368848
##########
_utils.sh:
##########
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+function check_variable_set {
+ variable=$1
+
+ if [ -z "${!variable:-}" ]; then
+ echo "${variable} was not set."
+ exit 1
+ fi
+}
+
+function create_pristine_source {
+ source_dir=$1
+ release_dir=$2
+
+ clone_dir="${release_dir}/tmp-clone"
+ clean_dir="${release_dir}/tmp-clean-clone"
+ # create a temporary git clone to ensure that we have a pristine source release
+ git clone "${source_dir}" "${clone_dir}"
+
+ rsync -a \
+ --exclude ".git" --exclude ".gitignore" --exclude ".gitattributes" --exclude ".gitmodules" --exclude ".github" \
+ --exclude ".idea" --exclude "*.iml" \
+ --exclude ".DS_Store" \
+ --exclude ".asf.yaml" \
+ --exclude "target" --exclude "tools/releasing/shared" \
Review Comment:
ideally the second exclusion would be more dynamic and figure out on its own what the `shared` directory is actually called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] zentol commented on a diff in pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1#discussion_r1011297983
##########
_init.sh:
##########
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# all scripts should contain this line + source ${SCRIPT_DIR}/_init.sh
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+export SHELLOPTS
+
+###########################
+
+MVN=${MVN:-mvn}
+
+if [ "$(uname)" == "Darwin" ]; then
Review Comment:
Doesn't hurt to have it I guess :shrug:
I suppose we just can't be sure this is the case in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] leonardBang commented on pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
leonardBang commented on PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1#issuecomment-1299982431
Thanks @zentol and @zentol for driving this, but I saw `dev` mail list received all PR updates information from this repo, is there some settings incorrect?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] MartijnVisser commented on a diff in pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on code in PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1#discussion_r1010743355
##########
README.md:
##########
@@ -1 +1,79 @@
-This repository contains utilities for [Apache Flink](https://flink.apache.org/) connectors.
\ No newline at end of file
+This is a collection of release utils for [Apache Flink](https://flink.apache.org/) connectors.
+
+# Integration
+
+The scripts assume that they are integrated into a connector repo as a submodule into the connector repo
+under `tools/releasing/<some directory>`.
+
+# Usage
+
+Some scripts rely on environment variables to be set.
+These are checked at the start of each script.
+Any instance of `${some_variable}` in this document refers to an environment variable that is used by the respective
+script.
+
+## check_environment.sh
+
+Runs some pre-release checks for the current environment, for example that all required programs are available.
+This should be run once at the start of the release process.
+
+## publish_snapshot_branch.sh
+
+Creates (and pushes!) a new snapshot branch for the current commit.
+The branch name is automatically determined from the version in the pom.
+This script should be called when work on a new major/minor version has started.
Review Comment:
```suggestion
This script should be called when work on a new major/minor version of the connector has started.
```
##########
_init.sh:
##########
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# all scripts should contain this line + source ${SCRIPT_DIR}/_init.sh
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+export SHELLOPTS
+
+###########################
+
+MVN=${MVN:-mvn}
+
+if [ "$(uname)" == "Darwin" ]; then
Review Comment:
While I have Darwin, I can also both run `sha512sum` and `shasum -a 512`. Don't think that's an issue for this script's purpose though
##########
check_environment.sh:
##########
@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
+
+source "${SCRIPT_DIR}/_init.sh"
+
+EXIT_CODE=0
+
+function check_program_available {
+ if program=$(command -v ${1}); then
+ printf "\t%-10s%s\n" "${1}" "using ${program}"
+ else
+ printf "\t%-10s%s\n" "${1}" "is not available."
+ EXIT_CODE=1
+ fi
+}
+
+echo "Checking program availability:"
+check_program_available git
+check_program_available tar
+check_program_available rsync
+check_program_available gpg
+check_program_available perl
+check_program_available sed
+check_program_available svn
+check_program_available ${MVN}
Review Comment:
Just to double check, any version of Maven for connectors should suffice, right? Doesn't need to be 3.2.5 like for Flink itself
##########
README.md:
##########
@@ -1 +1,79 @@
-This repository contains utilities for [Apache Flink](https://flink.apache.org/) connectors.
\ No newline at end of file
+This is a collection of release utils for [Apache Flink](https://flink.apache.org/) connectors.
+
+# Integration
+
+The scripts assume that they are integrated into a connector repo as a submodule into the connector repo
+under `tools/releasing/<some directory>`.
+
+# Usage
+
+Some scripts rely on environment variables to be set.
+These are checked at the start of each script.
+Any instance of `${some_variable}` in this document refers to an environment variable that is used by the respective
+script.
+
+## check_environment.sh
+
+Runs some pre-release checks for the current environment, for example that all required programs are available.
+This should be run once at the start of the release process.
+
+## publish_snapshot_branch.sh
+
+Creates (and pushes!) a new snapshot branch for the current commit.
+The branch name is automatically determined from the version in the pom.
+This script should be called when work on a new major/minor version has started.
+
+## update_branch_version.sh
+
+Updates the version in the poms of the current branch to `${NEW_VERSION}`.
+
+## stage_source_release.sh
+
+Creates a source release from the current branch and pushes it via `svn`
+to [dist.apache.org](https://dist.apache.org/repops/dist/dev/flink).
+The version is automatically determined from the version in the pom.
+The created `svn` directory will contain a `-rc${RC_NUM}` suffix.
+
+## stage_jars.sh
+
+Creates the jars from the current branch and deploys them to [repository.apache.org](https://repository.apache.org).
+The version will be suffixed with `-${FLINK_MINOR_VERSION}` to indicate the supported Flink version.
+If a particular version of a connector supports multiple Flink versions then this script should be called multiple
+times.
+
+## publish_git_tag.sh
+
+Creates a release tag for the current branch and pushes it to GitHub.
+The tag will be suffixed with `-rc${RC_NUM}`, if `${RC_NUM}` was set.
+This script should only be used _after_ the `-SNAPSHOT` version suffix was removd via `update_branch_version.sh`.
+
+## update_japicmp_configuration.sh
+
+Sets the japicmp reference version in the pom of the current branch to `${NEW_VERSION}`, enables compatibility checks
+for `@PublicEvolving` when used on snapshot branches an clears the list of exclusions.
+This should be called after a release on the associated snapshot branch. If it was a minor release it should
+additionally be called on the `main` branch.
+
+# Common workflow
+
+1. run `publish_snapshot_branch.sh`
+2. do some development work on the created snapshot branch
+3. checkout a specific commit to create a release from
+4. run `check_environment.sh`
Review Comment:
I think it makes more sense to have this as the first step and not the 4th. The `publish_snapshots_branch.sh` uses Maven and git, which are checked for availability in `check_environment.sh`.
##########
publish_git_tag.sh:
##########
@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
+
+source ${SCRIPT_DIR}/_init.sh
+source ${SCRIPT_DIR}/_utils.sh
+
+###########################
+
+RC_NUM=${RC_NUM:-none}
+
+###########################
+
+function create_release_tag {
+ cd "${SOURCE_DIR}"
+
+ version=$(get_pom_version)
+
+ tag=v${version}
+ if [ "$RC_NUM" != "none" ]; then
+ tag=${tag}-rc${RC_NUM}
+ fi
Review Comment:
Does it make sense to add a check here that verifies if the version doesn't contain SNAPSHOT anymore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] zentol merged pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
zentol merged PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] zentol commented on a diff in pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1#discussion_r1011297983
##########
_init.sh:
##########
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# all scripts should contain this line + source ${SCRIPT_DIR}/_init.sh
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+export SHELLOPTS
+
+###########################
+
+MVN=${MVN:-mvn}
+
+if [ "$(uname)" == "Darwin" ]; then
Review Comment:
Doesn't hurt to have it I guess :shrug:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] zentol commented on a diff in pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1#discussion_r1011369044
##########
README.md:
##########
@@ -1 +1,79 @@
-This repository contains utilities for [Apache Flink](https://flink.apache.org/) connectors.
\ No newline at end of file
+This is a collection of release utils for [Apache Flink](https://flink.apache.org/) connectors.
+
+# Integration
+
+The scripts assume that they are integrated into a connector repo as a submodule into the connector repo
+under `tools/releasing/<some directory>`.
Review Comment:
```suggestion
under `tools/releasing/shared`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-shared-utils] zentol commented on a diff in pull request #1: [FLINK-29472] Add first version of release scripts
Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #1:
URL: https://github.com/apache/flink-connector-shared-utils/pull/1#discussion_r1011301075
##########
README.md:
##########
@@ -1 +1,79 @@
-This repository contains utilities for [Apache Flink](https://flink.apache.org/) connectors.
\ No newline at end of file
+This is a collection of release utils for [Apache Flink](https://flink.apache.org/) connectors.
+
+# Integration
+
+The scripts assume that they are integrated into a connector repo as a submodule into the connector repo
+under `tools/releasing/<some directory>`.
+
+# Usage
+
+Some scripts rely on environment variables to be set.
+These are checked at the start of each script.
+Any instance of `${some_variable}` in this document refers to an environment variable that is used by the respective
+script.
+
+## check_environment.sh
+
+Runs some pre-release checks for the current environment, for example that all required programs are available.
+This should be run once at the start of the release process.
+
+## publish_snapshot_branch.sh
+
+Creates (and pushes!) a new snapshot branch for the current commit.
+The branch name is automatically determined from the version in the pom.
+This script should be called when work on a new major/minor version has started.
+
+## update_branch_version.sh
+
+Updates the version in the poms of the current branch to `${NEW_VERSION}`.
+
+## stage_source_release.sh
+
+Creates a source release from the current branch and pushes it via `svn`
+to [dist.apache.org](https://dist.apache.org/repops/dist/dev/flink).
+The version is automatically determined from the version in the pom.
+The created `svn` directory will contain a `-rc${RC_NUM}` suffix.
+
+## stage_jars.sh
+
+Creates the jars from the current branch and deploys them to [repository.apache.org](https://repository.apache.org).
+The version will be suffixed with `-${FLINK_MINOR_VERSION}` to indicate the supported Flink version.
+If a particular version of a connector supports multiple Flink versions then this script should be called multiple
+times.
+
+## publish_git_tag.sh
+
+Creates a release tag for the current branch and pushes it to GitHub.
+The tag will be suffixed with `-rc${RC_NUM}`, if `${RC_NUM}` was set.
+This script should only be used _after_ the `-SNAPSHOT` version suffix was removd via `update_branch_version.sh`.
+
+## update_japicmp_configuration.sh
+
+Sets the japicmp reference version in the pom of the current branch to `${NEW_VERSION}`, enables compatibility checks
+for `@PublicEvolving` when used on snapshot branches an clears the list of exclusions.
+This should be called after a release on the associated snapshot branch. If it was a minor release it should
+additionally be called on the `main` branch.
+
+# Common workflow
+
+1. run `publish_snapshot_branch.sh`
+2. do some development work on the created snapshot branch
+3. checkout a specific commit to create a release from
+4. run `check_environment.sh`
Review Comment:
Basically the issue is that creating a snapshot branch will sometimes not be done by a release manager.
For example, after the ES 3.0 release we have a v3.0 branch and a main branch on 4.0-SNAPSHOT. v3.1 would be created by _someone_ when we want to make some change that requires a new minor version.
So all the other things the check does just aren't required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org