You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2023/05/22 16:13:35 UTC
[GitHub] [hudi] yihua opened a new pull request, #8779: [HUDI-6247] Add bundle validation for release candidates
yihua opened a new pull request, #8779:
URL: https://github.com/apache/hudi/pull/8779
### Change Logs
This PR adds the bundle validation for the release candidates. By default, this is disabled. To enable the bundle validation for release candidates, makes the following changes to the job `validate-release-candidate-bundles` in `.github/workflows/bot.yml` by adding the corresponding information:
```
if: true
env:
HUDI_VERSION: 0.13.1-rc1
STAGING_REPO_NUM: 1123
```
### Impact
Makes bundle validation on release candidates much easier.
### Risk level
none
### Documentation Update
Adds README on how bundle validation on release candidates can be run.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #8779: [HUDI-6247] Add bundle validation for release candidates
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #8779:
URL: https://github.com/apache/hudi/pull/8779
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8779: [HUDI-6247] Add bundle validation for release candidates
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8779:
URL: https://github.com/apache/hudi/pull/8779#issuecomment-1557587781
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aff465a8e6b11d76be2f9025013ca4b8eaa9c04a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8779: [HUDI-6247] Add bundle validation for release candidates
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8779:
URL: https://github.com/apache/hudi/pull/8779#discussion_r1200947437
##########
packaging/bundle-validation/README.md:
##########
@@ -50,4 +50,20 @@ Note that for each library like Hive and Spark, the download and extraction happ
only one layer is generated to limit the size of the image. However, this makes repeated downloads when rebuilding the
image. If you need faster iteration for local build, you may use the `Dockerfile` under `base-dev/`, which uses `ADD`
instruction for downloads, which provides caching across builds. This increases the size of the generated image compared
-to `base/` and the image should only be used for development only and not be pushed to remote.
\ No newline at end of file
+to `base/` and the image should only be used for development only and not be pushed to remote.
+
+## Running Bundle Validation on a Release Candidate
+
+The bundle validation on a release candidate is specified in the Github Action job `validate-release-candidate-bundles`
+in `.github/workflows/bot.yml`. By default, this is disabled.
+
+To enable the bundle validation on a particular release candidate, makes the following changes to the job by fipping the
Review Comment:
we should update our release guide as well rgdn this step
##########
.github/workflows/bot.yml:
##########
@@ -210,3 +210,62 @@ jobs:
run: |
HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
./packaging/bundle-validation/ci_run.sh $HUDI_VERSION openjdk17
+
+ validate-release-candidate-bundles:
+ if: true
+ runs-on: ubuntu-latest
+ env:
+ HUDI_VERSION: 0.13.1-rc1
Review Comment:
by default can you set some invalid release(instead of 0.13.1 rc1), so that it will fail rather than succeeding by mistake.
##########
packaging/bundle-validation/ci_run.sh:
##########
@@ -96,14 +99,73 @@ fi
# Copy bundle jars to temp dir for mounting
TMP_JARS_DIR=/tmp/jars/$(date +%s)
mkdir -p $TMP_JARS_DIR
-cp ${GITHUB_WORKSPACE}/packaging/hudi-flink-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
-cp ${GITHUB_WORKSPACE}/packaging/hudi-hadoop-mr-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
-cp ${GITHUB_WORKSPACE}/packaging/hudi-kafka-connect-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
-cp ${GITHUB_WORKSPACE}/packaging/hudi-spark-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
-cp ${GITHUB_WORKSPACE}/packaging/hudi-utilities-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
-cp ${GITHUB_WORKSPACE}/packaging/hudi-utilities-slim-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
-cp ${GITHUB_WORKSPACE}/packaging/hudi-metaserver-server-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
-echo 'Validating jars below:'
+
+if [[ "$HUDI_VERSION" == *"SNAPSHOT" ]]; then
+ cp ${GITHUB_WORKSPACE}/packaging/hudi-flink-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
+ cp ${GITHUB_WORKSPACE}/packaging/hudi-hadoop-mr-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
+ cp ${GITHUB_WORKSPACE}/packaging/hudi-kafka-connect-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
+ cp ${GITHUB_WORKSPACE}/packaging/hudi-spark-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
+ cp ${GITHUB_WORKSPACE}/packaging/hudi-utilities-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
+ cp ${GITHUB_WORKSPACE}/packaging/hudi-utilities-slim-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
+ cp ${GITHUB_WORKSPACE}/packaging/hudi-metaserver-server-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
+ echo 'Validating jars below:'
+else
+ echo 'Adding environment variables for bundles in the release candidate'
+
+ HUDI_HADOOP_MR_BUNDLE_NAME=hudi-hadoop-mr-bundle
+ HUDI_KAFKA_CONNECT_BUNDLE_NAME=hudi-kafka-connect-bundle
+ HUDI_METASERVER_SERVER_BUNDLE_NAME=hudi-metaserver-server-bundle
+
+ if [[ ${SPARK_PROFILE} == 'spark' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark-bundle_2.11
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.11
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.11
+ elif [[ ${SPARK_PROFILE} == 'spark2.4' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark2.4-bundle_2.11
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.11
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.11
+ elif [[ ${SPARK_PROFILE} == 'spark3.1' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark3.1-bundle_2.12
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.12
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.12
+ elif [[ ${SPARK_PROFILE} == 'spark3.2' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark3.2-bundle_2.12
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.12
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.12
+ elif [[ ${SPARK_PROFILE} == 'spark3.3' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark3.3-bundle_2.12
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.12
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.12
+ elif [[ ${SPARK_PROFILE} == 'spark3' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark3-bundle_2.12
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.12
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.12
+ fi
+
+ if [[ ${FLINK_PROFILE} == 'flink1.13' ]]; then
+ HUDI_FLINK_BUNDLE_NAME=hudi-flink1.13-bundle
+ elif [[ ${FLINK_PROFILE} == 'flink1.14' ]]; then
+ HUDI_FLINK_BUNDLE_NAME=hudi-flink1.14-bundle
+ elif [[ ${FLINK_PROFILE} == 'flink1.15' ]]; then
+ HUDI_FLINK_BUNDLE_NAME=hudi-flink1.15-bundle
+ elif [[ ${FLINK_PROFILE} == 'flink1.16' ]]; then
+ HUDI_FLINK_BUNDLE_NAME=hudi-flink1.16-bundle
+ elif [[ ${FLINK_PROFILE} == 'flink1.17' ]]; then
+ HUDI_FLINK_BUNDLE_NAME=hudi-flink1.17-bundle
+ fi
+
+ echo "Downloading bundle jars from staging repo orgapachehudi-$STAGING_REPO_NUM ..."
+ REPO_BASE_URL=https://repository.apache.org/content/repositories/orgapachehudi-$STAGING_REPO_NUM/org/apache/hudi
+ wget -q $REPO_BASE_URL/$HUDI_FLINK_BUNDLE_NAME/$HUDI_VERSION/$HUDI_FLINK_BUNDLE_NAME-$HUDI_VERSION.jar -P $TMP_JARS_DIR/
+ wget -q $REPO_BASE_URL/$HUDI_HADOOP_MR_BUNDLE_NAME/$HUDI_VERSION/$HUDI_HADOOP_MR_BUNDLE_NAME-$HUDI_VERSION.jar -P $TMP_JARS_DIR/
+ wget -q $REPO_BASE_URL/$HUDI_KAFKA_CONNECT_BUNDLE_NAME/$HUDI_VERSION/$HUDI_KAFKA_CONNECT_BUNDLE_NAME-$HUDI_VERSION.jar -P $TMP_JARS_DIR/
+ wget -q $REPO_BASE_URL/$HUDI_SPARK_BUNDLE_NAME/$HUDI_VERSION/$HUDI_SPARK_BUNDLE_NAME-$HUDI_VERSION.jar -P $TMP_JARS_DIR/
+ wget -q $REPO_BASE_URL/$HUDI_UTILITIES_BUNDLE_NAME/$HUDI_VERSION/$HUDI_UTILITIES_BUNDLE_NAME-$HUDI_VERSION.jar -P $TMP_JARS_DIR/
+ wget -q $REPO_BASE_URL/$HUDI_UTILITIES_SLIM_BUNDLE_NAME/$HUDI_VERSION/$HUDI_UTILITIES_SLIM_BUNDLE_NAME-$HUDI_VERSION.jar -P $TMP_JARS_DIR/
+ wget -q $REPO_BASE_URL/$HUDI_METASERVER_SERVER_BUNDLE_NAME/$HUDI_VERSION/$HUDI_METASERVER_SERVER_BUNDLE_NAME-$HUDI_VERSION.jar -P $TMP_JARS_DIR/
+ echo "Downloaded these jars from $REPO_BASE_URL for validation:"
Review Comment:
do you think we should add a count validation on the output of
```
ls -l $TMP_JARS_DIR
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8779: [HUDI-6247] Add bundle validation for release candidates
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8779:
URL: https://github.com/apache/hudi/pull/8779#issuecomment-1557681057
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"triggerType" : "PUSH"
}, {
"hash" : "4fefd25e9d3251058dda43b1c0034ead6fc35498",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17271",
"triggerID" : "4fefd25e9d3251058dda43b1c0034ead6fc35498",
"triggerType" : "PUSH"
}, {
"hash" : "86c4d5baf362e91f796108e16b4a38b7c94e5439",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17273",
"triggerID" : "86c4d5baf362e91f796108e16b4a38b7c94e5439",
"triggerType" : "PUSH"
}, {
"hash" : "8e57681d55a418092cb4c03716ee11228313af05",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17274",
"triggerID" : "8e57681d55a418092cb4c03716ee11228313af05",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aff465a8e6b11d76be2f9025013ca4b8eaa9c04a UNKNOWN
* 86c4d5baf362e91f796108e16b4a38b7c94e5439 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17273)
* 8e57681d55a418092cb4c03716ee11228313af05 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17274)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8779: [HUDI-6247] Add bundle validation for release candidates
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8779:
URL: https://github.com/apache/hudi/pull/8779#discussion_r1201004113
##########
packaging/bundle-validation/README.md:
##########
@@ -50,4 +50,20 @@ Note that for each library like Hive and Spark, the download and extraction happ
only one layer is generated to limit the size of the image. However, this makes repeated downloads when rebuilding the
image. If you need faster iteration for local build, you may use the `Dockerfile` under `base-dev/`, which uses `ADD`
instruction for downloads, which provides caching across builds. This increases the size of the generated image compared
-to `base/` and the image should only be used for development only and not be pushed to remote.
\ No newline at end of file
+to `base/` and the image should only be used for development only and not be pushed to remote.
+
+## Running Bundle Validation on a Release Candidate
+
+The bundle validation on a release candidate is specified in the Github Action job `validate-release-candidate-bundles`
+in `.github/workflows/bot.yml`. By default, this is disabled.
+
+To enable the bundle validation on a particular release candidate, makes the following changes to the job by fipping the
Review Comment:
I'll put up a separate PR to update the release guide as a whole, as it's pretty out-dated and needs more work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8779: [HUDI-6247] Add bundle validation for release candidates
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8779:
URL: https://github.com/apache/hudi/pull/8779#issuecomment-1558254116
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"triggerType" : "PUSH"
}, {
"hash" : "4fefd25e9d3251058dda43b1c0034ead6fc35498",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17271",
"triggerID" : "4fefd25e9d3251058dda43b1c0034ead6fc35498",
"triggerType" : "PUSH"
}, {
"hash" : "86c4d5baf362e91f796108e16b4a38b7c94e5439",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17273",
"triggerID" : "86c4d5baf362e91f796108e16b4a38b7c94e5439",
"triggerType" : "PUSH"
}, {
"hash" : "8e57681d55a418092cb4c03716ee11228313af05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17274",
"triggerID" : "8e57681d55a418092cb4c03716ee11228313af05",
"triggerType" : "PUSH"
}, {
"hash" : "87fa4e51b432788ab40dd28203540babea48c258",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17278",
"triggerID" : "87fa4e51b432788ab40dd28203540babea48c258",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aff465a8e6b11d76be2f9025013ca4b8eaa9c04a UNKNOWN
* 87fa4e51b432788ab40dd28203540babea48c258 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17278)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8779: [HUDI-6247] Add bundle validation for release candidates
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8779:
URL: https://github.com/apache/hudi/pull/8779#issuecomment-1558029485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"triggerType" : "PUSH"
}, {
"hash" : "4fefd25e9d3251058dda43b1c0034ead6fc35498",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17271",
"triggerID" : "4fefd25e9d3251058dda43b1c0034ead6fc35498",
"triggerType" : "PUSH"
}, {
"hash" : "86c4d5baf362e91f796108e16b4a38b7c94e5439",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17273",
"triggerID" : "86c4d5baf362e91f796108e16b4a38b7c94e5439",
"triggerType" : "PUSH"
}, {
"hash" : "8e57681d55a418092cb4c03716ee11228313af05",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17274",
"triggerID" : "8e57681d55a418092cb4c03716ee11228313af05",
"triggerType" : "PUSH"
}, {
"hash" : "87fa4e51b432788ab40dd28203540babea48c258",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17278",
"triggerID" : "87fa4e51b432788ab40dd28203540babea48c258",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aff465a8e6b11d76be2f9025013ca4b8eaa9c04a UNKNOWN
* 8e57681d55a418092cb4c03716ee11228313af05 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17274)
* 87fa4e51b432788ab40dd28203540babea48c258 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17278)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8779: [HUDI-6247] Add bundle validation for release candidates
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8779:
URL: https://github.com/apache/hudi/pull/8779#discussion_r1201007409
##########
packaging/bundle-validation/ci_run.sh:
##########
@@ -96,14 +99,73 @@ fi
# Copy bundle jars to temp dir for mounting
TMP_JARS_DIR=/tmp/jars/$(date +%s)
mkdir -p $TMP_JARS_DIR
-cp ${GITHUB_WORKSPACE}/packaging/hudi-flink-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
-cp ${GITHUB_WORKSPACE}/packaging/hudi-hadoop-mr-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
-cp ${GITHUB_WORKSPACE}/packaging/hudi-kafka-connect-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
-cp ${GITHUB_WORKSPACE}/packaging/hudi-spark-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
-cp ${GITHUB_WORKSPACE}/packaging/hudi-utilities-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
-cp ${GITHUB_WORKSPACE}/packaging/hudi-utilities-slim-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
-cp ${GITHUB_WORKSPACE}/packaging/hudi-metaserver-server-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
-echo 'Validating jars below:'
+
+if [[ "$HUDI_VERSION" == *"SNAPSHOT" ]]; then
+ cp ${GITHUB_WORKSPACE}/packaging/hudi-flink-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
+ cp ${GITHUB_WORKSPACE}/packaging/hudi-hadoop-mr-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
+ cp ${GITHUB_WORKSPACE}/packaging/hudi-kafka-connect-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
+ cp ${GITHUB_WORKSPACE}/packaging/hudi-spark-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
+ cp ${GITHUB_WORKSPACE}/packaging/hudi-utilities-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
+ cp ${GITHUB_WORKSPACE}/packaging/hudi-utilities-slim-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
+ cp ${GITHUB_WORKSPACE}/packaging/hudi-metaserver-server-bundle/target/hudi-*-$HUDI_VERSION.jar $TMP_JARS_DIR/
+ echo 'Validating jars below:'
+else
+ echo 'Adding environment variables for bundles in the release candidate'
+
+ HUDI_HADOOP_MR_BUNDLE_NAME=hudi-hadoop-mr-bundle
+ HUDI_KAFKA_CONNECT_BUNDLE_NAME=hudi-kafka-connect-bundle
+ HUDI_METASERVER_SERVER_BUNDLE_NAME=hudi-metaserver-server-bundle
+
+ if [[ ${SPARK_PROFILE} == 'spark' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark-bundle_2.11
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.11
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.11
+ elif [[ ${SPARK_PROFILE} == 'spark2.4' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark2.4-bundle_2.11
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.11
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.11
+ elif [[ ${SPARK_PROFILE} == 'spark3.1' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark3.1-bundle_2.12
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.12
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.12
+ elif [[ ${SPARK_PROFILE} == 'spark3.2' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark3.2-bundle_2.12
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.12
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.12
+ elif [[ ${SPARK_PROFILE} == 'spark3.3' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark3.3-bundle_2.12
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.12
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.12
+ elif [[ ${SPARK_PROFILE} == 'spark3' ]]; then
+ HUDI_SPARK_BUNDLE_NAME=hudi-spark3-bundle_2.12
+ HUDI_UTILITIES_BUNDLE_NAME=hudi-utilities-bundle_2.12
+ HUDI_UTILITIES_SLIM_BUNDLE_NAME=hudi-utilities-slim-bundle_2.12
+ fi
+
+ if [[ ${FLINK_PROFILE} == 'flink1.13' ]]; then
+ HUDI_FLINK_BUNDLE_NAME=hudi-flink1.13-bundle
+ elif [[ ${FLINK_PROFILE} == 'flink1.14' ]]; then
+ HUDI_FLINK_BUNDLE_NAME=hudi-flink1.14-bundle
+ elif [[ ${FLINK_PROFILE} == 'flink1.15' ]]; then
+ HUDI_FLINK_BUNDLE_NAME=hudi-flink1.15-bundle
+ elif [[ ${FLINK_PROFILE} == 'flink1.16' ]]; then
+ HUDI_FLINK_BUNDLE_NAME=hudi-flink1.16-bundle
+ elif [[ ${FLINK_PROFILE} == 'flink1.17' ]]; then
+ HUDI_FLINK_BUNDLE_NAME=hudi-flink1.17-bundle
+ fi
+
+ echo "Downloading bundle jars from staging repo orgapachehudi-$STAGING_REPO_NUM ..."
+ REPO_BASE_URL=https://repository.apache.org/content/repositories/orgapachehudi-$STAGING_REPO_NUM/org/apache/hudi
+ wget -q $REPO_BASE_URL/$HUDI_FLINK_BUNDLE_NAME/$HUDI_VERSION/$HUDI_FLINK_BUNDLE_NAME-$HUDI_VERSION.jar -P $TMP_JARS_DIR/
+ wget -q $REPO_BASE_URL/$HUDI_HADOOP_MR_BUNDLE_NAME/$HUDI_VERSION/$HUDI_HADOOP_MR_BUNDLE_NAME-$HUDI_VERSION.jar -P $TMP_JARS_DIR/
+ wget -q $REPO_BASE_URL/$HUDI_KAFKA_CONNECT_BUNDLE_NAME/$HUDI_VERSION/$HUDI_KAFKA_CONNECT_BUNDLE_NAME-$HUDI_VERSION.jar -P $TMP_JARS_DIR/
+ wget -q $REPO_BASE_URL/$HUDI_SPARK_BUNDLE_NAME/$HUDI_VERSION/$HUDI_SPARK_BUNDLE_NAME-$HUDI_VERSION.jar -P $TMP_JARS_DIR/
+ wget -q $REPO_BASE_URL/$HUDI_UTILITIES_BUNDLE_NAME/$HUDI_VERSION/$HUDI_UTILITIES_BUNDLE_NAME-$HUDI_VERSION.jar -P $TMP_JARS_DIR/
+ wget -q $REPO_BASE_URL/$HUDI_UTILITIES_SLIM_BUNDLE_NAME/$HUDI_VERSION/$HUDI_UTILITIES_SLIM_BUNDLE_NAME-$HUDI_VERSION.jar -P $TMP_JARS_DIR/
+ wget -q $REPO_BASE_URL/$HUDI_METASERVER_SERVER_BUNDLE_NAME/$HUDI_VERSION/$HUDI_METASERVER_SERVER_BUNDLE_NAME-$HUDI_VERSION.jar -P $TMP_JARS_DIR/
+ echo "Downloaded these jars from $REPO_BASE_URL for validation:"
Review Comment:
I think it's not needed. If a jar is missing, the validation logic will fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8779: [HUDI-6247] Add bundle validation for release candidates
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8779:
URL: https://github.com/apache/hudi/pull/8779#discussion_r1201005064
##########
.github/workflows/bot.yml:
##########
@@ -210,3 +210,62 @@ jobs:
run: |
HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
./packaging/bundle-validation/ci_run.sh $HUDI_VERSION openjdk17
+
+ validate-release-candidate-bundles:
+ if: true
+ runs-on: ubuntu-latest
+ env:
+ HUDI_VERSION: 0.13.1-rc1
Review Comment:
Once GH Java CI passes, I'll turn off this by default (`if: false`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #8779: [HUDI-6247] Add bundle validation for release candidates
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #8779:
URL: https://github.com/apache/hudi/pull/8779#issuecomment-1557976981
Java CI passes: https://github.com/apache/hudi/actions/runs/5048657788/jobs/9057118563
Going to disable the flag and land (Azure CI is unaffected).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8779: [HUDI-6247] Add bundle validation for release candidates
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8779:
URL: https://github.com/apache/hudi/pull/8779#issuecomment-1557600463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"triggerType" : "PUSH"
}, {
"hash" : "4fefd25e9d3251058dda43b1c0034ead6fc35498",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fefd25e9d3251058dda43b1c0034ead6fc35498",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aff465a8e6b11d76be2f9025013ca4b8eaa9c04a UNKNOWN
* 4fefd25e9d3251058dda43b1c0034ead6fc35498 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8779: [HUDI-6247] Add bundle validation for release candidates
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8779:
URL: https://github.com/apache/hudi/pull/8779#issuecomment-1558017592
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"triggerType" : "PUSH"
}, {
"hash" : "4fefd25e9d3251058dda43b1c0034ead6fc35498",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17271",
"triggerID" : "4fefd25e9d3251058dda43b1c0034ead6fc35498",
"triggerType" : "PUSH"
}, {
"hash" : "86c4d5baf362e91f796108e16b4a38b7c94e5439",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17273",
"triggerID" : "86c4d5baf362e91f796108e16b4a38b7c94e5439",
"triggerType" : "PUSH"
}, {
"hash" : "8e57681d55a418092cb4c03716ee11228313af05",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17274",
"triggerID" : "8e57681d55a418092cb4c03716ee11228313af05",
"triggerType" : "PUSH"
}, {
"hash" : "87fa4e51b432788ab40dd28203540babea48c258",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "87fa4e51b432788ab40dd28203540babea48c258",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aff465a8e6b11d76be2f9025013ca4b8eaa9c04a UNKNOWN
* 86c4d5baf362e91f796108e16b4a38b7c94e5439 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17273)
* 8e57681d55a418092cb4c03716ee11228313af05 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17274)
* 87fa4e51b432788ab40dd28203540babea48c258 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8779: [HUDI-6247] Add bundle validation for release candidates
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8779:
URL: https://github.com/apache/hudi/pull/8779#issuecomment-1557669792
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"triggerType" : "PUSH"
}, {
"hash" : "4fefd25e9d3251058dda43b1c0034ead6fc35498",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17271",
"triggerID" : "4fefd25e9d3251058dda43b1c0034ead6fc35498",
"triggerType" : "PUSH"
}, {
"hash" : "86c4d5baf362e91f796108e16b4a38b7c94e5439",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17273",
"triggerID" : "86c4d5baf362e91f796108e16b4a38b7c94e5439",
"triggerType" : "PUSH"
}, {
"hash" : "8e57681d55a418092cb4c03716ee11228313af05",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8e57681d55a418092cb4c03716ee11228313af05",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aff465a8e6b11d76be2f9025013ca4b8eaa9c04a UNKNOWN
* 4fefd25e9d3251058dda43b1c0034ead6fc35498 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17271)
* 86c4d5baf362e91f796108e16b4a38b7c94e5439 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17273)
* 8e57681d55a418092cb4c03716ee11228313af05 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8779: [HUDI-6247] Add bundle validation for release candidates
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8779:
URL: https://github.com/apache/hudi/pull/8779#issuecomment-1557612358
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aff465a8e6b11d76be2f9025013ca4b8eaa9c04a",
"triggerType" : "PUSH"
}, {
"hash" : "4fefd25e9d3251058dda43b1c0034ead6fc35498",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17271",
"triggerID" : "4fefd25e9d3251058dda43b1c0034ead6fc35498",
"triggerType" : "PUSH"
}, {
"hash" : "86c4d5baf362e91f796108e16b4a38b7c94e5439",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "86c4d5baf362e91f796108e16b4a38b7c94e5439",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aff465a8e6b11d76be2f9025013ca4b8eaa9c04a UNKNOWN
* 4fefd25e9d3251058dda43b1c0034ead6fc35498 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17271)
* 86c4d5baf362e91f796108e16b4a38b7c94e5439 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org