You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/09/16 08:14:07 UTC
[GitHub] [hudi] yanghua opened a new pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
yanghua opened a new pull request #3674:
URL: https://github.com/apache/hudi/pull/3674
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua edited a comment on pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
yanghua edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-925613960
I received many reports about dependency conflict around Hudi, e.g. [here](https://github.com/apache/incubator-kyuubi/pull/994) and Chinese WeChat group. Considering Hudi depends on so many Hadoop ecosystem components. IMO, it's time to do the dependency governance.
In this PR, I provided a dependency utility script that can be used to search/diff dependencies of bundles when the contributors change the dependencies. In addition, I have pre-generated a dependency list for some bundles.
Usage:
```shell
# use -r option to replace the old file when we introduce new dependencies
./scripts/dependency.sh -p hudi-utilities-bundle_2.11 -r
```
I have two suggestions:
- build a Github workflow to check the dependency diff, and reviewers should consider any new dependency seriously before merging.
- based on the pre-generated dependency file to do the dependency governance, you can see there are some dependencies that exist in multiple versions for one bundle.
WDYT? @vinothchandar @xushiyan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#discussion_r719198647
##########
File path: scripts/dependency.sh
##########
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -eou pipefail
+set -x
+
+export LC_ALL=C
+
+PWD=$(cd "$(dirname "$0")"/.. || exit; pwd)
+
+function printUsage() {
+ echo "Usage: $(basename "${0}") [-p <artifactId>] -r " 2>&1
+ echo ' -r [OPTIONAL] to replace the old dependencyList file with new dependencies'
+ echo ' -p [MUST] to generate new dependencyList file for the specified module'
+}
+
+function build_classpath() {
+ mvn dependency:build-classpath -pl :${PL} |\
+ grep -E -v "INFO|WARNING" | \
+ tr ":" "\n" | \
+ awk -F '/' '{
+ artifact_id=$(NF-2);
+ version=$(NF-1);
+ jar_name=$NF;
+ classifier_start_index=length(artifact_id"-"version"-") + 1;
+ classifier_end_index=index(jar_name, ".jar") - 1;
+ classifier=substr(jar_name, classifier_start_index, classifier_end_index - classifier_start_index + 1);
+ print artifact_id"/"version"/"classifier"/"jar_name
+ }' | grep -v "hudi" | sort >> "${DEP_PR}"
+}
+
+function check_diff() {
+ set +e
+ the_diff=$(diff ${DEP} ${DEP_PR})
+ set -e
+ rm -rf "${DEP_PR}"
+ if [[ -n $the_diff ]]; then
+ echo "Dependency List Changed Detected: "
+ echo ${the_diff}
+ echo "To update the dependency file, refer to the usage:"
+ printUsage
+ exit 1
+ fi
+}
+
+if [[ ${#} -eq 0 ]]; then
+ printUsage
+fi
+
+PL=''
+REPLACE='false'
+
+while getopts "rp:" arg; do
+ case "${arg}" in
+ r)
+ REPLACE="true"
+ ;;
+ p)
+ PL=$OPTARG
+ ;;
+ ?)
+ printUsage
+ ;;
+ esac
+done
+
+shift "$(( OPTIND - 1 ))"
+
+# check must option
+if [ -z "$PL" ]; then
+ echo 'Missing -p argument' >&2
+ exit 1
+fi
+
+DEP_PR="${PWD}"/dev/dependencyList"${PL}".tmp
+DEP="${PWD}"/dev/dependencyList_"${PL}"
Review comment:
s.g. will add more in the follow-up PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-920690239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256",
"triggerID" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"triggerType" : "PUSH"
}, {
"hash" : "a844e075069d260f22f1a3244708560bebf0128b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2273",
"triggerID" : "a844e075069d260f22f1a3244708560bebf0128b",
"triggerType" : "PUSH"
}, {
"hash" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2321",
"triggerID" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"triggerType" : "PUSH"
}, {
"hash" : "7c634fd5d815d4643732d0c26144171c9dd8e64c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2333",
"triggerID" : "7c634fd5d815d4643732d0c26144171c9dd8e64c",
"triggerType" : "PUSH"
}, {
"hash" : "cf1f2dba2dd9ca560cd33ca012dbe5613e42a733",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2469",
"triggerID" : "cf1f2dba2dd9ca560cd33ca012dbe5613e42a733",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7c634fd5d815d4643732d0c26144171c9dd8e64c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2333)
* cf1f2dba2dd9ca560cd33ca012dbe5613e42a733 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2469)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3674: [WIP][HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-920690239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256",
"triggerID" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"triggerType" : "PUSH"
}, {
"hash" : "a844e075069d260f22f1a3244708560bebf0128b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2273",
"triggerID" : "a844e075069d260f22f1a3244708560bebf0128b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a844e075069d260f22f1a3244708560bebf0128b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2273)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#discussion_r716275952
##########
File path: scripts/dependency.sh
##########
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -eou pipefail
+set -x
+
+export LC_ALL=C
+
+PWD=$(cd "$(dirname "$0")"/.. || exit; pwd)
+
+function printUsage() {
+ echo "Usage: $(basename "${0}") [-p <artifactId>] -r " 2>&1
+ echo ' -r [OPTIONAL] to replace the old dependencyList file with new dependencies'
+ echo ' -p [MUST] to generate new dependencyList file for the specified module'
+}
Review comment:
is the flow for PR owners to run the script manually and submit the diff along with the PR? it could be hard to execute though
##########
File path: scripts/dependency.sh
##########
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -eou pipefail
+set -x
+
+export LC_ALL=C
+
+PWD=$(cd "$(dirname "$0")"/.. || exit; pwd)
+
+function printUsage() {
+ echo "Usage: $(basename "${0}") [-p <artifactId>] -r " 2>&1
+ echo ' -r [OPTIONAL] to replace the old dependencyList file with new dependencies'
+ echo ' -p [MUST] to generate new dependencyList file for the specified module'
+}
+
+function build_classpath() {
+ mvn dependency:build-classpath -pl :${PL} |\
+ grep -E -v "INFO|WARNING" | \
+ tr ":" "\n" | \
+ awk -F '/' '{
+ artifact_id=$(NF-2);
+ version=$(NF-1);
+ jar_name=$NF;
+ classifier_start_index=length(artifact_id"-"version"-") + 1;
+ classifier_end_index=index(jar_name, ".jar") - 1;
+ classifier=substr(jar_name, classifier_start_index, classifier_end_index - classifier_start_index + 1);
+ print artifact_id"/"version"/"classifier"/"jar_name
+ }' | grep -v "hudi" | sort >> "${DEP_PR}"
+}
+
+function check_diff() {
+ set +e
+ the_diff=$(diff ${DEP} ${DEP_PR})
+ set -e
+ rm -rf "${DEP_PR}"
+ if [[ -n $the_diff ]]; then
+ echo "Dependency List Changed Detected: "
+ echo ${the_diff}
+ echo "To update the dependency file, refer to the usage:"
+ printUsage
+ exit 1
+ fi
+}
+
+if [[ ${#} -eq 0 ]]; then
+ printUsage
+fi
+
+PL=''
+REPLACE='false'
+
+while getopts "rp:" arg; do
+ case "${arg}" in
+ r)
+ REPLACE="true"
+ ;;
+ p)
+ PL=$OPTARG
+ ;;
+ ?)
+ printUsage
+ ;;
+ esac
+done
+
+shift "$(( OPTIND - 1 ))"
+
+# check must option
+if [ -z "$PL" ]; then
+ echo 'Missing -p argument' >&2
+ exit 1
+fi
+
+DEP_PR="${PWD}"/dev/dependencyList"${PL}".tmp
+DEP="${PWD}"/dev/dependencyList_"${PL}"
Review comment:
better to manually add an extension like `.txt`. the scala version `2.11` for eg may lead to unintended file extension. also shall we run this for 2.12 and spark3 combinations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-920690239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256",
"triggerID" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"triggerType" : "PUSH"
}, {
"hash" : "a844e075069d260f22f1a3244708560bebf0128b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2273",
"triggerID" : "a844e075069d260f22f1a3244708560bebf0128b",
"triggerType" : "PUSH"
}, {
"hash" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2321",
"triggerID" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"triggerType" : "PUSH"
}, {
"hash" : "7c634fd5d815d4643732d0c26144171c9dd8e64c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2333",
"triggerID" : "7c634fd5d815d4643732d0c26144171c9dd8e64c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7c634fd5d815d4643732d0c26144171c9dd8e64c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2333)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#discussion_r716322867
##########
File path: scripts/dependency.sh
##########
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -eou pipefail
+set -x
+
+export LC_ALL=C
+
+PWD=$(cd "$(dirname "$0")"/.. || exit; pwd)
+
+function printUsage() {
+ echo "Usage: $(basename "${0}") [-p <artifactId>] -r " 2>&1
+ echo ' -r [OPTIONAL] to replace the old dependencyList file with new dependencies'
+ echo ' -p [MUST] to generate new dependencyList file for the specified module'
+}
+
+function build_classpath() {
+ mvn dependency:build-classpath -pl :${PL} |\
+ grep -E -v "INFO|WARNING" | \
+ tr ":" "\n" | \
+ awk -F '/' '{
+ artifact_id=$(NF-2);
+ version=$(NF-1);
+ jar_name=$NF;
+ classifier_start_index=length(artifact_id"-"version"-") + 1;
+ classifier_end_index=index(jar_name, ".jar") - 1;
+ classifier=substr(jar_name, classifier_start_index, classifier_end_index - classifier_start_index + 1);
+ print artifact_id"/"version"/"classifier"/"jar_name
Review comment:
> shall we follow some existing convention to show the dependences? for example
>
> https://search.maven.org/artifact/org.apache.hudi/hudi-flink-bundle_2.12/0.9.0/jar
>
> the identifier pattern is `group:artifact:version`
You can find the dependency is ordered by the artifact, right? IMO, it's easier to distinguish the different versions of one artifact. And view based on order artifacts is more suitable for human sense. Generally, People pay more attention to artifacts than group, right?
But, yes, I agree that it's better to add group information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua edited a comment on pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
yanghua edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-927459881
@xushiyan Thanks for sharing your thoughts. Let's discuss some points.
> i see the point here is to allow PR reviewer easily identify dep changes.
Yes, that's one of the purposes. Another one is to let the contributors or developers/users have a way to know and view the dependencies of those bundles if that way(meet conflict problems). So I output them into the codebase. Just like the Kyuubi has [done.](https://github.com/apache/incubator-kyuubi/pull/1162/files)
> it turns out to have quite some logic and not friendly to maintainers who are not familiar with bash
IMO, It's a tool just like other tools in the codebase. We do not need to spend much time to change or maintain it. And we can add more comments and a better usage guide. It's the first version in order to receive other inputs from you guys.
> why dependency:build-classpath not dependency:tree?
Because it's easier and more readable.
`build-classpath` looks like `ls` behavior and flatted to a dependencies list. However, `tree` contains nested `| \ + -` right? It's harder to deal with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#discussion_r716321234
##########
File path: scripts/dependency.sh
##########
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -eou pipefail
+set -x
+
+export LC_ALL=C
+
+PWD=$(cd "$(dirname "$0")"/.. || exit; pwd)
+
+function printUsage() {
+ echo "Usage: $(basename "${0}") [-p <artifactId>] -r " 2>&1
+ echo ' -r [OPTIONAL] to replace the old dependencyList file with new dependencies'
+ echo ' -p [MUST] to generate new dependencyList file for the specified module'
+}
+
+function build_classpath() {
+ mvn dependency:build-classpath -pl :${PL} |\
+ grep -E -v "INFO|WARNING" | \
+ tr ":" "\n" | \
+ awk -F '/' '{
+ artifact_id=$(NF-2);
+ version=$(NF-1);
+ jar_name=$NF;
+ classifier_start_index=length(artifact_id"-"version"-") + 1;
+ classifier_end_index=index(jar_name, ".jar") - 1;
+ classifier=substr(jar_name, classifier_start_index, classifier_end_index - classifier_start_index + 1);
+ print artifact_id"/"version"/"classifier"/"jar_name
Review comment:
> seeing double `/` before jar_name.
It is used to extract the `classifier ` value of the maven dependency. If not configured, it's an empty string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3674: [WIP][HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-920690239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256",
"triggerID" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"triggerType" : "PUSH"
}, {
"hash" : "a844e075069d260f22f1a3244708560bebf0128b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2273",
"triggerID" : "a844e075069d260f22f1a3244708560bebf0128b",
"triggerType" : "PUSH"
}, {
"hash" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a844e075069d260f22f1a3244708560bebf0128b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2273)
* e16725afd2d76d509abb7415166bfe4468ee6e58 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#discussion_r716325492
##########
File path: scripts/dependency.sh
##########
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -eou pipefail
+set -x
+
+export LC_ALL=C
+
+PWD=$(cd "$(dirname "$0")"/.. || exit; pwd)
+
+function printUsage() {
+ echo "Usage: $(basename "${0}") [-p <artifactId>] -r " 2>&1
+ echo ' -r [OPTIONAL] to replace the old dependencyList file with new dependencies'
+ echo ' -p [MUST] to generate new dependencyList file for the specified module'
+}
+
+function build_classpath() {
+ mvn dependency:build-classpath -pl :${PL} |\
+ grep -E -v "INFO|WARNING" | \
+ tr ":" "\n" | \
+ awk -F '/' '{
+ artifact_id=$(NF-2);
+ version=$(NF-1);
+ jar_name=$NF;
+ classifier_start_index=length(artifact_id"-"version"-") + 1;
+ classifier_end_index=index(jar_name, ".jar") - 1;
+ classifier=substr(jar_name, classifier_start_index, classifier_end_index - classifier_start_index + 1);
+ print artifact_id"/"version"/"classifier"/"jar_name
+ }' | grep -v "hudi" | sort >> "${DEP_PR}"
+}
+
+function check_diff() {
+ set +e
+ the_diff=$(diff ${DEP} ${DEP_PR})
+ set -e
+ rm -rf "${DEP_PR}"
+ if [[ -n $the_diff ]]; then
+ echo "Dependency List Changed Detected: "
+ echo ${the_diff}
+ echo "To update the dependency file, refer to the usage:"
+ printUsage
+ exit 1
+ fi
+}
+
+if [[ ${#} -eq 0 ]]; then
+ printUsage
+fi
+
+PL=''
+REPLACE='false'
+
+while getopts "rp:" arg; do
+ case "${arg}" in
+ r)
+ REPLACE="true"
+ ;;
+ p)
+ PL=$OPTARG
+ ;;
+ ?)
+ printUsage
+ ;;
+ esac
+done
+
+shift "$(( OPTIND - 1 ))"
+
+# check must option
+if [ -z "$PL" ]; then
+ echo 'Missing -p argument' >&2
+ exit 1
+fi
+
+DEP_PR="${PWD}"/dev/dependencyList"${PL}".tmp
+DEP="${PWD}"/dev/dependencyList_"${PL}"
Review comment:
> better to manually add an extension like `.txt`. the scala version `2.11` for eg may lead to unintended file extension.
reasonable, sounds good.
> also shall we run this for 2.12 and spark3 combinations?
I suggest involving all the bundles we provided.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-930860695
@xushiyan I have addresses some suggestions. Any new inputs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-925613960
I received many reports about dependency conflict around hudi, e.g. [here](https://github.com/apache/incubator-kyuubi/pull/994) and Chinese WeChat group. Considering Hudi depends on so many Hadoop ecosystem components. IMO, it's time to do the dependency governance.
In this PR, I provided a dependency utility script that can be used to search/diff dependencies of bundles when the contributors change the dependencys. What's more, I have pre-generated a dependency list for some bundles.
I have two suggestions:
- build a Github workflow to check the dependency diff, and reviewers should consider any new dependency seriously before merging.
- based on the pre-generated dependency file to do the dependency governance, you can see there are some dependencies that exist in multiple versions for one bundle.
WDYT? @vinothchandar @xushiyan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#discussion_r716324501
##########
File path: scripts/dependency.sh
##########
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -eou pipefail
+set -x
+
+export LC_ALL=C
+
+PWD=$(cd "$(dirname "$0")"/.. || exit; pwd)
+
+function printUsage() {
+ echo "Usage: $(basename "${0}") [-p <artifactId>] -r " 2>&1
+ echo ' -r [OPTIONAL] to replace the old dependencyList file with new dependencies'
+ echo ' -p [MUST] to generate new dependencyList file for the specified module'
+}
Review comment:
Yes, the practice that comes from the Kyuubi community is like this.
If the owner of PR changed the dependencies of maven. The contributor must install the project in its local. And run this script locally to override these dependency list files and submit the changes.
For Github workflow, we only add a new one to run this script and compare with the PR, if they are different, then show the diff and fail the workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3674: [WIP][HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-920690239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256",
"triggerID" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"triggerType" : "PUSH"
}, {
"hash" : "a844e075069d260f22f1a3244708560bebf0128b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a844e075069d260f22f1a3244708560bebf0128b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 61baa2fb144ba3521441b03c51b12da9ff9b11bf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256)
* a844e075069d260f22f1a3244708560bebf0128b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3674: [WIP][HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-920690239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256",
"triggerID" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"triggerType" : "PUSH"
}, {
"hash" : "a844e075069d260f22f1a3244708560bebf0128b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2273",
"triggerID" : "a844e075069d260f22f1a3244708560bebf0128b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 61baa2fb144ba3521441b03c51b12da9ff9b11bf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256)
* a844e075069d260f22f1a3244708560bebf0128b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2273)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3674: [WIP][HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-920690239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256",
"triggerID" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 61baa2fb144ba3521441b03c51b12da9ff9b11bf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3674: [WIP][HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-920690239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256",
"triggerID" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 61baa2fb144ba3521441b03c51b12da9ff9b11bf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua merged pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
yanghua merged pull request #3674:
URL: https://github.com/apache/hudi/pull/3674
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#discussion_r719181196
##########
File path: scripts/dependency.sh
##########
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -eou pipefail
+set -x
+
+export LC_ALL=C
+
+PWD=$(cd "$(dirname "$0")"/.. || exit; pwd)
+
+function printUsage() {
+ echo "Usage: $(basename "${0}") [-p <artifactId>] -r " 2>&1
+ echo ' -r [OPTIONAL] to replace the old dependencyList file with new dependencies'
+ echo ' -p [MUST] to generate new dependencyList file for the specified module'
+}
+
+function build_classpath() {
+ mvn dependency:build-classpath -pl :${PL} |\
+ grep -E -v "INFO|WARNING" | \
+ tr ":" "\n" | \
+ awk -F '/' '{
+ artifact_id=$(NF-2);
+ version=$(NF-1);
+ jar_name=$NF;
+ classifier_start_index=length(artifact_id"-"version"-") + 1;
+ classifier_end_index=index(jar_name, ".jar") - 1;
+ classifier=substr(jar_name, classifier_start_index, classifier_end_index - classifier_start_index + 1);
+ print artifact_id"/"version"/"classifier"/"jar_name
+ }' | grep -v "hudi" | sort >> "${DEP_PR}"
+}
+
+function check_diff() {
+ set +e
+ the_diff=$(diff ${DEP} ${DEP_PR})
+ set -e
+ rm -rf "${DEP_PR}"
+ if [[ -n $the_diff ]]; then
+ echo "Dependency List Changed Detected: "
+ echo ${the_diff}
+ echo "To update the dependency file, refer to the usage:"
+ printUsage
+ exit 1
+ fi
+}
+
+if [[ ${#} -eq 0 ]]; then
+ printUsage
+fi
+
+PL=''
+REPLACE='false'
+
+while getopts "rp:" arg; do
+ case "${arg}" in
+ r)
+ REPLACE="true"
+ ;;
+ p)
+ PL=$OPTARG
+ ;;
+ ?)
+ printUsage
+ ;;
+ esac
+done
+
+shift "$(( OPTIND - 1 ))"
+
+# check must option
+if [ -z "$PL" ]; then
+ echo 'Missing -p argument' >&2
+ exit 1
+fi
+
+DEP_PR="${PWD}"/dev/dependencyList"${PL}".tmp
+DEP="${PWD}"/dev/dependencyList_"${PL}"
Review comment:
yup all bundles should be covered
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3674: [WIP][HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-920690239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256",
"triggerID" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"triggerType" : "PUSH"
}, {
"hash" : "a844e075069d260f22f1a3244708560bebf0128b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2273",
"triggerID" : "a844e075069d260f22f1a3244708560bebf0128b",
"triggerType" : "PUSH"
}, {
"hash" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2321",
"triggerID" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e16725afd2d76d509abb7415166bfe4468ee6e58 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2321)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-920690239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256",
"triggerID" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"triggerType" : "PUSH"
}, {
"hash" : "a844e075069d260f22f1a3244708560bebf0128b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2273",
"triggerID" : "a844e075069d260f22f1a3244708560bebf0128b",
"triggerType" : "PUSH"
}, {
"hash" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2321",
"triggerID" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"triggerType" : "PUSH"
}, {
"hash" : "7c634fd5d815d4643732d0c26144171c9dd8e64c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2333",
"triggerID" : "7c634fd5d815d4643732d0c26144171c9dd8e64c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e16725afd2d76d509abb7415166bfe4468ee6e58 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2321)
* 7c634fd5d815d4643732d0c26144171c9dd8e64c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2333)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-927459881
@xushiyan Thanks for sharing your thoughts. Let's discuss some points.
> i see the point here is to allow PR reviewer easily identify dep changes.
Yes, that's one of the purposes. Another one is to let the contributors or developers/users have a way to know and view the dependencies of those bundles if that way(meet conflict problems). So I output them into the codebase. Just like the Kyuubi has [done.](https://github.com/apache/incubator-kyuubi/pull/1162/files)
> it turns out to have quite some logic and not friendly to maintainers who are not familiar with bash
IMO, It's a tool just like other tools in the codebase. We do not need to spend much time to change or maintain it. And we can add more comments and a better usage guide. It's the first version in order to receive other inputs from you guys.
> why dependency:build-classpath not dependency:tree?
Because it's easier and more readable.
`build-classpath` looks like `ls` behavior and flatted to a dependencies list. While `tree` contains nested `| \ + -` right? It's harder to deal with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-920690239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256",
"triggerID" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"triggerType" : "PUSH"
}, {
"hash" : "a844e075069d260f22f1a3244708560bebf0128b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2273",
"triggerID" : "a844e075069d260f22f1a3244708560bebf0128b",
"triggerType" : "PUSH"
}, {
"hash" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2321",
"triggerID" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"triggerType" : "PUSH"
}, {
"hash" : "7c634fd5d815d4643732d0c26144171c9dd8e64c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2333",
"triggerID" : "7c634fd5d815d4643732d0c26144171c9dd8e64c",
"triggerType" : "PUSH"
}, {
"hash" : "cf1f2dba2dd9ca560cd33ca012dbe5613e42a733",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cf1f2dba2dd9ca560cd33ca012dbe5613e42a733",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7c634fd5d815d4643732d0c26144171c9dd8e64c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2333)
* cf1f2dba2dd9ca560cd33ca012dbe5613e42a733 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3674: [WIP][HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-920690239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 61baa2fb144ba3521441b03c51b12da9ff9b11bf UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-920690239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256",
"triggerID" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"triggerType" : "PUSH"
}, {
"hash" : "a844e075069d260f22f1a3244708560bebf0128b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2273",
"triggerID" : "a844e075069d260f22f1a3244708560bebf0128b",
"triggerType" : "PUSH"
}, {
"hash" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2321",
"triggerID" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"triggerType" : "PUSH"
}, {
"hash" : "7c634fd5d815d4643732d0c26144171c9dd8e64c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7c634fd5d815d4643732d0c26144171c9dd8e64c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e16725afd2d76d509abb7415166bfe4468ee6e58 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2321)
* 7c634fd5d815d4643732d0c26144171c9dd8e64c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3674: [WIP][HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-920690239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256",
"triggerID" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"triggerType" : "PUSH"
}, {
"hash" : "a844e075069d260f22f1a3244708560bebf0128b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2273",
"triggerID" : "a844e075069d260f22f1a3244708560bebf0128b",
"triggerType" : "PUSH"
}, {
"hash" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2321",
"triggerID" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a844e075069d260f22f1a3244708560bebf0128b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2273)
* e16725afd2d76d509abb7415166bfe4468ee6e58 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2321)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#discussion_r716274428
##########
File path: scripts/dependency.sh
##########
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -eou pipefail
+set -x
+
+export LC_ALL=C
+
+PWD=$(cd "$(dirname "$0")"/.. || exit; pwd)
+
+function printUsage() {
+ echo "Usage: $(basename "${0}") [-p <artifactId>] -r " 2>&1
+ echo ' -r [OPTIONAL] to replace the old dependencyList file with new dependencies'
+ echo ' -p [MUST] to generate new dependencyList file for the specified module'
+}
+
+function build_classpath() {
+ mvn dependency:build-classpath -pl :${PL} |\
+ grep -E -v "INFO|WARNING" | \
+ tr ":" "\n" | \
+ awk -F '/' '{
+ artifact_id=$(NF-2);
+ version=$(NF-1);
+ jar_name=$NF;
+ classifier_start_index=length(artifact_id"-"version"-") + 1;
+ classifier_end_index=index(jar_name, ".jar") - 1;
+ classifier=substr(jar_name, classifier_start_index, classifier_end_index - classifier_start_index + 1);
+ print artifact_id"/"version"/"classifier"/"jar_name
Review comment:
shall we follow some existing convention to show the dependences? for example
https://search.maven.org/artifact/org.apache.hudi/hudi-flink-bundle_2.12/0.9.0/jar
the identifier pattern is `group:artifact:version`
##########
File path: scripts/dependency.sh
##########
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -eou pipefail
+set -x
+
+export LC_ALL=C
+
+PWD=$(cd "$(dirname "$0")"/.. || exit; pwd)
+
+function printUsage() {
+ echo "Usage: $(basename "${0}") [-p <artifactId>] -r " 2>&1
+ echo ' -r [OPTIONAL] to replace the old dependencyList file with new dependencies'
+ echo ' -p [MUST] to generate new dependencyList file for the specified module'
+}
+
+function build_classpath() {
+ mvn dependency:build-classpath -pl :${PL} |\
+ grep -E -v "INFO|WARNING" | \
+ tr ":" "\n" | \
+ awk -F '/' '{
+ artifact_id=$(NF-2);
+ version=$(NF-1);
+ jar_name=$NF;
+ classifier_start_index=length(artifact_id"-"version"-") + 1;
+ classifier_end_index=index(jar_name, ".jar") - 1;
+ classifier=substr(jar_name, classifier_start_index, classifier_end_index - classifier_start_index + 1);
+ print artifact_id"/"version"/"classifier"/"jar_name
Review comment:
seeing double `/` before jar_name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3674: [HUDI-2440] Add dependency change diff script for dependency governace
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3674:
URL: https://github.com/apache/hudi/pull/3674#issuecomment-920690239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2256",
"triggerID" : "61baa2fb144ba3521441b03c51b12da9ff9b11bf",
"triggerType" : "PUSH"
}, {
"hash" : "a844e075069d260f22f1a3244708560bebf0128b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2273",
"triggerID" : "a844e075069d260f22f1a3244708560bebf0128b",
"triggerType" : "PUSH"
}, {
"hash" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2321",
"triggerID" : "e16725afd2d76d509abb7415166bfe4468ee6e58",
"triggerType" : "PUSH"
}, {
"hash" : "7c634fd5d815d4643732d0c26144171c9dd8e64c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2333",
"triggerID" : "7c634fd5d815d4643732d0c26144171c9dd8e64c",
"triggerType" : "PUSH"
}, {
"hash" : "cf1f2dba2dd9ca560cd33ca012dbe5613e42a733",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2469",
"triggerID" : "cf1f2dba2dd9ca560cd33ca012dbe5613e42a733",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cf1f2dba2dd9ca560cd33ca012dbe5613e42a733 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2469)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org