You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2021/01/08 17:33:20 UTC
[GitHub] [hbase] saintstack opened a new pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
saintstack opened a new pull request #2862:
URL: https://github.com/apache/hbase/pull/2862
…rror: 'binary'"
Exclude hbase-shaded-testing-util*.jar from checkcompatibility; this
jar can not be unzipped on a case-insensitive filesystem. Added
some means of debug into the checkcompatibility to help when
cryptic failures.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
Posted by GitBox <gi...@apache.org>.
ndimiduk commented on a change in pull request #2862:
URL: https://github.com/apache/hbase/pull/2862#discussion_r554230076
##########
File path: dev-support/create-release/release-util.sh
##########
@@ -469,8 +469,18 @@ function generate_api_report {
local timing_token
timing_token="$(start_step)"
# Generate api report.
+ # Filter out some jar types. Filters are tricky. Python regex on
+ # file basename. Exclude the saved-aside original jars... they are
+ # not included in resulting artifact. Also, do not include the
+ # hbase-shaded-testing-util.* jars. This jar is unzip'able on mac
+ # os x as is because has it a META_INF/LICENSE file and then a
+ # META_INF/license directory for the included jar's licenses;
+ # it fails to unjar on mac os x which this tool does making its checks
+ # (Its exclusion should be fine; it is just an aggregate of other jars).
"${project}"/dev-support/checkcompatibility.py --annotation \
org.apache.yetus.audience.InterfaceAudience.Public \
+ -e "original-hbase.*.jar" \
+ -e "hbase-shaded-testing-util.*.jar" \
Review comment:
> Include/exclude are in the python script. Does jar files only. The jar files are subsequently passed to the wrapped perl script... which then does unzip... and fails with hbase-shaded-testing-util.
I meant exclude of unpacked archive content, not entire archives. And yes, that's what I feared, that the unpack happens in the perl script, "outside of our control."
I'm thinking the reason to retain the shaded jar is to notice if we break shading entirely. i.e., a change that adds a new jar to the shaded package would show in the report as loads of new classes. Perhaps we have other mechanisms in place already for catching these kinds of errors?
I like HBASE-25486.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
Posted by GitBox <gi...@apache.org>.
ndimiduk commented on a change in pull request #2862:
URL: https://github.com/apache/hbase/pull/2862#discussion_r554189525
##########
File path: dev-support/checkcompatibility.py
##########
@@ -241,6 +241,20 @@ def compare_results(tool_results, known_issues, compare_warnings):
return bool(unexpected_issues)
+def compare_tool_results_count(tool_results, check, issue_type, known_count):
+ """ Check problem counts are no more than the known count.
+ (This function exists just so can add in logging; previous was inlined
+ one-liner but this made it hard debugging)
+ """
+ # logging.info("known_count=%s, check key=%s, tool_results=%s, issue_type=%s",
+ # str(known_count), str(check), str(tool_results), str(issue_type))
+ try:
+ return tool_results[check][issue_type] > known_count
+ except KeyError as ke:
+ logging.info("KeyError %s", str(ke))
Review comment:
why log + swallow the error case here? Isn't it better to `raise` the error instead of just logging it? Or maybe calling code ignores thrown exceptions?
##########
File path: dev-support/create-release/release-util.sh
##########
@@ -469,8 +469,18 @@ function generate_api_report {
local timing_token
timing_token="$(start_step)"
# Generate api report.
+ # Filter out some jar types. Filters are tricky. Python regex on
+ # file basename. Exclude the saved-aside original jars... they are
+ # not included in resulting artifact. Also, do not include the
+ # hbase-shaded-testing-util.* jars. This jar is unzip'able on mac
+ # os x as is because has it a META_INF/LICENSE file and then a
+ # META_INF/license directory for the included jar's licenses;
+ # it fails to unjar on mac os x which this tool does making its checks
+ # (Its exclusion should be fine; it is just an aggregate of other jars).
"${project}"/dev-support/checkcompatibility.py --annotation \
org.apache.yetus.audience.InterfaceAudience.Public \
+ -e "original-hbase.*.jar" \
+ -e "hbase-shaded-testing-util.*.jar" \
Review comment:
excluding this jar is a problem -- the shaded jars are intended explicitly as our public interface to down-streamers, aren't they?
What if we add an exclude of the `[lL][iI][cC][eE][nN][sS][eE]` pattern instead? oh, I see the `jar` command doesn't support exclude patterns, only an include pattern. And where does this happen, in the perl script?
##########
File path: dev-support/create-release/release-util.sh
##########
@@ -469,8 +469,18 @@ function generate_api_report {
local timing_token
timing_token="$(start_step)"
# Generate api report.
+ # Filter out some jar types. Filters are tricky. Python regex on
+ # file basename. Exclude the saved-aside original jars... they are
+ # not included in resulting artifact. Also, do not include the
+ # hbase-shaded-testing-util.* jars. This jar is unzip'able on mac
+ # os x as is because has it a META_INF/LICENSE file and then a
+ # META_INF/license directory for the included jar's licenses;
+ # it fails to unjar on mac os x which this tool does making its checks
+ # (Its exclusion should be fine; it is just an aggregate of other jars).
"${project}"/dev-support/checkcompatibility.py --annotation \
org.apache.yetus.audience.InterfaceAudience.Public \
+ -e "original-hbase.*.jar" \
+ -e "hbase-shaded-testing-util.*.jar" \
Review comment:
> Include/exclude are in the python script. Does jar files only. The jar files are subsequently passed to the wrapped perl script... which then does unzip... and fails with hbase-shaded-testing-util.
I meant exclude of unpacked archive content, not entire archives. And yes, that's what I feared, that the unpack happens in the perl script, "outside of our control."
I'm thinking the reason to retain the shaded jar is to notice if we break shading entirely. i.e., a change that adds a new jar to the shaded package would show in the report as loads of new classes. Perhaps we have other mechanisms in place already for catching these kinds of errors?
I like HBASE-25486.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2862:
URL: https://github.com/apache/hbase/pull/2862#issuecomment-757029432
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | shelldocs | 0m 1s | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch |
| -0 :warning: | pylint | 0m 5s | The patch generated 1 new + 0 unchanged - 5 fixed = 1 total (was 5) |
| +1 :green_heart: | shellcheck | 0m 2s | There were no new shellcheck issues. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
||| _ Other Tests _ |
| +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? |
| | | 3m 14s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2862/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2862 |
| Optional Tests | dupname asflicense pylint shellcheck shelldocs |
| uname | Linux 6b8d6904897c 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 49aba57181 |
| pylint | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2862/2/artifact/yetus-general-check/output/diff-patch-pylint.txt |
| Max. process+thread count | 47 (vs. ulimit of 30000) |
| modules | C: U: |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2862/2/console |
| versions | git=2.17.1 maven=3.6.3 shellcheck=0.4.6 pylint=2.4.4 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack merged pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
Posted by GitBox <gi...@apache.org>.
saintstack merged pull request #2862:
URL: https://github.com/apache/hbase/pull/2862
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #2862:
URL: https://github.com/apache/hbase/pull/2862#discussion_r554232498
##########
File path: dev-support/create-release/release-util.sh
##########
@@ -469,8 +469,18 @@ function generate_api_report {
local timing_token
timing_token="$(start_step)"
# Generate api report.
+ # Filter out some jar types. Filters are tricky. Python regex on
+ # file basename. Exclude the saved-aside original jars... they are
+ # not included in resulting artifact. Also, do not include the
+ # hbase-shaded-testing-util.* jars. This jar is unzip'able on mac
+ # os x as is because has it a META_INF/LICENSE file and then a
+ # META_INF/license directory for the included jar's licenses;
+ # it fails to unjar on mac os x which this tool does making its checks
+ # (Its exclusion should be fine; it is just an aggregate of other jars).
"${project}"/dev-support/checkcompatibility.py --annotation \
org.apache.yetus.audience.InterfaceAudience.Public \
+ -e "original-hbase.*.jar" \
+ -e "hbase-shaded-testing-util.*.jar" \
Review comment:
> I'm thinking the reason to retain the shaded jar is to notice if we break shading entirely. i.e., a change that adds a new jar to the shaded package would show in the report as loads of new classes. Perhaps we have other mechanisms in place already for catching these kinds of errors?
This is a valid concern. Lets fix HBASE-25486 and get the shaded jar back in the loop.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2862:
URL: https://github.com/apache/hbase/pull/2862#issuecomment-756899279
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 13s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch |
| -0 :warning: | pylint | 0m 8s | The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5) |
| +1 :green_heart: | shellcheck | 0m 2s | There were no new shellcheck issues. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
||| _ Other Tests _ |
| +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? |
| | | 3m 58s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2862/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2862 |
| Optional Tests | dupname asflicense pylint shellcheck shelldocs |
| uname | Linux 49dafead7c6c 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 5c233e9785 |
| pylint | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2862/1/artifact/yetus-general-check/output/diff-patch-pylint.txt |
| Max. process+thread count | 47 (vs. ulimit of 30000) |
| modules | C: U: |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2862/1/console |
| versions | git=2.17.1 maven=3.6.3 shellcheck=0.4.6 pylint=2.4.4 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #2862:
URL: https://github.com/apache/hbase/pull/2862#discussion_r554224191
##########
File path: dev-support/create-release/release-util.sh
##########
@@ -469,8 +469,18 @@ function generate_api_report {
local timing_token
timing_token="$(start_step)"
# Generate api report.
+ # Filter out some jar types. Filters are tricky. Python regex on
+ # file basename. Exclude the saved-aside original jars... they are
+ # not included in resulting artifact. Also, do not include the
+ # hbase-shaded-testing-util.* jars. This jar is unzip'able on mac
+ # os x as is because has it a META_INF/LICENSE file and then a
+ # META_INF/license directory for the included jar's licenses;
+ # it fails to unjar on mac os x which this tool does making its checks
+ # (Its exclusion should be fine; it is just an aggregate of other jars).
"${project}"/dev-support/checkcompatibility.py --annotation \
org.apache.yetus.audience.InterfaceAudience.Public \
+ -e "original-hbase.*.jar" \
+ -e "hbase-shaded-testing-util.*.jar" \
Review comment:
Include/exclude are in the python script. Does jar files only. The jar files are subsequently passed to the wrapped perl script... which then does unzip... and fails with hbase-shaded-testing-util.
I think it ok excluding this shaded jar because it is made from other jars that are NOT excluded; if an api change violation, it'll be caught there, at the source. Testing the shade would be more comprehensive yes but exclude is fine for now given it allows us getting the RC ball rolling again. The reason-for-exclusion is for fix in HBASE-25486
##########
File path: dev-support/checkcompatibility.py
##########
@@ -241,6 +241,20 @@ def compare_results(tool_results, known_issues, compare_warnings):
return bool(unexpected_issues)
+def compare_tool_results_count(tool_results, check, issue_type, known_count):
+ """ Check problem counts are no more than the known count.
+ (This function exists just so can add in logging; previous was inlined
+ one-liner but this made it hard debugging)
+ """
+ # logging.info("known_count=%s, check key=%s, tool_results=%s, issue_type=%s",
+ # str(known_count), str(check), str(tool_results), str(issue_type))
+ try:
+ return tool_results[check][issue_type] > known_count
+ except KeyError as ke:
+ logging.info("KeyError %s", str(ke))
Review comment:
Let me remove this bit... This was a debug method trying to see what was on other side of exception.... Let me make it fail.
##########
File path: dev-support/create-release/release-util.sh
##########
@@ -469,8 +469,18 @@ function generate_api_report {
local timing_token
timing_token="$(start_step)"
# Generate api report.
+ # Filter out some jar types. Filters are tricky. Python regex on
+ # file basename. Exclude the saved-aside original jars... they are
+ # not included in resulting artifact. Also, do not include the
+ # hbase-shaded-testing-util.* jars. This jar is unzip'able on mac
+ # os x as is because has it a META_INF/LICENSE file and then a
+ # META_INF/license directory for the included jar's licenses;
+ # it fails to unjar on mac os x which this tool does making its checks
+ # (Its exclusion should be fine; it is just an aggregate of other jars).
"${project}"/dev-support/checkcompatibility.py --annotation \
org.apache.yetus.audience.InterfaceAudience.Public \
+ -e "original-hbase.*.jar" \
+ -e "hbase-shaded-testing-util.*.jar" \
Review comment:
> I'm thinking the reason to retain the shaded jar is to notice if we break shading entirely. i.e., a change that adds a new jar to the shaded package would show in the report as loads of new classes. Perhaps we have other mechanisms in place already for catching these kinds of errors?
This is a valid concern. Lets fix HBASE-25486 and get the shaded jar back in the loop.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2862:
URL: https://github.com/apache/hbase/pull/2862#issuecomment-757029157
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 3s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 2m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2862/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2862 |
| Optional Tests | |
| uname | Linux c44ba3ac04e1 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 49aba57181 |
| Max. process+thread count | 51 (vs. ulimit of 30000) |
| modules | C: U: |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2862/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2862:
URL: https://github.com/apache/hbase/pull/2862#issuecomment-756898266
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
Posted by GitBox <gi...@apache.org>.
ndimiduk commented on a change in pull request #2862:
URL: https://github.com/apache/hbase/pull/2862#discussion_r554189525
##########
File path: dev-support/checkcompatibility.py
##########
@@ -241,6 +241,20 @@ def compare_results(tool_results, known_issues, compare_warnings):
return bool(unexpected_issues)
+def compare_tool_results_count(tool_results, check, issue_type, known_count):
+ """ Check problem counts are no more than the known count.
+ (This function exists just so can add in logging; previous was inlined
+ one-liner but this made it hard debugging)
+ """
+ # logging.info("known_count=%s, check key=%s, tool_results=%s, issue_type=%s",
+ # str(known_count), str(check), str(tool_results), str(issue_type))
+ try:
+ return tool_results[check][issue_type] > known_count
+ except KeyError as ke:
+ logging.info("KeyError %s", str(ke))
Review comment:
why log + swallow the error case here? Isn't it better to `raise` the error instead of just logging it? Or maybe calling code ignores thrown exceptions?
##########
File path: dev-support/create-release/release-util.sh
##########
@@ -469,8 +469,18 @@ function generate_api_report {
local timing_token
timing_token="$(start_step)"
# Generate api report.
+ # Filter out some jar types. Filters are tricky. Python regex on
+ # file basename. Exclude the saved-aside original jars... they are
+ # not included in resulting artifact. Also, do not include the
+ # hbase-shaded-testing-util.* jars. This jar is unzip'able on mac
+ # os x as is because has it a META_INF/LICENSE file and then a
+ # META_INF/license directory for the included jar's licenses;
+ # it fails to unjar on mac os x which this tool does making its checks
+ # (Its exclusion should be fine; it is just an aggregate of other jars).
"${project}"/dev-support/checkcompatibility.py --annotation \
org.apache.yetus.audience.InterfaceAudience.Public \
+ -e "original-hbase.*.jar" \
+ -e "hbase-shaded-testing-util.*.jar" \
Review comment:
excluding this jar is a problem -- the shaded jars are intended explicitly as our public interface to down-streamers, aren't they?
What if we add an exclude of the `[lL][iI][cC][eE][nN][sS][eE]` pattern instead? oh, I see the `jar` command doesn't support exclude patterns, only an include pattern. And where does this happen, in the perl script?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack merged pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
Posted by GitBox <gi...@apache.org>.
saintstack merged pull request #2862:
URL: https://github.com/apache/hbase/pull/2862
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2862:
URL: https://github.com/apache/hbase/pull/2862#issuecomment-757029088
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 40s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 2m 5s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2862/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2862 |
| Optional Tests | |
| uname | Linux 8354fcc6ff2d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 49aba57181 |
| Max. process+thread count | 52 (vs. ulimit of 30000) |
| modules | C: U: |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2862/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #2862:
URL: https://github.com/apache/hbase/pull/2862#discussion_r554224607
##########
File path: dev-support/checkcompatibility.py
##########
@@ -241,6 +241,20 @@ def compare_results(tool_results, known_issues, compare_warnings):
return bool(unexpected_issues)
+def compare_tool_results_count(tool_results, check, issue_type, known_count):
+ """ Check problem counts are no more than the known count.
+ (This function exists just so can add in logging; previous was inlined
+ one-liner but this made it hard debugging)
+ """
+ # logging.info("known_count=%s, check key=%s, tool_results=%s, issue_type=%s",
+ # str(known_count), str(check), str(tool_results), str(issue_type))
+ try:
+ return tool_results[check][issue_type] > known_count
+ except KeyError as ke:
+ logging.info("KeyError %s", str(ke))
Review comment:
Let me remove this bit... This was a debug method trying to see what was on other side of exception.... Let me make it fail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #2862:
URL: https://github.com/apache/hbase/pull/2862#discussion_r554224191
##########
File path: dev-support/create-release/release-util.sh
##########
@@ -469,8 +469,18 @@ function generate_api_report {
local timing_token
timing_token="$(start_step)"
# Generate api report.
+ # Filter out some jar types. Filters are tricky. Python regex on
+ # file basename. Exclude the saved-aside original jars... they are
+ # not included in resulting artifact. Also, do not include the
+ # hbase-shaded-testing-util.* jars. This jar is unzip'able on mac
+ # os x as is because has it a META_INF/LICENSE file and then a
+ # META_INF/license directory for the included jar's licenses;
+ # it fails to unjar on mac os x which this tool does making its checks
+ # (Its exclusion should be fine; it is just an aggregate of other jars).
"${project}"/dev-support/checkcompatibility.py --annotation \
org.apache.yetus.audience.InterfaceAudience.Public \
+ -e "original-hbase.*.jar" \
+ -e "hbase-shaded-testing-util.*.jar" \
Review comment:
Include/exclude are in the python script. Does jar files only. The jar files are subsequently passed to the wrapped perl script... which then does unzip... and fails with hbase-shaded-testing-util.
I think it ok excluding this shaded jar because it is made from other jars that are NOT excluded; if an api change violation, it'll be caught there, at the source. Testing the shade would be more comprehensive yes but exclude is fine for now given it allows us getting the RC ball rolling again. The reason-for-exclusion is for fix in HBASE-25486
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2862: HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2862:
URL: https://github.com/apache/hbase/pull/2862#issuecomment-756898266
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org