You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/11/22 18:48:05 UTC
[GitHub] [hadoop] hchaverri opened a new pull request, #5159: HADOOP-18535. Implement token storage solution based on MySQL
hchaverri opened a new pull request, #5159:
URL: https://github.com/apache/hadoop/pull/5159
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
These changes follow the same logic used in the existing ZK based secret manager implementation.
1. Minor changes to DelegationTokenInformation class so it's easier to convert to byte array format.
2. SQLDelegationTokenSecretManager transforms tokens and delegation keys into expected format and delegates the storage operations to the implementation class.
3. SQLDelegationTokenSecretManagerImpl interacts with SQL storage.
4. DistributedSQLCounter is used to manage sequenceNumbers and delegationKeyIds in a distributed way.
5. Including SQL schema files to create sample database, tables and users.
### How was this patch tested?
1. Executed operations for token creation, renewal and cancellation. Compared results with existing ZK implementation.
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5159: HADOOP-18535. Implement token storage solution based on MySQL
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5159:
URL: https://github.com/apache/hadoop/pull/5159#issuecomment-1324577713
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 8s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 35s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 24s | | trunk passed |
| +1 :green_heart: | compile | 25m 51s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 22m 11s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 13s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 3s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 1m 0s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 24m 35s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 59s | | the patch passed |
| +1 :green_heart: | compile | 24m 49s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 24m 49s | | the patch passed |
| +1 :green_heart: | compile | 22m 10s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 10s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/2/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 20s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/2/artifact/out/results-checkstyle-root.txt) | root: The patch generated 7 new + 24 unchanged - 0 fixed = 31 total (was 24) |
| +1 :green_heart: | mvnsite | 3m 48s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | javadoc | 3m 5s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 2s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 44s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 24m 50s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 43s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 19m 5s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 34m 12s | | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 12s | | The patch does not generate ASF License warnings. |
| | | 292m 21s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5159 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint shellcheck shelldocs |
| uname | Linux aeb70bb10131 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0cd73b10ad085b27088c12af377d0f594fbd1b89 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/2/testReport/ |
| Max. process+thread count | 3144 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5159: HADOOP-18535. Implement token storage solution based on MySQL
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5159:
URL: https://github.com/apache/hadoop/pull/5159#issuecomment-1324359389
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 7s | | trunk passed |
| +1 :green_heart: | compile | 23m 21s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 20m 51s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 36s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 53s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 1m 22s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 22m 17s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 39s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 4s | | the patch passed |
| +1 :green_heart: | compile | 22m 44s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 22m 44s | | the patch passed |
| +1 :green_heart: | compile | 20m 53s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 53s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 6s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/1/artifact/out/results-checkstyle-root.txt) | root: The patch generated 7 new + 24 unchanged - 0 fixed = 31 total (was 24) |
| +1 :green_heart: | mvnsite | 4m 21s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 1m 18s | [/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/1/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt) | hadoop-common-project_hadoop-common-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | javadoc | 3m 46s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 50s | | hadoop-project has no data from spotbugs |
| -1 :x: | spotbugs | 1m 58s | [/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/1/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html) | hadoop-hdfs-project/hadoop-hdfs-rbf generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| +1 :green_heart: | shadedclient | 22m 32s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 49s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 19m 26s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 22m 52s | | hadoop-hdfs-rbf in the patch passed. |
| -1 :x: | asflicense | 1m 25s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/1/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 270m 59s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs-rbf |
| | org.apache.hadoop.hdfs.server.federation.router.security.token.DistributedSQLCounter.selectCounterValue(boolean, Connection) may fail to clean up java.sql.ResultSet Obligation to clean up resource created at DistributedSQLCounter.java:clean up java.sql.ResultSet Obligation to clean up resource created at DistributedSQLCounter.java:[line 66] is not discharged |
| | org.apache.hadoop.hdfs.server.federation.router.security.token.SQLDelegationTokenSecretManagerImpl.selectDelegationKey(int) may fail to clean up java.sql.ResultSet Obligation to clean up resource created at SQLDelegationTokenSecretManagerImpl.java:up java.sql.ResultSet Obligation to clean up resource created at SQLDelegationTokenSecretManagerImpl.java:[line 172] is not discharged |
| | org.apache.hadoop.hdfs.server.federation.router.security.token.SQLDelegationTokenSecretManagerImpl.selectTokenInfo(int, byte[]) may fail to clean up java.sql.ResultSet Obligation to clean up resource created at SQLDelegationTokenSecretManagerImpl.java:clean up java.sql.ResultSet Obligation to clean up resource created at SQLDelegationTokenSecretManagerImpl.java:[line 126] is not discharged |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5159 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint shellcheck shelldocs |
| uname | Linux 7d8a8ec565f2 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 54419bb7872cd05718a5a9b5d33c44f2c1372cc8 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/1/testReport/ |
| Max. process+thread count | 2398 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hchaverri closed pull request #5159: HADOOP-18535. Implement token storage solution based on MySQL
Posted by "hchaverri (via GitHub)" <gi...@apache.org>.
hchaverri closed pull request #5159: HADOOP-18535. Implement token storage solution based on MySQL
URL: https://github.com/apache/hadoop/pull/5159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org