You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/05/01 03:20:08 UTC
[GitHub] [hbase] busbey opened a new pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
busbey opened a new pull request #1620:
URL: https://github.com/apache/hbase/pull/1620
WIP that can get through the GPG signing test when building on mac.
includes some additional cleanup of the release script.
Need to get through a full build still.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-629727207
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 54s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 3m 6s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | |
| uname | Linux 9be806ee7c4b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 15627bb722 |
| Max. process+thread count | 52 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] asfgit closed pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #1620:
URL: https://github.com/apache/hbase/pull/1620
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-629861601
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 28s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 2m 39s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | |
| uname | Linux daa0ec5d60ed 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 15627bb722 |
| Max. process+thread count | 45 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-622231680
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 39s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch |
| -1 :x: | hadolint | 0m 1s | The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| -0 :warning: | shellcheck | 0m 2s | The patch generated 3 new + 89 unchanged - 9 fixed = 92 total (was 98) |
| -0 :warning: | whitespace | 0m 0s | The patch 5 line(s) with tabs. |
||| _ Other Tests _ |
| +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? |
| | | 2m 18s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | dupname asflicense shellcheck shelldocs hadolint |
| uname | Linux 416ed6fc17b4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 512d00e75d |
| hadolint | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/1/artifact/yetus-general-check/output/diff-patch-hadolint.txt |
| shellcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/1/artifact/yetus-general-check/output/diff-patch-shellcheck.txt |
| whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/1/artifact/yetus-general-check/output/whitespace-tabs.txt |
| Max. process+thread count | 52 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 hadolint=1.17.5-0-g443423c |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-648602777
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r426321582
##########
File path: dev-support/create-release/do-release-docker.sh
##########
@@ -102,12 +102,26 @@ if [ -d "$WORKDIR/output" ]; then
fi
fi
+if [ -f "${WORKDIR}/gpg-proxy.ssh.pid" ] || \
+ [ -f "${WORKDIR}/gpg-proxy.cid" ] || \
+ [ -f "${WORKDIR}/release.cid" ]; then
+ read -r -p "container/pid files from prior run exists. Overwrite and continue? [y/n] " ANSWER
+ if [ "$ANSWER" != "y" ]; then
+ error "Exiting."
+ fi
+fi
+
cd "$WORKDIR"
rm -rf "$WORKDIR/output"
+rm -rf "${WORKDIR}/gpg-proxy.ssh.pid" "${WORKDIR}/gpg-proxy.cid" "${WORKDIR}/release.cid"
Review comment:
so the clean up function should take care of them, but something can go wrong such that they don't get handled.
since docker will fail loudly if the passed container id file path exists, this ensures we have a smooth path back to running again even if something goes wrong.
also the latest version of the WIP has an option to purposefully leave those files in place at the end of execution in case someone needs to debug what the containers are doing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-648602921
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 37s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch |
| +1 :green_heart: | hadolint | 0m 3s | There were no new hadolint issues. |
| +1 :green_heart: | shellcheck | 0m 2s | There were no new shellcheck issues. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
||| _ Other Tests _ |
| +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? |
| | | 2m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/9/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | dupname asflicense shellcheck shelldocs hadolint |
| uname | Linux 7ad39e0febcb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b556343292 |
| Max. process+thread count | 51 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/9/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 hadolint=1.17.5-0-g443423c |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-639195328
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 1m 45s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/8/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | |
| uname | Linux 2e098cdb74b3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b2ec4c1ea0 |
| Max. process+thread count | 45 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/8/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r439168143
##########
File path: dev-support/create-release/README.txt
##########
@@ -37,15 +47,53 @@ $ sudo add-apt-repository -y \
stable"
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
-$ sudo usermod -a -G docker $USERID
+# Follow the post installation steps: https://docs.docker.com/engine/install/linux-postinstall/
+$ sudo usermod -aG docker $USER
# LOGOUT and then LOGIN again so $USERID shows as part of docker group
-# Copy up private key for $USERID export from laptop and import on gce.
-$ gpg --import stack.duboce.net.asc
-$ export GPG_TTY=$(tty) # https://github.com/keybase/keybase-issues/issues/2798
-$ eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400)
-$ export PROJECT="${PROJECT:-hbase}"
-$ git clone https://github.com/apache/${PROJECT}.git
-$ cd "${PROJECT}"
+# Test here by running docker's hello world as your build user
+$ docker run hello-world
+
+# Follow the GPG guide for forwarding your gpg-agent from your local machine to the VM
+# https://wiki.gnupg.org/AgentForwarding
+# On the VM find out the location of the gpg agent socket and extra socket
+$ gpgconf --list-dir agent-socket
+/run/user/1000/gnupg/S.gpg-agent
+$ gpgconf --list-dir agent-extra-socket
+/run/user/1000/gnupg/S.gpg-agent.extra
+# On the VM configure sshd to remove stale sockets
+$ sudo bash -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config'
+$ sudo systemctl restart ssh
+# logout of the VM
+
+# Do these steps on your local machine.
+# Export your public key and copy it to the VM.
+# Assuming 'example.gce.host' maps to your VM's external IP (or use the IP)
+$ gpg --export example@apache.org > ~/gpg.example.apache.pub
+$ scp ~/gpg.example.apache.pub example.gce.host:
+# ssh into the VM while forwarding the remote gpg socket locations found above to your local
+# gpg-agent's extra socket (this will restrict what commands the remote node is allowed to have
+# your agent handle. Note that the gpg guide above can help you set this up in your ssh config
+# rather than typing it in ssh like this every time.
+$ ssh -i ~/.ssh/my_id \
+ -R "/run/user/1000/gnupg/S.gpg-agent:$(gpgconf --list-dir agent-extra-socket)" \
+ -R "/run/user/1000/gnupg/S.gpg-agent.extra:$(gpgconf --list-dir agent-extra-socket)" \
+ example.gce.host
+
+# now in an SSH session on the VM with the socket forwarding
+# import your public key and test signing with the forwarding to your local agent.
+$ gpg --no-autostart --import gpg.example.apache.pub
+$ echo "foo" > foo.txt
+$ gpg --no-autostart --detach --armor --sign foo.txt
+$ gpg --no-autostart --verify foo.txt.asc
+
+# install git and clone the main project on the build machine
+$ sudo apt-get install -y git
+$ git clone https://github.com/apache/hbase.git
Review comment:
the original docs pointed at github so I figured it wasn't worth changing. I also haven't looked at the relative uptime of gitbox vs github lately. frankly I don't think it matters. Both are considered authoritative now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-648639329
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch |
| +1 :green_heart: | hadolint | 0m 3s | There were no new hadolint issues. |
| +1 :green_heart: | shellcheck | 0m 3s | There were no new shellcheck issues. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
||| _ Other Tests _ |
| +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? |
| | | 2m 13s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/10/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | dupname asflicense shellcheck shelldocs hadolint |
| uname | Linux 612a9920d620 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b556343292 |
| Max. process+thread count | 50 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/10/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 hadolint=1.17.5-0-g443423c |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-648637103
these two additional commits are both minor tweaks. @ndimiduk and @mattf-apache please shout if you'd like to give further feedback. barring that I'll go ahead with merging after the qabot comes back and I get time to squash+merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache edited a comment on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
mattf-apache edited a comment on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-644969177
@ndimiduk @busbey , both the creation of the `-r` option and the use of shared `objects` while cloning local repo, were part of [commit a9fefd7f533c](https://github.pie.apple.com/IPR/apache-hbase/commit/a9fefd7f533cf9bdb6369d7159ab0df3e01b357d) of PR #1725 (HBASE-24297 release scripts should be able to use an existing project clone). I didn't test with it, sorry.
Parenthetically, I am confused by the `-r .git` idiom. If I understand the GIT_REPO usage properly, it needs to point at an actual repository, while `.git` is the usual name of the git account metadata directory. Why would I point `-r` at it? The processing code doesn't shed any light for me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] apurtell commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
apurtell commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r435637248
##########
File path: dev-support/create-release/README.txt
##########
@@ -17,13 +17,23 @@ anomalies are explained up in JIRA.
See http://hbase.apache.org/book.html#maven.release
+Before starting an RC build, make sure your local gpg-agent has configs
+to properly handle your credentials, especially if you want to avoid
+typing the passphrase to your secret key.
+
+e.g. if you are going to run and step away, best to increase the TTL
+on caching the unlocked secret via ~/.gnupg/gpg-agent.conf
+ # in seconds, e.g. a day
+ default-cache-ttl 86400
+ max-cache-ttl 86400
+
Running a build on GCE is easy enough. Here are some notes if of use.
Create an instance. 4CPU/15G/10G disk seems to work well enough.
Once up, run the below to make your machine fit for RC building:
-# Presuming debian-compatible OS
-$ sudo apt-get install -y git openjdk-8-jdk maven gnupg gnupg-agent
-# Install docker
+# Presuming debian-compatible OS, do these steps on the VM
+# your VM username should be your ASF id, because it will show up in build artifacts.
+# Follow the docker install guide: https://docs.docker.com/engine/install/debian/
Review comment:
I would like to release without installing/requiring docker. Is that possible?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey edited a comment on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey edited a comment on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-643723774
> A question: It seems it will require the RM's gpg passphrase at signing time, which is
> many minutes into the build process. If the RM is off having a cup of coffee, how long
> will it wait before experiencing an ssh and/or gpg-agent timeout? If this is an issue, is there a way to shortcut it by signing a foo file locally, with the local gpg-agent cache timeout set to more than the default 2 hours?
If you've set up your gpg agent and unlocked your private key for it then you do not need to give the passphrase during the build. the README goes over increasing the timeout to a day instead of 2 hours.
We also test that the gpg agent can be connected to once the container launches and before we start building.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-625465391
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 39s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 2m 6s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | |
| uname | Linux 2d63e98e7d1e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2cafe81e9c |
| Max. process+thread count | 50 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/3/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-633096629
I think this is ready now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-625442526
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 41s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 37s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| -1 :x: | hadolint | 0m 3s | The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) |
| -0 :warning: | shellcheck | 0m 3s | The patch generated 12 new + 102 unchanged - 18 fixed = 114 total (was 120) |
| -0 :warning: | whitespace | 0m 0s | The patch 5 line(s) with tabs. |
||| _ Other Tests _ |
| +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? |
| | | 2m 52s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | dupname asflicense shellcheck shelldocs hadolint |
| uname | Linux 29f1e06866e7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2cafe81e9c |
| hadolint | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/2/artifact/yetus-general-check/output/diff-patch-hadolint.txt |
| shellcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/2/artifact/yetus-general-check/output/diff-patch-shellcheck.txt |
| whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/2/artifact/yetus-general-check/output/whitespace-tabs.txt |
| Max. process+thread count | 52 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 hadolint=1.17.5-0-g443423c |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r439862148
##########
File path: dev-support/create-release/hbase-rm/Dockerfile
##########
@@ -50,10 +50,15 @@ RUN wget -qO- "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&fi
tar xvz -C /opt
ENV YETUS_HOME /opt/apache-yetus-${YETUS_VERSION}
-WORKDIR /opt/hbase-rm/output
-
ARG UID
-RUN useradd -m -s /bin/bash -p hbase-rm -u $UID hbase-rm
-USER hbase-rm:hbase-rm
+ARG RM_USER
+RUN groupadd hbase-rm && \
+ useradd --create-home --shell /bin/bash -p hbase-rm -u $UID $RM_USER && \
+ mkdir /home/$RM_USER/.gnupg && \
+ chown -R $RM_USER:hbase-rm /home/$RM_USER && \
+ chmod -R 700 /home/$RM_USER
+
+USER $RM_USER:hbase-rm
+WORKDIR /home/$RM_USER/hbase-rm/
Review comment:
Yup.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-643723774
> A question: It seems it will require the RM's gpg passphrase at signing time, which is
> many minutes into the build process. If the RM is off having a cup of coffee, how long
> will it wait before experiencing an ssh and/or gpg-agent timeout? If this is an issue, is there a way to shortcut it by signing a foo file locally, with the local gpg-agent cache timeout set to more than the default 2 hours?
If you've set up your gpg agent and unlocked your private key for it then you do not need to give the passphrase during the build. the README goes over increasing the timeout to a day instead of 2 hours.
We also test that the gpg agent can be connected to one the container launches and before we start building.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r439169131
##########
File path: dev-support/create-release/mac-sshd-gpg-agent/Dockerfile
##########
@@ -0,0 +1,100 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Image for use on Mac boxes to get a gpg agent socket available
Review comment:
You could use this on other host OSes but it is only needed on the Mac because Docker Desktop for mac can't bind mount generic sockets.
there's notes in the `do-docker-build.sh` about use of this container on OS X vs what happens when we run on other OSes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r439580182
##########
File path: dev-support/create-release/README.txt
##########
@@ -37,15 +47,53 @@ $ sudo add-apt-repository -y \
stable"
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
-$ sudo usermod -a -G docker $USERID
+# Follow the post installation steps: https://docs.docker.com/engine/install/linux-postinstall/
+$ sudo usermod -aG docker $USER
# LOGOUT and then LOGIN again so $USERID shows as part of docker group
-# Copy up private key for $USERID export from laptop and import on gce.
-$ gpg --import stack.duboce.net.asc
-$ export GPG_TTY=$(tty) # https://github.com/keybase/keybase-issues/issues/2798
-$ eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400)
-$ export PROJECT="${PROJECT:-hbase}"
-$ git clone https://github.com/apache/${PROJECT}.git
-$ cd "${PROJECT}"
+# Test here by running docker's hello world as your build user
+$ docker run hello-world
+
+# Follow the GPG guide for forwarding your gpg-agent from your local machine to the VM
+# https://wiki.gnupg.org/AgentForwarding
+# On the VM find out the location of the gpg agent socket and extra socket
+$ gpgconf --list-dir agent-socket
+/run/user/1000/gnupg/S.gpg-agent
+$ gpgconf --list-dir agent-extra-socket
+/run/user/1000/gnupg/S.gpg-agent.extra
+# On the VM configure sshd to remove stale sockets
+$ sudo bash -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config'
+$ sudo systemctl restart ssh
+# logout of the VM
+
+# Do these steps on your local machine.
+# Export your public key and copy it to the VM.
+# Assuming 'example.gce.host' maps to your VM's external IP (or use the IP)
+$ gpg --export example@apache.org > ~/gpg.example.apache.pub
Review comment:
Yes, brilliant! (The added documentation, that is.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r439795504
##########
File path: dev-support/create-release/hbase-rm/Dockerfile
##########
@@ -50,10 +50,15 @@ RUN wget -qO- "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&fi
tar xvz -C /opt
ENV YETUS_HOME /opt/apache-yetus-${YETUS_VERSION}
-WORKDIR /opt/hbase-rm/output
-
ARG UID
-RUN useradd -m -s /bin/bash -p hbase-rm -u $UID hbase-rm
-USER hbase-rm:hbase-rm
+ARG RM_USER
+RUN groupadd hbase-rm && \
+ useradd --create-home --shell /bin/bash -p hbase-rm -u $UID $RM_USER && \
+ mkdir /home/$RM_USER/.gnupg && \
+ chown -R $RM_USER:hbase-rm /home/$RM_USER && \
+ chmod -R 700 /home/$RM_USER
+
+USER $RM_USER:hbase-rm
+WORKDIR /home/$RM_USER/hbase-rm/
Review comment:
you need the gpg directory in the user home directory. this clean up was just because I figured we should be consistent once there was a home directory.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-643814757
> If you've set up your gpg agent and unlocked your private key for it then you do not need to give the passphrase during the build. the README goes over increasing the timeout to a day instead of 2 hours.
Nice, I missed that. Thanks.
> yes your gpg agent has to be running on your local machine before hand. ...
> I could add an example of forcing a clean start of the gpg-agent, e.g. gpgconf --kill all && gpg-connect-agent /bye
> The test signature at container launch should ensure the key is unlocked prior to us building artifacts.
I recommend adding that example in the README. I think most people do not have gpg-agent running by default, may not know how to launch it (I was vague on it), and while it is true that the test signature at container launch will catch a missing agent timely, it won't tell the user how to fix the problem. -- So, maybe also have that test sig command catch failure and give an instructive error message. Please?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
ndimiduk commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-644993808
> Similarly, regarding the failure without `-r`, I don't have insights. The code in release-utils.sh:git_clone_overwrite() is supposed to url-encode the ASF_PASSWORD, and it was inherited from prior code that worked in-situ before PR #1725 . See https://github.pie.apple.com/IPR/apache-hbase/blob/8ff8e70edfbb50967cf123e6afdd9cc5dcfa4878/dev-support/create-release/release-build.sh#L116-L119
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-645028169
Righto. Nice catch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-632983426
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 1m 54s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | |
| uname | Linux 9aa66f2a0e5f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a9fefd7f53 |
| Max. process+thread count | 54 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/6/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
ndimiduk commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-644447084
Using it without `-r` doesn't work either.
```
Cloning into 'hbase'...
fatal: unable to access 'https://ndimiduk:hXXXXXXXXX@gitbox.apache.org/repos/asf/hbase.git/': Port number ended with 'h'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-644969177
@ndimiduk @busbey , both the creation of the `-r` option and the use of shared `objects` while cloning local repo, were part of [commit a9fefd7f533c](https://github.pie.apple.com/IPR/apache-hbase/commit/a9fefd7f533cf9bdb6369d7159ab0df3e01b357d) of PR #1725 (HBASE-24297 release scripts should be able to use an existing project clone). I didn't test with it.
Parenthetically, I am confused by the `-r .git` idiom. If I understand the GIT_REPO usage properly, it needs to point at an actual repository, while `.git` is the usual name of the git account metadata directory. Why would I point `-r` at it? The processing code doesn't shed any light for me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r439593215
##########
File path: dev-support/create-release/README.txt
##########
@@ -37,15 +47,53 @@ $ sudo add-apt-repository -y \
stable"
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
-$ sudo usermod -a -G docker $USERID
+# Follow the post installation steps: https://docs.docker.com/engine/install/linux-postinstall/
+$ sudo usermod -aG docker $USER
# LOGOUT and then LOGIN again so $USERID shows as part of docker group
-# Copy up private key for $USERID export from laptop and import on gce.
-$ gpg --import stack.duboce.net.asc
-$ export GPG_TTY=$(tty) # https://github.com/keybase/keybase-issues/issues/2798
-$ eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400)
-$ export PROJECT="${PROJECT:-hbase}"
-$ git clone https://github.com/apache/${PROJECT}.git
-$ cd "${PROJECT}"
+# Test here by running docker's hello world as your build user
+$ docker run hello-world
+
+# Follow the GPG guide for forwarding your gpg-agent from your local machine to the VM
+# https://wiki.gnupg.org/AgentForwarding
+# On the VM find out the location of the gpg agent socket and extra socket
+$ gpgconf --list-dir agent-socket
+/run/user/1000/gnupg/S.gpg-agent
+$ gpgconf --list-dir agent-extra-socket
+/run/user/1000/gnupg/S.gpg-agent.extra
+# On the VM configure sshd to remove stale sockets
+$ sudo bash -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config'
+$ sudo systemctl restart ssh
+# logout of the VM
+
+# Do these steps on your local machine.
+# Export your public key and copy it to the VM.
+# Assuming 'example.gce.host' maps to your VM's external IP (or use the IP)
+$ gpg --export example@apache.org > ~/gpg.example.apache.pub
+$ scp ~/gpg.example.apache.pub example.gce.host:
+# ssh into the VM while forwarding the remote gpg socket locations found above to your local
+# gpg-agent's extra socket (this will restrict what commands the remote node is allowed to have
+# your agent handle. Note that the gpg guide above can help you set this up in your ssh config
+# rather than typing it in ssh like this every time.
+$ ssh -i ~/.ssh/my_id \
+ -R "/run/user/1000/gnupg/S.gpg-agent:$(gpgconf --list-dir agent-extra-socket)" \
+ -R "/run/user/1000/gnupg/S.gpg-agent.extra:$(gpgconf --list-dir agent-extra-socket)" \
+ example.gce.host
Review comment:
@busbey , I'm not sure this is an issue, I'm just going by the docs. However:
Does this work correctly if gpg-agent is not already running on my local machine, given that the remote invocation of gpg on line 84 uses `--no-autostart`?
In other words, should the instructions include something at about line 72 that will guarantee the local gpg-agent is running, such as signing a foo file locally too?
This also would have the advantage of pre-loading the passphrase into the local host's gpg-agent cache, so maybe the RM doesn't have to be alert at the keyboard when signing time comes around.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
ndimiduk commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-647710176
> the object directory messages are a side effect of using the shared objects. It shouldn't be listed as an error since git then immediately checks the alternates we provide and finds what it needs.
I think the build failed for me due to this error. Will try it again for RC1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-632983548
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 33s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 2m 54s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | |
| uname | Linux b759ccc08827 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a9fefd7f53 |
| Max. process+thread count | 43 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/6/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-629861490
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 1m 53s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | |
| uname | Linux 726fd3d0fe94 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 15627bb722 |
| Max. process+thread count | 52 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r426321658
##########
File path: dev-support/create-release/do-release-docker.sh
##########
@@ -151,21 +200,53 @@ GIT_NAME=$GIT_NAME
GIT_EMAIL=$GIT_EMAIL
GPG_KEY=$GPG_KEY
ASF_PASSWORD=$ASF_PASSWORD
-GPG_PASSPHRASE=$GPG_PASSPHRASE
RELEASE_STEP=$RELEASE_STEP
RELEASE_STEP=$RELEASE_STEP
API_DIFF_TAG=$API_DIFF_TAG
EOF
-JAVA_VOL=
+JAVA_MOUNT=()
if [ -n "$JAVA" ]; then
echo "JAVA_HOME=/opt/hbase-java" >> "$ENVFILE"
- JAVA_VOL="--volume $JAVA:/opt/hbase-java"
+ JAVA_MOUNT=(--mount "type=bind,src=${JAVA},dest=/opt/hbase-java,readonly")
+fi
+
+GPG_PROXY_MOUNT=()
+if [ "${HOST_OS}" == "DARWIN" ]; then
+ GPG_PROXY_MOUNT=(--mount "type=volume,src=gpgagent,dst=/home/${USER}/.gnupg/")
+ echo "Setting up GPG agent proxy container needed on OS X."
+ echo " we should clean this up for you. If that fails the container ID is below and in " \
+ "gpg-proxy.cid"
+ #TODO the key pair used should be configurable
+ docker run --rm -p 62222:22 \
+ --detach --cidfile "${WORKDIR}/gpg-proxy.cid" \
+ --mount \
+ "type=bind,src=${HOME}/.ssh/id_rsa.pub,dst=/home/${USER}/.ssh/authorized_keys,readonly" \
+ "${GPG_PROXY_MOUNT[@]}" \
+ "org.apache.hbase/gpg-agent-proxy:${IMGTAG}"
+ echo "Launching ssh reverse tunnel from the container to gpg agent."
+ echo " we should clean this up for you. If that fails the PID is in gpg-proxy.ssh.pid"
+ ssh -p 62222 -R "/home/${USER}/.gnupg/S.gpg-agent:$(gpgconf --list-dir agent-extra-socket)" \
+ -i "${HOME}/.ssh/id_rsa" -N -n localhost &
+ echo $! > "${WORKDIR}/gpg-proxy.ssh.pid"
+else
+ # TODO this presumes we are still trying to make a local gpg-agent available to the container.
+ # add an option so that we can run the buid on a remote machine and get the forwarded
+ # gpg-agent in the container. Should look like the side-car container mount above.
+ # it is important not to do that for a local linux agent because we only want the container
+ # to get access to the restricted extra socket on our local gpg-agent.
+ GPG_PROXY_MOUNT=(--mount \
+ "type=bind,src=$(gpgconf --list-dir agent-extra-socket),dst=/home/${USER}/.gnupg/S.gpg-agent")
Review comment:
yeah. I think I can make this less complicated by giving docs on how to proxy your gpg-agent to a remote host. gonna work through that next now that I have local execution on my mac working.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r439168475
##########
File path: dev-support/create-release/README.txt
##########
@@ -37,15 +47,53 @@ $ sudo add-apt-repository -y \
stable"
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
-$ sudo usermod -a -G docker $USERID
+# Follow the post installation steps: https://docs.docker.com/engine/install/linux-postinstall/
+$ sudo usermod -aG docker $USER
# LOGOUT and then LOGIN again so $USERID shows as part of docker group
-# Copy up private key for $USERID export from laptop and import on gce.
-$ gpg --import stack.duboce.net.asc
-$ export GPG_TTY=$(tty) # https://github.com/keybase/keybase-issues/issues/2798
-$ eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400)
-$ export PROJECT="${PROJECT:-hbase}"
-$ git clone https://github.com/apache/${PROJECT}.git
-$ cd "${PROJECT}"
+# Test here by running docker's hello world as your build user
+$ docker run hello-world
+
+# Follow the GPG guide for forwarding your gpg-agent from your local machine to the VM
+# https://wiki.gnupg.org/AgentForwarding
+# On the VM find out the location of the gpg agent socket and extra socket
+$ gpgconf --list-dir agent-socket
+/run/user/1000/gnupg/S.gpg-agent
+$ gpgconf --list-dir agent-extra-socket
+/run/user/1000/gnupg/S.gpg-agent.extra
+# On the VM configure sshd to remove stale sockets
+$ sudo bash -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config'
+$ sudo systemctl restart ssh
+# logout of the VM
+
+# Do these steps on your local machine.
+# Export your public key and copy it to the VM.
+# Assuming 'example.gce.host' maps to your VM's external IP (or use the IP)
+$ gpg --export example@apache.org > ~/gpg.example.apache.pub
+$ scp ~/gpg.example.apache.pub example.gce.host:
+# ssh into the VM while forwarding the remote gpg socket locations found above to your local
+# gpg-agent's extra socket (this will restrict what commands the remote node is allowed to have
+# your agent handle. Note that the gpg guide above can help you set this up in your ssh config
+# rather than typing it in ssh like this every time.
+$ ssh -i ~/.ssh/my_id \
+ -R "/run/user/1000/gnupg/S.gpg-agent:$(gpgconf --list-dir agent-extra-socket)" \
+ -R "/run/user/1000/gnupg/S.gpg-agent.extra:$(gpgconf --list-dir agent-extra-socket)" \
+ example.gce.host
+
+# now in an SSH session on the VM with the socket forwarding
+# import your public key and test signing with the forwarding to your local agent.
+$ gpg --no-autostart --import gpg.example.apache.pub
+$ echo "foo" > foo.txt
+$ gpg --no-autostart --detach --armor --sign foo.txt
+$ gpg --no-autostart --verify foo.txt.asc
+
+# install git and clone the main project on the build machine
+$ sudo apt-get install -y git
+$ git clone https://github.com/apache/hbase.git
+# finally set up an output folder and launch a dry run.
$ mkdir ~/build
-$ ./dev-resources/create-release/do-release-docker.sh -d ~/build
-# etc.
+$ cd hbase
+$ ./dev-support/create-release/do-release-docker.sh -d ~/build
+
+# for building the main repo specifically you can save an extra download by pointing the build
+# to the local clone you just made
+$ ./dev-support/create-release/do-release-docker.sh -d ~/build -r .git
Review comment:
I dunno. I didn't want to force anyone into using my workflow so the goal was to have the default after these changes stay the same for as much as possible. FWIW I can't think of a reason not to take this approach when releasing the main project repo.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r429562396
##########
File path: dev-support/create-release/do-release-docker.sh
##########
@@ -151,21 +200,53 @@ GIT_NAME=$GIT_NAME
GIT_EMAIL=$GIT_EMAIL
GPG_KEY=$GPG_KEY
ASF_PASSWORD=$ASF_PASSWORD
-GPG_PASSPHRASE=$GPG_PASSPHRASE
RELEASE_STEP=$RELEASE_STEP
RELEASE_STEP=$RELEASE_STEP
API_DIFF_TAG=$API_DIFF_TAG
EOF
-JAVA_VOL=
+JAVA_MOUNT=()
if [ -n "$JAVA" ]; then
echo "JAVA_HOME=/opt/hbase-java" >> "$ENVFILE"
- JAVA_VOL="--volume $JAVA:/opt/hbase-java"
+ JAVA_MOUNT=(--mount "type=bind,src=${JAVA},dest=/opt/hbase-java,readonly")
+fi
+
+GPG_PROXY_MOUNT=()
+if [ "${HOST_OS}" == "DARWIN" ]; then
+ GPG_PROXY_MOUNT=(--mount "type=volume,src=gpgagent,dst=/home/${USER}/.gnupg/")
+ echo "Setting up GPG agent proxy container needed on OS X."
+ echo " we should clean this up for you. If that fails the container ID is below and in " \
+ "gpg-proxy.cid"
+ #TODO the key pair used should be configurable
+ docker run --rm -p 62222:22 \
+ --detach --cidfile "${WORKDIR}/gpg-proxy.cid" \
+ --mount \
+ "type=bind,src=${HOME}/.ssh/id_rsa.pub,dst=/home/${USER}/.ssh/authorized_keys,readonly" \
+ "${GPG_PROXY_MOUNT[@]}" \
+ "org.apache.hbase/gpg-agent-proxy:${IMGTAG}"
+ echo "Launching ssh reverse tunnel from the container to gpg agent."
+ echo " we should clean this up for you. If that fails the PID is in gpg-proxy.ssh.pid"
+ ssh -p 62222 -R "/home/${USER}/.gnupg/S.gpg-agent:$(gpgconf --list-dir agent-extra-socket)" \
+ -i "${HOME}/.ssh/id_rsa" -N -n localhost &
+ echo $! > "${WORKDIR}/gpg-proxy.ssh.pid"
+else
+ # TODO this presumes we are still trying to make a local gpg-agent available to the container.
+ # add an option so that we can run the buid on a remote machine and get the forwarded
+ # gpg-agent in the container. Should look like the side-car container mount above.
+ # it is important not to do that for a local linux agent because we only want the container
+ # to get access to the restricted extra socket on our local gpg-agent.
+ GPG_PROXY_MOUNT=(--mount \
+ "type=bind,src=$(gpgconf --list-dir agent-extra-socket),dst=/home/${USER}/.gnupg/S.gpg-agent")
Review comment:
I updated this note after working out instructions for the README so that we can treat forwarding from a VM to docker the same as forwarding a local linux run to docker.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-639195437
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch |
| +1 :green_heart: | hadolint | 0m 1s | There were no new hadolint issues. |
| +1 :green_heart: | shellcheck | 0m 4s | There were no new shellcheck issues. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
||| _ Other Tests _ |
| +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? |
| | | 2m 14s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/8/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | dupname asflicense shellcheck shelldocs hadolint |
| uname | Linux 8bad3f4b484b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b2ec4c1ea0 |
| Max. process+thread count | 47 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/8/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 hadolint=1.17.5-0-g443423c |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-648639086
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 37s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 1m 46s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/10/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | |
| uname | Linux 74f5e59a6074 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b556343292 |
| Max. process+thread count | 45 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/10/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
ndimiduk commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r439096517
##########
File path: dev-support/create-release/mac-sshd-gpg-agent/Dockerfile
##########
@@ -0,0 +1,100 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Image for use on Mac boxes to get a gpg agent socket available
Review comment:
This is unique to mac as the host os? What about a linux host?
##########
File path: dev-support/create-release/README.txt
##########
@@ -37,15 +47,53 @@ $ sudo add-apt-repository -y \
stable"
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
-$ sudo usermod -a -G docker $USERID
+# Follow the post installation steps: https://docs.docker.com/engine/install/linux-postinstall/
+$ sudo usermod -aG docker $USER
# LOGOUT and then LOGIN again so $USERID shows as part of docker group
-# Copy up private key for $USERID export from laptop and import on gce.
-$ gpg --import stack.duboce.net.asc
-$ export GPG_TTY=$(tty) # https://github.com/keybase/keybase-issues/issues/2798
-$ eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400)
-$ export PROJECT="${PROJECT:-hbase}"
-$ git clone https://github.com/apache/${PROJECT}.git
-$ cd "${PROJECT}"
+# Test here by running docker's hello world as your build user
+$ docker run hello-world
+
+# Follow the GPG guide for forwarding your gpg-agent from your local machine to the VM
+# https://wiki.gnupg.org/AgentForwarding
+# On the VM find out the location of the gpg agent socket and extra socket
+$ gpgconf --list-dir agent-socket
+/run/user/1000/gnupg/S.gpg-agent
+$ gpgconf --list-dir agent-extra-socket
+/run/user/1000/gnupg/S.gpg-agent.extra
+# On the VM configure sshd to remove stale sockets
+$ sudo bash -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config'
+$ sudo systemctl restart ssh
+# logout of the VM
+
+# Do these steps on your local machine.
+# Export your public key and copy it to the VM.
+# Assuming 'example.gce.host' maps to your VM's external IP (or use the IP)
+$ gpg --export example@apache.org > ~/gpg.example.apache.pub
+$ scp ~/gpg.example.apache.pub example.gce.host:
+# ssh into the VM while forwarding the remote gpg socket locations found above to your local
+# gpg-agent's extra socket (this will restrict what commands the remote node is allowed to have
+# your agent handle. Note that the gpg guide above can help you set this up in your ssh config
+# rather than typing it in ssh like this every time.
+$ ssh -i ~/.ssh/my_id \
+ -R "/run/user/1000/gnupg/S.gpg-agent:$(gpgconf --list-dir agent-extra-socket)" \
+ -R "/run/user/1000/gnupg/S.gpg-agent.extra:$(gpgconf --list-dir agent-extra-socket)" \
+ example.gce.host
+
+# now in an SSH session on the VM with the socket forwarding
+# import your public key and test signing with the forwarding to your local agent.
+$ gpg --no-autostart --import gpg.example.apache.pub
+$ echo "foo" > foo.txt
+$ gpg --no-autostart --detach --armor --sign foo.txt
+$ gpg --no-autostart --verify foo.txt.asc
+
+# install git and clone the main project on the build machine
+$ sudo apt-get install -y git
+$ git clone https://github.com/apache/hbase.git
Review comment:
nit: should we point to gitbox instead?
##########
File path: dev-support/create-release/release-util.sh
##########
@@ -381,8 +383,6 @@ function configure_maven {
<password>${env.ASF_PASSWORD}</password></server>
<server><id>apache.releases.https</id><username>${env.ASF_USERNAME}</username>
<password>${env.ASF_PASSWORD}</password></server>
- <server><id>gpg.passphrase</id>
- <passphrase>${env.GPG_PASSPHRASE}</passphrase></server>
Review comment:
👍
##########
File path: dev-support/create-release/README.txt
##########
@@ -37,15 +47,53 @@ $ sudo add-apt-repository -y \
stable"
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
-$ sudo usermod -a -G docker $USERID
+# Follow the post installation steps: https://docs.docker.com/engine/install/linux-postinstall/
+$ sudo usermod -aG docker $USER
# LOGOUT and then LOGIN again so $USERID shows as part of docker group
-# Copy up private key for $USERID export from laptop and import on gce.
-$ gpg --import stack.duboce.net.asc
-$ export GPG_TTY=$(tty) # https://github.com/keybase/keybase-issues/issues/2798
-$ eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400)
-$ export PROJECT="${PROJECT:-hbase}"
-$ git clone https://github.com/apache/${PROJECT}.git
-$ cd "${PROJECT}"
+# Test here by running docker's hello world as your build user
+$ docker run hello-world
+
+# Follow the GPG guide for forwarding your gpg-agent from your local machine to the VM
+# https://wiki.gnupg.org/AgentForwarding
+# On the VM find out the location of the gpg agent socket and extra socket
+$ gpgconf --list-dir agent-socket
+/run/user/1000/gnupg/S.gpg-agent
+$ gpgconf --list-dir agent-extra-socket
+/run/user/1000/gnupg/S.gpg-agent.extra
+# On the VM configure sshd to remove stale sockets
+$ sudo bash -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config'
+$ sudo systemctl restart ssh
+# logout of the VM
+
+# Do these steps on your local machine.
+# Export your public key and copy it to the VM.
+# Assuming 'example.gce.host' maps to your VM's external IP (or use the IP)
+$ gpg --export example@apache.org > ~/gpg.example.apache.pub
+$ scp ~/gpg.example.apache.pub example.gce.host:
+# ssh into the VM while forwarding the remote gpg socket locations found above to your local
+# gpg-agent's extra socket (this will restrict what commands the remote node is allowed to have
+# your agent handle. Note that the gpg guide above can help you set this up in your ssh config
+# rather than typing it in ssh like this every time.
+$ ssh -i ~/.ssh/my_id \
+ -R "/run/user/1000/gnupg/S.gpg-agent:$(gpgconf --list-dir agent-extra-socket)" \
+ -R "/run/user/1000/gnupg/S.gpg-agent.extra:$(gpgconf --list-dir agent-extra-socket)" \
+ example.gce.host
+
+# now in an SSH session on the VM with the socket forwarding
+# import your public key and test signing with the forwarding to your local agent.
+$ gpg --no-autostart --import gpg.example.apache.pub
+$ echo "foo" > foo.txt
+$ gpg --no-autostart --detach --armor --sign foo.txt
+$ gpg --no-autostart --verify foo.txt.asc
+
+# install git and clone the main project on the build machine
+$ sudo apt-get install -y git
+$ git clone https://github.com/apache/hbase.git
+# finally set up an output folder and launch a dry run.
$ mkdir ~/build
-$ ./dev-resources/create-release/do-release-docker.sh -d ~/build
-# etc.
+$ cd hbase
+$ ./dev-support/create-release/do-release-docker.sh -d ~/build
+
+# for building the main repo specifically you can save an extra download by pointing the build
+# to the local clone you just made
+$ ./dev-support/create-release/do-release-docker.sh -d ~/build -r .git
Review comment:
nice! should this be the default ?
##########
File path: dev-support/create-release/release-util.sh
##########
@@ -436,6 +436,7 @@ function git_clone_overwrite {
}
# Writes report into cwd!
+# TODO should have option for maintenance release that include LimitedPrivate in report
Review comment:
👍 file an issue?
##########
File path: dev-support/create-release/README.txt
##########
@@ -37,15 +47,53 @@ $ sudo add-apt-repository -y \
stable"
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
-$ sudo usermod -a -G docker $USERID
+# Follow the post installation steps: https://docs.docker.com/engine/install/linux-postinstall/
+$ sudo usermod -aG docker $USER
# LOGOUT and then LOGIN again so $USERID shows as part of docker group
-# Copy up private key for $USERID export from laptop and import on gce.
-$ gpg --import stack.duboce.net.asc
-$ export GPG_TTY=$(tty) # https://github.com/keybase/keybase-issues/issues/2798
-$ eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400)
-$ export PROJECT="${PROJECT:-hbase}"
-$ git clone https://github.com/apache/${PROJECT}.git
-$ cd "${PROJECT}"
+# Test here by running docker's hello world as your build user
+$ docker run hello-world
+
+# Follow the GPG guide for forwarding your gpg-agent from your local machine to the VM
+# https://wiki.gnupg.org/AgentForwarding
+# On the VM find out the location of the gpg agent socket and extra socket
+$ gpgconf --list-dir agent-socket
+/run/user/1000/gnupg/S.gpg-agent
+$ gpgconf --list-dir agent-extra-socket
+/run/user/1000/gnupg/S.gpg-agent.extra
+# On the VM configure sshd to remove stale sockets
+$ sudo bash -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config'
+$ sudo systemctl restart ssh
+# logout of the VM
+
+# Do these steps on your local machine.
+# Export your public key and copy it to the VM.
+# Assuming 'example.gce.host' maps to your VM's external IP (or use the IP)
+$ gpg --export example@apache.org > ~/gpg.example.apache.pub
Review comment:
really helpful docs here, thank you.
##########
File path: dev-support/create-release/do-release-docker.sh
##########
@@ -162,15 +219,15 @@ GIT_NAME=$GIT_NAME
GIT_EMAIL=$GIT_EMAIL
GPG_KEY=$GPG_KEY
ASF_PASSWORD=$ASF_PASSWORD
-GPG_PASSPHRASE=$GPG_PASSPHRASE
RELEASE_STEP=$RELEASE_STEP
API_DIFF_TAG=$API_DIFF_TAG
+HOST_OS=$HOST_OS
EOF
-JAVA_VOL=()
+JAVA_MOUNT=()
if [ -n "$JAVA" ]; then
echo "JAVA_HOME=/opt/hbase-java" >> "$ENVFILE"
- JAVA_VOL=(--volume "$JAVA:/opt/hbase-java")
+ JAVA_MOUNT=(--mount "type=bind,src=${JAVA},dst=/opt/hbase-java,readonly")
Review comment:
do we actually need java in the host environment? why not install it in the docker image?
How is it the mounted java version is compatible with the docker environment? If you're running with Mac as the host and the docker image is linux...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-633002302
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | shelldocs | 0m 1s | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch |
| +1 :green_heart: | hadolint | 0m 1s | There were no new hadolint issues. |
| +1 :green_heart: | shellcheck | 0m 4s | There were no new shellcheck issues. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
||| _ Other Tests _ |
| +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? |
| | | 2m 11s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/7/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | dupname asflicense shellcheck shelldocs hadolint |
| uname | Linux f6f88f8d60c0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a9fefd7f53 |
| Max. process+thread count | 48 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/7/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 hadolint=1.17.5-0-g443423c |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-639194124
force push just rebased to current master and took care of a conflict.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-625465596
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ndimiduk edited a comment on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
ndimiduk edited a comment on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-644993808
> Similarly, regarding the failure without `-r`, I don't have insights. The code in release-utils.sh:git_clone_overwrite() is supposed to url-encode the ASF_PASSWORD, and it was inherited from prior code that worked in-situ before PR #1725 . See https://github.pie.apple.com/IPR/apache-hbase/blob/8ff8e70edfbb50967cf123e6afdd9cc5dcfa4878/dev-support/create-release/release-build.sh#L116-L119
I tracked this down to a character that the library helpfully does not encode by default. See PR#1907.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r435642013
##########
File path: dev-support/create-release/README.txt
##########
@@ -17,13 +17,23 @@ anomalies are explained up in JIRA.
See http://hbase.apache.org/book.html#maven.release
+Before starting an RC build, make sure your local gpg-agent has configs
+to properly handle your credentials, especially if you want to avoid
+typing the passphrase to your secret key.
+
+e.g. if you are going to run and step away, best to increase the TTL
+on caching the unlocked secret via ~/.gnupg/gpg-agent.conf
+ # in seconds, e.g. a day
+ default-cache-ttl 86400
+ max-cache-ttl 86400
+
Running a build on GCE is easy enough. Here are some notes if of use.
Create an instance. 4CPU/15G/10G disk seems to work well enough.
Once up, run the below to make your machine fit for RC building:
-# Presuming debian-compatible OS
-$ sudo apt-get install -y git openjdk-8-jdk maven gnupg gnupg-agent
-# Install docker
+# Presuming debian-compatible OS, do these steps on the VM
+# your VM username should be your ASF id, because it will show up in build artifacts.
+# Follow the docker install guide: https://docs.docker.com/engine/install/debian/
Review comment:
sorry, I meant a non-docker release build. long day :p
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-622231594
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 1m 54s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | |
| uname | Linux c2369aeb012c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 512d00e75d |
| Max. process+thread count | 52 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
ndimiduk commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-644248138
I was able to use this patch over the weekend to build an rc as dry run. all the gpg passthrough worked well for me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-648640031
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 16s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 3m 25s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/10/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | |
| uname | Linux 24983de37e6a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b556343292 |
| Max. process+thread count | 48 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/10/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-622231649
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 40s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 2m 7s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | |
| uname | Linux 31c4b83de04f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 512d00e75d |
| Max. process+thread count | 54 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
busbey commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-629884049
sorry for the lag here. missed that I had comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-633096717
Mind taking a look @mattf-apache?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-645125135
> error: object directory /home/vagrant/repos/hbase/.git/objects does not exist; check .git/objects/info/alternates.
the object directory messages are a side effect of using the shared objects. It shouldn't be listed as an error since git then immediately checks the alternates we provide and finds what it needs.
> Parenthetically, I am confused by the -r .git idiom. If I understand the GIT_REPO usage properly, it needs to point at an actual repository, while .git is the usual name of the git account metadata directory. Why would I point -r at it? The processing code doesn't shed any light for me.
the `.git` metadata directory is itself a repository, albeit one only on the local system. using it for a release candidate means the tag generated will be in that local repository. If the RM wants the tag published somewhere else then they need to push it to the other remote.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r418643409
##########
File path: dev-support/create-release/do-release-docker.sh
##########
@@ -102,12 +102,26 @@ if [ -d "$WORKDIR/output" ]; then
fi
fi
+if [ -f "${WORKDIR}/gpg-proxy.ssh.pid" ] || \
+ [ -f "${WORKDIR}/gpg-proxy.cid" ] || \
+ [ -f "${WORKDIR}/release.cid" ]; then
+ read -r -p "container/pid files from prior run exists. Overwrite and continue? [y/n] " ANSWER
+ if [ "$ANSWER" != "y" ]; then
+ error "Exiting."
+ fi
+fi
+
cd "$WORKDIR"
rm -rf "$WORKDIR/output"
+rm -rf "${WORKDIR}/gpg-proxy.ssh.pid" "${WORKDIR}/gpg-proxy.cid" "${WORKDIR}/release.cid"
Review comment:
Move to a trap/signal handler? Follow-on.
##########
File path: dev-support/create-release/do-release-docker.sh
##########
@@ -102,12 +102,26 @@ if [ -d "$WORKDIR/output" ]; then
fi
fi
+if [ -f "${WORKDIR}/gpg-proxy.ssh.pid" ] || \
+ [ -f "${WORKDIR}/gpg-proxy.cid" ] || \
+ [ -f "${WORKDIR}/release.cid" ]; then
+ read -r -p "container/pid files from prior run exists. Overwrite and continue? [y/n] " ANSWER
+ if [ "$ANSWER" != "y" ]; then
+ error "Exiting."
+ fi
+fi
+
cd "$WORKDIR"
rm -rf "$WORKDIR/output"
+rm -rf "${WORKDIR}/gpg-proxy.ssh.pid" "${WORKDIR}/gpg-proxy.cid" "${WORKDIR}/release.cid"
Review comment:
All these clean ups should be in a cleanup handler... Not your issue.
##########
File path: dev-support/create-release/do-release-docker.sh
##########
@@ -102,12 +102,26 @@ if [ -d "$WORKDIR/output" ]; then
fi
fi
+if [ -f "${WORKDIR}/gpg-proxy.ssh.pid" ] || \
+ [ -f "${WORKDIR}/gpg-proxy.cid" ] || \
+ [ -f "${WORKDIR}/release.cid" ]; then
+ read -r -p "container/pid files from prior run exists. Overwrite and continue? [y/n] " ANSWER
+ if [ "$ANSWER" != "y" ]; then
+ error "Exiting."
+ fi
+fi
+
cd "$WORKDIR"
rm -rf "$WORKDIR/output"
+rm -rf "${WORKDIR}/gpg-proxy.ssh.pid" "${WORKDIR}/gpg-proxy.cid" "${WORKDIR}/release.cid"
Review comment:
Hows this relate to the below nice cleanup function?
##########
File path: dev-support/create-release/do-release-docker.sh
##########
@@ -151,21 +200,53 @@ GIT_NAME=$GIT_NAME
GIT_EMAIL=$GIT_EMAIL
GPG_KEY=$GPG_KEY
ASF_PASSWORD=$ASF_PASSWORD
-GPG_PASSPHRASE=$GPG_PASSPHRASE
RELEASE_STEP=$RELEASE_STEP
RELEASE_STEP=$RELEASE_STEP
API_DIFF_TAG=$API_DIFF_TAG
EOF
-JAVA_VOL=
+JAVA_MOUNT=()
if [ -n "$JAVA" ]; then
echo "JAVA_HOME=/opt/hbase-java" >> "$ENVFILE"
- JAVA_VOL="--volume $JAVA:/opt/hbase-java"
+ JAVA_MOUNT=(--mount "type=bind,src=${JAVA},dest=/opt/hbase-java,readonly")
+fi
+
+GPG_PROXY_MOUNT=()
+if [ "${HOST_OS}" == "DARWIN" ]; then
+ GPG_PROXY_MOUNT=(--mount "type=volume,src=gpgagent,dst=/home/${USER}/.gnupg/")
+ echo "Setting up GPG agent proxy container needed on OS X."
+ echo " we should clean this up for you. If that fails the container ID is below and in " \
+ "gpg-proxy.cid"
+ #TODO the key pair used should be configurable
+ docker run --rm -p 62222:22 \
+ --detach --cidfile "${WORKDIR}/gpg-proxy.cid" \
+ --mount \
+ "type=bind,src=${HOME}/.ssh/id_rsa.pub,dst=/home/${USER}/.ssh/authorized_keys,readonly" \
+ "${GPG_PROXY_MOUNT[@]}" \
+ "org.apache.hbase/gpg-agent-proxy:${IMGTAG}"
+ echo "Launching ssh reverse tunnel from the container to gpg agent."
+ echo " we should clean this up for you. If that fails the PID is in gpg-proxy.ssh.pid"
+ ssh -p 62222 -R "/home/${USER}/.gnupg/S.gpg-agent:$(gpgconf --list-dir agent-extra-socket)" \
+ -i "${HOME}/.ssh/id_rsa" -N -n localhost &
+ echo $! > "${WORKDIR}/gpg-proxy.ssh.pid"
+else
+ # TODO this presumes we are still trying to make a local gpg-agent available to the container.
+ # add an option so that we can run the buid on a remote machine and get the forwarded
+ # gpg-agent in the container. Should look like the side-car container mount above.
+ # it is important not to do that for a local linux agent because we only want the container
+ # to get access to the restricted extra socket on our local gpg-agent.
+ GPG_PROXY_MOUNT=(--mount \
+ "type=bind,src=$(gpgconf --list-dir agent-extra-socket),dst=/home/${USER}/.gnupg/S.gpg-agent")
Review comment:
my head hurts
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -79,17 +78,7 @@ if [[ -z "$ASF_PASSWORD" ]]; then
stty -echo && printf "ASF password: " && read ASF_PASSWORD && printf '\n' && stty echo
fi
-# Read in the GPG passphrase
-if [[ -z "$GPG_PASSPHRASE" ]]; then
- echo 'The environment variable GPG_PASSPHRASE is not set. Enter the passphrase to'
- echo 'unlock the GPG signing key that will be used to sign the release!'
- echo
- stty -echo && printf "GPG passphrase: " && read GPG_PASSPHRASE && printf '\n' && stty echo
- export GPG_PASSPHRASE
- export GPG_TTY=$(tty)
-fi
-
-for env in ASF_USERNAME GPG_PASSPHRASE GPG_KEY; do
Review comment:
This is gone because require agent?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-629727114
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch |
| +1 :green_heart: | hadolint | 0m 3s | There were no new hadolint issues. |
| -0 :warning: | shellcheck | 0m 2s | The patch generated 6 new + 5 unchanged - 0 fixed = 11 total (was 5) |
| -0 :warning: | whitespace | 0m 0s | The patch 6 line(s) with tabs. |
||| _ Other Tests _ |
| +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? |
| | | 2m 0s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | dupname asflicense shellcheck shelldocs hadolint |
| uname | Linux f1a0a30f2407 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 15627bb722 |
| shellcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/4/artifact/yetus-general-check/output/diff-patch-shellcheck.txt |
| whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/4/artifact/yetus-general-check/output/whitespace-tabs.txt |
| Max. process+thread count | 47 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 hadolint=1.17.5-0-g443423c |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-639195819
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 24s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 3m 46s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/8/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | |
| uname | Linux 3fc0b19a8d4a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b2ec4c1ea0 |
| Max. process+thread count | 49 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/8/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-644975779
Similarly, regarding the failure without `-r`, I don't have insights. The code in release-utils.sh:git_clone_overwrite() is supposed to url-encode the ASF_PASSWORD, and it was inherited from prior code that worked in-situ before PR #1725 . See https://github.pie.apple.com/IPR/apache-hbase/blob/8ff8e70edfbb50967cf123e6afdd9cc5dcfa4878/dev-support/create-release/release-build.sh#L116-L119
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-634270546
Looking...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r426321691
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -79,17 +78,7 @@ if [[ -z "$ASF_PASSWORD" ]]; then
stty -echo && printf "ASF password: " && read ASF_PASSWORD && printf '\n' && stty echo
fi
-# Read in the GPG passphrase
-if [[ -z "$GPG_PASSPHRASE" ]]; then
- echo 'The environment variable GPG_PASSPHRASE is not set. Enter the passphrase to'
- echo 'unlock the GPG signing key that will be used to sign the release!'
- echo
- stty -echo && printf "GPG passphrase: " && read GPG_PASSPHRASE && printf '\n' && stty echo
- export GPG_PASSPHRASE
- export GPG_TTY=$(tty)
-fi
-
-for env in ASF_USERNAME GPG_PASSPHRASE GPG_KEY; do
Review comment:
yep! no more GPG_PASSPHRASE because that should get specified by you to your local gpg-agent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
busbey commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-629885166
shellcheck and whitespace complaints should be easy enough to clean up. will get to it after I get PRs up for all the commits on the branch that are not this particular jira.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r435642448
##########
File path: dev-support/create-release/README.txt
##########
@@ -17,13 +17,23 @@ anomalies are explained up in JIRA.
See http://hbase.apache.org/book.html#maven.release
+Before starting an RC build, make sure your local gpg-agent has configs
+to properly handle your credentials, especially if you want to avoid
+typing the passphrase to your secret key.
+
+e.g. if you are going to run and step away, best to increase the TTL
+on caching the unlocked secret via ~/.gnupg/gpg-agent.conf
+ # in seconds, e.g. a day
+ default-cache-ttl 86400
+ max-cache-ttl 86400
+
Running a build on GCE is easy enough. Here are some notes if of use.
Create an instance. 4CPU/15G/10G disk seems to work well enough.
Once up, run the below to make your machine fit for RC building:
-# Presuming debian-compatible OS
-$ sudo apt-get install -y git openjdk-8-jdk maven gnupg gnupg-agent
-# Install docker
+# Presuming debian-compatible OS, do these steps on the VM
+# your VM username should be your ASF id, because it will show up in build artifacts.
+# Follow the docker install guide: https://docs.docker.com/engine/install/debian/
Review comment:
I am reasonably certain we provide useful error messages if you are missing build requirements when doing a non-docker release build. I expect probably we only give one error at a time though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-632983446
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch |
| +1 :green_heart: | hadolint | 0m 1s | There were no new hadolint issues. |
| +1 :green_heart: | shellcheck | 0m 4s | There were no new shellcheck issues. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
||| _ Other Tests _ |
| +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? |
| | | 2m 4s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/6/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | dupname asflicense shellcheck shelldocs hadolint |
| uname | Linux 64676be4b319 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a9fefd7f53 |
| Max. process+thread count | 48 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/6/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 hadolint=1.17.5-0-g443423c |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r439795903
##########
File path: dev-support/create-release/README.txt
##########
@@ -37,15 +47,53 @@ $ sudo add-apt-repository -y \
stable"
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
-$ sudo usermod -a -G docker $USERID
+# Follow the post installation steps: https://docs.docker.com/engine/install/linux-postinstall/
+$ sudo usermod -aG docker $USER
# LOGOUT and then LOGIN again so $USERID shows as part of docker group
-# Copy up private key for $USERID export from laptop and import on gce.
-$ gpg --import stack.duboce.net.asc
-$ export GPG_TTY=$(tty) # https://github.com/keybase/keybase-issues/issues/2798
-$ eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400)
-$ export PROJECT="${PROJECT:-hbase}"
-$ git clone https://github.com/apache/${PROJECT}.git
-$ cd "${PROJECT}"
+# Test here by running docker's hello world as your build user
+$ docker run hello-world
+
+# Follow the GPG guide for forwarding your gpg-agent from your local machine to the VM
+# https://wiki.gnupg.org/AgentForwarding
+# On the VM find out the location of the gpg agent socket and extra socket
+$ gpgconf --list-dir agent-socket
+/run/user/1000/gnupg/S.gpg-agent
+$ gpgconf --list-dir agent-extra-socket
+/run/user/1000/gnupg/S.gpg-agent.extra
+# On the VM configure sshd to remove stale sockets
+$ sudo bash -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config'
+$ sudo systemctl restart ssh
+# logout of the VM
+
+# Do these steps on your local machine.
+# Export your public key and copy it to the VM.
+# Assuming 'example.gce.host' maps to your VM's external IP (or use the IP)
+$ gpg --export example@apache.org > ~/gpg.example.apache.pub
+$ scp ~/gpg.example.apache.pub example.gce.host:
+# ssh into the VM while forwarding the remote gpg socket locations found above to your local
+# gpg-agent's extra socket (this will restrict what commands the remote node is allowed to have
+# your agent handle. Note that the gpg guide above can help you set this up in your ssh config
+# rather than typing it in ssh like this every time.
+$ ssh -i ~/.ssh/my_id \
+ -R "/run/user/1000/gnupg/S.gpg-agent:$(gpgconf --list-dir agent-extra-socket)" \
+ -R "/run/user/1000/gnupg/S.gpg-agent.extra:$(gpgconf --list-dir agent-extra-socket)" \
+ example.gce.host
Review comment:
yes your gpg agent has to be running on your local machine before hand. The remote invocations of gpg have to use `--no-autostart` because if the gpg agent starts on that machine it'll overwrite the socket that we forwarded.
I could add an example of forcing a clean start of the gpg-agent, e.g. `gpgconf --kill all && gpg-connect-agent /bye`
The test signature at container launch should ensure the key is unlocked prior to us building artifacts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r439580182
##########
File path: dev-support/create-release/README.txt
##########
@@ -37,15 +47,53 @@ $ sudo add-apt-repository -y \
stable"
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
-$ sudo usermod -a -G docker $USERID
+# Follow the post installation steps: https://docs.docker.com/engine/install/linux-postinstall/
+$ sudo usermod -aG docker $USER
# LOGOUT and then LOGIN again so $USERID shows as part of docker group
-# Copy up private key for $USERID export from laptop and import on gce.
-$ gpg --import stack.duboce.net.asc
-$ export GPG_TTY=$(tty) # https://github.com/keybase/keybase-issues/issues/2798
-$ eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400)
-$ export PROJECT="${PROJECT:-hbase}"
-$ git clone https://github.com/apache/${PROJECT}.git
-$ cd "${PROJECT}"
+# Test here by running docker's hello world as your build user
+$ docker run hello-world
+
+# Follow the GPG guide for forwarding your gpg-agent from your local machine to the VM
+# https://wiki.gnupg.org/AgentForwarding
+# On the VM find out the location of the gpg agent socket and extra socket
+$ gpgconf --list-dir agent-socket
+/run/user/1000/gnupg/S.gpg-agent
+$ gpgconf --list-dir agent-extra-socket
+/run/user/1000/gnupg/S.gpg-agent.extra
+# On the VM configure sshd to remove stale sockets
+$ sudo bash -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config'
+$ sudo systemctl restart ssh
+# logout of the VM
+
+# Do these steps on your local machine.
+# Export your public key and copy it to the VM.
+# Assuming 'example.gce.host' maps to your VM's external IP (or use the IP)
+$ gpg --export example@apache.org > ~/gpg.example.apache.pub
Review comment:
Yes, brilliant!
##########
File path: dev-support/create-release/README.txt
##########
@@ -17,13 +17,23 @@ anomalies are explained up in JIRA.
See http://hbase.apache.org/book.html#maven.release
+Before starting an RC build, make sure your local gpg-agent has configs
+to properly handle your credentials, especially if you want to avoid
+typing the passphrase to your secret key.
+
+e.g. if you are going to run and step away, best to increase the TTL
+on caching the unlocked secret via ~/.gnupg/gpg-agent.conf
+ # in seconds, e.g. a day
+ default-cache-ttl 86400
+ max-cache-ttl 86400
+
Running a build on GCE is easy enough. Here are some notes if of use.
Create an instance. 4CPU/15G/10G disk seems to work well enough.
Once up, run the below to make your machine fit for RC building:
-# Presuming debian-compatible OS
-$ sudo apt-get install -y git openjdk-8-jdk maven gnupg gnupg-agent
Review comment:
Nice minimization. The JDK is indeed unneeded on the host, unless the user for some reason chooses to use the `-j` option. And maven is not needed.
##########
File path: dev-support/create-release/do-release.sh
##########
@@ -42,27 +43,41 @@ fi
# If running in docker, import and then cache keys.
if [ "$RUNNING_IN_DOCKER" = "1" ]; then
- # Run gpg agent.
- eval "$(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase \
- --default-cache-ttl=86400 --max-cache-ttl=86400)"
- echo "GPG Version: $(gpg --version)"
- # Inside docker, need to import the GPG keyfile stored in the current directory.
- # (On workstation, assume GPG has access to keychain/cache with key_id already imported.)
- echo "$GPG_PASSPHRASE" | $GPG --passphrase-fd 0 --import "$SELF/gpg.key"
+ # when Docker Desktop for mac is running under load there is a delay before the mounted volume
+ # becomes available. if we do not pause then we may try to use the gpg-agent socket before docker
+ # has got it ready and we will not think there is a gpg-agent.
+ if [ "${HOST_OS}" == "DARWIN" ]; then
+ sleep 5
+ fi
+ # in docker our working dir is set to where all of our scripts are held
+ # and we want default output to go into the "output" directory that should be in there.
+ if [ -d "output" ]; then
+ cd output
+ fi
+ echo "GPG Version: $("${GPG}" "${GPG_ARGS[@]}" --version)"
+ # Inside docker, need to import the GPG key stored in the current directory.
+ $GPG "${GPG_ARGS[@]}" --import "$SELF/gpg.key.public"
# We may need to adjust the path since JAVA_HOME may be overridden by the driver script.
if [ -n "$JAVA_HOME" ]; then
+ echo "Using JAVA_HOME from host."
export PATH="$JAVA_HOME/bin:$PATH"
else
# JAVA_HOME for the openjdk package.
- export JAVA_HOME=/usr
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
fi
else
# Outside docker, need to ask for information about the release.
get_release_info
fi
+
GPG_TTY="$(tty)"
export GPG_TTY
+echo "Testing gpg signing."
+echo "foo" > gpg_test.txt
+"${GPG}" "${GPG_ARGS[@]}" --detach --armor --sign gpg_test.txt
+# In --batch mode we have to be explicit about what we are verifying
+"${GPG}" "${GPG_ARGS[@]}" --verify gpg_test.txt.asc gpg_test.txt
Review comment:
Good.
##########
File path: dev-support/create-release/release-util.sh
##########
@@ -381,8 +383,6 @@ function configure_maven {
<password>${env.ASF_PASSWORD}</password></server>
<server><id>apache.releases.https</id><username>${env.ASF_USERNAME}</username>
<password>${env.ASF_PASSWORD}</password></server>
- <server><id>gpg.passphrase</id>
- <passphrase>${env.GPG_PASSPHRASE}</passphrase></server>
Review comment:
@ndimiduk , do note this was NOT writing the passphrase literally into the maven settings.xml file.
That got taken care of already; it is just a reference to the environment variable which held the passphrase.
##########
File path: dev-support/create-release/README.txt
##########
@@ -37,15 +47,53 @@ $ sudo add-apt-repository -y \
stable"
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
-$ sudo usermod -a -G docker $USERID
+# Follow the post installation steps: https://docs.docker.com/engine/install/linux-postinstall/
+$ sudo usermod -aG docker $USER
# LOGOUT and then LOGIN again so $USERID shows as part of docker group
-# Copy up private key for $USERID export from laptop and import on gce.
-$ gpg --import stack.duboce.net.asc
-$ export GPG_TTY=$(tty) # https://github.com/keybase/keybase-issues/issues/2798
-$ eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400)
-$ export PROJECT="${PROJECT:-hbase}"
-$ git clone https://github.com/apache/${PROJECT}.git
-$ cd "${PROJECT}"
+# Test here by running docker's hello world as your build user
+$ docker run hello-world
+
+# Follow the GPG guide for forwarding your gpg-agent from your local machine to the VM
+# https://wiki.gnupg.org/AgentForwarding
+# On the VM find out the location of the gpg agent socket and extra socket
+$ gpgconf --list-dir agent-socket
+/run/user/1000/gnupg/S.gpg-agent
+$ gpgconf --list-dir agent-extra-socket
+/run/user/1000/gnupg/S.gpg-agent.extra
+# On the VM configure sshd to remove stale sockets
+$ sudo bash -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config'
+$ sudo systemctl restart ssh
+# logout of the VM
+
+# Do these steps on your local machine.
+# Export your public key and copy it to the VM.
+# Assuming 'example.gce.host' maps to your VM's external IP (or use the IP)
+$ gpg --export example@apache.org > ~/gpg.example.apache.pub
+$ scp ~/gpg.example.apache.pub example.gce.host:
+# ssh into the VM while forwarding the remote gpg socket locations found above to your local
+# gpg-agent's extra socket (this will restrict what commands the remote node is allowed to have
+# your agent handle. Note that the gpg guide above can help you set this up in your ssh config
+# rather than typing it in ssh like this every time.
+$ ssh -i ~/.ssh/my_id \
+ -R "/run/user/1000/gnupg/S.gpg-agent:$(gpgconf --list-dir agent-extra-socket)" \
+ -R "/run/user/1000/gnupg/S.gpg-agent.extra:$(gpgconf --list-dir agent-extra-socket)" \
+ example.gce.host
Review comment:
@busbey , I'm not sure this is an issue, I'm just going by the docs. However:
Does this work correctly if gpg-agent is not already running on my local machine, given that the remote invocation of gpg on line 84 uses `--no-autostart`?
In other words, should the instructions include something at about line 72 that will guarantee the local gpg-agent is running, such as signing a foo file locally too?
##########
File path: dev-support/create-release/do-release.sh
##########
@@ -17,6 +17,7 @@
# limitations under the License.
#
+set -e
Review comment:
No harm, but it is set for you in line 29 when release-util.sh is sourced.
Maybe it's good to have it evident here too.
##########
File path: dev-support/create-release/hbase-rm/Dockerfile
##########
@@ -50,10 +50,15 @@ RUN wget -qO- "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&fi
tar xvz -C /opt
ENV YETUS_HOME /opt/apache-yetus-${YETUS_VERSION}
-WORKDIR /opt/hbase-rm/output
-
ARG UID
-RUN useradd -m -s /bin/bash -p hbase-rm -u $UID hbase-rm
-USER hbase-rm:hbase-rm
+ARG RM_USER
+RUN groupadd hbase-rm && \
+ useradd --create-home --shell /bin/bash -p hbase-rm -u $UID $RM_USER && \
+ mkdir /home/$RM_USER/.gnupg && \
+ chown -R $RM_USER:hbase-rm /home/$RM_USER && \
+ chmod -R 700 /home/$RM_USER
+
+USER $RM_USER:hbase-rm
+WORKDIR /home/$RM_USER/hbase-rm/
Review comment:
Nice improvement. Was this mandatory for the ssh tunneled gpg-agent to work?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-629861697
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 38s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch |
| +1 :green_heart: | hadolint | 0m 1s | There were no new hadolint issues. |
| -0 :warning: | shellcheck | 0m 4s | The patch generated 6 new + 5 unchanged - 0 fixed = 11 total (was 5) |
| -0 :warning: | whitespace | 0m 0s | The patch 6 line(s) with tabs. |
||| _ Other Tests _ |
| +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? |
| | | 3m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | dupname asflicense shellcheck shelldocs hadolint |
| uname | Linux 27abd214f267 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 15627bb722 |
| shellcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/artifact/yetus-general-check/output/diff-patch-shellcheck.txt |
| whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/artifact/yetus-general-check/output/whitespace-tabs.txt |
| Max. process+thread count | 52 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 hadolint=1.17.5-0-g443423c |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-647047924
Thanks for the explanation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
ndimiduk commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-644437760
@busbey @mattf-apache I'm trying to use this to create an RC for real, using `-r .git`. Does this mean anything to you?
```
$ cat ~/build-rc0/output/tag.log
...
21:57:58 [INFO] ------------------------------------------------------------------------
21:57:58 [INFO] BUILD SUCCESS
21:57:58 [INFO] ------------------------------------------------------------------------
21:57:58 [INFO] Total time: 5.609 s
21:57:58 [INFO] Finished at: 2020-06-15T21:57:58Z
21:57:58 [INFO] ------------------------------------------------------------------------
ESC[0m
+ git commit -a -m 'Preparing development version 2.3.1-SNAPSHOT'
error: object directory /home/vagrant/repos/hbase/.git/objects does not exist; check .git/objects/info/alternates.
error: object directory /home/vagrant/repos/hbase/.git/objects does not exist; check .git/objects/info/alternates.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-633002270
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 1m 45s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | |
| uname | Linux 89677f9c25a2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a9fefd7f53 |
| Max. process+thread count | 45 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/7/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r439168692
##########
File path: dev-support/create-release/do-release-docker.sh
##########
@@ -162,15 +219,15 @@ GIT_NAME=$GIT_NAME
GIT_EMAIL=$GIT_EMAIL
GPG_KEY=$GPG_KEY
ASF_PASSWORD=$ASF_PASSWORD
-GPG_PASSPHRASE=$GPG_PASSPHRASE
RELEASE_STEP=$RELEASE_STEP
API_DIFF_TAG=$API_DIFF_TAG
+HOST_OS=$HOST_OS
EOF
-JAVA_VOL=()
+JAVA_MOUNT=()
if [ -n "$JAVA" ]; then
echo "JAVA_HOME=/opt/hbase-java" >> "$ENVFILE"
- JAVA_VOL=(--volume "$JAVA:/opt/hbase-java")
+ JAVA_MOUNT=(--mount "type=bind,src=${JAVA},dst=/opt/hbase-java,readonly")
Review comment:
we do install it in the docker image, that's the default. I have only done my java builds via the docker image java. I just updated this to use the preferred cli options instead of the legacy ones. I presume it was important for someone.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-625442675
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-648636537
> > yes your gpg agent has to be running on your local machine before hand. ...
> > I could add an example of forcing a clean start of the gpg-agent, e.g. gpgconf --kill all && gpg-connect-agent /bye
> > The test signature at container launch should ensure the key is unlocked prior to us building artifacts.
>
> I recommend adding that example in the README. I think most people do not have gpg-agent running by default, may not know how to launch it (I was vague on it), and while it is true that the test signature at container launch will catch a missing agent timely, it won't tell the user how to fix the problem. -- So, maybe also have that test sig command catch failure and give an instructive error message. Please?
@mattf-apache latest commit adds in an example in the README and some error handling to point to it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache edited a comment on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
mattf-apache edited a comment on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-643814757
> If you've set up your gpg agent and unlocked your private key for it then you do not need to give the passphrase during the build. the README goes over increasing the timeout to a day instead of 2 hours.
Nice, I missed that on my earlier read-thru. Thanks.
> yes your gpg agent has to be running on your local machine before hand. ...
> I could add an example of forcing a clean start of the gpg-agent, e.g. gpgconf --kill all && gpg-connect-agent /bye
> The test signature at container launch should ensure the key is unlocked prior to us building artifacts.
I recommend adding that example in the README. I think most people do not have gpg-agent running by default, may not know how to launch it (I was vague on it), and while it is true that the test signature at container launch will catch a missing agent timely, it won't tell the user how to fix the problem. -- So, maybe also have that test sig command catch failure and give an instructive error message. Please?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-629726953
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. |
| -1 :x: | docker | 0m 0s | Docker command '/usr/bin/docker' is too old ( < 17.0). |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/4/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r435641768
##########
File path: dev-support/create-release/README.txt
##########
@@ -17,13 +17,23 @@ anomalies are explained up in JIRA.
See http://hbase.apache.org/book.html#maven.release
+Before starting an RC build, make sure your local gpg-agent has configs
+to properly handle your credentials, especially if you want to avoid
+typing the passphrase to your secret key.
+
+e.g. if you are going to run and step away, best to increase the TTL
+on caching the unlocked secret via ~/.gnupg/gpg-agent.conf
+ # in seconds, e.g. a day
+ default-cache-ttl 86400
+ max-cache-ttl 86400
+
Running a build on GCE is easy enough. Here are some notes if of use.
Create an instance. 4CPU/15G/10G disk seems to work well enough.
Once up, run the below to make your machine fit for RC building:
-# Presuming debian-compatible OS
-$ sudo apt-get install -y git openjdk-8-jdk maven gnupg gnupg-agent
-# Install docker
+# Presuming debian-compatible OS, do these steps on the VM
+# your VM username should be your ASF id, because it will show up in build artifacts.
+# Follow the docker install guide: https://docs.docker.com/engine/install/debian/
Review comment:
yes, since HBASE-24318 / #1643 you can run `do-release.sh` instead of `do-release-docker.sh`.
I believe it's not recommended because of needed build set up. I don't know if we document the steps for doing a non-release build. if we don't I'd rather do that in a different jira.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-643025027
> I'd like to try this out from a vagrant/VirtualBox vm, instead of going to GCE. thoughts?
I do not know of any reason that would not work. I only did GCE because Stack's instructions were already for GCE and I figured I would be less likely to trip on something that way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-633002289
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 36s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch |
||| _ Other Tests _ |
| | | 1m 57s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1620 |
| Optional Tests | |
| uname | Linux 20660a30492a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / a9fefd7f53 |
| Max. process+thread count | 52 (vs. ulimit of 12500) |
| modules | C: U: |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/7/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1620: HBASE-23339 Release scripts should use forwarded gpg-agent
Posted by GitBox <gi...@apache.org>.
busbey commented on pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#issuecomment-648600803
force push was just me rebasing to the current HEAD with the changes as they were. the next commit is a minor issue I found while doing non-docker builds.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org