You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "damccorm (via GitHub)" <gi...@apache.org> on 2023/05/31 14:05:49 UTC

[GitHub] [beam] damccorm opened a new pull request, #26950: [WIP] Publish Docker artifacts from build_release_candidate.yml

damccorm opened a new pull request, #26950:
URL: https://github.com/apache/beam/pull/26950

   There are 2 problems/tasks that need to be addressed before this can be published/merged:
   
   - [ ] Ask infra to add `DOCKERHUB_USERNAME` and `DOCKERHUB_TOKEN` secrets to the beam repo (hopefully we can do this for a release before removing a bunch of people from the docker group)
   - [ ] Right now, this workflow runs out of space most of the way through ([example](https://github.com/damccorm/beam/actions/runs/5122878704/jobs/9212582481)). We need a docker prune step injected before proceeding.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on pull request #26950: Publish Docker artifacts from build_release_candidate.yml

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #26950:
URL: https://github.com/apache/beam/pull/26950#issuecomment-1622247189

   From the logs:
   
   ```
   Run docker/login-action@v2
     with:
       password: ***
       ecr: auto
       logout: true
   ```
   
   looks like password is, but maybe not username? I followed up in https://issues.apache.org/jira/browse/INFRA-24750


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on pull request #26950: [WIP] Publish Docker artifacts from build_release_candidate.yml

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #26950:
URL: https://github.com/apache/beam/pull/26950#issuecomment-1614792422

   @Abacn could you try using this instead of the docker push step when you reach that part of the build_rc script? I'll update the docs once we see that its working


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on a diff in pull request #26950: Publish Docker artifacts from build_release_candidate.yml

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #26950:
URL: https://github.com/apache/beam/pull/26950#discussion_r1248009002


##########
.github/workflows/build_release_candidate.yml:
##########
@@ -130,3 +134,43 @@ jobs:
             svn add --force .
             svn status
             svn commit -m "Staging Java artifacts for Apache Beam ${{ github.event.inputs.RELEASE }} RC${{ github.event.inputs.RC }}" --non-interactive --username ${{ github.event.inputs.APACHE_ID }} --password ${{ github.event.inputs.APACHE_PASSWORD }} 
+        
+      
+  stage_docker:
+    if: ${{github.event.inputs.STAGE_DOCKER_ARTIFACTS == 'yes'}}
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+        with:
+          ref: "v${{ github.event.inputs.RELEASE }}-RC${{ github.event.inputs.RC }}"
+          repository: apache/beam
+      - name: Install Java 11
+        uses: actions/setup-java@v3
+        with:
+          distribution: 'temurin'
+          java-version: '11'
+      - name: Install Python 3.8
+        uses: actions/setup-python@v3
+        with:
+          python-version: '3.8'
+      - run: echo $JAVA_HOME
+      - run: echo "JAVA11_HOME=${JAVA_HOME}" >> "$GITHUB_OUTPUT"
+        id: export-java11
+      - name: Install Java 8
+        uses: actions/setup-java@v3
+        with:
+          distribution: 'temurin'
+          java-version: '8'
+      - name: Remove default github maven configuration
+        # This step is a workaround to avoid a decryption issue of Beam's
+        # net.linguica.gradle.maven.settings plugin and github's provided maven
+        # settings.xml file
+        run: rm ~/.m2/settings.xml

Review Comment:
   I added a comment to remove this step if we move to self-hosted and added the `|| true` in case the hosted runner setup ever changes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on pull request #26950: [WIP] Publish Docker artifacts from build_release_candidate.yml

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #26950:
URL: https://github.com/apache/beam/pull/26950#issuecomment-1614783186

   Run RAT PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26950: Publish Docker artifacts from build_release_candidate.yml

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26950:
URL: https://github.com/apache/beam/pull/26950#issuecomment-1614819467

   Thank you @damccorm sure will do


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #26950: [WIP] Publish Docker artifacts from build_release_candidate.yml

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26950:
URL: https://github.com/apache/beam/pull/26950#issuecomment-1614785610

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on pull request #26950: Publish Docker artifacts from build_release_candidate.yml

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #26950:
URL: https://github.com/apache/beam/pull/26950#issuecomment-1623616517

   Ah, that's because https://github.com/apache/beam/commit/4d22675b2bfc6488f482006b9847d053c2489227 didn't make it in to the cut. We should just use the script to run the docker push step locally in that case (it might do the same thing to your local box, so you might need to do a docker prune in the middle and take some of the tasks independently).
   
   It should be idempotent, so it is safe to just run it locally. If we have an RC2, lets cherry pick that commit in so we can verify it works.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26950: Publish Docker artifacts from build_release_candidate.yml

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26950:
URL: https://github.com/apache/beam/pull/26950#issuecomment-1622237674

   triggered: https://github.com/apache/beam/actions/runs/5467495256/jobs/9953880281
   
   failing due to "Error: Username and password required"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on pull request #26950: [WIP] Publish Docker artifacts from build_release_candidate.yml

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #26950:
URL: https://github.com/apache/beam/pull/26950#issuecomment-1614783449

   R: @Abacn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on a diff in pull request #26950: Publish Docker artifacts from build_release_candidate.yml

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #26950:
URL: https://github.com/apache/beam/pull/26950#discussion_r1248068220


##########
.github/workflows/build_release_candidate.yml:
##########
@@ -130,3 +134,43 @@ jobs:
             svn add --force .
             svn status
             svn commit -m "Staging Java artifacts for Apache Beam ${{ github.event.inputs.RELEASE }} RC${{ github.event.inputs.RC }}" --non-interactive --username ${{ github.event.inputs.APACHE_ID }} --password ${{ github.event.inputs.APACHE_PASSWORD }} 
+        
+      
+  stage_docker:
+    if: ${{github.event.inputs.STAGE_DOCKER_ARTIFACTS == 'yes'}}
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+        with:
+          ref: "v${{ github.event.inputs.RELEASE }}-RC${{ github.event.inputs.RC }}"
+          repository: apache/beam
+      - name: Install Java 11
+        uses: actions/setup-java@v3
+        with:
+          distribution: 'temurin'
+          java-version: '11'
+      - name: Install Python 3.8
+        uses: actions/setup-python@v3
+        with:
+          python-version: '3.8'
+      - run: echo $JAVA_HOME
+      - run: echo "JAVA11_HOME=${JAVA_HOME}" >> "$GITHUB_OUTPUT"
+        id: export-java11
+      - name: Install Java 8
+        uses: actions/setup-java@v3
+        with:
+          distribution: 'temurin'
+          java-version: '8'
+      - name: Remove default github maven configuration
+        # This step is a workaround to avoid a decryption issue of Beam's
+        # net.linguica.gradle.maven.settings plugin and github's provided maven
+        # settings.xml file
+        run: rm ~/.m2/settings.xml

Review Comment:
   Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26950: Publish Docker artifacts from build_release_candidate.yml

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26950:
URL: https://github.com/apache/beam/pull/26950#issuecomment-1622705195

   update: now it fails due to "No space left on device : '/home/runner/runners/2.305.0/_diag/Worker_20230705-213934-utc.log' at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 ..."
   
   example run: https://github.com/apache/beam/actions/runs/5469204932


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on a diff in pull request #26950: Publish Docker artifacts from build_release_candidate.yml

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #26950:
URL: https://github.com/apache/beam/pull/26950#discussion_r1248002449


##########
.github/workflows/build_release_candidate.yml:
##########
@@ -130,3 +134,43 @@ jobs:
             svn add --force .
             svn status
             svn commit -m "Staging Java artifacts for Apache Beam ${{ github.event.inputs.RELEASE }} RC${{ github.event.inputs.RC }}" --non-interactive --username ${{ github.event.inputs.APACHE_ID }} --password ${{ github.event.inputs.APACHE_PASSWORD }} 
+        
+      
+  stage_docker:
+    if: ${{github.event.inputs.STAGE_DOCKER_ARTIFACTS == 'yes'}}
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+        with:
+          ref: "v${{ github.event.inputs.RELEASE }}-RC${{ github.event.inputs.RC }}"
+          repository: apache/beam
+      - name: Install Java 11
+        uses: actions/setup-java@v3
+        with:
+          distribution: 'temurin'
+          java-version: '11'
+      - name: Install Python 3.8
+        uses: actions/setup-python@v3
+        with:
+          python-version: '3.8'
+      - run: echo $JAVA_HOME
+      - run: echo "JAVA11_HOME=${JAVA_HOME}" >> "$GITHUB_OUTPUT"
+        id: export-java11
+      - name: Install Java 8
+        uses: actions/setup-java@v3
+        with:
+          distribution: 'temurin'
+          java-version: '8'
+      - name: Remove default github maven configuration
+        # This step is a workaround to avoid a decryption issue of Beam's
+        # net.linguica.gradle.maven.settings plugin and github's provided maven
+        # settings.xml file
+        run: rm ~/.m2/settings.xml

Review Comment:
   if we migrate to self hosted runner in the future, will remove setting causing problems?
   
   if not, and the self hosted runner does not have this file by default, probably add `|| true` could avoid failure of this step



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm merged pull request #26950: Publish Docker artifacts from build_release_candidate.yml

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #26950:
URL: https://github.com/apache/beam/pull/26950


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org