You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "fpiesche (via GitHub)" <gi...@apache.org> on 2023/08/12 13:09:26 UTC
[GitHub] [tika-docker] fpiesche opened a new pull request, #19: Add Github CI workflows for multi-arch Docker images
fpiesche opened a new pull request, #19:
URL: https://github.com/apache/tika-docker/pull/19
With the increasing popularity of people running homelab servers and entire clusters on ARM SBCs such as the Raspberry Pi (hello) it might be useful to have multi-architecture images that run on x86, ARM64 and possibly even 32-bit ARM machines such as the Raspberry Pi 2. I've added a Github workflow that takes the TIKA_VERSION build arg for the Docker images as an input parameter and will build Docker images for Tika for all of these architectures (with others easy to add as long as the base images support them) and upload them as a single multi-architecture manifest via Docker's BuildKit functionality.
As an example:
https://hub.docker.com/r/florianpiesche/tika-minimal/tags
https://hub.docker.com/r/florianpiesche/tika-full/tags
This significantly simplifies the build process for new images - new Docker images for arbitrary versions of Tika can be built and pushed out to multiple registries from Github's web UI literally at the click of a button.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "nextgens (via GitHub)" <gi...@apache.org>.
nextgens commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1757727522
@lewismc any chance you could get this merged please?
It would be great to have arm64 images
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "bartek (via GitHub)" <gi...@apache.org>.
bartek commented on code in PR #19:
URL: https://github.com/apache/tika-docker/pull/19#discussion_r1520746810
##########
.github/workflows/docker-build.yaml:
##########
@@ -0,0 +1,183 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Build Docker images
+
+on:
+ # Allow manual runs.
+ workflow_dispatch:
+ inputs:
+ TIKA_VERSION:
+ description: The Tika version to build the image from
+ type: string
+ default: 2.8.0
+ PUSH_IMAGE:
+ description: Push image to registry
+ type: boolean
+ default: true
+ push:
+ tags:
+ - "*.*.*.*"
+
+env:
+ # Further platforms can be specified here as long as the base image supports them
+ PLATFORMS: linux/arm/v7,linux/arm64/v8,linux/amd64
+ IMAGE_NAME: tika
+ EXPECTED_USER: "35002:35002"
+
+jobs:
+ build-image:
+ name: Build ${{ matrix.IMAGE_TYPE }} Docker image
+ runs-on: ubuntu-latest
+
+ # Set up a local registry so the multi-arch images can be stored for testing
+ services:
+ registry:
+ image: registry:2
+ ports:
+ - 5000:5000
+
+ # This will build both full and minimal images with each run of the workflow
+ strategy:
+ matrix:
+ IMAGE_TYPE: ["full", "minimal"]
+
+ env:
+ TEST_CONTAINER_NAME: tika-${{ matrix.IMAGE_TYPE }}-test
+ TEST_IMAGE_NAME: localhost:5000/tika/tika-${{ matrix.IMAGE_TYPE }}:test
+
+ steps:
+ - name: Check out repo
+ uses: actions/checkout@v4
+
+ # Determine Tika version to build from current tag name
+ - name: "[SETUP] Determine build configuration"
+ shell: bash
+ run: |
+ echo "::group::Determine whether image should be pushed"
+ if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
+ # If the workflow was run manually we want whatever the PUSH_IMAGE input is
+ PUSH_IMAGE="${{ inputs.PUSH_IMAGE }}"
+ elif [[ "${{ github.event_name }}" == "create" ]] && [[ "${{ github.ref }}" == "refs/tags/"* ]]; then
+ # If the workflow was triggered by the creation of a new tag
+ PUSH_IMAGE="true"
+ elif [[ "${{ github.event_name }}" == "push" ]] && [[ "${{ github.ref }}" == "refs/tags/"* ]]; then
+ # If the workflow was triggered by push of a new tag
+ PUSH_IMAGE="true"
+ elif [[ "${{ github.event_name }}" == "release" ]]; then
+ # If the workflow was triggered by the creation of a new GitHub release
+ PUSH_IMAGE="true"
+ else
+ PUSH_IMAGE="false"
+ fi
+ echo "PUSH_IMAGE=${PUSH_IMAGE}" >> $GITHUB_ENV
+ echo "Push image? ${PUSH_IMAGE}"
+ echo "::endgroup::"
+
+ echo "::group::Determine Tika version to build from"
+ if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
+ TIKA_VERSION="${{ inputs.TIKA_VERSION }}"
+ else
+ IMAGE_VERSION=$(git describe --tags --abbrev=0)
+ TIKA_VERSION=$(echo ${IMAGE_VERSION} | sed 's/\(.*\)..\d*$/\1/')
+ fi
+ echo "Building Tika version ${TIKA_VERSION}"
+ echo "TIKA_VERSION=${TIKA_VERSION}" >> $GITHUB_ENV
+ echo "::endgroup::"
+
+ # QEMU and BuildX are required for multi-architecture support
+ - name: "[SETUP] Set up QEMU"
Review Comment:
You could pass `platforms` to this action to limit the amount of platform binaries being installed. https://github.com/docker/setup-qemu-action
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2120479707
I opened: https://issues.apache.org/jira/browse/TIKA-4258 to track this on our JIRA. I also opened an issue on infra.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images -- TIKA-4258 [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2120574718
How's this for a proposed way forward?
We basically keep our current workflow on the release manager's laptop/hardware. We modify our build scripts to build a single-arch image, run our usual tests and then do a second call to docker buildx where we build multiarch images and then deploy to dockerhub?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images -- TIKA-4258 [tika-docker]
Posted by "hegerdes (via GitHub)" <gi...@apache.org>.
hegerdes commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2120869924
> Wow...it looks like it actually worked?!
>
> Can you all give this a shot? https://hub.docker.com/layers/apache/tika/2.9.2-alpha-multi-arch/images/sha256-b8b6e02e3e9f98ddae33b74881f4ead7846ee12352d53149098857378bb3393d?context=repo
Nice, thx 👍
Runs just fine on a raspberry pi 4 (arm64)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1794778767
@fpiesche and fellow devs...IIUC, to release this from the Apache account, I'd have to add my personal dockerhub username and password to the secrets in Apache account, where anyone with collaborator status could use it.
Is there a better solution? Should I make the release from my personal fork?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "fpiesche (via GitHub)" <gi...@apache.org>.
fpiesche commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1986672882
Sorry for dropping off the radar - I've had a lot of Life coming at me over the past few months so until fairly recently a lot of my personal github stuff fell by the wayside. It's getting late here but I'll make some time to sort out that regex tomorrow.
As for the Docker username/token, adding your personal account/token to the github repo as secrets would indeed be the approach with this workflow. Does the ASF maybe have the ability or a process for setting up org-level accounts for things like accessing project Docker repos for builds (so eg. to have an `apache-tika` account that's controlled by the ASF and just has a Docker Hub access token for external CI processes etc)?
I don't have a Docker subscription myself so I honestly have no idea how the Docker Hub CI works or how it would be configured to build on ARM - this workflow wouldn't apply for that process at all :thinking: I had set the GH workflow up on my personal repository as running builds on GitHub's CI and then pushing them to Docker Hub and GHCR from there is how I handle most of my hobby Docker projects, but as those have just me working on them I hadn't run into the token sharing problem...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images -- TIKA-4258 [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2120845395
Wow...it looks like it actually worked?!
Can you all give this a shot? https://hub.docker.com/layers/apache/tika/2.9.2-alpha-multi-arch/images/sha256-b8b6e02e3e9f98ddae33b74881f4ead7846ee12352d53149098857378bb3393d?context=repo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images -- TIKA-4258 [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison closed pull request #19: Add Github CI workflows for multi-arch Docker images -- TIKA-4258
URL: https://github.com/apache/tika-docker/pull/19
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "fpiesche (via GitHub)" <gi...@apache.org>.
fpiesche commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1793029617
With this most recent set of changes:
* The workflow will automatically trigger if a new version tag (of format `*.*.*.*`) is created
* The build will run and locally store the image, then
* run the newly-built image
* as per the `docker-tool.sh`, check that the service is responding to http requests and running as the expected user
* If tests have passed, the image will be pushed to the remote repositories with the following tags:
* the name of the latest tag
* the Tika version (so e.g. the image tagged `2.8.0` will always be the latest build out of the `2.8.0.x` tags in the git repository)
* `latest`
And finally, here's an example build [triggered by creating a new GitHub release](https://github.com/fpiesche/tika-docker/actions/runs/6749676453)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2120433486
@nextgens , I don't have enough knowledge to move forward on this PR alone. If there's a simpler way to achieve multi-arch with fewer mods to our current process, I'd be more than happy to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2120452143
Pinged asf infra on credentials and how to do this for an asf project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika-docker] fpiesche commented on pull request #19: Add Github CI workflows for multi-arch Docker images
Posted by "fpiesche (via GitHub)" <gi...@apache.org>.
fpiesche commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1681833331
@stumpylog my paperless-ngx installation is in fact what I created this PR for :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "fpiesche (via GitHub)" <gi...@apache.org>.
fpiesche commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1793274083
A good opportunity to show how Dependabot PRs work! ;) after enabling Dependabot on my fork:
https://github.com/fpiesche/tika-docker/pulls?q=is%3Apr+is%3Aclosed
None of these needed any changes before merging. Manually-run example build after closing the lot, to check the update hasn't broken anything: https://github.com/fpiesche/tika-docker/actions/runs/6751065869
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images -- TIKA-4258 [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2120807457
Let's add other registries on a later ticket?
How's this look? https://github.com/apache/tika-docker/pull/21
I haven't tested it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "nextgens (via GitHub)" <gi...@apache.org>.
nextgens commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2120096349
Any chance we could have multi-arch for 2.9.2 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2120444104
Let me ping infra at asf to see what we need to do to get this working as an action. I _think_ that's the blocker for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1795096590
And, sorry for the noise, but we recently changed our main branch to {{main}} -- https://issues.apache.org/jira/browse/TIKA-4163
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "stumpylog (via GitHub)" <gi...@apache.org>.
stumpylog commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2120440136
If be happy to help with reviewing it, but I don't see there is a simpler way. Using actions is the best way to achieve this end goal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1757745667
This would be a fantastic modernization of our release process with the added benefit of multi-arch. Thank you!
I have two small concerns/questions:
1) I'm not sure we want dependabot nagging us for base image updates up to daily. Or, if it does, is the expectation that we'll make releases up to daily. This would be an up to daily chore that I'm not sure we have time for.
2) How can we run the tests before pushing to docker hub?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1757756032
more questions...sorry.
3. I'm sorry for not grocking this.. is this building both the minimal and full images? Or, would we do that manually?
4. We moved to versioning of docker images of {tika-version}.{docker.version}. For example, 2.9.0.1 would be the second docker release for Tika 2.9.0. Does this handle that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika-docker] mpdude commented on pull request #19: Add Github CI workflows for multi-arch Docker images
Posted by "mpdude (via GitHub)" <gi...@apache.org>.
mpdude commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1676060806
👏🏻
Had this on my todo list for next week and wanted to come up with a suggestion as well. Kudos!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika-docker] fpiesche commented on pull request #19: Add Github CI workflows for multi-arch Docker images
Posted by "fpiesche (via GitHub)" <gi...@apache.org>.
fpiesche commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1680915176
Added some more comments documenting what various bits of the workflow and dependabot configuration file do while I was at it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images -- TIKA-4258 [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2123275951
At long last, I think we're all set. Thank you @fpiesche for opening this and for all of your work on it! I'm sorry I took the more daft option, but here we are. If you still want, please do open a separate issue for dependabot.
Thank you to all who helped with this and tested the alpha release!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images -- TIKA-4258 [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2120501490
It looks like Airflow at least has moved away from github actions and moved towards a release manager building locally and pushing to dockerhub -- https://cwiki.apache.org/confluence/display/INFRA/Github+Actions+to+DockerHub
Can I just use buildx along these lines to use our current workflow without github actions? https://developers.redhat.com/articles/2023/11/03/how-build-multi-architecture-container-images#docker_buildx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images -- TIKA-4258 [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2120876458
> I think building multiarch with buildx requires QEMU, but as long as that's available on the host doing the builds just running buildx should be perfectly fine - that's all the github workflow does after all!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images -- TIKA-4258 [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2120577030
> If securing the credentials required for dockerhub is the only concern, I think using github container registry instead may be a great solution. https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry
>
> If you still want the images to be on dockerhub you could sync them (locally or otherwise) using a tool such as https://github.com/regclient/regclient/. We use it in Mailu, see https://github.com/Mailu/Mailu/blob/master/.github/workflows/mirror.yml#L35
Awesome. Thank you. ASF infra has a way to do the auth. My current thinking is not to rework our workflow into github actions, but rather see if we can tweak our current workflow to get multi-arch images.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images -- TIKA-4258 [tika-docker]
Posted by "nextgens (via GitHub)" <gi...@apache.org>.
nextgens commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2120530390
If securing the credentials required for dockerhub is the only concern, I think using github container registry instead may be a great solution.
https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry
If you still want the images to be on dockerhub you could sync them (locally or otherwise) using a tool such as https://github.com/regclient/regclient/. We use it in Mailu, see https://github.com/Mailu/Mailu/blob/master/.github/workflows/mirror.yml#L35
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1986632975
We could definitely use some help. This is not an area of strength for me and has fallen off my plate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "bartek (via GitHub)" <gi...@apache.org>.
bartek commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1989856635
Thank you @fpiesche!
Yes, it sounds like there ideally is an existing process within the ASF to handle credentials which would be part of a CI/upload to Docker Hub. I would guess that this has been already handled given there is a process to publish the existing amd64 images?
Aside, I added one tiny comment on the PR :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1768131857
Wow, thank you @fpiesche! All looks good.
For tests, this is our current [release process](https://cwiki.apache.org/confluence/display/TIKA/Release+Process+for+tika-docker). Our rudimentary tests are here: https://github.com/apache/tika-docker/blob/master/docker-tool.sh#L39 We'd always welcome more tests!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "fpiesche (via GitHub)" <gi...@apache.org>.
fpiesche commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1766929188
I'll put some work into some of your concerns. To answer your questions for now:
1) Dependabot will check for base image updates on a daily basis, *but* will only open a PR if the latest base image tag has changed. This won't actually happen with the current setup as the specified tag is not a versionable tag. What this would do is if e.g. the Dockerfiles switched to using `ubuntu:23.10` instead of `ubuntu:jammy`, dependabot would open a PR to update this to 24.04 once that image is released, and again 24.10 once that drops, etc. Dependabot does not check for whether an already-existing tag has had a new image deployed (as this wouldn't need changes to the Dockerfiles, just a rebuild). Similarly, the `github-actions` section will ensure that if a new version of any of the Actions used (e.g. `docker/build-push-action`) is released, Dependabot will open a PR to update the workflows to use this. CI builds can then run to verify that the builds still work, after which the PR could just be merged and the workflows thus kept up to date. This *could* be an up to dail
y task, but updates to GitHub actions are fairly infrequent as well.
2) This is a change I can make - what are the tests you'd want to run? Are they unit tests, or do they involve bringing up the built image and running some tests on the build to ensure it's working properly? Either should be fairly easy to implement.
3) Currently the images are tagged with just the Tika version - however I've checked and since you already create tags in this GitHub repo with the proper version numbers, this is just a matter of changing the tag parameter in line 81 of the workflow. I'll change this in a bit...
4) This is correct. Images will be built as `tika-full` and `tika-minimal` respectively (lines 77/78 of the workflow) and pushed under the `latest` tag and the version tag to the GitHub container registry and Docker Hub (ie. they'll be accessible as `ghcr.io/apache/tika-minimal|tika-latest` and `docker.io/${{secrets.DOCKERHUB_USERNAME}}/tika-minimal|tika-latest`). Tagging the most recent stable release of an image as `latest` is the de facto standard with most Docker images; development images usually use `edge` as a generic "most recent development build" option. The Docker metadata action can [specify further possible tags](https://github.com/docker/metadata-action#tags-input), e.g. git commit SHA sums, arbitrary text, build numbers etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on code in PR #19:
URL: https://github.com/apache/tika-docker/pull/19#discussion_r1383363528
##########
.github/workflows/docker-build.yaml:
##########
@@ -0,0 +1,183 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Build Docker images
+
+on:
+ # Allow manual runs.
+ workflow_dispatch:
+ inputs:
+ TIKA_VERSION:
+ description: The Tika version to build the image from
+ type: string
+ default: 2.8.0
+ PUSH_IMAGE:
+ description: Push image to registry
+ type: boolean
+ default: true
+ push:
+ tags:
+ - "*.*.*.*"
+
+env:
+ # Further platforms can be specified here as long as the base image supports them
+ PLATFORMS: linux/arm/v7,linux/arm64/v8,linux/amd64
+ IMAGE_NAME: tika
+ EXPECTED_USER: "35002:35002"
+
+jobs:
+ build-image:
+ name: Build ${{ matrix.IMAGE_TYPE }} Docker image
+ runs-on: ubuntu-latest
+
+ # Set up a local registry so the multi-arch images can be stored for testing
+ services:
+ registry:
+ image: registry:2
+ ports:
+ - 5000:5000
+
+ # This will build both full and minimal images with each run of the workflow
+ strategy:
+ matrix:
+ IMAGE_TYPE: ["full", "minimal"]
+
+ env:
+ TEST_CONTAINER_NAME: tika-${{ matrix.IMAGE_TYPE }}-test
+ TEST_IMAGE_NAME: localhost:5000/tika/tika-${{ matrix.IMAGE_TYPE }}:test
+
+ steps:
+ - name: Check out repo
+ uses: actions/checkout@v4
+
+ # Determine Tika version to build from current tag name
+ - name: "[SETUP] Determine build configuration"
+ shell: bash
+ run: |
+ echo "::group::Determine whether image should be pushed"
+ if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
+ # If the workflow was run manually we want whatever the PUSH_IMAGE input is
+ PUSH_IMAGE="${{ inputs.PUSH_IMAGE }}"
+ elif [[ "${{ github.event_name }}" == "create" ]] && [[ "${{ github.ref }}" == "refs/tags/"* ]]; then
+ # If the workflow was triggered by the creation of a new tag
+ PUSH_IMAGE="true"
+ elif [[ "${{ github.event_name }}" == "push" ]] && [[ "${{ github.ref }}" == "refs/tags/"* ]]; then
+ # If the workflow was triggered by push of a new tag
+ PUSH_IMAGE="true"
+ elif [[ "${{ github.event_name }}" == "release" ]]; then
+ # If the workflow was triggered by the creation of a new GitHub release
+ PUSH_IMAGE="true"
+ else
+ PUSH_IMAGE="false"
+ fi
+ echo "PUSH_IMAGE=${PUSH_IMAGE}" >> $GITHUB_ENV
+ echo "Push image? ${PUSH_IMAGE}"
+ echo "::endgroup::"
+
+ echo "::group::Determine Tika version to build from"
+ if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
+ TIKA_VERSION="${{ inputs.TIKA_VERSION }}"
+ else
+ IMAGE_VERSION=$(git describe --tags --abbrev=0)
+ TIKA_VERSION=$(echo ${IMAGE_VERSION} | sed 's/\(.*\)..\d*$/\1/')
Review Comment:
I _think_ this regex works by accident, how about something like: `TIKA_VERSION=$(echo ${IMAGE_VERSION} | sed -r 's/^([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+(-beta)?$/\1/')` That would allow for a -beta as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images [tika-docker]
Posted by "bartek (via GitHub)" <gi...@apache.org>.
bartek commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1986589544
Hello! :wave:
This PR has been kicking around for some time and I'm interested in official arm images (right now we are building our own internally).
Is anything blocking this PR from being merged? I see @lewismc requested changes but there's been many updates since.
Does this need shepherding through? It's not clear if @fpiesche is active on this work anymore?
Thank you! Happy to help as necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika-docker] stumpylog commented on pull request #19: Add Github CI workflows for multi-arch Docker images
Posted by "stumpylog (via GitHub)" <gi...@apache.org>.
stumpylog commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-1681499287
Awesome, this would let us archive https://github.com/paperless-ngx/tika, which existed solely for providing multi-arch images
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images -- TIKA-4258 [tika-docker]
Posted by "fpiesche (via GitHub)" <gi...@apache.org>.
fpiesche commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2120688287
I think building multiarch with buildx requires QEMU, but as long as that's available on the host doing the builds just running buildx should be perfectly fine - that's all the github workflow does after all!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Github CI workflows for multi-arch Docker images -- TIKA-4258 [tika-docker]
Posted by "nextgens (via GitHub)" <gi...@apache.org>.
nextgens commented on PR #19:
URL: https://github.com/apache/tika-docker/pull/19#issuecomment-2122027255
I have just tried ``apache/tika:2.9.2-alpha-multi-arch-full`` on an Ampere A1 (arm64) box and that seems to work fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@tika.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org