[jira] [Updated] (SOLR-15250) Dockerfile for local builds that can also serve as template for 'official' docker images

["Chris M. Hostetter (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris M. Hostetter updated SOLR-15250:
--------------------------------------
    Attachment: SOLR-15250.patch
        Status: Open  (was: Open)

I started experimenting with this idea based on the "tepid" (at best) initial response from the docker-library team in [https://github.com/docker-solr/docker-solr/issues/368] (and the complete lack of response to my follow up questions seeking more clarifications).

My impression is that we're not going to be able to continue to pursue the approach advocated in SOLR-15129 for "official" {{_/solr}} docker images.

The attached patch starts with a hybrid of a couple of existing ideas:
 * From SOLR-15129: put all the docker specific scripts needed in the final docker image into the released solr.tgz
 ** Anything/Everything solr related that is needed in the final docker image comes from solr.tgz
 * From the existing docker-solr Dockerfile.template:
 ** add in SHA512 validation of the solr.tgz, based on a build ARG
 ** add in GPG validation logic, but with the option to skip it based on some build ARGs
 ** loop over a configured set of "download urls" until we find the solr.tgz in our context
 * Similar to SOLR-15127: if the solr.tgz is already in the build context we can skip this download loop
 ** when building from gradle, use a build context that does in fact include the locally built solr.tgz – so no download is attempted

Completely new additions:
 * gradle task implementing the "templating" logic for building the {{Dockerfile.official}}
 ** looks for a few specific {{ARG}} declarations with blank values, and fills them in with the expected values based on current build properties
 ** fails the build if the expected {{ARG}} lines aren't found exactly once

----
If people like this approach, the next steps would be:
 * get sign off from docker-library folks on using multistage builds in our official Dockerfile
 ** the core ideas in this jira don't _require_ using a multistage build, but since the Dockerfile on the main branch already uses multi-stage builds to keep the images size down I left it in when incorporating the additions from the exsting docker-solr {{Dockerfile-varsolr.template}}
 ** If that's a deal breaker with docker-library we should rip it out sooner rather then later
 * add new gradle tasks/scripts to incorporate validation of the "generated" {{Dockerfile.official}} using a mock download server (see nocommits in build.gradle)
 * decide if we want to keep docker-solr's current approach of including the Release Manager's public key ID directly in the Dockerfile and fetching that key from third party keyserver directories – OR – if we want to make the GPG validation based on the canonical {{KEYS}} file for the Solr project
 ** I'm in favor of switching to downloading our {{KEYS}} file – not only because I like reducing our dependency on third-party services, but also because it would help us simplify how much templating we have to do (the {{KEYS}} url is fixed and wouldn't change per release like the RM's public key ID)
 ** If we want to keep docker-solr's current approach, then the templating logic needs to know what the RM's GPG key ID is ... which would probably be blocked until Solr's overall gradle build gets updated with release helper tasks (similar to what the ant build has on branch_8x) for signing the release artifacts – so we can have our docker templating task depend on those and look at the key used.
 *** I suspect the only reason that docker-solr works that way in the 8.x releases is because for a long time we had a unique {{KEYS}} file per release – but since that shouldn't be an issue for 9.x releases we can switch to a single canonical URL for all pulic keys of all Release Managers
 * regardless of what we decide regarding how we get the keys for GPG validation, we once the gradle build supports signing solr.tgz, we can have our Dockerfile validation tasks check all the GPG logic in the Dockerfile as well
 ** again by using mock server to override the solr.tgz.asc url (which would need to become a new ARG)
 ** this validation could be run manualy by Release Managers – but if we have a way to generate a 'mock key' then even Jenkins/CI builds can validate the logic
 * decide on & resolve a few other misc nocommits: better name for new gradle task(s); possible support for stripping _some_ specific comments in the Dockerfile template that won't be relevant in the final {{Dockerfile.official}}; if we should rename the Dockerfile(s) for clarity; and where exactly the docker-scripts should live in the solr.tgz (I'm not a fan of where SOLR-15129 suggests putting them)

> Dockerfile for local builds that can also serve as template for 'official' docker images
> ----------------------------------------------------------------------------------------
>
>                 Key: SOLR-15250
>                 URL: https://issues.apache.org/jira/browse/SOLR-15250
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Chris M. Hostetter
>            Assignee: Chris M. Hostetter
>            Priority: Major
>         Attachments: SOLR-15250.patch
>
>
> This issue tracks PoC work experimenting with the idea of the following workflow:
> For Users:
>  * a {{Dockerfile}} (or {{Dockerfile.local}}) in our git repo that can be used (directly or via gradle) to build docker images directly from a local solr.tgz (in the docker build context)
> For Release Manager:
>  * The exact same {{Dockerfile}} serves as a "template" that gradle tasks use to generate a {{build/Dockerfile.official}} via some very simple substitutions to fill in ARG defaults based on the "official" solr-VERSION.tgz for this release
>  * This {{Dockerfile.official}} can then be committed to the docker-solr github repo (or some similar new 9.x+ repo) and can be build with a a completely empty build context – in which it downloads (and validates) the official solr-VERSION.tgz (based on the ARG values that were filled in during the release)
>  * Automated tests can help us "validate" that the generated {{Dockerfile.official}} will work _prior_ to officially publishing release artifacts, by using a "mock" download server to host the local solr-VERSION.tgz file
> The driving goal being that the Dockerfile used for official {{_/solr}} builds should be as close as possible to identical to the Dockerfile used for "local" builds by users – given the constraints put on us by the docker-library team.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)