You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2018/01/29 21:39:49 UTC
[GitHub] mrutkows closed pull request #15: Add the specs to implement the release process
mrutkows closed pull request #15: Add the specs to implement the release process
URL: https://github.com/apache/incubator-openwhisk-release/pull/15
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/README.md b/README.md
index 51c2a00..0471157 100644
--- a/README.md
+++ b/README.md
@@ -34,6 +34,13 @@ repository and the release package, and adding Licensing headers to each source
## Release distribution
We need to upload all artifacts to project?s subdirectory in Apache channel. This step needs to be implemented in Travis build.
+# Specifications to implement the above release process
+
+To find the specifications about how to implement the Apache Release process for OpenWhisk project set, please visit the
+following page.
+
+- [General Specification](docs/general_spec.md)
+
# Reference
[Apache release policy](http://www.apache.org/legal/release-policy.html)
diff --git a/docs/general_spec.md b/docs/general_spec.md
new file mode 100644
index 0000000..5fad19f
--- /dev/null
+++ b/docs/general_spec.md
@@ -0,0 +1,142 @@
+# General plan to implement the Apache release process for OpenWhisk
+
+The purpose of this repository is to provide a one-stop entry tool to manage the release process of OpenWhisk in an automated
+way. 1,000 people will probably have 1,000 ideas about how to fulfill it, since we are living in such an innovative world.
+We believe they are all the valid routes, directing to our final destination, as long as they are compliant with the Apache
+release process. This specification just gives us one proposal, which may not be perfect, but will be proven to be capable of
+leading OpenWhisk team to the paradise, the release of shipped, packaged, Apache-compliant, able-to-run OpenWhisk source
+code and compiled artifacts.
+
+Long story short, Apache generally needs two type of packages to be publicly available after the declaration of the release:
+source code and the compiled artifacts. We would like to divide the work into two major phases: release of source code and
+release of compiled artifacts.
+
+## Release of OpenWhisk artifacts
+
+Please click on the work items listed as belows to access the detailed information:
+- [1. Download the source code](work_items.md#1-download-the-source-code-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/16))
+- [2. Verify the compliance of the source code](work_items.md#2-verify-the-compliance-of-the-source-code-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/17))
+- [3. Deploy OpenWhisk based on the source code](work_items.md#3-deploy-openwhisk-based-on-the-source-code-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/18))
+- [4. Test each OpenWhisk component](work_items.md#4-test-each-openwhisk-component-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/19))
+- [5. Add NOTICE, LICENSE, and CHANGELOG to each OpenWhisk project](work_items.md#5-add-notice-license-and-changelog-to-each-openwhisk-project-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/13))
+- [6. Package the artifacts](work_items.md#6-package-the-artifacts-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/20))
+- [7. Upload the artifacts to staging repository](work_items.md#7-upload-the-artifacts-into-staging-repository-for-release-vote-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/21))
+ - After the above 7 steps, note will be sent out to the commonuty for vote. Before we reach enough votes, Step 1 to 7
+ can be re-iterated to update the artifacts in staging repository.
+
+- [8. Release the artifacts](work_items.md#8-release-the-artifacts-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/22))
+
+# Proposed implementation
+## How to automate the release process for OpenWhisk?
+
+Travis CI is proposed as the pipeline to walk through the download, verify, build and publish processes.
+
+As we can predict, OpenWhisk is going to release continuously in the future, so we propose to create directories in a
+hierarchical structure with configuration files inside to map different releases and sub-releases of OpenWhisk. One major
+advantage of this plan is to provide the release manager a clear and straightforward overview for the historic releases
+of OpenWhisk, which is convenient to navigate and manage. Once this repository is checked-out, the footprint
+of each release's configuration files can be found on the master branch.
+
+OpenWhisk consists of dozens of projects, which means we need to manage a series of hash values(or tags) for each OpenWhisk
+release or sub-release. We propose to use a JSON-format configuration file to maintain all the information. Since Travis
+is used to run all the jobs, it will parse the configuration file, download the source code, based on the hash values(or tags)
+saved for each repository.
+
+The following diagram shows the structure of the repository:
+```
+|--incubator-openwhisk-release--|
+ |--releases--|
+ |--current--| # the only directory to work with before proposing any release or sub-release
+ |--config.json--|
+ |--NOTICE--|
+ |--LICENSE--|
+ |--CHANGELOG--|
+ |--...--|
+ |--incubator--| # example with no sub-release
+ |--config.json--|
+ |--NOTICE--|
+ |--LICENSE--|
+ |--CHANGELOG--|
+ |--...--|
+ |--1.0--| # example with sub-releases
+ |--milestone-1--|
+ |--config.json--|
+ |--NOTICE--|
+ |--LICENSE--|
+ |--CHANGELOG--|
+ |--...--|
+ |--milestone-2--|
+ |--config.json--|
+ |--NOTICE--|
+ |--LICENSE--|
+ |--CHANGELOG--|
+ |--...--|
+ |--...--|
+```
+
+This OpenWhisk release project is a Github project, and it is targeting the release managers of OpenWhisk as the users.
+We would like to keep the development process of this repository the same as we contribute to any other Github project:
+Release manager creates a fork of this repository, clones the source code locally, changes the code after switching to
+a new branch, and submits the change as a pull request to the master branch of this repository.
+
+The directory called _releases_ is created to maintain the configurations for all the releases. The directory called _current_
+under _releases_ is the only directory to work on, before release manager proposing any release or sub-release. The release
+manager use _current_ as the working directory to try out the hash values or tags to be picked from OpenWhisk repositories.
+When OpenWhisk is about to have a new release, a directory named after the release name will be created, to host all the
+configuration files for this release. If there is a sub-release, a directory named after the sub-release name will be
+created under the corresponding directory with the release name, to host all the configuration files for this sub-release.
+
+## How does the release manager work on this repository to propose a OpenWhisk release/candidate?
+
+Since the directory _current_ is the only place to work on before any release or sub-release is determined, the release
+manager can add or change a certain OpenWhisk project's name, with a hash value or a tag by editing the configuration
+file, change the NOTICE, LICENSE, CHANGELOG, or any other necessary release document within this folder.
+
+The configuration files under the directory _current_ will help the release manager find out a combo of OpenWhisk projects
+with hash values or tags as a qualified candidate to release. Each time a change is submitted as a pull request to master
+branch, Travis build will be triggered to verify that combination of hash values or tags mapping to different OpenWhisk
+projects. If the Travis build passes, the aggregation of OpenWhisk projects works.
+
+Once the release manager feels confident in a certain the project set, a release can be proposed. For example, the next release
+name of OpenWhisk is _incubator_, and there is no sub-release name. We can copy all the files under _releases/current_ into
+the directory _releases/incubator_(create one if it is missing), and submit this change as a PR to this master branch. When
+this PR is merged into master and becomes a valid commit, the release manager can tag it with the name _incubator_, in order
+to kick off a Travis build to publish the packages into Apache/staging repository.
+
+## TBD:
+**We can publish the packaged artifacts into staging repository before sending out the note for the community to vote on.
+After we reach an agreement for this release, move the packaged artifacts from the staging repository to the public
+Apache directory for release.** (Thinking in progress)
+
+## How do we make Travis builds select different configuration files as input, based on different releases?
+
+There are three types of event-driven Travis builds for this repository, we can take full advantage of.
+
+* **PR-based Travis build**: the Travis build triggered by a pull request. Pull request is a change request, so we do not
+need this type of build to pack and publish the packages. This type of Travis build pulls down the source based on the defined
+hash values of each repository of OpenWhisk projects, verify the code compliance, deploy the OpenWhisk environment, and test
+the OpenWhisk environment. It always runs based on the configuration files under **_releases/current_**.
+
+* **Push-based Travis build**: the Travis build triggered by a push into master branch. A push event to the master is the
+affirmation of a pull request, this type of build will definitely do the same thing as PR-based Travis build does, based on
+the configuration files under **_releases/current_**. However, it still does not need to pack and publish the packages.
+
+* **Tag-based Travis build**: the Travis build triggered by git tag. Adding/changing a git tag can kick off the Travis build,
+which will be used by the release manager to trigger a release for OpenWhisk. This type of build needs to pack and publish the
+packages into the Apache directory/staging repository, in addition to what PR-based Travis build or push-based Travis
+build has done. We propose to have a stringent naming convention for the git tags. It must include the same release name
+as the OpenWhisk release name, and the sub-release name, so that this type of Travis build can pick up and find the
+configuration files under **the directory named after the release name**.
+
+For example, if the major release of OpenWhisk is _1.0_, and the sub-release is milestone-1, the release manager should
+create a directory named _1.0_ under _releases_, and a directory _milestone-1_ under _1.0_. After copy all the configuration
+files under _releases_/_current_ (because the release manager has used this directory to develop a new groups of hashes or
+tags for the coming release) into _releases_/_1.0_/_milestone-1_, the release manager submit a PR with the changes. When
+the PR is merged, a new commit in master branch is generated. The new tag named _1.0_-_milestone-1_ needs to be created,
+to kick off the Travis build, which will run based on the configuration files under _releases_/_1.0_/_milestone-1_.
+
+There are plenty of Travis built-in variables we can use to identify the type of Travis build. We pick up different source
+code of OpenWhisk as the input to the Travis build pipeline, according to the type.
+
+
+
diff --git a/docs/work_items.md b/docs/work_items.md
new file mode 100644
index 0000000..d9cb41e
--- /dev/null
+++ b/docs/work_items.md
@@ -0,0 +1,107 @@
+# Identified work items
+
+Before we pushing any release packages into the staging repository or the Apache directory. The source code of all OpenWhisk
+projects are hosted under [apache folder at Github](https://github.com/apache). Release managers are responsible for picking
+up valid hash values or tags for each OpenWhisk project to be released. As we described on the [General Specification](general_specs.md)
+page, a JSON-format configuration file is used to save the project names with the hash values or tags. Then, source code
+can be downloaded based on the configuration, and verified the compliance either by some existing tools or programs. If
+the source code meets the requirements, we can proceed to deploy the OpenWhisk services and other client tools, and run
+the test cases to check whether the services are up and running, whether the client tools are ready to use, etc.
+
+This release of the source code packages need to have the following features implemented.
+
+## 1. Download the source code ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/16))
+
+All the source code OpenWhisk project is located at the Github website. If a configuration file is in place, we should
+be able to download the source code, based on the hash values or tags, defined for each specified repository. This
+feature can be implemented in a bash script, which is able to run both on a local machine or in Travis build.
+
+The script can be named "download_source_code.sh". When this repository is cloned, go to the home directory of this repository,
+and run ./download_source_code.sh in a terminal. The source code of OpenWhisk projects should be downloaded to either
+the specified place or the default place. When a Travis build is kicked off, this script should be called in order to
+download the source code. It is better to support both Ubuntu and Mac systems.
+
+**Work bulletins**:
+- Download the source code into a local workspace, based on predefined project hashes or tags
+- By default, clone the source code with all the git history into a default directory
+
+## 2. Verify the compliance of the source code ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/17))
+
+Each source code file needs to have Apache licensing header at the top. We need to implement the verification in the
+script able to run both locally and in Travis build.
+
+**Work bulletins**:
+- Check if all the source code files have valid licensing headers
+ - Investigate what tools can be used to valid the headers, like Apache RAT, etc
+ - Implement in the script to check the headers
+- Other verifications to be determined
+
+## 3. Deploy OpenWhisk based on the source code ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/18))
+
+Build the source code and deploy the OpenWhisk environment. We need to implement the following bulletins in the script.
+
+**Work bulletins**:
+- Build the images for OpenWhisk services
+- Build the images for all the runtimes
+- Build the CLI, wskdeploy, and any other client tool
+- Build the image for api gateway
+- Deploy OpenWhisk services
+- Install catalog
+
+
+## 4. Test each OpenWhisk component ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/19))
+
+This step is optional for the release process, but the release manager needs to make sure the code we deliver is in
+good shape in terms of functionality.
+
+**Work bulletins**:
+- Test OpenWhisk services by running the test cases under OpenWhisk repository.
+- Test OpenWhisk catalog by running the test cases under OpenWhisk catalog repository.
+- Test the CLI, wskdeploy, and any other client tool by running the test cases under each repository.
+- Test api gateway by running the test cases under OpenWhisk api gateway
+- ...
+
+## 5. Add NOTICE, LICENSE, and CHANGELOG to each OpenWhisk project ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/13))
+
+We need to prepare these files when release managers are developing the release repository. As we investigated, Apache
+Tentacles can create the LICENSE and NOTICE report. The current question is: can we run this tools locally, to generate
+the NOTICE and LICENSE, before we upload artifacts to any repository, like the staging repository?(TBD)
+
+**Work bulletins**:
+- Investigate how to generate NOTICE, LICENSE, and CHANGELOG, unless we have to manually edit them.
+ - For files with static content, we can add them into the existing project.
+ - For files with dynamic content for different releases, we can either manually edit them or use tools to generate them.
+
+## 6. Package the artifacts ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/20))
+
+Each OpenWhisk project needs to have one compiled package, and one source code package. This is probably where the Maven
+release plugin can play out. As we have already downloaded the source code of each OpenWhisk repository, Maven command
+can be used to generate all the artifacts.
+
+One challenge we can think of is that OpenWhisk consists of projects based on different programming languages individually,
+we need to figure out how to package the artifacts differently for different languages, by using the similar or
+consistent build environment. OpenWhisk core is based on Scala, CLI is based on Go, wskdeploy is based on Go, api gateway
+is based on lua, runtime project may be based on its native runtime language, etc.
+
+**Work bulletins**:
+- Investigate how to package the artifacts with existing tools, like Gradle, Maven, any other building plugins, etc.
+- Investigate how the tools works on the source code differently, according to different languages
+- Implement the script to package the artifacts
+
+## 7. Upload the artifacts into staging repository for release vote ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/21))
+
+Upload the artifacts including source code, compiled packages, etc into the staging repository for vote.
+
+**Work bulletins**:
+- Investigate how to upload the artifacts to a staging repository with existing tools, like Gradle, Maven, any other
+building plugins, etc
+- Implement the script to upload the artifacts to the staging repository
+- Validate the artifacts in the staging repository
+
+## 8. Release the artifacts ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/22))
+
+When we reach an agreement on the candidate located in the staging repository, the artifacts need to be move to the Apache\
+directory for release.
+
+**Work bulletins**:
+- Migrate artifacts from the staging repository to Apache directory
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services