You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@openwhisk.apache.org by mr...@apache.org on 2018/01/29 21:39:46 UTC
[incubator-openwhisk-release] branch master updated: Add the specs
to implement the release process (#15)
This is an automated email from the ASF dual-hosted git repository.
mrutkowski pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-openwhisk-release.git
The following commit(s) were added to refs/heads/master by this push:
new ddb7b5c Add the specs to implement the release process (#15)
ddb7b5c is described below
commit ddb7b5c237e1adb696bffdae1f1f0d65ec1e9e45
Author: Vincent <sh...@us.ibm.com>
AuthorDate: Mon Jan 29 16:39:44 2018 -0500
Add the specs to implement the release process (#15)
---
README.md | 7 +++
docs/general_spec.md | 142 +++++++++++++++++++++++++++++++++++++++++++++++++++
docs/work_items.md | 107 ++++++++++++++++++++++++++++++++++++++
3 files changed, 256 insertions(+)
diff --git a/README.md b/README.md
index 51c2a00..0471157 100644
--- a/README.md
+++ b/README.md
@@ -34,6 +34,13 @@ repository and the release package, and adding Licensing headers to each source
## Release distribution
We need to upload all artifacts to project’s subdirectory in Apache channel. This step needs to be implemented in Travis build.
+# Specifications to implement the above release process
+
+To find the specifications about how to implement the Apache Release process for OpenWhisk project set, please visit the
+following page.
+
+- [General Specification](docs/general_spec.md)
+
# Reference
[Apache release policy](http://www.apache.org/legal/release-policy.html)
diff --git a/docs/general_spec.md b/docs/general_spec.md
new file mode 100644
index 0000000..5fad19f
--- /dev/null
+++ b/docs/general_spec.md
@@ -0,0 +1,142 @@
+# General plan to implement the Apache release process for OpenWhisk
+
+The purpose of this repository is to provide a one-stop entry tool to manage the release process of OpenWhisk in an automated
+way. 1,000 people will probably have 1,000 ideas about how to fulfill it, since we are living in such an innovative world.
+We believe they are all the valid routes, directing to our final destination, as long as they are compliant with the Apache
+release process. This specification just gives us one proposal, which may not be perfect, but will be proven to be capable of
+leading OpenWhisk team to the paradise, the release of shipped, packaged, Apache-compliant, able-to-run OpenWhisk source
+code and compiled artifacts.
+
+Long story short, Apache generally needs two type of packages to be publicly available after the declaration of the release:
+source code and the compiled artifacts. We would like to divide the work into two major phases: release of source code and
+release of compiled artifacts.
+
+## Release of OpenWhisk artifacts
+
+Please click on the work items listed as belows to access the detailed information:
+- [1. Download the source code](work_items.md#1-download-the-source-code-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/16))
+- [2. Verify the compliance of the source code](work_items.md#2-verify-the-compliance-of-the-source-code-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/17))
+- [3. Deploy OpenWhisk based on the source code](work_items.md#3-deploy-openwhisk-based-on-the-source-code-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/18))
+- [4. Test each OpenWhisk component](work_items.md#4-test-each-openwhisk-component-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/19))
+- [5. Add NOTICE, LICENSE, and CHANGELOG to each OpenWhisk project](work_items.md#5-add-notice-license-and-changelog-to-each-openwhisk-project-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/13))
+- [6. Package the artifacts](work_items.md#6-package-the-artifacts-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/20))
+- [7. Upload the artifacts to staging repository](work_items.md#7-upload-the-artifacts-into-staging-repository-for-release-vote-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/21))
+ - After the above 7 steps, note will be sent out to the commonuty for vote. Before we reach enough votes, Step 1 to 7
+ can be re-iterated to update the artifacts in staging repository.
+
+- [8. Release the artifacts](work_items.md#8-release-the-artifacts-open-issue) ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/22))
+
+# Proposed implementation
+## How to automate the release process for OpenWhisk?
+
+Travis CI is proposed as the pipeline to walk through the download, verify, build and publish processes.
+
+As we can predict, OpenWhisk is going to release continuously in the future, so we propose to create directories in a
+hierarchical structure with configuration files inside to map different releases and sub-releases of OpenWhisk. One major
+advantage of this plan is to provide the release manager a clear and straightforward overview for the historic releases
+of OpenWhisk, which is convenient to navigate and manage. Once this repository is checked-out, the footprint
+of each release's configuration files can be found on the master branch.
+
+OpenWhisk consists of dozens of projects, which means we need to manage a series of hash values(or tags) for each OpenWhisk
+release or sub-release. We propose to use a JSON-format configuration file to maintain all the information. Since Travis
+is used to run all the jobs, it will parse the configuration file, download the source code, based on the hash values(or tags)
+saved for each repository.
+
+The following diagram shows the structure of the repository:
+```
+|--incubator-openwhisk-release--|
+ |--releases--|
+ |--current--| # the only directory to work with before proposing any release or sub-release
+ |--config.json--|
+ |--NOTICE--|
+ |--LICENSE--|
+ |--CHANGELOG--|
+ |--...--|
+ |--incubator--| # example with no sub-release
+ |--config.json--|
+ |--NOTICE--|
+ |--LICENSE--|
+ |--CHANGELOG--|
+ |--...--|
+ |--1.0--| # example with sub-releases
+ |--milestone-1--|
+ |--config.json--|
+ |--NOTICE--|
+ |--LICENSE--|
+ |--CHANGELOG--|
+ |--...--|
+ |--milestone-2--|
+ |--config.json--|
+ |--NOTICE--|
+ |--LICENSE--|
+ |--CHANGELOG--|
+ |--...--|
+ |--...--|
+```
+
+This OpenWhisk release project is a Github project, and it is targeting the release managers of OpenWhisk as the users.
+We would like to keep the development process of this repository the same as we contribute to any other Github project:
+Release manager creates a fork of this repository, clones the source code locally, changes the code after switching to
+a new branch, and submits the change as a pull request to the master branch of this repository.
+
+The directory called _releases_ is created to maintain the configurations for all the releases. The directory called _current_
+under _releases_ is the only directory to work on, before release manager proposing any release or sub-release. The release
+manager use _current_ as the working directory to try out the hash values or tags to be picked from OpenWhisk repositories.
+When OpenWhisk is about to have a new release, a directory named after the release name will be created, to host all the
+configuration files for this release. If there is a sub-release, a directory named after the sub-release name will be
+created under the corresponding directory with the release name, to host all the configuration files for this sub-release.
+
+## How does the release manager work on this repository to propose a OpenWhisk release/candidate?
+
+Since the directory _current_ is the only place to work on before any release or sub-release is determined, the release
+manager can add or change a certain OpenWhisk project's name, with a hash value or a tag by editing the configuration
+file, change the NOTICE, LICENSE, CHANGELOG, or any other necessary release document within this folder.
+
+The configuration files under the directory _current_ will help the release manager find out a combo of OpenWhisk projects
+with hash values or tags as a qualified candidate to release. Each time a change is submitted as a pull request to master
+branch, Travis build will be triggered to verify that combination of hash values or tags mapping to different OpenWhisk
+projects. If the Travis build passes, the aggregation of OpenWhisk projects works.
+
+Once the release manager feels confident in a certain the project set, a release can be proposed. For example, the next release
+name of OpenWhisk is _incubator_, and there is no sub-release name. We can copy all the files under _releases/current_ into
+the directory _releases/incubator_(create one if it is missing), and submit this change as a PR to this master branch. When
+this PR is merged into master and becomes a valid commit, the release manager can tag it with the name _incubator_, in order
+to kick off a Travis build to publish the packages into Apache/staging repository.
+
+## TBD:
+**We can publish the packaged artifacts into staging repository before sending out the note for the community to vote on.
+After we reach an agreement for this release, move the packaged artifacts from the staging repository to the public
+Apache directory for release.** (Thinking in progress)
+
+## How do we make Travis builds select different configuration files as input, based on different releases?
+
+There are three types of event-driven Travis builds for this repository, we can take full advantage of.
+
+* **PR-based Travis build**: the Travis build triggered by a pull request. Pull request is a change request, so we do not
+need this type of build to pack and publish the packages. This type of Travis build pulls down the source based on the defined
+hash values of each repository of OpenWhisk projects, verify the code compliance, deploy the OpenWhisk environment, and test
+the OpenWhisk environment. It always runs based on the configuration files under **_releases/current_**.
+
+* **Push-based Travis build**: the Travis build triggered by a push into master branch. A push event to the master is the
+affirmation of a pull request, this type of build will definitely do the same thing as PR-based Travis build does, based on
+the configuration files under **_releases/current_**. However, it still does not need to pack and publish the packages.
+
+* **Tag-based Travis build**: the Travis build triggered by git tag. Adding/changing a git tag can kick off the Travis build,
+which will be used by the release manager to trigger a release for OpenWhisk. This type of build needs to pack and publish the
+packages into the Apache directory/staging repository, in addition to what PR-based Travis build or push-based Travis
+build has done. We propose to have a stringent naming convention for the git tags. It must include the same release name
+as the OpenWhisk release name, and the sub-release name, so that this type of Travis build can pick up and find the
+configuration files under **the directory named after the release name**.
+
+For example, if the major release of OpenWhisk is _1.0_, and the sub-release is milestone-1, the release manager should
+create a directory named _1.0_ under _releases_, and a directory _milestone-1_ under _1.0_. After copy all the configuration
+files under _releases_/_current_ (because the release manager has used this directory to develop a new groups of hashes or
+tags for the coming release) into _releases_/_1.0_/_milestone-1_, the release manager submit a PR with the changes. When
+the PR is merged, a new commit in master branch is generated. The new tag named _1.0_-_milestone-1_ needs to be created,
+to kick off the Travis build, which will run based on the configuration files under _releases_/_1.0_/_milestone-1_.
+
+There are plenty of Travis built-in variables we can use to identify the type of Travis build. We pick up different source
+code of OpenWhisk as the input to the Travis build pipeline, according to the type.
+
+
+
diff --git a/docs/work_items.md b/docs/work_items.md
new file mode 100644
index 0000000..d9cb41e
--- /dev/null
+++ b/docs/work_items.md
@@ -0,0 +1,107 @@
+# Identified work items
+
+Before we pushing any release packages into the staging repository or the Apache directory. The source code of all OpenWhisk
+projects are hosted under [apache folder at Github](https://github.com/apache). Release managers are responsible for picking
+up valid hash values or tags for each OpenWhisk project to be released. As we described on the [General Specification](general_specs.md)
+page, a JSON-format configuration file is used to save the project names with the hash values or tags. Then, source code
+can be downloaded based on the configuration, and verified the compliance either by some existing tools or programs. If
+the source code meets the requirements, we can proceed to deploy the OpenWhisk services and other client tools, and run
+the test cases to check whether the services are up and running, whether the client tools are ready to use, etc.
+
+This release of the source code packages need to have the following features implemented.
+
+## 1. Download the source code ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/16))
+
+All the source code OpenWhisk project is located at the Github website. If a configuration file is in place, we should
+be able to download the source code, based on the hash values or tags, defined for each specified repository. This
+feature can be implemented in a bash script, which is able to run both on a local machine or in Travis build.
+
+The script can be named "download_source_code.sh". When this repository is cloned, go to the home directory of this repository,
+and run ./download_source_code.sh in a terminal. The source code of OpenWhisk projects should be downloaded to either
+the specified place or the default place. When a Travis build is kicked off, this script should be called in order to
+download the source code. It is better to support both Ubuntu and Mac systems.
+
+**Work bulletins**:
+- Download the source code into a local workspace, based on predefined project hashes or tags
+- By default, clone the source code with all the git history into a default directory
+
+## 2. Verify the compliance of the source code ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/17))
+
+Each source code file needs to have Apache licensing header at the top. We need to implement the verification in the
+script able to run both locally and in Travis build.
+
+**Work bulletins**:
+- Check if all the source code files have valid licensing headers
+ - Investigate what tools can be used to valid the headers, like Apache RAT, etc
+ - Implement in the script to check the headers
+- Other verifications to be determined
+
+## 3. Deploy OpenWhisk based on the source code ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/18))
+
+Build the source code and deploy the OpenWhisk environment. We need to implement the following bulletins in the script.
+
+**Work bulletins**:
+- Build the images for OpenWhisk services
+- Build the images for all the runtimes
+- Build the CLI, wskdeploy, and any other client tool
+- Build the image for api gateway
+- Deploy OpenWhisk services
+- Install catalog
+
+
+## 4. Test each OpenWhisk component ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/19))
+
+This step is optional for the release process, but the release manager needs to make sure the code we deliver is in
+good shape in terms of functionality.
+
+**Work bulletins**:
+- Test OpenWhisk services by running the test cases under OpenWhisk repository.
+- Test OpenWhisk catalog by running the test cases under OpenWhisk catalog repository.
+- Test the CLI, wskdeploy, and any other client tool by running the test cases under each repository.
+- Test api gateway by running the test cases under OpenWhisk api gateway
+- ...
+
+## 5. Add NOTICE, LICENSE, and CHANGELOG to each OpenWhisk project ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/13))
+
+We need to prepare these files when release managers are developing the release repository. As we investigated, Apache
+Tentacles can create the LICENSE and NOTICE report. The current question is: can we run this tools locally, to generate
+the NOTICE and LICENSE, before we upload artifacts to any repository, like the staging repository?(TBD)
+
+**Work bulletins**:
+- Investigate how to generate NOTICE, LICENSE, and CHANGELOG, unless we have to manually edit them.
+ - For files with static content, we can add them into the existing project.
+ - For files with dynamic content for different releases, we can either manually edit them or use tools to generate them.
+
+## 6. Package the artifacts ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/20))
+
+Each OpenWhisk project needs to have one compiled package, and one source code package. This is probably where the Maven
+release plugin can play out. As we have already downloaded the source code of each OpenWhisk repository, Maven command
+can be used to generate all the artifacts.
+
+One challenge we can think of is that OpenWhisk consists of projects based on different programming languages individually,
+we need to figure out how to package the artifacts differently for different languages, by using the similar or
+consistent build environment. OpenWhisk core is based on Scala, CLI is based on Go, wskdeploy is based on Go, api gateway
+is based on lua, runtime project may be based on its native runtime language, etc.
+
+**Work bulletins**:
+- Investigate how to package the artifacts with existing tools, like Gradle, Maven, any other building plugins, etc.
+- Investigate how the tools works on the source code differently, according to different languages
+- Implement the script to package the artifacts
+
+## 7. Upload the artifacts into staging repository for release vote ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/21))
+
+Upload the artifacts including source code, compiled packages, etc into the staging repository for vote.
+
+**Work bulletins**:
+- Investigate how to upload the artifacts to a staging repository with existing tools, like Gradle, Maven, any other
+building plugins, etc
+- Implement the script to upload the artifacts to the staging repository
+- Validate the artifacts in the staging repository
+
+## 8. Release the artifacts ([open issue](https://github.com/apache/incubator-openwhisk-release/issues/22))
+
+When we reach an agreement on the candidate located in the staging repository, the artifacts need to be move to the Apache\
+directory for release.
+
+**Work bulletins**:
+- Migrate artifacts from the staging repository to Apache directory
--
To stop receiving notification emails like this one, please contact
mrutkowski@apache.org.