You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ozone.apache.org by ad...@apache.org on 2022/07/14 11:30:00 UTC
[ozone] branch master updated: HDDS-6994. Update Contributing guide (#3593)
This is an automated email from the ASF dual-hosted git repository.
adoroszlai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ozone.git
The following commit(s) were added to refs/heads/master by this push:
new 912ff42bd1 HDDS-6994. Update Contributing guide (#3593)
912ff42bd1 is described below
commit 912ff42bd1b60c13e4d4fd55ddfa3fde07540ad9
Author: Doroszlai, Attila <64...@users.noreply.github.com>
AuthorDate: Thu Jul 14 13:29:56 2022 +0200
HDDS-6994. Update Contributing guide (#3593)
---
CONTRIBUTING.md | 146 +++++++++++----------
.../network-tests/src/test/blockade/README.md | 4 +-
2 files changed, 82 insertions(+), 68 deletions(-)
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index af9958941f..72fc08546f 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -32,7 +32,7 @@ If you have any questions, please don't hesitate to contact
* **chat**: You can find the #ozone channel at the ASF slack. Invite link is [here](http://s.apache.org/slack-invite)
* **meeting**: [We have weekly meetings](https://cwiki.apache.org/confluence/display/OZONE/Ozone+Community+Calls) which is open to anybody. Feel free to join and ask any questions
-## Building from the source code
+## Building from source
### Requirements
@@ -40,100 +40,103 @@ Requirements to compile the code:
* Unix System
* JDK 1.8 or higher
-* Maven 3.5 or later
+* Maven 3.6 or later
* Internet connection for first build (to fetch all Maven and Ozone dependencies)
-Additional requirements to run your first pseudo cluster:
+(Standard development tools such as make, gcc, etc. are required.)
-* docker
-* docker-compose
+### Build the project
-Additional requirements to execute different type of tests:
+After installing the requirements (especially Maven) build is as simple as:
-* [Robot framework](https://robotframework.org/) (for executing acceptance tests)
-* docker-compose (to start pseudo cluster, also used for blockade and acceptance tests)
-* [blockade](https://pypi.org/project/blockade/) To execute network fault-injection testing.
-* [jq](https://stedolan.github.io/jq/) (for executing acceptance tests)
+```
+mvn clean verify -DskipTests
+```
-Optional dependencies:
+### Useful Maven build options
-* [hugo](https://gohugo.io/) to include the documentation in the web ui.
+ * Use `-DskipShade` to skip shaded Ozone FS jar file creation. Saves time, but you can't test integration with other software that uses Ozone as a Hadoop-compatible file system.
+ * Use `-DskipRecon` to skip building Recon Web UI. It saves about 2 minutes.
+ * Use `-Pdist` to build the binary tarball, similar to the one that gets released
-(Standard development tools such as make, gcc, etc. are required.)
+## Running Ozone in Docker
-### Build the project
+Additional requirements for running Ozone in pseudo cluster (including acceptance tests):
-After installing the requirements (especially maven) the build is as simple as:
-
-```
-mvn clean install -DskipTests
-```
+* docker
+* docker-compose
+* [jq](https://stedolan.github.io/jq/) (utility used heavily by acceptance tests)
-And you can start your first cluster:
+After building Ozone locally, you can start your first pseudo cluster:
```
cd hadoop-ozone/dist/target/ozone-*-SNAPSHOT/compose/ozone
-docker-compose up -d --scale datanode=3
+OZONE_REPLICATION_FACTOR=3 ./run.sh -d
```
-### Helper scripts
-
-`hadoop-ozone/dev-support/checks` directory contains helper scripts to build and check your code. (Including findbugs and checkstyle). Use them if you don't know the exact maven goals / parameters.
-
-These scripts are executed by the CI servers, so it's always good to run them locally before creating a PR.
+See more details in the [README](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/compose/ozone/README.md) and in the [docs](https://ozone.apache.org/docs/current/start.html).
-### Maven build options:
-
- * Use `-DskipShade` to exclude ozonefs jar file creation from the release. It's way more faster, but you can't test Hadoop Compatible file system.
- * Use `-DskipRecon` to exclude the Recon build (Web UI and monitoring) from the build. It saves about 2 additional minutes.
- * Use `-Pdist` to build a distribution (Without this profile you won't have the final tar file)
- * Use `-Pdocker-build` to build a docker image which includes Ozone
- * Use `-Ddocker.image=repo/name` to define the name of your docker image
- * USe `-Pdocker-push` to push the created docker image to the docker registry
-
## Contribute your modifications
-We use github pull requests instead of uploading patches to JIRA. The main contribution workflow is as follows:
-
- 1. Fork `apache/ozone` github repository (first time)
- 2. Create a new Jira in HDDS project (eg. HDDS-1234)
- 3. Create a local branch for your contribution (eg. `git checkout -b HDDS-1234`)
- 4. Create your commits and push your branches to your personal fork.
- 5. Create a pull request on github UI
- * Please include the Jira link, problem description and testing instruction
- 6. Set the Jira to "Patch Available" state
- 7. Address any review comments if applicable by pushing new commits to the PR.
- 8. When addressing review comments, there is no need to squash your commits. This makes it easy for reviewers to only review the incremental changes. The committer will take care to squash all your commits before merging to master.
+We use GitHub pull requests for contributing changes to the repository. The main workflow is as follows:
+
+ 1. Fork [`apache/ozone`](https://github.com/apache/ozone) repository (first time) and clone it to your local machine
+ 2. Enable the `build-branch` GitHub Actions workflow (defined in `.github/workflows/post-commit.yml`) in your fork
+ 3. Ensure a Jira issue corresponding to the change exists in the [HDDS project](https://issues.apache.org/jira/projects/HDDS/) (eg. HDDS-1234)
+ * Please search Jira before creating a new issue, someone might have already reported the same.
+ * If this is your first issue, you might not be able to assign it to yourself. If so, please make a comment in the issue, indicating that your are working on it.
+ 4. Create a local branch for your contribution (eg. `git checkout -b HDDS-1234`)
+ 5. Make your changes locally.
+ * For complex changes, committing each logical part is recommended.
+ 6. Push your changes to your fork of Ozone
+ 7. Wait for the `build-branch` workflow to complete successfully for your commit.
+ 8. Create a pull request for your changes
+ * Please include the Jira link, problem description and testing instruction (follow the [template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md))
+ 9. Set the Jira issue to "Patch Available" state
+ 10. Address any review comments if applicable
+ * Create new, incremental commits in your branch. This makes it easy for reviewers to only review the new changes. The committer will take care to squash all your commits when merging the pull request.
+ * Push your commits in a batch, when no more changes are expected. This reduces the burden on automated CI checks.
+ * If you need to bring your PR up-to-date with the base branch (usually `master`), e.g. to resolve conflicts, please do so by merge, not rebase: `git merge --no-edit origin/master`.
+ * In general, please try to avoid force-push when updating the PR. Here are some great articles that explain why:
+ * https://developers.mattermost.com/blog/submitting-great-prs/#4-avoid-force-pushing
+ * https://www.freecodecamp.org/news/optimize-pull-requests-for-reviewer-happiness#request-a-review
## Code convention and tests
-We follow the code convention of Hadoop project (2 spaces instead of tabs, 80 char line width, ASF licence headers). The code checked with checkstyle, findbugs and various test frameworks.
+Basic code conventions followed by Ozone:
-Please don't post / commit any code with any code violations (all checks are not checking the introduced violations as checks in Hadoop but all the available violations).
+ * 2 spaces indentation
+ * 80-char line length limit
+ * Apache license header required in most files
+ * no `@author` tags, authorship is indicated by Git history
-### Check your contribution
+These are checked by tools like Checkstyle and RAT.
-The easiest way to check your contribution is using the simplified shell scripts under `hadoop-ozone/dev-support/checks`. The problems will be printed out on the standard output.
+### Check your contribution
-For example:
-```
-hadoop-ozone/dev-support/checks/rat.sh
-hadoop-ozone/dev-support/checks/checkstyle.sh
-hadoop-ozone/dev-support/checks/findbugs.sh
-```
+The [`hadoop-ozone/dev-support/checks` directory](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks) contains scripts to build and check Ozone. Most of these are executed by CI for every commit and pull request. Running them before creating a pull request is strongly recommended. This can be achieved by enabling the `build-branch` workflow in your fork and letting GitHub run all of the checks, but most of the checks can also be run locally.
-Execution of rat and checkstyle are very fast. Findbugs is slightly slower. Executing unit.sh takes about 30 minutes.
+ 1. `build.sh`: compiles Ozone
+ 2. quick checks (less than 2 minutes)
+ * `author.sh`: checks for `@author` tags
+ * `bats.sh`: unit test for shell scripts
+ * `rat.sh`: checks for Apache license header
+ * `docs.sh`: sanity checks for [Ozone documentation](https://github.com/apache/ozone/tree/master/hadoop-hdds/docs)
+ * `dependency.sh`: compares list of jars in build output with known list
+ * `checkstyle.sh`: Checkstyle
+ 3. moderate (around 10 minutes)
+ * `findbugs.sh`: SpotBugs
+ * `kubernetes.sh`: very limited set of tests run in Kubernetes environment
+ 4. slow (around 1 hour or more)
+ * `unit.sh`: pure unit tests
+ * `integration.sh`: Java-based tests using single JVM "mini cluster"
+ * `acceptance.sh`: rather complete set of tests in Docker Compose-based environment
-The same scripts are executed by the github PR checker.
+The set of tests run by `integration` and `acceptance` may be limited via arguments, please see the scripts for details. This is used by CI to run them in multiple splits to avoid taking too much time.
-It's always good practice (and fast) to test with the related docker-compose based pseudo clusters:
+Some scripts require third-party tools, but most of these are installed during the first run, if needed.
-```
-cd hadoop-ozone/dist/target/ozone-*-SNAPSHOT/compose/ozone
-./test.sh
-```
-
-(To test S3 use `compose/ozones3`, to test security use `compose/ozonsecure`, etc.
+Most scripts (except `build.sh`) output results in `target/<name>`, e.g. `target/docs`.
### False positive findbugs violation
@@ -149,7 +152,7 @@ private synchronized void addEntryTodnsToUuidMap(
## Using IDE
-As Ozone uses Apache Maven it can be developed from any IDE. As IntelliJ is a common choice, here are some suggestions to use it for Ozone development:
+As Ozone uses Apache Maven it can be developed from any IDE. IntelliJ IDEA is a common choice, here are some suggestions to use it for Ozone development.
### Run Ozone from IntelliJ
@@ -186,12 +189,23 @@ Checkstyle plugin may help to detect violations directly from the IDE.
### Common problems
+#### Too large generated classes
+
IntelliJ may not pick up protoc generated classes as they can be very huge. If the protoc files can't be compiled try the following:
1. Open _Help_ -> _Edit custom properties_ menu.
2. Add `idea.max.intellisense.filesize=10000` entry
3. Restart your IDE
+#### Bad class file
+
+Sometimes during incremental build IDEA encounters the following error:
+
+`bad class file: hadoop-hdds/common/target/classes/org/apache/hadoop/ozone/common/ChunkBufferImplWithByteBufferList$1.class`
+
+Usually this can be fixed by removing the class file (outside of the IDE), but sometimes only by a full Rebuild.
+
## CI
+
The Ozone project uses Github Actions for its CI system. The configuration is described in detail [here](.github/ci.md).
diff --git a/hadoop-ozone/fault-injection-test/network-tests/src/test/blockade/README.md b/hadoop-ozone/fault-injection-test/network-tests/src/test/blockade/README.md
index 7fb62b338e..ef27ee4126 100644
--- a/hadoop-ozone/fault-injection-test/network-tests/src/test/blockade/README.md
+++ b/hadoop-ozone/fault-injection-test/network-tests/src/test/blockade/README.md
@@ -15,8 +15,8 @@
## Blockade Tests
Following python packages need to be installed before running the tests :
-1. blockade
-2. pytest==3.2.0
+1. [blockade](https://pypi.org/project/blockade/)
+2. [pytest==3.2.0](https://pypi.org/project/pytest/3.2.0/)
Running test as part of the maven build:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@ozone.apache.org
For additional commands, e-mail: commits-help@ozone.apache.org