You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@brooklyn.apache.org by he...@apache.org on 2016/02/01 22:32:34 UTC
[1/2] brooklyn-docs git commit: docs for using submodules
Repository: brooklyn-docs
Updated Branches:
refs/heads/master 6e86cb020 -> f1da1755a
docs for using submodules
Project: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/repo
Commit: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/commit/6a8aef76
Tree: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/tree/6a8aef76
Diff: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/diff/6a8aef76
Branch: refs/heads/master
Commit: 6a8aef762895be69141fbaef2866b8553cd13cf2
Parents: 6e86cb0
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Authored: Mon Feb 1 21:08:43 2016 +0000
Committer: Alex Heneveld <al...@cloudsoftcorp.com>
Committed: Mon Feb 1 21:08:43 2016 +0000
----------------------------------------------------------------------
guide/dev/code/index.md | 132 ++++++++++++++++----------------
guide/dev/code/no-submodules.md | 20 +++++
guide/dev/code/structure.md | 59 ++++++++++++++
guide/dev/code/submodules.md | 144 +++++++++++++++++++++++++++++++++++
guide/dev/index.md | 5 +-
5 files changed, 290 insertions(+), 70 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/6a8aef76/guide/dev/code/index.md
----------------------------------------------------------------------
diff --git a/guide/dev/code/index.md b/guide/dev/code/index.md
index bf2e22a..a589edf 100644
--- a/guide/dev/code/index.md
+++ b/guide/dev/code/index.md
@@ -1,91 +1,87 @@
---
-title: About the Code
+title: Get the Code
layout: website-normal
+children:
+- submodules.md
+- no-submodules.md
---
## The Basics
-Brooklyn is available at [GitHub apache/incubator-brooklyn](http://github.com/apache/incubator-brooklyn). Check it out using:
+The Apache Brooklyn source code is available at [GitHub apache/brooklyn](http://github.com/apache/brooklyn),
+together with many [`brooklyn-*` sub-module projects](https://github.com/apache?query=brooklyn).
+Checkout and build all the submodules with:
{% highlight bash %}
-git clone git@github.com:apache/incubator-brooklyn.git
+git clone http://github.com/apache/brooklyn/
cd brooklyn
-{% endhighlight %}
-
-Build it with:
+git submodule init
+git submodule update --remote --merge --recursive
-{% highlight bash %}
mvn clean install
{% endhighlight %}
-And launch it with:
+This will produce an artifact in `brooklyn-dist/dist/brooklyn-dist-0.9.0-SNAPSHOT-dist.tar.gz` <!-- BROOKLYN_VERSION -->
+which you can use [in the usual way](../../start/running.html).
+Some options which may be useful:
-{% highlight bash %}
-cd usage/dist/target/brooklyn-dist/
-bin/brooklyn launch
-{% endhighlight %}
+* Use `--depth 1` with `git clone` to skip the history (much faster but your `git log` will be incomplete)
+* Use `-DskipTests` with `mvn` to skip tests (again much faster but it won't catch failures)
+* Drop `--recursive` on the `git clone` and set up some submodules manually
+ (e.g. with `git submodule init && git submodule update --remote [module]`)
+ or avoid submodules, following the directions below
+
+Thereafter to update the code in submodules, we strongly recommend doing this:
+
+ git pull && git submodule update --remote --merge --recursive
+
+This merges the latest upstream changes into the current branch of each sub-module on your local machine,
+giving nice errors on conflicts.
+It's fine also to do branching and pulling in each submodule,
+but running `update` without these parameters can cause chaos!
+This chaos -- and a `git sup` alias for this command -- are described in the [submodules](submodules.html) page.
+
+
+### If You Can't Stand Submodules
+
+[These instructions](no-submodules.html) can help setting up a local environment
+which does not rely on submodules.
+
+
+### Contributing a Small Change
+
+If you're making a small change in one project, consider just using that project.
+Whether you use this uber-project or not, to [contribute]({{ site.path.website }}/developers/how-to-contribute.html)
+you'll need to follow the usual fork->work->push->pull-request process.
+
+To understand where you might want to make your change,
+look at the [code structure](structure.html).
+
+
+### Bigger and Regular Changes
+
+Regular contributors will typically have their own fork for each of the submodule projects,
+and will probably want some other settings and tips [as described here](submodules.html).
+
+
+## History, Tags, and Workflow
+
+There are branches for each released version and tags for various other milestones.
+
+As described in more detail [here](submodules.html), we primarily use submodule remote branch tracking
+rather than submodule SHA1 ID's.
-{% comment %}
-TODO examples
-Plenty of examples are in the **examples** sub-dir,
-described [here]({{site.path.guide}}/use/examples).
-{% endcomment %}
-
-Information on using Brooklyn -- configuring locations (in `brooklyn.properties`)
-and adding new projects to a catalog -- can be found in the [User's Guide]({{site.path.guide}}).
-This document is intended to help people become familiar with the codebase.
-
-## Project Structure
-
-Brooklyn is split into the following projects and sub-projects:
-
-* **``camp``**: the components for a server which speaks with the CAMP REST API and understands the CAMP YAML plan language
-* **``api``**: the pure-Java interfaces for interacting with the system
-* **``core``**: the base class implementations for entities and applications, entity traits, locations, policies, sensor and effector support, tasks, and more
-* **``locations``**: specific location integrations
- * **``jclouds``**: integration with many cloud APIs and providers via Apache jclouds
-* **``policies``**: collection of useful policies for automating entity activity
-* **``software``**: entities which are mainly launched by launched software processes on machines, and collections thereof
- * **``base``**: software process lifecycle abstract classes and drivers (e.g. SSH)
- * **``webapp``**: web servers (JBoss, Tomcat), load-balancers (Nginx), and DNS (Geoscaling)
- * **``database``**: relational databases (SQL)
- * **``nosql``**: datastores other than RDBMS/SQL (often better in distributed environments)
- * **``messaging``**: messaging systems, including Qpid, Apache MQ, RabbitMQ
- * **``monitoring``**: monitoring tools, including Monit
- * **``osgi``**: OSGi servers
- * **...**
-* **``utils``**: projects with lower level utilities
- * **common**: Utility classes and methods developed for Brooklyn but not dependent on Brooklyn
- * **groovy**: Groovy extensions and utility classes and methods developed for Brooklyn but not dependent on Brooklyn
- * **jmx/jmxmp-ssl-agent**: An agent implementation that can be attached to a Java process, to give expose secure JMXMP
- * **jmx/jmxrmi-agent**: An agent implementation that can be attached to a Java process, to give expose JMX-RMI without requiring all high-number ports to be open
- * **rest-swagger**: Swagger REST API utility classes and methods developed for Brooklyn but not dependent on Brooklyn
- * **test-support**: Test utility classes and methods developed for Brooklyn but not dependent on Brooklyn
-* **``usage``**: projects which make Brooklyn easier to use, either for end-users or Brooklyn developers
- * **all**: maven project to supply a shaded JAR (containing all dependencies) for convenience
- * **archetypes**: A maven archetype for easily generating the structure of new downstream projects
- * **camp**: Brooklyn bindings for the CAMP REST API
- * **cli**: backing implementation for Brooklyn's command line interface
- * **dist**: builds brooklyn as a downloadable .zip and .tar.gz
- * **jsgui**: Javascript web-app for the brooklyn management web console (builds a WAR)
- * **launcher**: for launching brooklyn, either using a main method or invoked from the cli project
- * **logback-includes**: Various helpful logback XML files that can be included; does not contain logback.xml
- * **logback-xml**: Contains a logback.xml that references the include files in brooklyn-logback-includes
- * **rest-api**: The API classes for the Brooklyn REST api
- * **rest-client**: A client Java implementation for using the Brooklyn REST API
- * **rest-server**: The server-side implementation of the Brooklyn REST API
- * **scripts**: various scripts useful for building, updating, etc. (see comments in the scripts)
- * **qa**: longevity and stress tests
- * **test-support**: provides Brooklyn-specific support for tests, used by nearly all projects in scope ``test``
-* **``docs``**: the markdown source code for this documentation
-* **``examples``**: some canonical examples
-* **``sandbox``**: various projects, entities and policies which the Brooklyn Project is incubating
+The history prior to `0.9.0` is imported from the legacy `incubator-brooklyn` repo for reference and history only.
+Visit that repo to build those versions; they are not intended to build here.
+(Although this works:
+`mkdir merged ; for x in brooklyn-* ; do pushd $x ; git checkout 0.8.0-incubating ; cp -r * ../merged ; popd ; cd merged ; mvn clean install`.)
## Next Steps
-If you're interested in building and editing the code, check out:
+If you're interested in building and editing the code, you probably want to become familiar with these:
+* [Product structure](structure.html)
* [Maven setup](../env/maven-build.html)
* [IDE setup](../env/ide/)
* [Tests](tests.html)
http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/6a8aef76/guide/dev/code/no-submodules.md
----------------------------------------------------------------------
diff --git a/guide/dev/code/no-submodules.md b/guide/dev/code/no-submodules.md
new file mode 100644
index 0000000..4516653
--- /dev/null
+++ b/guide/dev/code/no-submodules.md
@@ -0,0 +1,20 @@
+---
+title: Not Using Submodules
+layout: website-normal
+---
+
+If you don't want to use submodules, this will achieve the same result:
+
+{% highlight bash %}
+mkdir apache-brooklyn
+cd apache-brooklyn
+git clone http://github.com/apache/brooklyn/
+git clone http://github.com/apache/brooklyn-ui/
+git clone http://github.com/apache/brooklyn-server/
+git clone http://github.com/apache/brooklyn-client/
+git clone http://github.com/apache/brooklyn-docs/
+git clone http://github.com/apache/brooklyn-library/
+git clone http://github.com/apache/brooklyn-dist/
+ln -s brooklyn/pom.xml .
+mvn clean install
+{% endhighlight %}
http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/6a8aef76/guide/dev/code/structure.md
----------------------------------------------------------------------
diff --git a/guide/dev/code/structure.md b/guide/dev/code/structure.md
new file mode 100644
index 0000000..bf1139b
--- /dev/null
+++ b/guide/dev/code/structure.md
@@ -0,0 +1,59 @@
+---
+title: Code Structure
+layout: website-normal
+---
+
+Brooklyn is split into the following subprojects:
+
+* **brooklyn-server**:
+ * **api**: the pure-Java interfaces for interacting with the system
+ * **camp**: the components for a server which speaks with the CAMP REST API and understands the CAMP YAML plan language
+ * **core**: the base class implementations for entities and applications, entity traits, locations, policies, sensor and effector support, tasks, and more
+ * **karaf**: OSGi support
+ * **launcher**: for launching brooklyn, either using a main method or invoked from the cli project
+ * **locations**: specific location integrations
+ * **jclouds**: integration with many cloud APIs and providers via Apache jclouds
+ * **logging**: how we enable configurable logging
+ * **logback-includes**: Various helpful logback XML files that can be included; does not contain logback.xml
+ * **logback-xml**: Contains a logback.xml that references the include files in brooklyn-logback-includes
+ * **parent**: a meta-project parent to collect dependencies and other maven configuration for re-use
+ * **policy**: collection of useful policies for automating entity activity
+ * **rest**: supporting the REST API
+ * **rest-api**: The API classes for the Brooklyn REST api
+ * **rest-client**: A client Java implementation for using the Brooklyn REST API
+ * **rest-server**: The server-side implementation of the Brooklyn REST API
+ * **server-cli**: implementation of the Brooklyn *server* command line interface; not to be confused with the client CLI
+ * **software**: support frameworks for creating entities which mainly launch software processes on machines
+ * **base**: software process lifecycle abstract classes and drivers (e.g. SSH)
+ * **winrm**: support for connecting to Windows machines
+ * **test-framework**: provides Brooklyn entities for building YAML tests for other entities
+ * **test-support**: provides Brooklyn-specific support for Java TestNG tests, used by nearly all projects in scope ``test``, building on `utils/test-support`
+ * **utils**: projects with lower level utilities
+ * **common**: Utility classes and methods developed for Brooklyn but not dependent on Brooklyn
+ * **groovy**: Groovy extensions and utility classes and methods developed for Brooklyn but not dependent on Brooklyn
+ * **jmx/jmxmp-ssl-agent**: An agent implementation that can be attached to a Java process, to give expose secure JMXMP
+ * **jmx/jmxrmi-agent**: An agent implementation that can be attached to a Java process, to give expose JMX-RMI without requiring all high-number ports to be open
+ * **rest-swagger**: Swagger REST API utility classes and methods developed for Brooklyn but not dependent on Brooklyn
+ * **test-support**: Test utility classes and methods developed for Brooklyn but not dependent on Brooklyn
+
+* **brooklyn-ui**: Javascript web-app for the brooklyn management web console (builds a WAR)
+
+* **brooklyn-library**: a library of useful blueprints
+ * **examples**: some canonical examples
+ * **qa**: longevity and stress tests
+ * **sandbox**: experimental items
+ * **software**: blueprints for software processes
+ * **webapp**: web servers (JBoss, Tomcat), load-balancers (Nginx), and DNS (Geoscaling)
+ * **database**: relational databases (SQL)
+ * **nosql**: datastores other than RDBMS/SQL (often better in distributed environments)
+ * **messaging**: messaging systems, including Qpid, Apache MQ, RabbitMQ
+ * **monitoring**: monitoring tools, including Monit
+ * **osgi**: OSGi servers
+
+* **brooklyn-docs**: the markdown source code for this documentation
+
+* **brooklyn-dist**: projects for packaging Brooklyn and making it easier to consume
+ * **all**: maven project to supply a shaded JAR (containing all dependencies) for convenience
+ * **archetypes**: A maven archetype for easily generating the structure of new downstream projects
+ * **dist**: builds brooklyn as a downloadable .zip and .tar.gz
+ * **scripts**: various scripts useful for building, updating, etc. (see comments in the scripts)
http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/6a8aef76/guide/dev/code/submodules.md
----------------------------------------------------------------------
diff --git a/guide/dev/code/submodules.md b/guide/dev/code/submodules.md
new file mode 100644
index 0000000..06380c2
--- /dev/null
+++ b/guide/dev/code/submodules.md
@@ -0,0 +1,144 @@
+---
+title: Git Submodules
+layout: website-normal
+---
+
+## Setting Up Forks
+
+If you're contributing regularly, you'll want your submodules to pull from the `apache` repo
+and push to your fork.
+You'll need to create a fork and the set up your local submodules should then have:
+
+* your fork as the `origin` remote repo
+* the Apache github repo as the `upstream` remote repo
+* optionally the repo at the Apache git server as the `apache` remote repo
+
+This can be done automatically with the following commands:
+
+{% highlight bash %}
+ hub fork
+ git remote add apache https://git-wip-us.apache.org/repos/asf/$x
+ for x in brooklyn-* ; do
+ pushd $x
+ hub fork
+ git remote add apache https://git-wip-us.apache.org/repos/asf/$x
+ popd
+ done
+{% endhighlight %}
+
+This requires the command-line tool `hub` [from github](https://github.com/github/hub) (or `sudo npm install -g hub`),
+run in the directory of the uber-project checked out earlier.
+
+You can then pull and update from upstream, push to origin, and create pull requests in each sub-project.
+Cross-project changes will require multiple PRs:
+try to minimise these, especially where one depends on another,
+and especially especially where two depend on each other -- that is normally a sign of broken backwards compatibility!
+Open the PRs in dependency order and assist reviewers by including the URLs of any upstream dependency PRs
+in the dependent PR to help reviewers
+(dependency PRs will then include a "mention" comment of the dependent PR).
+
+
+## Things You Should Know
+
+Our submodules track **branches**, rather than specific commits,
+although due to the way `git` works there are still references to specific commits.
+
+We track `master` for the master branch and the version branch for other official branches,
+starting with `0.9.0`.
+We update the uber-project recorded SHA reference to subprojects on releases but not regularly --
+that just creates noise and is unnecessary with the `--remote` option on `submodule update`.
+In fact, `git submodule update --remote --merge` pretty much works well;
+the `git sup` alias (below) makes it even easier.
+
+On the other hand, `git status` is not very nice in the uber-project:
+it will show a "new commits" message for submodules,
+unless you're exactly at the uber-project's recorded reference.
+Ignore these.
+It will tell you if you have uncommitted changes,
+but it's not very useful for telling whether you're up to date or if you have newer changes committed
+in the subproject or in your push branch.
+If you go in to each sub-project, `git status` works better, but it can be confusing
+to track which branch each subproject is one.
+A `summary` script is provided below which solves these issues,
+showing useful status across all subprojects.
+
+
+### Pitfalls of Submodules
+
+Submodules can be confusing; if you get stuck the references at the bottom may be useful.
+You can also work [without submodules](no-submodules.html).
+
+Some of the things to be careful of are:
+
+* **Don't copy submodule directories.** This doesn't act as you'd expect;
+ its `.git` record simply points at the parent project's `.git` folder,
+ which in turn points back at it. So if you copy it and make changes in the copy,
+ it's rather surprising where those changes actually get made.
+ Worse, `git` doesn't report errors; you'll only notice it when you see files change bizarrely.
+
+* **Be careful committing in the uber-project.**
+ You can update commit IDs, but if these accidentally point to an ID that isn't committed,
+ everyone else sees errors.
+ It's useful to do this on release (and update the target branch then also)
+ and maybe occasionally at other milestones but so much at other times as these ID's
+ very quickly become stale on `master`.
+
+
+### Useful Aliases
+
+{% highlight bash %}
+git config --global alias.sup 'submodule update --remote --merge --recursive'
+git config --global alias.sdiff '!git diff && git submodule foreach "git diff"'
+{% endhighlight %}
+
+
+### Getting a Summary of Submodules
+
+In addition, the `git-summary` script [here]() makes working with submodules much more enjoyable,
+simply install and use `git ss` in the uber-project to see the status of each submodule:
+
+{% highlight bash %}
+curl https://gist.githubusercontent.com/ahgittin/6399a29df1229a37b092/raw/05f99aa95a5e8eb541bb79c6707324e26fc0f579/git-summary.sh \
+ | sudo tee /usr/local/bin/git-summary
+sudo chmod 755 /usr/local/bin/git-summary
+git config --global alias.ss '!git-summary ; echo ; git submodule foreach --quiet "git summary"'
+{% endhighlight %}
+
+
+### Other Handy Commands
+
+{% highlight bash %}
+# run a git command (eg pull) in each submodule
+git submodule foreach 'git pull'
+
+# iterate across submodules in bash, e.g. doing git status
+for x in brooklyn-* ; do pushd $x ; git status ; popd ; done
+{% endhighlight %}
+
+
+### Legacy Incubator Pull Requests
+
+If you need to apply code changes made pre-graduation, against the incubator repository,
+splitting it up into submodules, it's fairly straightforward:
+
+1. In the incubator codebase, start at its final state, in `master`.
+2. `git checkout -b making-a-diff`
+3. Merge or rebase the required commits.
+ Ensure conflicts are resolved, but don't worry about commit messages.
+4. `git diff > /tmp/diff`
+5. Go to the new `brooklyn` uber-project directory.
+ Ensure you are at master and all subprojects updated (`git sup`).
+6. `patch -p1 < /tmp/diff`
+7. Run `git ss` to inspect the changes.
+8. Test it, commit each changed project on a branch and create pull requests.
+ Where applicable, record the original author(s) and message(s) in the commit.
+
+
+
+## Git Submodule References
+
+* [1] [Git SCM Book](https://git-scm.com/book/en/v2/Git-Tools-Submodules)
+* [2] [Medium blog: Mastering Git Submodules](https://medium.com/@porteneuve/mastering-git-submodules-34c65e940407#.r7677prhv)
+* [3] `git submodule --help` section on `update`
+* [4] [StackOverflow: Git Submodules Branch Tag](http://stackoverflow.com/questions/1777854/git-submodules-specify-a-branch-tag/18797720#18797720)
+
http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/6a8aef76/guide/dev/index.md
----------------------------------------------------------------------
diff --git a/guide/dev/index.md b/guide/dev/index.md
index 2c294b6..3fb28dd 100644
--- a/guide/dev/index.md
+++ b/guide/dev/index.md
@@ -6,14 +6,15 @@ breadcrumbs:
- /website/documentation/index.md
- /guide/dev/index.md
children:
-- env/
- code/
-- { link: "http://github.com/apache/incubator-brooklyn", title: "GitHub" }
+- env/
+- code/structure.md
- code/tests.md
- code/licensing.md
- tips/
- tips/logging.md
- tips/debugging-remote-brooklyn.md
+- { link: "http://github.com/apache/incubator-brooklyn", title: "GitHub" }
- { link: "https://brooklyn.apache.org/v/latest/misc/javadoc", title: "Javadoc" }
---
[2/2] brooklyn-docs git commit: This closes #1
Posted by he...@apache.org.
This closes #1
Project: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/repo
Commit: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/commit/f1da1755
Tree: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/tree/f1da1755
Diff: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/diff/f1da1755
Branch: refs/heads/master
Commit: f1da1755a15ab0d68556fa27bf78920666cb837d
Parents: 6e86cb0 6a8aef7
Author: Alex Heneveld <al...@cloudsoftcorp.com>
Authored: Mon Feb 1 21:31:34 2016 +0000
Committer: Alex Heneveld <al...@cloudsoftcorp.com>
Committed: Mon Feb 1 21:31:34 2016 +0000
----------------------------------------------------------------------
guide/dev/code/index.md | 132 ++++++++++++++++----------------
guide/dev/code/no-submodules.md | 20 +++++
guide/dev/code/structure.md | 59 ++++++++++++++
guide/dev/code/submodules.md | 144 +++++++++++++++++++++++++++++++++++
guide/dev/index.md | 5 +-
5 files changed, 290 insertions(+), 70 deletions(-)
----------------------------------------------------------------------