[PROPOSAL] Revamp of website/docs

[Thomas Bouron <th...@cloudsoft.io>]

Hello Brooklyners, and welcome into this new decade! (i.e. Happy New Year!)

To start off nicely, I would like to make a proposal regarding the Brooklyn
website/doc.

*Background*
Currently, we have 2 separate projects in `brooklyn-docs`:
- the docs are using `gitbooks` and sources live in the `master` branch.
- the rest of the website is using an ancient version `Jekyll` and sources
live in the `website` branch.

We did this because:
1. the ancient version of Jekyll requires a very specific version of ruby
which is a pain to setup locally and on the CI. Meaning it was virtually
impossible to contribute the docs alone.
2. if there is a change in the docs, it didn't make sense to update the
website part and vice-versa.
3. we wanted to automatically publish the changes live as soon as PRs were
merged.

*Issue*
The problem with this approach is that we need to have 2 CI jobs to build 2
different parts and to somehow push these live. We never managed to
automatically publish updates to website/docs through Jenkins.
We also picked a tool that is unfortunately not maintained anymore
(gitbooks) as their team is pushing for a commercial offering. Our Jekyll
setup is very old and impossible to upgrade currently, as it's using custom
plugins that would need to be rewritten.
And finally, our setup is, in my view, way too complicated and clunky to
use.

*Proposal*
I took a look at different solutions and found one that seemed to tick all
the boxes, it's called vuepress [1] and let us to:
- reconciled the website and docs under the same branch as it supports
multiple layouts, hence build both in one go.
- have a custom navigation, based on whatever configuration we write.
- defined custom blocks that can use hardcoded HTML or vue templates (we do
have custom blocks for the blueprint tours, tips, alerts, etc...)
- have custom search like gitbooks + be responsive and SEO friendly.
- have a toolchain that is maintained, open source and easy to use.

On top of that, having the full website build in one go will enable us to
push, from Jenkins, the compiled files to another branch (like `asf-pages`)
and leverage a new service called `.asf.yaml` [2] which takes care of
publishing it.

WDYT?

Best.

[1] https://vuepress.vuejs.org/
<https://vuepress.vuejs.org/guide/#how-it-works>
[2]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127405038#id-.asf.yamlfeaturesforgitrepositories-WebSiteDeploymentServiceforGitRepositories


-- 
Thomas Bouron
Senior Software Engineer

*Cloudsoft <https://cloudsoft.io/> *| Bringing Business to the Cloud

GitHub: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron

Re: [PROPOSAL] Revamp of website/docs

[Geoff Macartney <ge...@gmail.com>]

Hi Thomas et al.

Keeping it brief:

1. I do think it would be good to bring the documentation/website process
up to date, definitely getting rid of of Jekyll and the ancient Ruby, which
is a complete pain.
2. +1 to Aled's point that documenting things on one clear page would help,
but I also have a feeling that the structure and process isn't simple
enough at the moment to document simply. (Guess any doc improvement would
still help though.)
3. It would also be good (Aled's point 2) to proceed in steps where
possible rather than a big bang.
4. I don't know vuepress either, I don't mind what we choose so long as
it's simple and stable. An experiment with vuepress would be valuable -
Thomas do you think porting from current text to vuepress is going to be
simple or painful?
5. I do find the current website/docs model difficult to work with; I'd
like to have a model that had the source for both in the
brooklyn-docs/master branch, clearly separated into docs/ and website/
folders so you weren't forced to change between branches. docs/ would get
published on each Brooklyn release.  The website would get generated not
from website/ in master, instead we would still have the live "website"
branch, from which the website would be generated using the .asf.yaml
file.   The live website would only get updated whenever we decided to
merge to it from master.
5b. *OR* ! would it be easier to just pull the website out into a separate
repo, brooklyn-website?

Agree also we should bear downstream projects in mind as we plan - would it
be worth thinking about any restructuring to create clearer "hooks" into
which any downstream projects could slot? A good clear "southbound API"
design for brooklyn-docs could be useful for downstream projects.

Geoff






On Tue, 7 Jan 2020 at 16:15, Richard Downer <ri...@apache.org> wrote:

> Hi Thomas,
>
> Agree with a lot of what you say - we've made some decisions that in
> hindsight were bad ones that are now causing pain (even though they were
> the correct decisions at the time - back when Brooklyn first started,
> before Apache, Jekyll was just about the only static site generator with a
> lot of traction).
>
> I've a couple of points to make:
>
> In your description about why website is on a separate branch, you've
> omitted that website and docs have big differences in their lifecycles.
> There's always only one branch of the website, but there's multiple
> versions of the documentation. Mixing website and docs has the problem of
> making multiple versions of the website. Only one branch will contain the
> "real" website and all others will necessarily be out-of-date copies. For
> that reason I'd prefer that the website and the docs are kept in distinct
> branches.
>
> I don't have any particular view on which static site generator is best
> other than to say I'm wary of fast-moving targets. Jekyll and Ruby bit us
> like that - you have to keep your content and infrastructure up-to-date -
> stand still for too long and suddenly it's a pain building your tooling and
> a hefty job to upgrade. I'm very wary of the Node.js, Yarn, NPM ecosystem
> for that reason and I wouldn't want us to be in a similar position to now a
> year or two down the line. I'd be happier with a generator written in Go as
> it usually has simpler distributions and setup (native static-linked
> binaries) that have a longer shelf-life.
>
> Richard.
>
>
> On Tue, 7 Jan 2020 at 09:58, Thomas Bouron <th...@cloudsoft.io>
> wrote:
>
> > Hello Brooklyners, and welcome into this new decade! (i.e. Happy New
> Year!)
> >
> > To start off nicely, I would like to make a proposal regarding the
> Brooklyn
> > website/doc.
> >
> > *Background*
> > Currently, we have 2 separate projects in `brooklyn-docs`:
> > - the docs are using `gitbooks` and sources live in the `master` branch.
> > - the rest of the website is using an ancient version `Jekyll` and
> sources
> > live in the `website` branch.
> >
> > We did this because:
> > 1. the ancient version of Jekyll requires a very specific version of ruby
> > which is a pain to setup locally and on the CI. Meaning it was virtually
> > impossible to contribute the docs alone.
> > 2. if there is a change in the docs, it didn't make sense to update the
> > website part and vice-versa.
> > 3. we wanted to automatically publish the changes live as soon as PRs
> were
> > merged.
> >
> > *Issue*
> > The problem with this approach is that we need to have 2 CI jobs to
> build 2
> > different parts and to somehow push these live. We never managed to
> > automatically publish updates to website/docs through Jenkins.
> > We also picked a tool that is unfortunately not maintained anymore
> > (gitbooks) as their team is pushing for a commercial offering. Our Jekyll
> > setup is very old and impossible to upgrade currently, as it's using
> custom
> > plugins that would need to be rewritten.
> > And finally, our setup is, in my view, way too complicated and clunky to
> > use.
> >
> > *Proposal*
> > I took a look at different solutions and found one that seemed to tick
> all
> > the boxes, it's called vuepress [1] and let us to:
> > - reconciled the website and docs under the same branch as it supports
> > multiple layouts, hence build both in one go.
> > - have a custom navigation, based on whatever configuration we write.
> > - defined custom blocks that can use hardcoded HTML or vue templates (we
> do
> > have custom blocks for the blueprint tours, tips, alerts, etc...)
> > - have custom search like gitbooks + be responsive and SEO friendly.
> > - have a toolchain that is maintained, open source and easy to use.
> >
> > On top of that, having the full website build in one go will enable us to
> > push, from Jenkins, the compiled files to another branch (like
> `asf-pages`)
> > and leverage a new service called `.asf.yaml` [2] which takes care of
> > publishing it.
> >
> > WDYT?
> >
> > Best.
> >
> > [1] https://vuepress.vuejs.org/
> > <https://vuepress.vuejs.org/guide/#how-it-works>
> > [2]
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127405038#id-.asf.yamlfeaturesforgitrepositories-WebSiteDeploymentServiceforGitRepositories
> >
> >
> > --
> > Thomas Bouron
> > Senior Software Engineer
> >
> > *Cloudsoft <https://cloudsoft.io/> *| Bringing Business to the Cloud
> >
> > GitHub: https://github.com/tbouron
> > Twitter: https://twitter.com/eltibouron
> >
>

Re: [PROPOSAL] Revamp of website/docs

[Richard Downer <ri...@apache.org>]

Hi Thomas,

Agree with a lot of what you say - we've made some decisions that in
hindsight were bad ones that are now causing pain (even though they were
the correct decisions at the time - back when Brooklyn first started,
before Apache, Jekyll was just about the only static site generator with a
lot of traction).

I've a couple of points to make:

In your description about why website is on a separate branch, you've
omitted that website and docs have big differences in their lifecycles.
There's always only one branch of the website, but there's multiple
versions of the documentation. Mixing website and docs has the problem of
making multiple versions of the website. Only one branch will contain the
"real" website and all others will necessarily be out-of-date copies. For
that reason I'd prefer that the website and the docs are kept in distinct
branches.

I don't have any particular view on which static site generator is best
other than to say I'm wary of fast-moving targets. Jekyll and Ruby bit us
like that - you have to keep your content and infrastructure up-to-date -
stand still for too long and suddenly it's a pain building your tooling and
a hefty job to upgrade. I'm very wary of the Node.js, Yarn, NPM ecosystem
for that reason and I wouldn't want us to be in a similar position to now a
year or two down the line. I'd be happier with a generator written in Go as
it usually has simpler distributions and setup (native static-linked
binaries) that have a longer shelf-life.

Richard.


On Tue, 7 Jan 2020 at 09:58, Thomas Bouron <th...@cloudsoft.io>
wrote:

> Hello Brooklyners, and welcome into this new decade! (i.e. Happy New Year!)
>
> To start off nicely, I would like to make a proposal regarding the Brooklyn
> website/doc.
>
> *Background*
> Currently, we have 2 separate projects in `brooklyn-docs`:
> - the docs are using `gitbooks` and sources live in the `master` branch.
> - the rest of the website is using an ancient version `Jekyll` and sources
> live in the `website` branch.
>
> We did this because:
> 1. the ancient version of Jekyll requires a very specific version of ruby
> which is a pain to setup locally and on the CI. Meaning it was virtually
> impossible to contribute the docs alone.
> 2. if there is a change in the docs, it didn't make sense to update the
> website part and vice-versa.
> 3. we wanted to automatically publish the changes live as soon as PRs were
> merged.
>
> *Issue*
> The problem with this approach is that we need to have 2 CI jobs to build 2
> different parts and to somehow push these live. We never managed to
> automatically publish updates to website/docs through Jenkins.
> We also picked a tool that is unfortunately not maintained anymore
> (gitbooks) as their team is pushing for a commercial offering. Our Jekyll
> setup is very old and impossible to upgrade currently, as it's using custom
> plugins that would need to be rewritten.
> And finally, our setup is, in my view, way too complicated and clunky to
> use.
>
> *Proposal*
> I took a look at different solutions and found one that seemed to tick all
> the boxes, it's called vuepress [1] and let us to:
> - reconciled the website and docs under the same branch as it supports
> multiple layouts, hence build both in one go.
> - have a custom navigation, based on whatever configuration we write.
> - defined custom blocks that can use hardcoded HTML or vue templates (we do
> have custom blocks for the blueprint tours, tips, alerts, etc...)
> - have custom search like gitbooks + be responsive and SEO friendly.
> - have a toolchain that is maintained, open source and easy to use.
>
> On top of that, having the full website build in one go will enable us to
> push, from Jenkins, the compiled files to another branch (like `asf-pages`)
> and leverage a new service called `.asf.yaml` [2] which takes care of
> publishing it.
>
> WDYT?
>
> Best.
>
> [1] https://vuepress.vuejs.org/
> <https://vuepress.vuejs.org/guide/#how-it-works>
> [2]
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127405038#id-.asf.yamlfeaturesforgitrepositories-WebSiteDeploymentServiceforGitRepositories
>
>
> --
> Thomas Bouron
> Senior Software Engineer
>
> *Cloudsoft <https://cloudsoft.io/> *| Bringing Business to the Cloud
>
> GitHub: https://github.com/tbouron
> Twitter: https://twitter.com/eltibouron
>

Re: [PROPOSAL] Revamp of website/docs

[Aled Sage <al...@gmail.com>]

Hi Thomas,

Sounds really interesting.

I'd view the docs issues slightly differently: what are our biggest pain 
points and what should be the first incremental steps to solve those?

I think the biggest pain points (and thus biggest bang for buck) are:

_*1. How to build and release the existing docs / website is not well 
documented*_

There is not a good web-page describing how one would build the docs 
locally (e.g. to test it), or a good web-page saying how to release this 
to make it go live. Writing a page that describes the current process 
would greatly improve things. [if there is already, apologies and please 
point me at it!]

Thomas, I believe you previously created a docker container for building 
the docs (with the right jekyll version etc) - is that right? Can you 
point us at that please?

_*2. Simplify releasing the docs (manually)*_

Switching to .asf.yaml sounds good. Could we do this now, before we 
switch to vuepress?

It feels like this would not be too much work to document / set up, and 
a lot of this would be reusable when/if we switch to vuepress.

I think setting this up so it can be done manually would be a 
pre-requisite before trying to automate it in CI/jenkins, so is worth 
doing first.

---

_*Affect on Downstream Projects*_

I know of at least one downstream project that embeds the brooklyn docs 
within its own docs - I know that switching to vuepress would impact 
that. We should be slightly conservative about changes to the docs 
system. But saying that, this should not force us to stick with old tech 
stacks - it's not as strict as an API contract!

Perhaps the first step for vuepress would be a spike, or sharing some 
more examples - giving us a better idea about the total effort required, 
and thus letting downstream projects know when it would be likely to 
land and what the new system would look like.

_*Alternatives to vuepress?*_

Can anyone else comment on pros/cons and alternatives? I'm not familiar 
enough with the docs domain to say whether vuepress is the best choice. 
It sounds like a much better choice than the current tech, but I don't 
know what the plausible alternatives are.

vuepress github repo [1] looks promising: 15.2k stars, 281 contributors, 
1820 commits since April 2018, and 8 commits in the last 30 days from 8 
different people.

[1] https://github.com/vuejs/vuepress


On 07/01/2020 09:58, Thomas Bouron wrote:
> Hello Brooklyners, and welcome into this new decade! (i.e. Happy New Year!)
>
> To start off nicely, I would like to make a proposal regarding the Brooklyn
> website/doc.
>
> *Background*
> Currently, we have 2 separate projects in `brooklyn-docs`:
> - the docs are using `gitbooks` and sources live in the `master` branch.
> - the rest of the website is using an ancient version `Jekyll` and sources
> live in the `website` branch.
>
> We did this because:
> 1. the ancient version of Jekyll requires a very specific version of ruby
> which is a pain to setup locally and on the CI. Meaning it was virtually
> impossible to contribute the docs alone.
> 2. if there is a change in the docs, it didn't make sense to update the
> website part and vice-versa.
> 3. we wanted to automatically publish the changes live as soon as PRs were
> merged.
>
> *Issue*
> The problem with this approach is that we need to have 2 CI jobs to build 2
> different parts and to somehow push these live. We never managed to
> automatically publish updates to website/docs through Jenkins.
> We also picked a tool that is unfortunately not maintained anymore
> (gitbooks) as their team is pushing for a commercial offering. Our Jekyll
> setup is very old and impossible to upgrade currently, as it's using custom
> plugins that would need to be rewritten.
> And finally, our setup is, in my view, way too complicated and clunky to
> use.
>
> *Proposal*
> I took a look at different solutions and found one that seemed to tick all
> the boxes, it's called vuepress [1] and let us to:
> - reconciled the website and docs under the same branch as it supports
> multiple layouts, hence build both in one go.
> - have a custom navigation, based on whatever configuration we write.
> - defined custom blocks that can use hardcoded HTML or vue templates (we do
> have custom blocks for the blueprint tours, tips, alerts, etc...)
> - have custom search like gitbooks + be responsive and SEO friendly.
> - have a toolchain that is maintained, open source and easy to use.
>
> On top of that, having the full website build in one go will enable us to
> push, from Jenkins, the compiled files to another branch (like `asf-pages`)
> and leverage a new service called `.asf.yaml` [2] which takes care of
> publishing it.
>
> WDYT?
>
> Best.
>
> [1] https://vuepress.vuejs.org/
> <https://vuepress.vuejs.org/guide/#how-it-works>
> [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127405038#id-.asf.yamlfeaturesforgitrepositories-WebSiteDeploymentServiceforGitRepositories
>
>