[DISCUSS] Move user manual source to Accumulo website repo

[Mike Walch <mw...@apache.org>]

For 2.0, I would like convert user manual source from asciidoc to markdown
and move it to the accumulo-website to be served using Jekyll.  While I
will put these changes up for review, I would like to see if anyone has any
major objections or suggestions before I start work on it (I do not want to
spend a lot of time doing a tedious conversion if someone is going to -1
the change). Below is the issue if you want to comment in JIRA:

https://issues.apache.org/jira/browse/ACCUMULO-4630

Re: [DISCUSS] Move user manual source to Accumulo website repo

[Christopher <ct...@apache.org>]

+1 - I'm in favor of this.

On Tue, Apr 25, 2017 at 3:32 PM Mike Walch <mw...@apache.org> wrote:

> For 2.0, I would like convert user manual source from asciidoc to markdown
> and move it to the accumulo-website to be served using Jekyll.  While I
> will put these changes up for review, I would like to see if anyone has any
> major objections or suggestions before I start work on it (I do not want to
> spend a lot of time doing a tedious conversion if someone is going to -1
> the change). Below is the issue if you want to comment in JIRA:
>
> https://issues.apache.org/jira/browse/ACCUMULO-4630
>

Re: [DISCUSS] Move user manual source to Accumulo website repo

[Christopher <ct...@apache.org>]

I added another con to the JIRA:

* Harder to provide offline docs (especially as a single file).

I don't know if that matters to anybody (it doesn't to me or the users I
support). It's also possible to clone the `accumulo-website` repo as a
workaround.

On Tue, Apr 25, 2017 at 3:57 PM Josh Elser <jo...@gmail.com> wrote:

> copy-pasting from the JIRA issue because I was looking for these in-thread:
>
> <snip>
> Pros
> * Easier to link between pages and external Javadocs
> * Documentation can be broken up into distinct pages which is easier to
> read and better for SEO.
> * Easier to update documentation after releases. Only one commit necessary.
> * Jekyll+Markdown is more customizable and becoming more of a standard
> than asciidoc.
> * Documentation changes that affect multiple releases can be made with
> one PR.
>
> Cons
> * Documentation will no longer ship with tarball
> * Developers cannot update code and docs in one PR
> </snip>
>
> Mike Walch wrote:
> > For 2.0, I would like convert user manual source from asciidoc to
> markdown
> > and move it to the accumulo-website to be served using Jekyll.  While I
> > will put these changes up for review, I would like to see if anyone has
> any
> > major objections or suggestions before I start work on it (I do not want
> to
> > spend a lot of time doing a tedious conversion if someone is going to -1
> > the change). Below is the issue if you want to comment in JIRA:
> >
> > https://issues.apache.org/jira/browse/ACCUMULO-4630
> >
>

Re: [DISCUSS] Move user manual source to Accumulo website repo

[Christopher <ct...@apache.org>]

On Tue, Apr 25, 2017 at 4:43 PM Mike Walch <mw...@apache.org> wrote:

> This change is also part of a larger plan of mine to refactor the
> documentation.  After the user manual is moved to the website, I would like
> to refactor it and make it simpler.  In the future, I would like to see
> Accumulo documentation split between the user manual and the blog.
>
> User manual
> * Simple, easy to read documentation
> * Constantly reviewed for accuracy.
> * Must be changed to reflect any changes to Accumulo.
>
> Accumulo blog
> * Detailed posts about new features, designs, testing, applications
> * Accurate at the time the post was published.
> * Doesn't need to be reviewed after initial post or changed if Accumulo
> changes.
>
> The user manual and blog could complement each other.  If you create a new
> feature, you should add a simple overview and instructions on how to use it
> to the user manual.  A blog post could be written to announce the feature
> and describe the underlying motivation, design & implementation. The new
> section in the user manual and the blog post could link to each other.
>
>
I like most of this idea. I think our current user manual is far too wordy,
and the important technical bits are buried in explanations, examples, and
user stories, all of which can distract from the main points. Some of the
structural and density issues could be improved without the split, but I
think a split is better, because it gives us an opportunity to really
engage with the story-telling about how Accumulo features can be used
without worrying about disrupting or trying to fit into the rigid structure
of the technical docs that make it easy to navigate and useful.

For example: the user manual could say "a context class path provides
classloader isolation for a table's configured iterators; to use it, set X
property on your table", while one or more blog posts supplement this with
information like "this is how the context class path feature can be used to
do A/B testing... it might also be useful for these other things...".

The only nit I would have in what you described is that I think the links
should be one way: from the blogs to the manual. I think this improves
quality of our docs and eases navigation. A basic "see website for
examples" is probably fine, but "see Christopher Tubbs' for an explanation
on how to do X" is probably an indication that either 1) we could do a
better job in our inline explanation, or 2) what we are trying to explain,
or at least to the extent we are trying to explain it, is probably out of
scope.

Re: [DISCUSS] Move user manual source to Accumulo website repo

[Josh Elser <jo...@gmail.com>]

Mike Walch wrote:
> This change is also part of a larger plan of mine to refactor the
> documentation.  After the user manual is moved to the website, I would like
> to refactor it and make it simpler.  In the future, I would like to see
> Accumulo documentation split between the user manual and the blog.
>
> User manual
> * Simple, easy to read documentation
> * Constantly reviewed for accuracy.
> * Must be changed to reflect any changes to Accumulo.
>
> Accumulo blog
> * Detailed posts about new features, designs, testing, applications
> * Accurate at the time the post was published.
> * Doesn't need to be reviewed after initial post or changed if Accumulo
> changes.
>
> The user manual and blog could complement each other.  If you create a new
> feature, you should add a simple overview and instructions on how to use it
> to the user manual.  A blog post could be written to announce the feature
> and describe the underlying motivation, design&  implementation. The new
> section in the user manual and the blog post could link to each other.

I'm thinking about how I would apply the above to a new feature 
(re-using data-center replication in my head...), and I'm a little lost 
as to the type of content that I'd include in the user manual and what 
I'd leave for the blog.

* Problem statement -- Blog
* Architecture/design -- Blog
* Shell commands/Java API -- user manual
* Example workflow - Blog (and user manual?)
* Important notes/details - ??

As a user, I'm used to referring to a user manual to get understanding 
on the high level implementation of a feature, the API I use to access 
the feature, a simple example to confirm that I read the API correctly, 
and any "gotchas" I need to know about.

A blog post is inherently nicer to read in passing, but I would find it 
frustrating if, when using the feature, I have to jump between two 
places to get necessary information.

Re: [DISCUSS] Move user manual source to Accumulo website repo

[Mike Walch <mw...@apache.org>]

Faster doc deployment is also a goal of this change.

For older minor release, the documentation should updated if its incorrect
or if the update is related to a bugfix.  Every other change (like
grammar/syntax, documenting existing features) should be up to the
developer. If they want to spend the time, it's OK but not required.

On Wed, Apr 26, 2017 at 10:51 AM Josh Elser <jo...@gmail.com> wrote:

> Thanks, Mike. I this helped solidify some of the confusion over the
> ambiguity of how this would work as a developer that I had.
>
> Mike Walch wrote:
> > I was thinking that each minor release (2.0, 2.1, etc) would have its own
> > copy of the user manual markdown files in the master branch of the
> > accumulo-website repo.  We actually do this now in that a user manual
> html
> > file exists in master for each minor release. Currently, if you need to
> > update documentation for 1.6, 1.7&  1.8, you need to update each minor
> > release branch of Accumulo (I understand merging makes this easy),
> generate
> > 3 user manual html files, check them into master branch of
> > accumulo-website. Developers simplify it by not immediately pushing user
> > manual changes to the website and waiting for the next bug fix release
> > which could be several months away.
>
> Is faster doc-deployment also a goal of this change? I'd agree that
> we've been relatively "lazy" in making changes like you point out; but,
> that's probably also in part trying to avoid the premature docs
> publishing you mention below.
>
> Aside: HBase has recently automated their workflow so that when a change
> to their "user manual" is committed, Jenkins will automatically
> build+publish the generated HTML to their website. That might be
> something we could consider looking at for either our current workflow
> or this new workflow you're proposing. This hinged on some stuff from
> infra (ability to commit changes safely from ASF jenkins), but I believe
> this is all worked out now.
>
> > One potential issue that I do see with moving user manual source to the
> > accumulo-website repo is that if you make a code change to the 2.0 branch
> > for an upcoming 2.0.3 bug fix release, you might not want the 2.0
> > documentation updated for that change until 2.0.3 is released.
> Developers
> > will need to delay changing user manual until after release or add "since
> > 2.0.3" to the new text in the user manual.  However, new features should
> > not added in bugfix releases so I don't think this will be as common as
> > correcting bad documentation which needs to go out right of way.
>
> That's a good edge-case that we'll want/need to keep in mind.
>
> Since you've thought about this (at least more than I have), might you
> be able to outline the high-level steps for our common docs cases in
> this "new world" you're proposing? I think our doc additions typically
> fall into the following buckets: new features, grammar/syntax fixes,
> FAQ/HOWTO for common administrative actions in production. e.g. a new
> feature would only land in one manual, whereas grammar/syntax would
> (likely) land in all. I'm wondering if we can compose a simple workflow.
> I also understand if you haven't thought that far ahead yet.
>

Re: [DISCUSS] Move user manual source to Accumulo website repo

[Josh Elser <jo...@gmail.com>]

Thanks, Mike. I this helped solidify some of the confusion over the 
ambiguity of how this would work as a developer that I had.

Mike Walch wrote:
> I was thinking that each minor release (2.0, 2.1, etc) would have its own
> copy of the user manual markdown files in the master branch of the
> accumulo-website repo.  We actually do this now in that a user manual html
> file exists in master for each minor release. Currently, if you need to
> update documentation for 1.6, 1.7&  1.8, you need to update each minor
> release branch of Accumulo (I understand merging makes this easy), generate
> 3 user manual html files, check them into master branch of
> accumulo-website. Developers simplify it by not immediately pushing user
> manual changes to the website and waiting for the next bug fix release
> which could be several months away.

Is faster doc-deployment also a goal of this change? I'd agree that 
we've been relatively "lazy" in making changes like you point out; but, 
that's probably also in part trying to avoid the premature docs 
publishing you mention below.

Aside: HBase has recently automated their workflow so that when a change 
to their "user manual" is committed, Jenkins will automatically 
build+publish the generated HTML to their website. That might be 
something we could consider looking at for either our current workflow 
or this new workflow you're proposing. This hinged on some stuff from 
infra (ability to commit changes safely from ASF jenkins), but I believe 
this is all worked out now.

> One potential issue that I do see with moving user manual source to the
> accumulo-website repo is that if you make a code change to the 2.0 branch
> for an upcoming 2.0.3 bug fix release, you might not want the 2.0
> documentation updated for that change until 2.0.3 is released.  Developers
> will need to delay changing user manual until after release or add "since
> 2.0.3" to the new text in the user manual.  However, new features should
> not added in bugfix releases so I don't think this will be as common as
> correcting bad documentation which needs to go out right of way.

That's a good edge-case that we'll want/need to keep in mind.

Since you've thought about this (at least more than I have), might you 
be able to outline the high-level steps for our common docs cases in 
this "new world" you're proposing? I think our doc additions typically 
fall into the following buckets: new features, grammar/syntax fixes, 
FAQ/HOWTO for common administrative actions in production. e.g. a new 
feature would only land in one manual, whereas grammar/syntax would 
(likely) land in all. I'm wondering if we can compose a simple workflow. 
I also understand if you haven't thought that far ahead yet.

Re: [DISCUSS] Move user manual source to Accumulo website repo

[Mike Walch <mw...@apache.org>]

I was thinking that each minor release (2.0, 2.1, etc) would have its own
copy of the user manual markdown files in the master branch of the
accumulo-website repo.  We actually do this now in that a user manual html
file exists in master for each minor release. Currently, if you need to
update documentation for 1.6, 1.7 & 1.8, you need to update each minor
release branch of Accumulo (I understand merging makes this easy), generate
3 user manual html files, check them into master branch of
accumulo-website. Developers simplify it by not immediately pushing user
manual changes to the website and waiting for the next bug fix release
which could be several months away.

One potential issue that I do see with moving user manual source to the
accumulo-website repo is that if you make a code change to the 2.0 branch
for an upcoming 2.0.3 bug fix release, you might not want the 2.0
documentation updated for that change until 2.0.3 is released.  Developers
will need to delay changing user manual until after release or add "since
2.0.3" to the new text in the user manual.  However, new features should
not added in bugfix releases so I don't think this will be as common as
correcting bad documentation which needs to go out right of way.

On Tue, Apr 25, 2017 at 5:19 PM Josh Elser <jo...@gmail.com> wrote:

> I can respect the desire to separate technical documentation from
> examples. However, I think using a single branch of a repository to
> track the technical documentation is going to induce a larger burden on us.
>
> When I make a commit to 1.7 to change the user manual, I most of the
> time would just merge that commit through to 1.8/2.0 as it generally
> applies to all branches. Conversely, if a change only applies to
> specific versions, I can easily no-op merge the commit through to other
> branches (or avoid certain branches via cherry-pick).
>
> My understanding is that we would have a single branch of the website.
> How would we handle multiple versions of documentation via a single
> branch? Would `master` of accumulo-website include $x copies of the
> user-manual?
>
> Mike Walch wrote:
> > This change is also part of a larger plan of mine to refactor the
> > documentation.  After the user manual is moved to the website, I would
> like
> > to refactor it and make it simpler.  In the future, I would like to see
> > Accumulo documentation split between the user manual and the blog.
> >
> > User manual
> > * Simple, easy to read documentation
> > * Constantly reviewed for accuracy.
> > * Must be changed to reflect any changes to Accumulo.
> >
> > Accumulo blog
> > * Detailed posts about new features, designs, testing, applications
> > * Accurate at the time the post was published.
> > * Doesn't need to be reviewed after initial post or changed if Accumulo
> > changes.
> >
> > The user manual and blog could complement each other.  If you create a
> new
> > feature, you should add a simple overview and instructions on how to use
> it
> > to the user manual.  A blog post could be written to announce the feature
> > and describe the underlying motivation, design&  implementation. The new
> > section in the user manual and the blog post could link to each other.
> >
> > On Tue, Apr 25, 2017 at 3:57 PM Josh Elser<jo...@gmail.com>  wrote:
> >
> >> copy-pasting from the JIRA issue because I was looking for these
> in-thread:
> >>
> >> <snip>
> >> Pros
> >> * Easier to link between pages and external Javadocs
> >> * Documentation can be broken up into distinct pages which is easier to
> >> read and better for SEO.
> >> * Easier to update documentation after releases. Only one commit
> necessary.
> >> * Jekyll+Markdown is more customizable and becoming more of a standard
> >> than asciidoc.
> >> * Documentation changes that affect multiple releases can be made with
> >> one PR.
> >>
> >> Cons
> >> * Documentation will no longer ship with tarball
> >> * Developers cannot update code and docs in one PR
> >> </snip>
> >>
> >> Mike Walch wrote:
> >>> For 2.0, I would like convert user manual source from asciidoc to
> >> markdown
> >>> and move it to the accumulo-website to be served using Jekyll.  While I
> >>> will put these changes up for review, I would like to see if anyone has
> >> any
> >>> major objections or suggestions before I start work on it (I do not
> want
> >> to
> >>> spend a lot of time doing a tedious conversion if someone is going to
> -1
> >>> the change). Below is the issue if you want to comment in JIRA:
> >>>
> >>> https://issues.apache.org/jira/browse/ACCUMULO-4630
> >>>
> >
>

Re: [DISCUSS] Move user manual source to Accumulo website repo

[Josh Elser <jo...@gmail.com>]

I can respect the desire to separate technical documentation from 
examples. However, I think using a single branch of a repository to 
track the technical documentation is going to induce a larger burden on us.

When I make a commit to 1.7 to change the user manual, I most of the 
time would just merge that commit through to 1.8/2.0 as it generally 
applies to all branches. Conversely, if a change only applies to 
specific versions, I can easily no-op merge the commit through to other 
branches (or avoid certain branches via cherry-pick).

My understanding is that we would have a single branch of the website. 
How would we handle multiple versions of documentation via a single 
branch? Would `master` of accumulo-website include $x copies of the 
user-manual?

Mike Walch wrote:
> This change is also part of a larger plan of mine to refactor the
> documentation.  After the user manual is moved to the website, I would like
> to refactor it and make it simpler.  In the future, I would like to see
> Accumulo documentation split between the user manual and the blog.
>
> User manual
> * Simple, easy to read documentation
> * Constantly reviewed for accuracy.
> * Must be changed to reflect any changes to Accumulo.
>
> Accumulo blog
> * Detailed posts about new features, designs, testing, applications
> * Accurate at the time the post was published.
> * Doesn't need to be reviewed after initial post or changed if Accumulo
> changes.
>
> The user manual and blog could complement each other.  If you create a new
> feature, you should add a simple overview and instructions on how to use it
> to the user manual.  A blog post could be written to announce the feature
> and describe the underlying motivation, design&  implementation. The new
> section in the user manual and the blog post could link to each other.
>
> On Tue, Apr 25, 2017 at 3:57 PM Josh Elser<jo...@gmail.com>  wrote:
>
>> copy-pasting from the JIRA issue because I was looking for these in-thread:
>>
>> <snip>
>> Pros
>> * Easier to link between pages and external Javadocs
>> * Documentation can be broken up into distinct pages which is easier to
>> read and better for SEO.
>> * Easier to update documentation after releases. Only one commit necessary.
>> * Jekyll+Markdown is more customizable and becoming more of a standard
>> than asciidoc.
>> * Documentation changes that affect multiple releases can be made with
>> one PR.
>>
>> Cons
>> * Documentation will no longer ship with tarball
>> * Developers cannot update code and docs in one PR
>> </snip>
>>
>> Mike Walch wrote:
>>> For 2.0, I would like convert user manual source from asciidoc to
>> markdown
>>> and move it to the accumulo-website to be served using Jekyll.  While I
>>> will put these changes up for review, I would like to see if anyone has
>> any
>>> major objections or suggestions before I start work on it (I do not want
>> to
>>> spend a lot of time doing a tedious conversion if someone is going to -1
>>> the change). Below is the issue if you want to comment in JIRA:
>>>
>>> https://issues.apache.org/jira/browse/ACCUMULO-4630
>>>
>

Re: [DISCUSS] Move user manual source to Accumulo website repo

[Mike Walch <mw...@apache.org>]

This change is also part of a larger plan of mine to refactor the
documentation.  After the user manual is moved to the website, I would like
to refactor it and make it simpler.  In the future, I would like to see
Accumulo documentation split between the user manual and the blog.

User manual
* Simple, easy to read documentation
* Constantly reviewed for accuracy.
* Must be changed to reflect any changes to Accumulo.

Accumulo blog
* Detailed posts about new features, designs, testing, applications
* Accurate at the time the post was published.
* Doesn't need to be reviewed after initial post or changed if Accumulo
changes.

The user manual and blog could complement each other.  If you create a new
feature, you should add a simple overview and instructions on how to use it
to the user manual.  A blog post could be written to announce the feature
and describe the underlying motivation, design & implementation. The new
section in the user manual and the blog post could link to each other.

On Tue, Apr 25, 2017 at 3:57 PM Josh Elser <jo...@gmail.com> wrote:

> copy-pasting from the JIRA issue because I was looking for these in-thread:
>
> <snip>
> Pros
> * Easier to link between pages and external Javadocs
> * Documentation can be broken up into distinct pages which is easier to
> read and better for SEO.
> * Easier to update documentation after releases. Only one commit necessary.
> * Jekyll+Markdown is more customizable and becoming more of a standard
> than asciidoc.
> * Documentation changes that affect multiple releases can be made with
> one PR.
>
> Cons
> * Documentation will no longer ship with tarball
> * Developers cannot update code and docs in one PR
> </snip>
>
> Mike Walch wrote:
> > For 2.0, I would like convert user manual source from asciidoc to
> markdown
> > and move it to the accumulo-website to be served using Jekyll.  While I
> > will put these changes up for review, I would like to see if anyone has
> any
> > major objections or suggestions before I start work on it (I do not want
> to
> > spend a lot of time doing a tedious conversion if someone is going to -1
> > the change). Below is the issue if you want to comment in JIRA:
> >
> > https://issues.apache.org/jira/browse/ACCUMULO-4630
> >
>

Re: [DISCUSS] Move user manual source to Accumulo website repo

[Josh Elser <jo...@gmail.com>]

copy-pasting from the JIRA issue because I was looking for these in-thread:

<snip>
Pros
* Easier to link between pages and external Javadocs
* Documentation can be broken up into distinct pages which is easier to 
read and better for SEO.
* Easier to update documentation after releases. Only one commit necessary.
* Jekyll+Markdown is more customizable and becoming more of a standard 
than asciidoc.
* Documentation changes that affect multiple releases can be made with 
one PR.

Cons
* Documentation will no longer ship with tarball
* Developers cannot update code and docs in one PR
</snip>

Mike Walch wrote:
> For 2.0, I would like convert user manual source from asciidoc to markdown
> and move it to the accumulo-website to be served using Jekyll.  While I
> will put these changes up for review, I would like to see if anyone has any
> major objections or suggestions before I start work on it (I do not want to
> spend a lot of time doing a tedious conversion if someone is going to -1
> the change). Below is the issue if you want to comment in JIRA:
>
> https://issues.apache.org/jira/browse/ACCUMULO-4630
>