[lucy-dev] Standalone documentation files

[Nick Wellnhofer <we...@aevum.de>]

Lucifers,

The issue of standalone Markdown documentation files has already been discussed:

     https://issues.apache.org/jira/browse/CLOWNFISH-26

I'd like to get this feature into 0.5.0, mainly to provide documentation of 
the Clownfish header language as POD and HTML. The main problem is that we 
don't have a way to tell which parcel a Markdown document belongs to. We 
decided to solve this by reworking the layout of our core source directories, 
but this is a major change for a later release.

It's possible to work-around this limitation by

* requiring links from Markdown documents to classes to
   explicitly add the name of the parcel.
* changing the directory structure of the HTML docs to use
   fully-qualified class names instead of parcel/struct_sym.

Here's an initial implementation for POD:

     https://github.com/nwellnhof/lucy-clownfish/commits/standalone_docs
     https://github.com/nwellnhof/lucy/commits/standalone_docs

The generated POD can be reviewed here:

     https://github.com/nwellnhof/lucy/tree/generated_pod/perl/lib/Lucy/Docs

I added a new class CFCDocument for Markdown files. This is needed to simplify 
linking between these files. I decided to use the same link syntax as for classes:

     [](cfish:Tutorial)

This means that a couple of documentation files had to be renamed to avoid 
names clashes (Lucy::Docs::Tutorial::FieldType vs. Lucy::Plan::FieldType, for 
example).

We also lost the feature to link to Perl packages that aren't Clownfish 
classes. This was used for `Lucy::Simple`, `LucyX::Remote::ClusterSearcher`, 
and `Parse::RecDescent`. Maybe we could a add a syntax for linking to 
arbitrary classes of the host language.

The generated POD also lost the NAME section and DESCRIPTION header. I don't 
think this is a problem.

To adapt the Lucy tutorial and cookbook to other host languages, some parts of 
the documentation will have to be reworded or moved to comments in 
language-specific code blocks.

Nick

Re: [lucy-dev] Standalone documentation files

[Marvin Humphrey <ma...@rectangular.com>]

On Wed, Jul 8, 2015 at 10:09 AM, Nick Wellnhofer <we...@aevum.de> wrote:
> Lucifers,
>
> The issue of standalone Markdown documentation files has already been
> discussed:
>
>     https://issues.apache.org/jira/browse/CLOWNFISH-26
>
> I'd like to get this feature into 0.5.0, mainly to provide documentation of
> the Clownfish header language as POD and HTML. The main problem is that we
> don't have a way to tell which parcel a Markdown document belongs to. We
> decided to solve this by reworking the layout of our core source
> directories, but this is a major change for a later release.
>
> It's possible to work-around this limitation by
>
> * requiring links from Markdown documents to classes to
>   explicitly add the name of the parcel.
> * changing the directory structure of the HTML docs to use
>   fully-qualified class names instead of parcel/struct_sym.
>
> Here's an initial implementation for POD:
>
>     https://github.com/nwellnhof/lucy-clownfish/commits/standalone_docs
>     https://github.com/nwellnhof/lucy/commits/standalone_docs
>
> The generated POD can be reviewed here:
>
>     https://github.com/nwellnhof/lucy/tree/generated_pod/perl/lib/Lucy/Docs

Great work, Nick!  +1 to merge these branches.

The workarounds are fine with me.  Whatever it takes...

The one comment I'd make is that in principle I think Clownfish should move
away from deep class names rather than depending on them more heavily.
It's odd in the first place that Clownfish won't let you have both `Foo` and
`Blah::Foo` in the same parcel.  Flat packages map better to some hosts and
work fine in all hosts -- for illustration:

*   Python: `lucy.Indexer`
*   Go: `lucy.Indexer`
*   Perl: `Lucy::Indexer`
*   Ruby: `Lucy::Indexer`
*   Java: `org.apache.lucy.Indexer`

We can discuss this more after 0.5.0 when we change the directory layout.

> I added a new class CFCDocument for Markdown files. This is needed to
> simplify linking between these files. I decided to use the same link syntax
> as for classes:
>
>     [](cfish:Tutorial)

+1, perfect!

> This means that a couple of documentation files had to be renamed to avoid
> names clashes (Lucy::Docs::Tutorial::FieldType vs. Lucy::Plan::FieldType,
> for example).

Right, to e.g. `Lucy::Docs::Tutorial::FieldTypeTutorial`.  That works, though
in the future I think we should encourage Lucy::FieldTypeTutorial instead.

> We also lost the feature to link to Perl packages that aren't Clownfish
> classes. This was used for `Lucy::Simple`, `LucyX::Remote::ClusterSearcher`,
> and `Parse::RecDescent`. Maybe we could a add a syntax for linking to
> arbitrary classes of the host language.

In the meantime, standard HTML links are an option.

> The generated POD also lost the NAME section and DESCRIPTION header. I don't
> think this is a problem.

So, this means that at the top of the document, you now see this...

    Near real-time index updates

... when before you saw this:

    Lucy::Docs::Cookbook::FastUpdates - Near real-time index updates.

Those aren't quite the same.  However, I don't disagree with the proposed
technical solution -- I just would have composed different titles.  For
example:

    Near-real-time Lucy Index Updates

> To adapt the Lucy tutorial and cookbook to other host languages, some parts
> of the documentation will have to be reworded or moved to comments in
> language-specific code blocks.

This was always planned.  I look forward to adding some more examples!

Marvin Humphrey