Solr Ref Guide builds, precommit, and a plea

[Cassandra Targett <ct...@apache.org>]

There were some recent changes to the way the Solr Ref Guide gets built
made about a month ago, but stuff moves fast throughout the project so I
suspect it's likely a few have missed some details.

The first, and most important, change is now when you run "ant precommit",
internal (page to page) & javadoc links in the Ref Guide are checked and
validated. This used to only run when you specifically built the Ref Guide
- now it happens with precommit. This was all done with SOLR-12134.

This means that if you didn't set up your env to build the HTML and you
don't want to build the PDF before editing docs, you don't have to. Run
precommit like you're already supposed to for every commit. It will fail if
your doc edits are messed up.

This was one step in the process of merging the Ref Guide build with the
main build. There's more to do, but I don't have a concrete plan in mind.

The second new thing that's worth mentioning is that the Ref Guide now uses
variables in place of "hard coded" version numbers for 3rd party
components. These variables pull their values from ivy-versions.properties
so they are accurate for each release without human intervention. This
includes the versions of Tika, ZooKeeper, Log4J, OpenNLP, Velocity, Commons
Codec, and Dropwizard today - more can be added if they are needed.

Finally - and this likely applies to more than just the Ref Guide - let's
try to use branches for big stuff [1]. Branches are cheap and if things go
awry you don't stall everyone trying to use precommit for 12-18 hours or
more. We can set up Jenkins jobs for a branch if you want it, but master
really isn't the place for stuff that's not done yet, hasn't been tested
locally, and in a state where it's kinda broken, even if it's "just docs".
It also makes it easier for all of us to collaborate with each other, which
is never a bad thing.

Hope this info helps -
Cassandra

[1] And by big, I mean things on the order of adding an entire new section
of multiple new pages or reorganizing content in multiple pages,
particularly when you know it's going to take several weeks or months for
you to finally achieve your vision.

Re: Solr Ref Guide builds, precommit, and a plea

[Cassandra Targett <ca...@gmail.com>]

Just to close the loop (I was out for a couple days)....I hear you that
linking is a bit confusing, let me try to explain a bit and hopefully it
will help.

Nearly all the rules are there because of the PDF version which is the only
official release of the Ref Guide. To generate the PDF, every page of the
Guide is assembled into one massive page and then converted to PDF. If you
think about an online corollary to that - if we put the whole Guide into
one single HTML page - then it should become clear why every single link
between pages needs to be unique.

If you want to link to another page, the pattern is
<<page-name.adoc#page-name,Page Name>>. Someday we hopefully won't have to
do the repetition there (the "page-name.adoc#page-name" part), but since
it's unintuitive, one of the validation rules checks to catch cases where
people may have missed doing it.

Here again if we think about one single massive page of the whole Ref Guide
as the paradigm, "page-name.adoc" doesn't exist in that scenario - only
"SolrRefGuide-all.adoc" does - so the only thing left to reference a link
is the anchor. So, you must have an anchor in the link definition or the
one single page won't know where within itself to link. It would be nice if
the PDF could be smarter about it without the unintuitive pattern, but it's
not yet.

The other issue you ran into was that one page was an orphan (had no
parent). We don't allow that because pages shouldn't be added until they
have a place in the overall page hierarchy. The assumption is that it only
happens accidentally and an orphan should be corrected as soon as possible,
so build/precommit fails on that case.

All these validation rules are in place to try to find common mistakes with
the Ref Guide pages and ensure what comes out of the builds is as close to
release-able as possible without human page-by-page review (which we used
to need to do).

I have no doubt there is room for improvement to all these processes, so
contributions are welcome if there's a way to make it easier/clearer.

On Thu, May 10, 2018 at 1:46 PM Joel Bernstein <jo...@gmail.com> wrote:

> I see that the ref-guide build was also failing. When I made the first fix
> to ref-guide build I didn't see any errors for a while figured that build
> was now working. But then a new set of errors were generated. I didn't
> realize that the ref-guide fails in stages.
>
> I think in general I'm finding all the strictness in the refguide hard
> figure out. Things like internal linking have taken a long time to get
> right even though they work fine when viewed on github, the build often
> still fails due to the links
>
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Thu, May 10, 2018 at 2:33 PM, Joel Bernstein <jo...@gmail.com>
> wrote:
>
>> Ah, just saw all the errors caused by the refguide change that I made. I
>> hadn't realized that precommit had been setup for refguide changes. I
>> apologize for the noise this caused. I originally got one error reported
>> for the ref guide build and fixed that one, and figured I was done. Next
>> time I'll run precommit before pushing out ref guide changes.
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>> On Thu, May 10, 2018 at 10:30 AM, Cassandra Targett <ct...@apache.org>
>> wrote:
>>
>>> There were some recent changes to the way the Solr Ref Guide gets built
>>> made about a month ago, but stuff moves fast throughout the project so I
>>> suspect it's likely a few have missed some details.
>>>
>>> The first, and most important, change is now when you run "ant
>>> precommit", internal (page to page) & javadoc links in the Ref Guide are
>>> checked and validated. This used to only run when you specifically built
>>> the Ref Guide - now it happens with precommit. This was all done with
>>> SOLR-12134.
>>>
>>> This means that if you didn't set up your env to build the HTML and you
>>> don't want to build the PDF before editing docs, you don't have to. Run
>>> precommit like you're already supposed to for every commit. It will fail if
>>> your doc edits are messed up.
>>>
>>> This was one step in the process of merging the Ref Guide build with the
>>> main build. There's more to do, but I don't have a concrete plan in mind.
>>>
>>> The second new thing that's worth mentioning is that the Ref Guide now
>>> uses variables in place of "hard coded" version numbers for 3rd party
>>> components. These variables pull their values from ivy-versions.properties
>>> so they are accurate for each release without human intervention. This
>>> includes the versions of Tika, ZooKeeper, Log4J, OpenNLP, Velocity, Commons
>>> Codec, and Dropwizard today - more can be added if they are needed.
>>>
>>> Finally - and this likely applies to more than just the Ref Guide -
>>> let's try to use branches for big stuff [1]. Branches are cheap and if
>>> things go awry you don't stall everyone trying to use precommit for 12-18
>>> hours or more. We can set up Jenkins jobs for a branch if you want it, but
>>> master really isn't the place for stuff that's not done yet, hasn't been
>>> tested locally, and in a state where it's kinda broken, even if it's "just
>>> docs". It also makes it easier for all of us to collaborate with each
>>> other, which is never a bad thing.
>>>
>>> Hope this info helps -
>>> Cassandra
>>>
>>> [1] And by big, I mean things on the order of adding an entire new
>>> section of multiple new pages or reorganizing content in multiple pages,
>>> particularly when you know it's going to take several weeks or months for
>>> you to finally achieve your vision.
>>>
>>
>>
>

Re: Solr Ref Guide builds, precommit, and a plea

[Joel Bernstein <jo...@gmail.com>]

I see that the ref-guide build was also failing. When I made the first fix
to ref-guide build I didn't see any errors for a while figured that build
was now working. But then a new set of errors were generated. I didn't
realize that the ref-guide fails in stages.

I think in general I'm finding all the strictness in the refguide hard
figure out. Things like internal linking have taken a long time to get
right even though they work fine when viewed on github, the build often
still fails due to the links



Joel Bernstein
http://joelsolr.blogspot.com/

On Thu, May 10, 2018 at 2:33 PM, Joel Bernstein <jo...@gmail.com> wrote:

> Ah, just saw all the errors caused by the refguide change that I made. I
> hadn't realized that precommit had been setup for refguide changes. I
> apologize for the noise this caused. I originally got one error reported
> for the ref guide build and fixed that one, and figured I was done. Next
> time I'll run precommit before pushing out ref guide changes.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Thu, May 10, 2018 at 10:30 AM, Cassandra Targett <ct...@apache.org>
> wrote:
>
>> There were some recent changes to the way the Solr Ref Guide gets built
>> made about a month ago, but stuff moves fast throughout the project so I
>> suspect it's likely a few have missed some details.
>>
>> The first, and most important, change is now when you run "ant
>> precommit", internal (page to page) & javadoc links in the Ref Guide are
>> checked and validated. This used to only run when you specifically built
>> the Ref Guide - now it happens with precommit. This was all done with
>> SOLR-12134.
>>
>> This means that if you didn't set up your env to build the HTML and you
>> don't want to build the PDF before editing docs, you don't have to. Run
>> precommit like you're already supposed to for every commit. It will fail if
>> your doc edits are messed up.
>>
>> This was one step in the process of merging the Ref Guide build with the
>> main build. There's more to do, but I don't have a concrete plan in mind.
>>
>> The second new thing that's worth mentioning is that the Ref Guide now
>> uses variables in place of "hard coded" version numbers for 3rd party
>> components. These variables pull their values from ivy-versions.properties
>> so they are accurate for each release without human intervention. This
>> includes the versions of Tika, ZooKeeper, Log4J, OpenNLP, Velocity, Commons
>> Codec, and Dropwizard today - more can be added if they are needed.
>>
>> Finally - and this likely applies to more than just the Ref Guide - let's
>> try to use branches for big stuff [1]. Branches are cheap and if things go
>> awry you don't stall everyone trying to use precommit for 12-18 hours or
>> more. We can set up Jenkins jobs for a branch if you want it, but master
>> really isn't the place for stuff that's not done yet, hasn't been tested
>> locally, and in a state where it's kinda broken, even if it's "just docs".
>> It also makes it easier for all of us to collaborate with each other, which
>> is never a bad thing.
>>
>> Hope this info helps -
>> Cassandra
>>
>> [1] And by big, I mean things on the order of adding an entire new
>> section of multiple new pages or reorganizing content in multiple pages,
>> particularly when you know it's going to take several weeks or months for
>> you to finally achieve your vision.
>>
>
>

Re: Solr Ref Guide builds, precommit, and a plea

[Joel Bernstein <jo...@gmail.com>]

Ah, just saw all the errors caused by the refguide change that I made. I
hadn't realized that precommit had been setup for refguide changes. I
apologize for the noise this caused. I originally got one error reported
for the ref guide build and fixed that one, and figured I was done. Next
time I'll run precommit before pushing out ref guide changes.

Joel Bernstein
http://joelsolr.blogspot.com/

On Thu, May 10, 2018 at 10:30 AM, Cassandra Targett <ct...@apache.org>
wrote:

> There were some recent changes to the way the Solr Ref Guide gets built
> made about a month ago, but stuff moves fast throughout the project so I
> suspect it's likely a few have missed some details.
>
> The first, and most important, change is now when you run "ant precommit",
> internal (page to page) & javadoc links in the Ref Guide are checked and
> validated. This used to only run when you specifically built the Ref Guide
> - now it happens with precommit. This was all done with SOLR-12134.
>
> This means that if you didn't set up your env to build the HTML and you
> don't want to build the PDF before editing docs, you don't have to. Run
> precommit like you're already supposed to for every commit. It will fail if
> your doc edits are messed up.
>
> This was one step in the process of merging the Ref Guide build with the
> main build. There's more to do, but I don't have a concrete plan in mind.
>
> The second new thing that's worth mentioning is that the Ref Guide now
> uses variables in place of "hard coded" version numbers for 3rd party
> components. These variables pull their values from ivy-versions.properties
> so they are accurate for each release without human intervention. This
> includes the versions of Tika, ZooKeeper, Log4J, OpenNLP, Velocity, Commons
> Codec, and Dropwizard today - more can be added if they are needed.
>
> Finally - and this likely applies to more than just the Ref Guide - let's
> try to use branches for big stuff [1]. Branches are cheap and if things go
> awry you don't stall everyone trying to use precommit for 12-18 hours or
> more. We can set up Jenkins jobs for a branch if you want it, but master
> really isn't the place for stuff that's not done yet, hasn't been tested
> locally, and in a state where it's kinda broken, even if it's "just docs".
> It also makes it easier for all of us to collaborate with each other, which
> is never a bad thing.
>
> Hope this info helps -
> Cassandra
>
> [1] And by big, I mean things on the order of adding an entire new
> section of multiple new pages or reorganizing content in multiple pages,
> particularly when you know it's going to take several weeks or months for
> you to finally achieve your vision.
>