You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Michael Miklavcic <mi...@gmail.com> on 2018/12/20 19:47:00 UTC
[DISCUSS] Metron documentation improvements
We recently had our site-book doc generation break due to our not including
it in the Travis build. The fix for a broken build seems simple enough -
add it to our build process and assuming it doesn't cause build timeout
issues, we should be good to go.
Beyond that, there are additional issues with the existing process. We have
a step in our PR review for validating that the docs are rendering
properly. I know I've gone back and corrected issues with broken images or
incorrectly rendering pages at least a few times now. On one hand, we might
say this is simply a matter of being better about validating documentation
during the review process. That may be true, but rather than fight upstream
like a salmon, I would prefer to simplify things, automate what we can, and
use technology to work with us. Based on this conversation on METRON-1950 -
(
https://lists.apache.org/thread.html/e2acf91efc5f51ba0e26d76b00ca02415d3c6ee0adee74a037ab2beb@%3Cdev.metron.apache.org%3E),
I'd like to open up a general convo about improvements to our documentation
generation.
*Current Issues:*
1. Duplicated effort - have to check pages render in Github and the
Doxia-generated site-book
2. Inconsistent model - what works for Github markdown may not work for
Doxia, and vice versa
3. Github is part of our workflow and easy to check, Doxia requires an
extra separate step - suffers unintentional bugs due to #2.
4. Images have to be manually added to the site rendering code for
copying to the "images" folder, and explicit src ref replacements have to
be included for all affected pages/links as well.
5. Page links and images are not validated - this currently requires
manual review and intervention during PR review and whenever we create a
new Metron release.
6. Failed site-book build is not validated. Broken build does not fail
Travis
*Options and Solutions:*
1. Otto has already brought up using Ascii doc as one option for solving
a number of these issues.
2. For issue #5, we can write a scraper that validates links or use
tooling like Cypress for this.
3. For issue #6, we can add site-book building to our Travis runs. It's
pretty quick to generate and will catch the more egregious rendering bugs.
I plan to look at this presently.
Mike
Re: [DISCUSS] Metron documentation improvements
Posted by Michael Miklavcic <mi...@gmail.com>.
I spent a bit of time getting this to run. I currently get a host of 404's
and am going to need to dig in a bit and see just what they are. In the
meantime, I'm going to commit https://github.com/apache/metron/pull/1310
and have created a Jira to track adding site-book link validation to our
Travis checks - https://issues.apache.org/jira/browse/METRON-1952. For the
curious, I landed on the following modified commands to get this running:
nohup python -m SimpleHTTPServer 2>&1 &
wget -Dlocalhost --spider -e robots=off -w 1 -r -p http://localhost:8000
2>&1 > /dev/null | grep -C 2 -E "404"
if [ $? -eq 0 ]; then
echo FAILED;
else
echo PASSED;
fi
http.server is the impl for Python 3. Simple tweak using SimpleHTTPServer
for Python 2.
Cheers,
Mike
On Thu, Dec 20, 2018 at 1:24 PM Casey Stella <ce...@gmail.com> wrote:
> You will want to add a -Dlocalhost to the wget to ensure you're not
> checking domains linked from our docs and turn travis into google. ;)
>
>
> On Thu, Dec 20, 2018 at 3:19 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
> > Well golly, I love this. I'll give that a whirl!
> >
> > On Thu, Dec 20, 2018 at 1:08 PM Casey Stella <ce...@gmail.com> wrote:
> >
> > > I definitely agree with option 3; that's a no-brainer IMO. I thought
> for
> > > sure this was already happening, honestly.
> > >
> > > As for 2, we could even script the broken link check by:
> > >
> > > - Serving up the site locally via python with `python -m
> http.server`
> > > from the site-book output directory
> > > - Looking at the output of wget --spider -e robots=off -w 1 -r -p
> > > http://localhost:8000 for 404s
> > >
> > >
> > > As for 1, I'm fine with it if that's where we want to go. I'm a +0
> > >
> > > On Thu, Dec 20, 2018 at 2:47 PM Michael Miklavcic <
> > > michael.miklavcic@gmail.com> wrote:
> > >
> > > > We recently had our site-book doc generation break due to our not
> > > including
> > > > it in the Travis build. The fix for a broken build seems simple
> enough
> > -
> > > > add it to our build process and assuming it doesn't cause build
> timeout
> > > > issues, we should be good to go.
> > > >
> > > > Beyond that, there are additional issues with the existing process.
> We
> > > have
> > > > a step in our PR review for validating that the docs are rendering
> > > > properly. I know I've gone back and corrected issues with broken
> images
> > > or
> > > > incorrectly rendering pages at least a few times now. On one hand, we
> > > might
> > > > say this is simply a matter of being better about validating
> > > documentation
> > > > during the review process. That may be true, but rather than fight
> > > upstream
> > > > like a salmon, I would prefer to simplify things, automate what we
> can,
> > > and
> > > > use technology to work with us. Based on this conversation on
> > > METRON-1950 -
> > > > (
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/e2acf91efc5f51ba0e26d76b00ca02415d3c6ee0adee74a037ab2beb@%3Cdev.metron.apache.org%3E
> > > > ),
> > > > I'd like to open up a general convo about improvements to our
> > > documentation
> > > > generation.
> > > >
> > > > *Current Issues:*
> > > >
> > > > 1. Duplicated effort - have to check pages render in Github and
> the
> > > > Doxia-generated site-book
> > > > 2. Inconsistent model - what works for Github markdown may not
> work
> > > for
> > > > Doxia, and vice versa
> > > > 3. Github is part of our workflow and easy to check, Doxia
> requires
> > an
> > > > extra separate step - suffers unintentional bugs due to #2.
> > > > 4. Images have to be manually added to the site rendering code for
> > > > copying to the "images" folder, and explicit src ref replacements
> > have
> > > > to
> > > > be included for all affected pages/links as well.
> > > > 5. Page links and images are not validated - this currently
> requires
> > > > manual review and intervention during PR review and whenever we
> > > create a
> > > > new Metron release.
> > > > 6. Failed site-book build is not validated. Broken build does not
> > fail
> > > > Travis
> > > >
> > > > *Options and Solutions:*
> > > >
> > > > 1. Otto has already brought up using Ascii doc as one option for
> > > solving
> > > > a number of these issues.
> > > > 2. For issue #5, we can write a scraper that validates links or
> use
> > > > tooling like Cypress for this.
> > > > 3. For issue #6, we can add site-book building to our Travis runs.
> > > It's
> > > > pretty quick to generate and will catch the more egregious
> rendering
> > > > bugs.
> > > > I plan to look at this presently.
> > > >
> > > > Mike
> > > >
> > >
> >
>
Re: [DISCUSS] Metron documentation improvements
Posted by Casey Stella <ce...@gmail.com>.
You will want to add a -Dlocalhost to the wget to ensure you're not
checking domains linked from our docs and turn travis into google. ;)
On Thu, Dec 20, 2018 at 3:19 PM Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:
> Well golly, I love this. I'll give that a whirl!
>
> On Thu, Dec 20, 2018 at 1:08 PM Casey Stella <ce...@gmail.com> wrote:
>
> > I definitely agree with option 3; that's a no-brainer IMO. I thought for
> > sure this was already happening, honestly.
> >
> > As for 2, we could even script the broken link check by:
> >
> > - Serving up the site locally via python with `python -m http.server`
> > from the site-book output directory
> > - Looking at the output of wget --spider -e robots=off -w 1 -r -p
> > http://localhost:8000 for 404s
> >
> >
> > As for 1, I'm fine with it if that's where we want to go. I'm a +0
> >
> > On Thu, Dec 20, 2018 at 2:47 PM Michael Miklavcic <
> > michael.miklavcic@gmail.com> wrote:
> >
> > > We recently had our site-book doc generation break due to our not
> > including
> > > it in the Travis build. The fix for a broken build seems simple enough
> -
> > > add it to our build process and assuming it doesn't cause build timeout
> > > issues, we should be good to go.
> > >
> > > Beyond that, there are additional issues with the existing process. We
> > have
> > > a step in our PR review for validating that the docs are rendering
> > > properly. I know I've gone back and corrected issues with broken images
> > or
> > > incorrectly rendering pages at least a few times now. On one hand, we
> > might
> > > say this is simply a matter of being better about validating
> > documentation
> > > during the review process. That may be true, but rather than fight
> > upstream
> > > like a salmon, I would prefer to simplify things, automate what we can,
> > and
> > > use technology to work with us. Based on this conversation on
> > METRON-1950 -
> > > (
> > >
> > >
> >
> https://lists.apache.org/thread.html/e2acf91efc5f51ba0e26d76b00ca02415d3c6ee0adee74a037ab2beb@%3Cdev.metron.apache.org%3E
> > > ),
> > > I'd like to open up a general convo about improvements to our
> > documentation
> > > generation.
> > >
> > > *Current Issues:*
> > >
> > > 1. Duplicated effort - have to check pages render in Github and the
> > > Doxia-generated site-book
> > > 2. Inconsistent model - what works for Github markdown may not work
> > for
> > > Doxia, and vice versa
> > > 3. Github is part of our workflow and easy to check, Doxia requires
> an
> > > extra separate step - suffers unintentional bugs due to #2.
> > > 4. Images have to be manually added to the site rendering code for
> > > copying to the "images" folder, and explicit src ref replacements
> have
> > > to
> > > be included for all affected pages/links as well.
> > > 5. Page links and images are not validated - this currently requires
> > > manual review and intervention during PR review and whenever we
> > create a
> > > new Metron release.
> > > 6. Failed site-book build is not validated. Broken build does not
> fail
> > > Travis
> > >
> > > *Options and Solutions:*
> > >
> > > 1. Otto has already brought up using Ascii doc as one option for
> > solving
> > > a number of these issues.
> > > 2. For issue #5, we can write a scraper that validates links or use
> > > tooling like Cypress for this.
> > > 3. For issue #6, we can add site-book building to our Travis runs.
> > It's
> > > pretty quick to generate and will catch the more egregious rendering
> > > bugs.
> > > I plan to look at this presently.
> > >
> > > Mike
> > >
> >
>
Re: [DISCUSS] Metron documentation improvements
Posted by Michael Miklavcic <mi...@gmail.com>.
Well golly, I love this. I'll give that a whirl!
On Thu, Dec 20, 2018 at 1:08 PM Casey Stella <ce...@gmail.com> wrote:
> I definitely agree with option 3; that's a no-brainer IMO. I thought for
> sure this was already happening, honestly.
>
> As for 2, we could even script the broken link check by:
>
> - Serving up the site locally via python with `python -m http.server`
> from the site-book output directory
> - Looking at the output of wget --spider -e robots=off -w 1 -r -p
> http://localhost:8000 for 404s
>
>
> As for 1, I'm fine with it if that's where we want to go. I'm a +0
>
> On Thu, Dec 20, 2018 at 2:47 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
> > We recently had our site-book doc generation break due to our not
> including
> > it in the Travis build. The fix for a broken build seems simple enough -
> > add it to our build process and assuming it doesn't cause build timeout
> > issues, we should be good to go.
> >
> > Beyond that, there are additional issues with the existing process. We
> have
> > a step in our PR review for validating that the docs are rendering
> > properly. I know I've gone back and corrected issues with broken images
> or
> > incorrectly rendering pages at least a few times now. On one hand, we
> might
> > say this is simply a matter of being better about validating
> documentation
> > during the review process. That may be true, but rather than fight
> upstream
> > like a salmon, I would prefer to simplify things, automate what we can,
> and
> > use technology to work with us. Based on this conversation on
> METRON-1950 -
> > (
> >
> >
> https://lists.apache.org/thread.html/e2acf91efc5f51ba0e26d76b00ca02415d3c6ee0adee74a037ab2beb@%3Cdev.metron.apache.org%3E
> > ),
> > I'd like to open up a general convo about improvements to our
> documentation
> > generation.
> >
> > *Current Issues:*
> >
> > 1. Duplicated effort - have to check pages render in Github and the
> > Doxia-generated site-book
> > 2. Inconsistent model - what works for Github markdown may not work
> for
> > Doxia, and vice versa
> > 3. Github is part of our workflow and easy to check, Doxia requires an
> > extra separate step - suffers unintentional bugs due to #2.
> > 4. Images have to be manually added to the site rendering code for
> > copying to the "images" folder, and explicit src ref replacements have
> > to
> > be included for all affected pages/links as well.
> > 5. Page links and images are not validated - this currently requires
> > manual review and intervention during PR review and whenever we
> create a
> > new Metron release.
> > 6. Failed site-book build is not validated. Broken build does not fail
> > Travis
> >
> > *Options and Solutions:*
> >
> > 1. Otto has already brought up using Ascii doc as one option for
> solving
> > a number of these issues.
> > 2. For issue #5, we can write a scraper that validates links or use
> > tooling like Cypress for this.
> > 3. For issue #6, we can add site-book building to our Travis runs.
> It's
> > pretty quick to generate and will catch the more egregious rendering
> > bugs.
> > I plan to look at this presently.
> >
> > Mike
> >
>
Re: [DISCUSS] Metron documentation improvements
Posted by Casey Stella <ce...@gmail.com>.
I definitely agree with option 3; that's a no-brainer IMO. I thought for
sure this was already happening, honestly.
As for 2, we could even script the broken link check by:
- Serving up the site locally via python with `python -m http.server`
from the site-book output directory
- Looking at the output of wget --spider -e robots=off -w 1 -r -p
http://localhost:8000 for 404s
As for 1, I'm fine with it if that's where we want to go. I'm a +0
On Thu, Dec 20, 2018 at 2:47 PM Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:
> We recently had our site-book doc generation break due to our not including
> it in the Travis build. The fix for a broken build seems simple enough -
> add it to our build process and assuming it doesn't cause build timeout
> issues, we should be good to go.
>
> Beyond that, there are additional issues with the existing process. We have
> a step in our PR review for validating that the docs are rendering
> properly. I know I've gone back and corrected issues with broken images or
> incorrectly rendering pages at least a few times now. On one hand, we might
> say this is simply a matter of being better about validating documentation
> during the review process. That may be true, but rather than fight upstream
> like a salmon, I would prefer to simplify things, automate what we can, and
> use technology to work with us. Based on this conversation on METRON-1950 -
> (
>
> https://lists.apache.org/thread.html/e2acf91efc5f51ba0e26d76b00ca02415d3c6ee0adee74a037ab2beb@%3Cdev.metron.apache.org%3E
> ),
> I'd like to open up a general convo about improvements to our documentation
> generation.
>
> *Current Issues:*
>
> 1. Duplicated effort - have to check pages render in Github and the
> Doxia-generated site-book
> 2. Inconsistent model - what works for Github markdown may not work for
> Doxia, and vice versa
> 3. Github is part of our workflow and easy to check, Doxia requires an
> extra separate step - suffers unintentional bugs due to #2.
> 4. Images have to be manually added to the site rendering code for
> copying to the "images" folder, and explicit src ref replacements have
> to
> be included for all affected pages/links as well.
> 5. Page links and images are not validated - this currently requires
> manual review and intervention during PR review and whenever we create a
> new Metron release.
> 6. Failed site-book build is not validated. Broken build does not fail
> Travis
>
> *Options and Solutions:*
>
> 1. Otto has already brought up using Ascii doc as one option for solving
> a number of these issues.
> 2. For issue #5, we can write a scraper that validates links or use
> tooling like Cypress for this.
> 3. For issue #6, we can add site-book building to our Travis runs. It's
> pretty quick to generate and will catch the more egregious rendering
> bugs.
> I plan to look at this presently.
>
> Mike
>