You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Casey Stella <ce...@gmail.com> on 2017/01/12 23:19:13 UTC
Re: [PROPOSAL] up-to-date versioned documentation
So, I do think this would be better than what we currently do. I like a
few things in particular:
- I don't like the wiki one bit.
- We have a LOT of documentation in the README.md's and it's sometimes
poorly organized
- I like a documentation preprocessing pipeline to be present. For
instance, a major ask is all of the stellar functions in one place. That's
solved by updating an index manually in the READMEs and keeping it in sync
with the annotation. I'd like to make a stellar annotation -> markdown
generator as part of the build and this would be nice for such a task.
My only concern is that the html generation/viewer seems like a fair amount
of engineering. Are you sure there isn't something easier that we could
conform to? I'm sure we aren't the only project in the world that has this
particular issue. Is there something like a maven site plugin or
something? Just a thought. I'll come back with more :)
Great ideas! Keep them coming!
Casey
On Thu, Jan 12, 2017 at 6:05 PM, Matt Foley <ma...@apache.org> wrote:
> We currently have three forms of documentation, with the following
> advantages and disadvantages:
>
> || Docs || Pro || Con ||
> | CWiki |
> Easy to edit, no special tools required, don't have to be a
> developer to contribute, google and wiki search |
> Not versioned, no review process, distant from the code, obsolete content
> tends to accumulate |
> | Site |
> Versioned and reviewed, only committers can edit, google search |
> Yet another arcane toolset must be learned, only web programmers
> feel comfortable contributing, "asf-site" branch not related to code
> versions, distant from the code, tends to go obsolete due to
> non-maintenance |
> | README.md |
> Versioned and reviewed, only committers can edit, tied to code
> versions, highly local to the code being documented |
> Non-developers don't know about them, may be scared by github, poor
> scoring in google search, no high-level presentation |
>
> Various discussion threads indicate the developer community likes
> README-based docs, and it's easy to see why from the above. I propose this
> extension to the README-based documentation, to address their disadvantages:
>
> 1. Produce a script that gathers the README.md files from all code
> subdirectories into a hierarchical list. The script would have an
> exclusion list for non-user-content, which at this point would consist of
> [site/*, build_utils/*]. The hierarchy would be sorted depth-first. The
> resulting hierarchical list at this time (with six added README files to
> complete the hierarchy) would be:
>
> ./README.md
> ./metron-analytics/README.md <== (need file here)
> ./metron-analytics/metron-maas-service/README.md
> ./metron-analytics/metron-profiler/README.md
> ./metron-analytics/metron-profiler-client/README.md
> ./metron-analytics/metron-statistics/README.md
> ./metron-deployment/README.md
> ./metron-deployment/amazon-ec2/README.md
> ./metron-deployment/packaging/README.md <== (need file here)
> ./metron-deployment/packaging/ambari/README.md <== (need file here)
> ./metron-deployment/packaging/docker/ansible-docker/README.md
> ./metron-deployment/packaging/docker/rpm-docker/README.md
> ./metron-deployment/packer-build/README.md
> ./metron-deployment/roles/ <== (need file here)
> ./metron-deployment/roles/kibana/README.md
> ./metron-deployment/roles/monit/README.md
> ./metron-deployment/roles/opentaxii/README.md
> ./metron-deployment/roles/pcap_replay/README.md
> ./metron-deployment/roles/sensor-test-mode/README.md
> ./metron-deployment/vagrant/README.md <== (need file here)
> ./metron-deployment/vagrant/codelab-platform/README.md
> ./metron-deployment/vagrant/fastcapa-test-platform/README.md
> ./metron-deployment/vagrant/full-dev-platform/README.md
> ./metron-deployment/vagrant/quick-dev-platform/README.md
> ./metron-platform/README.md
> ./metron-platform/metron-api/README.md
> ./metron-platform/metron-common/README.md
> ./metron-platform/metron-data-management/README.md
> ./metron-platform/metron-enrichment/README.md
> ./metron-platform/metron-indexing/README.md
> ./metron-platform/metron-management/README.md
> ./metron-platform/metron-parsers/README.md
> ./metron-platform/metron-pcap-backend/README.md
> ./metron-sensors/README.md <== (need file here)
> ./metron-sensors/fastcapa/README.md
> ./metron-sensors/pycapa/README.md
>
> 2. Arrange to run this script as part of the build process, and commit the
> resulting hierarchy list to someplace in the versioned and branched ./site/
> subdirectory.
>
> 3. Produce a "doc reader" web page that takes in this hierarchy of .md
> pages, and presents a LHS doc tree of links, and a main display area for a
> currently selected file. If we want to get fancy, this page would also
> provide: (a) telescoping (collapse/expand) of the doc tree; (b) floating
> next/prev/up/home buttons in the display area.
>
> #4. Add to this web page a pull-down menu that selects among all the
> release versions of Metron, and (if not running in the Apache site) a
> SNAPSHOT version for the current filesystem version (for developer
> preview). Let it re-write the file paths per release version to the proper
> release tag in github. This web page will therefore be
> version-independent. Put it in the asf-site branch of the Apache site, as
> the new "docs" sub-site from the home web page. Update the list of
> releases at each release, or if we want to get fancy, teach it to read the
> release tags from github.
>
> 5. As part of the release process, the release manager (a) assures the
> release is tagged in github with a consistent naming convention, and (b)
> submits the new hierarchy of links to google search (there's an api for
> that).
>
> 6. Deprecate the use of cwiki for anything but long-lived
> demonstrations/tutorials that are unlikely to go obsolete.
>
>
> Do folks feel this would be a good contribution to the visibility,
> timeliness, and usability of our docs?
> Is this an adequate solution for the current problems?
>
> Thanks,
> --Matt
>
>
>
Re: [PROPOSAL] up-to-date versioned documentation
Posted by Matt Foley <ma...@apache.org>.
The Spark docs sure are pretty. I suspect there’s a lot of person-weeks of work behind the content. I don’t know how hard it was to set up the infrastructure, but the instructions for generating the site mention an impressive list of tools needed.
The Falcon docs site seems much more straightforward, and reasonably pretty too. I can take a little time to understand it better.
Thanks,
--Matt
On 1/12/17, 6:19 PM, "Kyle Richardson" <ky...@gmail.com> wrote:
Matt, thanks for pulling this together. I completely agree that we need to
go all in on either cwiki or the README.md's. I think the wiki is poorly
updated and can cause confusion for new users and devs. My preference is
certainly for the README.md's.
I like your approach but also agree that we shouldn't need to roll our own
here. I really like the Spark documentation that Mike pointed out. Any way
we can duplicate/adapt their approach?
-Kyle
On Thu, Jan 12, 2017 at 7:19 PM, Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:
> Casey, Matt - These guys are using doxia
> https://github.com/apache/falcon/tree/master/docs
>
> Honestly, I kind of like Spark's approach -
> https://github.com/apache/spark/tree/master/docs
>
> Mike
>
> On Thu, Jan 12, 2017 at 4:48 PM, Matt Foley <ma...@apache.org> wrote:
>
> > I’m ambivalent; I think we’d end up tied to the doxia processing
> pipeline,
> > which is “yet another arcane toolset” to learn. Using .md as the input
> > format decreases the dependency, but we’d still be dependent on it.
> >
> > I had anticipated that the web page would be a write-once thing that
> would
> > be only a couple days for an experienced Web developer. But I was going
> to
> > get an estimate from some co-workers before actually trying to get it
> > implemented. And the script is a few hours of work with find and awk.
> >
> > On the other hand, doxia is certainly an expectable solution. Is setting
> > up that infrastructure less work than developing the web page? Or is it
> > actually just a matter of a few lines in pom.xml?
> >
> >
> > On 1/12/17, 3:24 PM, "Casey Stella" <ce...@gmail.com> wrote:
> >
> > Just a followup thought that's a bit more constructive, maybe we
> could
> > migrate the README.md's into a site directory and use doxia markdown
> > (example here <https://github.com/larrycai/doxia-markdown-demo>) to
> > generate the site as part of the build to resolve 1 through 3?
> >
> > On Thu, Jan 12, 2017 at 6:19 PM, Casey Stella <ce...@gmail.com>
> > wrote:
> >
> > > So, I do think this would be better than what we currently do. I
> > like a
> > > few things in particular:
> > >
> > > - I don't like the wiki one bit.
> > > - We have a LOT of documentation in the README.md's and it's
> > sometimes
> > > poorly organized
> > > - I like a documentation preprocessing pipeline to be present.
> > For
> > > instance, a major ask is all of the stellar functions in one
> > place. That's
> > > solved by updating an index manually in the READMEs and keeping
> > it in sync
> > > with the annotation. I'd like to make a stellar annotation ->
> > markdown
> > > generator as part of the build and this would be nice for such a
> > task.
> > >
> > > My only concern is that the html generation/viewer seems like a
> fair
> > > amount of engineering. Are you sure there isn't something easier
> > that we
> > > could conform to? I'm sure we aren't the only project in the world
> > that
> > > has this particular issue. Is there something like a maven site
> > plugin or
> > > something? Just a thought. I'll come back with more :)
> > >
> > > Great ideas! Keep them coming!
> > >
> > > Casey
> > >
> > > On Thu, Jan 12, 2017 at 6:05 PM, Matt Foley <ma...@apache.org>
> > wrote:
> > >
> > >> We currently have three forms of documentation, with the following
> > >> advantages and disadvantages:
> > >>
> > >> || Docs || Pro || Con ||
> > >> | CWiki |
> > >> Easy to edit, no special tools required, don't have to be a
> > >> developer to contribute, google and wiki search |
> > >> Not versioned, no review process, distant from the code, obsolete
> > content
> > >> tends to accumulate |
> > >> | Site |
> > >> Versioned and reviewed, only committers can edit, google
> > search |
> > >> Yet another arcane toolset must be learned, only web
> > programmers
> > >> feel comfortable contributing, "asf-site" branch not related to
> code
> > >> versions, distant from the code, tends to go obsolete due to
> > >> non-maintenance |
> > >> | README.md |
> > >> Versioned and reviewed, only committers can edit, tied to
> code
> > >> versions, highly local to the code being documented |
> > >> Non-developers don't know about them, may be scared by
> > github, poor
> > >> scoring in google search, no high-level presentation |
> > >>
> > >> Various discussion threads indicate the developer community likes
> > >> README-based docs, and it's easy to see why from the above. I
> > propose this
> > >> extension to the README-based documentation, to address their
> > disadvantages:
> > >>
> > >> 1. Produce a script that gathers the README.md files from all code
> > >> subdirectories into a hierarchical list. The script would have an
> > >> exclusion list for non-user-content, which at this point would
> > consist of
> > >> [site/*, build_utils/*]. The hierarchy would be sorted
> > depth-first. The
> > >> resulting hierarchical list at this time (with six added README
> > files to
> > >> complete the hierarchy) would be:
> > >>
> > >> ./README.md
> > >> ./metron-analytics/README.md <== (need file here)
> > >> ./metron-analytics/metron-maas-service/README.md
> > >> ./metron-analytics/metron-profiler/README.md
> > >> ./metron-analytics/metron-profiler-client/README.md
> > >> ./metron-analytics/metron-statistics/README.md
> > >> ./metron-deployment/README.md
> > >> ./metron-deployment/amazon-ec2/README.md
> > >> ./metron-deployment/packaging/README.md <== (need file here)
> > >> ./metron-deployment/packaging/ambari/README.md <== (need file
> here)
> > >> ./metron-deployment/packaging/docker/ansible-docker/README.md
> > >> ./metron-deployment/packaging/docker/rpm-docker/README.md
> > >> ./metron-deployment/packer-build/README.md
> > >> ./metron-deployment/roles/ <== (need file here)
> > >> ./metron-deployment/roles/kibana/README.md
> > >> ./metron-deployment/roles/monit/README.md
> > >> ./metron-deployment/roles/opentaxii/README.md
> > >> ./metron-deployment/roles/pcap_replay/README.md
> > >> ./metron-deployment/roles/sensor-test-mode/README.md
> > >> ./metron-deployment/vagrant/README.md <== (need file here)
> > >> ./metron-deployment/vagrant/codelab-platform/README.md
> > >> ./metron-deployment/vagrant/fastcapa-test-platform/README.md
> > >> ./metron-deployment/vagrant/full-dev-platform/README.md
> > >> ./metron-deployment/vagrant/quick-dev-platform/README.md
> > >> ./metron-platform/README.md
> > >> ./metron-platform/metron-api/README.md
> > >> ./metron-platform/metron-common/README.md
> > >> ./metron-platform/metron-data-management/README.md
> > >> ./metron-platform/metron-enrichment/README.md
> > >> ./metron-platform/metron-indexing/README.md
> > >> ./metron-platform/metron-management/README.md
> > >> ./metron-platform/metron-parsers/README.md
> > >> ./metron-platform/metron-pcap-backend/README.md
> > >> ./metron-sensors/README.md <== (need file here)
> > >> ./metron-sensors/fastcapa/README.md
> > >> ./metron-sensors/pycapa/README.md
> > >>
> > >> 2. Arrange to run this script as part of the build process, and
> > commit
> > >> the resulting hierarchy list to someplace in the versioned and
> > branched
> > >> ./site/ subdirectory.
> > >>
> > >> 3. Produce a "doc reader" web page that takes in this hierarchy of
> > .md
> > >> pages, and presents a LHS doc tree of links, and a main display
> > area for a
> > >> currently selected file. If we want to get fancy, this page would
> > also
> > >> provide: (a) telescoping (collapse/expand) of the doc tree; (b)
> > floating
> > >> next/prev/up/home buttons in the display area.
> > >>
> > >> #4. Add to this web page a pull-down menu that selects among all
> the
> > >> release versions of Metron, and (if not running in the Apache
> site)
> > a
> > >> SNAPSHOT version for the current filesystem version (for developer
> > >> preview). Let it re-write the file paths per release version to
> > the proper
> > >> release tag in github. This web page will therefore be
> > >> version-independent. Put it in the asf-site branch of the Apache
> > site, as
> > >> the new "docs" sub-site from the home web page. Update the list
> of
> > >> releases at each release, or if we want to get fancy, teach it to
> > read the
> > >> release tags from github.
> > >>
> > >> 5. As part of the release process, the release manager (a) assures
> > the
> > >> release is tagged in github with a consistent naming convention,
> > and (b)
> > >> submits the new hierarchy of links to google search (there's an
> api
> > for
> > >> that).
> > >>
> > >> 6. Deprecate the use of cwiki for anything but long-lived
> > >> demonstrations/tutorials that are unlikely to go obsolete.
> > >>
> > >>
> > >> Do folks feel this would be a good contribution to the visibility,
> > >> timeliness, and usability of our docs?
> > >> Is this an adequate solution for the current problems?
> > >>
> > >> Thanks,
> > >> --Matt
> > >>
> > >>
> > >>
> > >
> >
> >
> >
> >
>
Re: [PROPOSAL] up-to-date versioned documentation
Posted by Kyle Richardson <ky...@gmail.com>.
Matt, thanks for pulling this together. I completely agree that we need to
go all in on either cwiki or the README.md's. I think the wiki is poorly
updated and can cause confusion for new users and devs. My preference is
certainly for the README.md's.
I like your approach but also agree that we shouldn't need to roll our own
here. I really like the Spark documentation that Mike pointed out. Any way
we can duplicate/adapt their approach?
-Kyle
On Thu, Jan 12, 2017 at 7:19 PM, Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:
> Casey, Matt - These guys are using doxia
> https://github.com/apache/falcon/tree/master/docs
>
> Honestly, I kind of like Spark's approach -
> https://github.com/apache/spark/tree/master/docs
>
> Mike
>
> On Thu, Jan 12, 2017 at 4:48 PM, Matt Foley <ma...@apache.org> wrote:
>
> > I’m ambivalent; I think we’d end up tied to the doxia processing
> pipeline,
> > which is “yet another arcane toolset” to learn. Using .md as the input
> > format decreases the dependency, but we’d still be dependent on it.
> >
> > I had anticipated that the web page would be a write-once thing that
> would
> > be only a couple days for an experienced Web developer. But I was going
> to
> > get an estimate from some co-workers before actually trying to get it
> > implemented. And the script is a few hours of work with find and awk.
> >
> > On the other hand, doxia is certainly an expectable solution. Is setting
> > up that infrastructure less work than developing the web page? Or is it
> > actually just a matter of a few lines in pom.xml?
> >
> >
> > On 1/12/17, 3:24 PM, "Casey Stella" <ce...@gmail.com> wrote:
> >
> > Just a followup thought that's a bit more constructive, maybe we
> could
> > migrate the README.md's into a site directory and use doxia markdown
> > (example here <https://github.com/larrycai/doxia-markdown-demo>) to
> > generate the site as part of the build to resolve 1 through 3?
> >
> > On Thu, Jan 12, 2017 at 6:19 PM, Casey Stella <ce...@gmail.com>
> > wrote:
> >
> > > So, I do think this would be better than what we currently do. I
> > like a
> > > few things in particular:
> > >
> > > - I don't like the wiki one bit.
> > > - We have a LOT of documentation in the README.md's and it's
> > sometimes
> > > poorly organized
> > > - I like a documentation preprocessing pipeline to be present.
> > For
> > > instance, a major ask is all of the stellar functions in one
> > place. That's
> > > solved by updating an index manually in the READMEs and keeping
> > it in sync
> > > with the annotation. I'd like to make a stellar annotation ->
> > markdown
> > > generator as part of the build and this would be nice for such a
> > task.
> > >
> > > My only concern is that the html generation/viewer seems like a
> fair
> > > amount of engineering. Are you sure there isn't something easier
> > that we
> > > could conform to? I'm sure we aren't the only project in the world
> > that
> > > has this particular issue. Is there something like a maven site
> > plugin or
> > > something? Just a thought. I'll come back with more :)
> > >
> > > Great ideas! Keep them coming!
> > >
> > > Casey
> > >
> > > On Thu, Jan 12, 2017 at 6:05 PM, Matt Foley <ma...@apache.org>
> > wrote:
> > >
> > >> We currently have three forms of documentation, with the following
> > >> advantages and disadvantages:
> > >>
> > >> || Docs || Pro || Con ||
> > >> | CWiki |
> > >> Easy to edit, no special tools required, don't have to be a
> > >> developer to contribute, google and wiki search |
> > >> Not versioned, no review process, distant from the code, obsolete
> > content
> > >> tends to accumulate |
> > >> | Site |
> > >> Versioned and reviewed, only committers can edit, google
> > search |
> > >> Yet another arcane toolset must be learned, only web
> > programmers
> > >> feel comfortable contributing, "asf-site" branch not related to
> code
> > >> versions, distant from the code, tends to go obsolete due to
> > >> non-maintenance |
> > >> | README.md |
> > >> Versioned and reviewed, only committers can edit, tied to
> code
> > >> versions, highly local to the code being documented |
> > >> Non-developers don't know about them, may be scared by
> > github, poor
> > >> scoring in google search, no high-level presentation |
> > >>
> > >> Various discussion threads indicate the developer community likes
> > >> README-based docs, and it's easy to see why from the above. I
> > propose this
> > >> extension to the README-based documentation, to address their
> > disadvantages:
> > >>
> > >> 1. Produce a script that gathers the README.md files from all code
> > >> subdirectories into a hierarchical list. The script would have an
> > >> exclusion list for non-user-content, which at this point would
> > consist of
> > >> [site/*, build_utils/*]. The hierarchy would be sorted
> > depth-first. The
> > >> resulting hierarchical list at this time (with six added README
> > files to
> > >> complete the hierarchy) would be:
> > >>
> > >> ./README.md
> > >> ./metron-analytics/README.md <== (need file here)
> > >> ./metron-analytics/metron-maas-service/README.md
> > >> ./metron-analytics/metron-profiler/README.md
> > >> ./metron-analytics/metron-profiler-client/README.md
> > >> ./metron-analytics/metron-statistics/README.md
> > >> ./metron-deployment/README.md
> > >> ./metron-deployment/amazon-ec2/README.md
> > >> ./metron-deployment/packaging/README.md <== (need file here)
> > >> ./metron-deployment/packaging/ambari/README.md <== (need file
> here)
> > >> ./metron-deployment/packaging/docker/ansible-docker/README.md
> > >> ./metron-deployment/packaging/docker/rpm-docker/README.md
> > >> ./metron-deployment/packer-build/README.md
> > >> ./metron-deployment/roles/ <== (need file here)
> > >> ./metron-deployment/roles/kibana/README.md
> > >> ./metron-deployment/roles/monit/README.md
> > >> ./metron-deployment/roles/opentaxii/README.md
> > >> ./metron-deployment/roles/pcap_replay/README.md
> > >> ./metron-deployment/roles/sensor-test-mode/README.md
> > >> ./metron-deployment/vagrant/README.md <== (need file here)
> > >> ./metron-deployment/vagrant/codelab-platform/README.md
> > >> ./metron-deployment/vagrant/fastcapa-test-platform/README.md
> > >> ./metron-deployment/vagrant/full-dev-platform/README.md
> > >> ./metron-deployment/vagrant/quick-dev-platform/README.md
> > >> ./metron-platform/README.md
> > >> ./metron-platform/metron-api/README.md
> > >> ./metron-platform/metron-common/README.md
> > >> ./metron-platform/metron-data-management/README.md
> > >> ./metron-platform/metron-enrichment/README.md
> > >> ./metron-platform/metron-indexing/README.md
> > >> ./metron-platform/metron-management/README.md
> > >> ./metron-platform/metron-parsers/README.md
> > >> ./metron-platform/metron-pcap-backend/README.md
> > >> ./metron-sensors/README.md <== (need file here)
> > >> ./metron-sensors/fastcapa/README.md
> > >> ./metron-sensors/pycapa/README.md
> > >>
> > >> 2. Arrange to run this script as part of the build process, and
> > commit
> > >> the resulting hierarchy list to someplace in the versioned and
> > branched
> > >> ./site/ subdirectory.
> > >>
> > >> 3. Produce a "doc reader" web page that takes in this hierarchy of
> > .md
> > >> pages, and presents a LHS doc tree of links, and a main display
> > area for a
> > >> currently selected file. If we want to get fancy, this page would
> > also
> > >> provide: (a) telescoping (collapse/expand) of the doc tree; (b)
> > floating
> > >> next/prev/up/home buttons in the display area.
> > >>
> > >> #4. Add to this web page a pull-down menu that selects among all
> the
> > >> release versions of Metron, and (if not running in the Apache
> site)
> > a
> > >> SNAPSHOT version for the current filesystem version (for developer
> > >> preview). Let it re-write the file paths per release version to
> > the proper
> > >> release tag in github. This web page will therefore be
> > >> version-independent. Put it in the asf-site branch of the Apache
> > site, as
> > >> the new "docs" sub-site from the home web page. Update the list
> of
> > >> releases at each release, or if we want to get fancy, teach it to
> > read the
> > >> release tags from github.
> > >>
> > >> 5. As part of the release process, the release manager (a) assures
> > the
> > >> release is tagged in github with a consistent naming convention,
> > and (b)
> > >> submits the new hierarchy of links to google search (there's an
> api
> > for
> > >> that).
> > >>
> > >> 6. Deprecate the use of cwiki for anything but long-lived
> > >> demonstrations/tutorials that are unlikely to go obsolete.
> > >>
> > >>
> > >> Do folks feel this would be a good contribution to the visibility,
> > >> timeliness, and usability of our docs?
> > >> Is this an adequate solution for the current problems?
> > >>
> > >> Thanks,
> > >> --Matt
> > >>
> > >>
> > >>
> > >
> >
> >
> >
> >
>
Re: [PROPOSAL] up-to-date versioned documentation
Posted by Michael Miklavcic <mi...@gmail.com>.
Casey, Matt - These guys are using doxia
https://github.com/apache/falcon/tree/master/docs
Honestly, I kind of like Spark's approach -
https://github.com/apache/spark/tree/master/docs
Mike
On Thu, Jan 12, 2017 at 4:48 PM, Matt Foley <ma...@apache.org> wrote:
> I’m ambivalent; I think we’d end up tied to the doxia processing pipeline,
> which is “yet another arcane toolset” to learn. Using .md as the input
> format decreases the dependency, but we’d still be dependent on it.
>
> I had anticipated that the web page would be a write-once thing that would
> be only a couple days for an experienced Web developer. But I was going to
> get an estimate from some co-workers before actually trying to get it
> implemented. And the script is a few hours of work with find and awk.
>
> On the other hand, doxia is certainly an expectable solution. Is setting
> up that infrastructure less work than developing the web page? Or is it
> actually just a matter of a few lines in pom.xml?
>
>
> On 1/12/17, 3:24 PM, "Casey Stella" <ce...@gmail.com> wrote:
>
> Just a followup thought that's a bit more constructive, maybe we could
> migrate the README.md's into a site directory and use doxia markdown
> (example here <https://github.com/larrycai/doxia-markdown-demo>) to
> generate the site as part of the build to resolve 1 through 3?
>
> On Thu, Jan 12, 2017 at 6:19 PM, Casey Stella <ce...@gmail.com>
> wrote:
>
> > So, I do think this would be better than what we currently do. I
> like a
> > few things in particular:
> >
> > - I don't like the wiki one bit.
> > - We have a LOT of documentation in the README.md's and it's
> sometimes
> > poorly organized
> > - I like a documentation preprocessing pipeline to be present.
> For
> > instance, a major ask is all of the stellar functions in one
> place. That's
> > solved by updating an index manually in the READMEs and keeping
> it in sync
> > with the annotation. I'd like to make a stellar annotation ->
> markdown
> > generator as part of the build and this would be nice for such a
> task.
> >
> > My only concern is that the html generation/viewer seems like a fair
> > amount of engineering. Are you sure there isn't something easier
> that we
> > could conform to? I'm sure we aren't the only project in the world
> that
> > has this particular issue. Is there something like a maven site
> plugin or
> > something? Just a thought. I'll come back with more :)
> >
> > Great ideas! Keep them coming!
> >
> > Casey
> >
> > On Thu, Jan 12, 2017 at 6:05 PM, Matt Foley <ma...@apache.org>
> wrote:
> >
> >> We currently have three forms of documentation, with the following
> >> advantages and disadvantages:
> >>
> >> || Docs || Pro || Con ||
> >> | CWiki |
> >> Easy to edit, no special tools required, don't have to be a
> >> developer to contribute, google and wiki search |
> >> Not versioned, no review process, distant from the code, obsolete
> content
> >> tends to accumulate |
> >> | Site |
> >> Versioned and reviewed, only committers can edit, google
> search |
> >> Yet another arcane toolset must be learned, only web
> programmers
> >> feel comfortable contributing, "asf-site" branch not related to code
> >> versions, distant from the code, tends to go obsolete due to
> >> non-maintenance |
> >> | README.md |
> >> Versioned and reviewed, only committers can edit, tied to code
> >> versions, highly local to the code being documented |
> >> Non-developers don't know about them, may be scared by
> github, poor
> >> scoring in google search, no high-level presentation |
> >>
> >> Various discussion threads indicate the developer community likes
> >> README-based docs, and it's easy to see why from the above. I
> propose this
> >> extension to the README-based documentation, to address their
> disadvantages:
> >>
> >> 1. Produce a script that gathers the README.md files from all code
> >> subdirectories into a hierarchical list. The script would have an
> >> exclusion list for non-user-content, which at this point would
> consist of
> >> [site/*, build_utils/*]. The hierarchy would be sorted
> depth-first. The
> >> resulting hierarchical list at this time (with six added README
> files to
> >> complete the hierarchy) would be:
> >>
> >> ./README.md
> >> ./metron-analytics/README.md <== (need file here)
> >> ./metron-analytics/metron-maas-service/README.md
> >> ./metron-analytics/metron-profiler/README.md
> >> ./metron-analytics/metron-profiler-client/README.md
> >> ./metron-analytics/metron-statistics/README.md
> >> ./metron-deployment/README.md
> >> ./metron-deployment/amazon-ec2/README.md
> >> ./metron-deployment/packaging/README.md <== (need file here)
> >> ./metron-deployment/packaging/ambari/README.md <== (need file here)
> >> ./metron-deployment/packaging/docker/ansible-docker/README.md
> >> ./metron-deployment/packaging/docker/rpm-docker/README.md
> >> ./metron-deployment/packer-build/README.md
> >> ./metron-deployment/roles/ <== (need file here)
> >> ./metron-deployment/roles/kibana/README.md
> >> ./metron-deployment/roles/monit/README.md
> >> ./metron-deployment/roles/opentaxii/README.md
> >> ./metron-deployment/roles/pcap_replay/README.md
> >> ./metron-deployment/roles/sensor-test-mode/README.md
> >> ./metron-deployment/vagrant/README.md <== (need file here)
> >> ./metron-deployment/vagrant/codelab-platform/README.md
> >> ./metron-deployment/vagrant/fastcapa-test-platform/README.md
> >> ./metron-deployment/vagrant/full-dev-platform/README.md
> >> ./metron-deployment/vagrant/quick-dev-platform/README.md
> >> ./metron-platform/README.md
> >> ./metron-platform/metron-api/README.md
> >> ./metron-platform/metron-common/README.md
> >> ./metron-platform/metron-data-management/README.md
> >> ./metron-platform/metron-enrichment/README.md
> >> ./metron-platform/metron-indexing/README.md
> >> ./metron-platform/metron-management/README.md
> >> ./metron-platform/metron-parsers/README.md
> >> ./metron-platform/metron-pcap-backend/README.md
> >> ./metron-sensors/README.md <== (need file here)
> >> ./metron-sensors/fastcapa/README.md
> >> ./metron-sensors/pycapa/README.md
> >>
> >> 2. Arrange to run this script as part of the build process, and
> commit
> >> the resulting hierarchy list to someplace in the versioned and
> branched
> >> ./site/ subdirectory.
> >>
> >> 3. Produce a "doc reader" web page that takes in this hierarchy of
> .md
> >> pages, and presents a LHS doc tree of links, and a main display
> area for a
> >> currently selected file. If we want to get fancy, this page would
> also
> >> provide: (a) telescoping (collapse/expand) of the doc tree; (b)
> floating
> >> next/prev/up/home buttons in the display area.
> >>
> >> #4. Add to this web page a pull-down menu that selects among all the
> >> release versions of Metron, and (if not running in the Apache site)
> a
> >> SNAPSHOT version for the current filesystem version (for developer
> >> preview). Let it re-write the file paths per release version to
> the proper
> >> release tag in github. This web page will therefore be
> >> version-independent. Put it in the asf-site branch of the Apache
> site, as
> >> the new "docs" sub-site from the home web page. Update the list of
> >> releases at each release, or if we want to get fancy, teach it to
> read the
> >> release tags from github.
> >>
> >> 5. As part of the release process, the release manager (a) assures
> the
> >> release is tagged in github with a consistent naming convention,
> and (b)
> >> submits the new hierarchy of links to google search (there's an api
> for
> >> that).
> >>
> >> 6. Deprecate the use of cwiki for anything but long-lived
> >> demonstrations/tutorials that are unlikely to go obsolete.
> >>
> >>
> >> Do folks feel this would be a good contribution to the visibility,
> >> timeliness, and usability of our docs?
> >> Is this an adequate solution for the current problems?
> >>
> >> Thanks,
> >> --Matt
> >>
> >>
> >>
> >
>
>
>
>
Re: [PROPOSAL] up-to-date versioned documentation
Posted by Matt Foley <ma...@apache.org>.
I’m ambivalent; I think we’d end up tied to the doxia processing pipeline, which is “yet another arcane toolset” to learn. Using .md as the input format decreases the dependency, but we’d still be dependent on it.
I had anticipated that the web page would be a write-once thing that would be only a couple days for an experienced Web developer. But I was going to get an estimate from some co-workers before actually trying to get it implemented. And the script is a few hours of work with find and awk.
On the other hand, doxia is certainly an expectable solution. Is setting up that infrastructure less work than developing the web page? Or is it actually just a matter of a few lines in pom.xml?
On 1/12/17, 3:24 PM, "Casey Stella" <ce...@gmail.com> wrote:
Just a followup thought that's a bit more constructive, maybe we could
migrate the README.md's into a site directory and use doxia markdown
(example here <https://github.com/larrycai/doxia-markdown-demo>) to
generate the site as part of the build to resolve 1 through 3?
On Thu, Jan 12, 2017 at 6:19 PM, Casey Stella <ce...@gmail.com> wrote:
> So, I do think this would be better than what we currently do. I like a
> few things in particular:
>
> - I don't like the wiki one bit.
> - We have a LOT of documentation in the README.md's and it's sometimes
> poorly organized
> - I like a documentation preprocessing pipeline to be present. For
> instance, a major ask is all of the stellar functions in one place. That's
> solved by updating an index manually in the READMEs and keeping it in sync
> with the annotation. I'd like to make a stellar annotation -> markdown
> generator as part of the build and this would be nice for such a task.
>
> My only concern is that the html generation/viewer seems like a fair
> amount of engineering. Are you sure there isn't something easier that we
> could conform to? I'm sure we aren't the only project in the world that
> has this particular issue. Is there something like a maven site plugin or
> something? Just a thought. I'll come back with more :)
>
> Great ideas! Keep them coming!
>
> Casey
>
> On Thu, Jan 12, 2017 at 6:05 PM, Matt Foley <ma...@apache.org> wrote:
>
>> We currently have three forms of documentation, with the following
>> advantages and disadvantages:
>>
>> || Docs || Pro || Con ||
>> | CWiki |
>> Easy to edit, no special tools required, don't have to be a
>> developer to contribute, google and wiki search |
>> Not versioned, no review process, distant from the code, obsolete content
>> tends to accumulate |
>> | Site |
>> Versioned and reviewed, only committers can edit, google search |
>> Yet another arcane toolset must be learned, only web programmers
>> feel comfortable contributing, "asf-site" branch not related to code
>> versions, distant from the code, tends to go obsolete due to
>> non-maintenance |
>> | README.md |
>> Versioned and reviewed, only committers can edit, tied to code
>> versions, highly local to the code being documented |
>> Non-developers don't know about them, may be scared by github, poor
>> scoring in google search, no high-level presentation |
>>
>> Various discussion threads indicate the developer community likes
>> README-based docs, and it's easy to see why from the above. I propose this
>> extension to the README-based documentation, to address their disadvantages:
>>
>> 1. Produce a script that gathers the README.md files from all code
>> subdirectories into a hierarchical list. The script would have an
>> exclusion list for non-user-content, which at this point would consist of
>> [site/*, build_utils/*]. The hierarchy would be sorted depth-first. The
>> resulting hierarchical list at this time (with six added README files to
>> complete the hierarchy) would be:
>>
>> ./README.md
>> ./metron-analytics/README.md <== (need file here)
>> ./metron-analytics/metron-maas-service/README.md
>> ./metron-analytics/metron-profiler/README.md
>> ./metron-analytics/metron-profiler-client/README.md
>> ./metron-analytics/metron-statistics/README.md
>> ./metron-deployment/README.md
>> ./metron-deployment/amazon-ec2/README.md
>> ./metron-deployment/packaging/README.md <== (need file here)
>> ./metron-deployment/packaging/ambari/README.md <== (need file here)
>> ./metron-deployment/packaging/docker/ansible-docker/README.md
>> ./metron-deployment/packaging/docker/rpm-docker/README.md
>> ./metron-deployment/packer-build/README.md
>> ./metron-deployment/roles/ <== (need file here)
>> ./metron-deployment/roles/kibana/README.md
>> ./metron-deployment/roles/monit/README.md
>> ./metron-deployment/roles/opentaxii/README.md
>> ./metron-deployment/roles/pcap_replay/README.md
>> ./metron-deployment/roles/sensor-test-mode/README.md
>> ./metron-deployment/vagrant/README.md <== (need file here)
>> ./metron-deployment/vagrant/codelab-platform/README.md
>> ./metron-deployment/vagrant/fastcapa-test-platform/README.md
>> ./metron-deployment/vagrant/full-dev-platform/README.md
>> ./metron-deployment/vagrant/quick-dev-platform/README.md
>> ./metron-platform/README.md
>> ./metron-platform/metron-api/README.md
>> ./metron-platform/metron-common/README.md
>> ./metron-platform/metron-data-management/README.md
>> ./metron-platform/metron-enrichment/README.md
>> ./metron-platform/metron-indexing/README.md
>> ./metron-platform/metron-management/README.md
>> ./metron-platform/metron-parsers/README.md
>> ./metron-platform/metron-pcap-backend/README.md
>> ./metron-sensors/README.md <== (need file here)
>> ./metron-sensors/fastcapa/README.md
>> ./metron-sensors/pycapa/README.md
>>
>> 2. Arrange to run this script as part of the build process, and commit
>> the resulting hierarchy list to someplace in the versioned and branched
>> ./site/ subdirectory.
>>
>> 3. Produce a "doc reader" web page that takes in this hierarchy of .md
>> pages, and presents a LHS doc tree of links, and a main display area for a
>> currently selected file. If we want to get fancy, this page would also
>> provide: (a) telescoping (collapse/expand) of the doc tree; (b) floating
>> next/prev/up/home buttons in the display area.
>>
>> #4. Add to this web page a pull-down menu that selects among all the
>> release versions of Metron, and (if not running in the Apache site) a
>> SNAPSHOT version for the current filesystem version (for developer
>> preview). Let it re-write the file paths per release version to the proper
>> release tag in github. This web page will therefore be
>> version-independent. Put it in the asf-site branch of the Apache site, as
>> the new "docs" sub-site from the home web page. Update the list of
>> releases at each release, or if we want to get fancy, teach it to read the
>> release tags from github.
>>
>> 5. As part of the release process, the release manager (a) assures the
>> release is tagged in github with a consistent naming convention, and (b)
>> submits the new hierarchy of links to google search (there's an api for
>> that).
>>
>> 6. Deprecate the use of cwiki for anything but long-lived
>> demonstrations/tutorials that are unlikely to go obsolete.
>>
>>
>> Do folks feel this would be a good contribution to the visibility,
>> timeliness, and usability of our docs?
>> Is this an adequate solution for the current problems?
>>
>> Thanks,
>> --Matt
>>
>>
>>
>
Re: [PROPOSAL] up-to-date versioned documentation
Posted by Casey Stella <ce...@gmail.com>.
Just a followup thought that's a bit more constructive, maybe we could
migrate the README.md's into a site directory and use doxia markdown
(example here <https://github.com/larrycai/doxia-markdown-demo>) to
generate the site as part of the build to resolve 1 through 3?
On Thu, Jan 12, 2017 at 6:19 PM, Casey Stella <ce...@gmail.com> wrote:
> So, I do think this would be better than what we currently do. I like a
> few things in particular:
>
> - I don't like the wiki one bit.
> - We have a LOT of documentation in the README.md's and it's sometimes
> poorly organized
> - I like a documentation preprocessing pipeline to be present. For
> instance, a major ask is all of the stellar functions in one place. That's
> solved by updating an index manually in the READMEs and keeping it in sync
> with the annotation. I'd like to make a stellar annotation -> markdown
> generator as part of the build and this would be nice for such a task.
>
> My only concern is that the html generation/viewer seems like a fair
> amount of engineering. Are you sure there isn't something easier that we
> could conform to? I'm sure we aren't the only project in the world that
> has this particular issue. Is there something like a maven site plugin or
> something? Just a thought. I'll come back with more :)
>
> Great ideas! Keep them coming!
>
> Casey
>
> On Thu, Jan 12, 2017 at 6:05 PM, Matt Foley <ma...@apache.org> wrote:
>
>> We currently have three forms of documentation, with the following
>> advantages and disadvantages:
>>
>> || Docs || Pro || Con ||
>> | CWiki |
>> Easy to edit, no special tools required, don't have to be a
>> developer to contribute, google and wiki search |
>> Not versioned, no review process, distant from the code, obsolete content
>> tends to accumulate |
>> | Site |
>> Versioned and reviewed, only committers can edit, google search |
>> Yet another arcane toolset must be learned, only web programmers
>> feel comfortable contributing, "asf-site" branch not related to code
>> versions, distant from the code, tends to go obsolete due to
>> non-maintenance |
>> | README.md |
>> Versioned and reviewed, only committers can edit, tied to code
>> versions, highly local to the code being documented |
>> Non-developers don't know about them, may be scared by github, poor
>> scoring in google search, no high-level presentation |
>>
>> Various discussion threads indicate the developer community likes
>> README-based docs, and it's easy to see why from the above. I propose this
>> extension to the README-based documentation, to address their disadvantages:
>>
>> 1. Produce a script that gathers the README.md files from all code
>> subdirectories into a hierarchical list. The script would have an
>> exclusion list for non-user-content, which at this point would consist of
>> [site/*, build_utils/*]. The hierarchy would be sorted depth-first. The
>> resulting hierarchical list at this time (with six added README files to
>> complete the hierarchy) would be:
>>
>> ./README.md
>> ./metron-analytics/README.md <== (need file here)
>> ./metron-analytics/metron-maas-service/README.md
>> ./metron-analytics/metron-profiler/README.md
>> ./metron-analytics/metron-profiler-client/README.md
>> ./metron-analytics/metron-statistics/README.md
>> ./metron-deployment/README.md
>> ./metron-deployment/amazon-ec2/README.md
>> ./metron-deployment/packaging/README.md <== (need file here)
>> ./metron-deployment/packaging/ambari/README.md <== (need file here)
>> ./metron-deployment/packaging/docker/ansible-docker/README.md
>> ./metron-deployment/packaging/docker/rpm-docker/README.md
>> ./metron-deployment/packer-build/README.md
>> ./metron-deployment/roles/ <== (need file here)
>> ./metron-deployment/roles/kibana/README.md
>> ./metron-deployment/roles/monit/README.md
>> ./metron-deployment/roles/opentaxii/README.md
>> ./metron-deployment/roles/pcap_replay/README.md
>> ./metron-deployment/roles/sensor-test-mode/README.md
>> ./metron-deployment/vagrant/README.md <== (need file here)
>> ./metron-deployment/vagrant/codelab-platform/README.md
>> ./metron-deployment/vagrant/fastcapa-test-platform/README.md
>> ./metron-deployment/vagrant/full-dev-platform/README.md
>> ./metron-deployment/vagrant/quick-dev-platform/README.md
>> ./metron-platform/README.md
>> ./metron-platform/metron-api/README.md
>> ./metron-platform/metron-common/README.md
>> ./metron-platform/metron-data-management/README.md
>> ./metron-platform/metron-enrichment/README.md
>> ./metron-platform/metron-indexing/README.md
>> ./metron-platform/metron-management/README.md
>> ./metron-platform/metron-parsers/README.md
>> ./metron-platform/metron-pcap-backend/README.md
>> ./metron-sensors/README.md <== (need file here)
>> ./metron-sensors/fastcapa/README.md
>> ./metron-sensors/pycapa/README.md
>>
>> 2. Arrange to run this script as part of the build process, and commit
>> the resulting hierarchy list to someplace in the versioned and branched
>> ./site/ subdirectory.
>>
>> 3. Produce a "doc reader" web page that takes in this hierarchy of .md
>> pages, and presents a LHS doc tree of links, and a main display area for a
>> currently selected file. If we want to get fancy, this page would also
>> provide: (a) telescoping (collapse/expand) of the doc tree; (b) floating
>> next/prev/up/home buttons in the display area.
>>
>> #4. Add to this web page a pull-down menu that selects among all the
>> release versions of Metron, and (if not running in the Apache site) a
>> SNAPSHOT version for the current filesystem version (for developer
>> preview). Let it re-write the file paths per release version to the proper
>> release tag in github. This web page will therefore be
>> version-independent. Put it in the asf-site branch of the Apache site, as
>> the new "docs" sub-site from the home web page. Update the list of
>> releases at each release, or if we want to get fancy, teach it to read the
>> release tags from github.
>>
>> 5. As part of the release process, the release manager (a) assures the
>> release is tagged in github with a consistent naming convention, and (b)
>> submits the new hierarchy of links to google search (there's an api for
>> that).
>>
>> 6. Deprecate the use of cwiki for anything but long-lived
>> demonstrations/tutorials that are unlikely to go obsolete.
>>
>>
>> Do folks feel this would be a good contribution to the visibility,
>> timeliness, and usability of our docs?
>> Is this an adequate solution for the current problems?
>>
>> Thanks,
>> --Matt
>>
>>
>>
>