You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streams.apache.org by Steve Blackmon <sb...@apache.org> on 2016/04/27 22:28:48 UTC
Re: Streams Web Site, Maven Sites, misc Documentation - proposed next steps
All,
I think we are pretty close to pulling all of the work on STREAMS-401 (new web page) together into a pretty slick package. Thanks for your patience and feedback.
Some new additions include diagrams on the architecture page and within many modules depicting streams components and data flow among them and with third-party systems, a 5-step tutorial to run a useful stream, consistent look-and-feel across all three repos, integrated breadcrumbs, and rudimentary github and twitter integrations.
Feel free to browse these links to see where we’re at.
http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-master/index.html
http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-project/index.html
http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-examples/index.html
All of the menu link now resolve, or will resolve as soon as everything is promoted. If you get any Not Available errors that you CAN’T resolve by substituting ‘latest’ for ‘0.3-incubating-SNAPSHOT’ in the url, please let me know.
Additionally if you identify any major issues or have any last suggestions please let me know before I kickoff the process to merge the STREAMS-401 branches and promote the site later this week.
Steve Blackmon
sblackmon@apache.org
On Thu, Mar 03, 2016 at 9:48 AM Ate Douma
<
mailto:Ate Douma <at...@douma.nu>
> wrote:
+1 to everything below :)
Good stuff Steve!
On 2016-03-01 23:50, Steve Blackmon wrote:
> Following up:
>
> I took a pass at cleaning up the release markdown files and they now look
> mostly the same as they did under apache cms.
>
> I also confirmed that site-deploy can write to a sub-directory of
> streams.incubator.apache.org using scm:svn capability.
>
>
http://streams.incubator.apache.org
/site/0.3-incubating-SNAPSHOT/streams-master/index.html
>
> Recall that streams-project and streams-examples each publish schema
> artifacts to their maven sites, so maintaining prior versions is important.
> streams-master doesn't currently, but since it's a parent of those other
> two and prior experience has led to believe we're better off using the same
> scheme and setting the streams-master
in the pom to be a sibling of
> streams-project and streams-example rather that publishing it to a
> different URL entirely. This is why the version an artifactId are being
> included in the above url.
>
> Brings up two issues -
> resolving the latest release and/or snapshot version at any given time
> ensuring a visitor to streams.incubator.apache.org lands on the
> streams-master site for that version
>
> While symlinks are a viable way to resolving the latest release and/or
> snapshot version while keeping previous release versions around
>
> i.e. forwarding
>
http://streams.incubator.apache.org
/site/latest/streams-master/index.html
> to
>
http://streams.incubator.apache.org
/site/0.3-incubating-SNAPSHOT/streams-master/index.html
>
> they won't help us forward
http://streams.incubator.apache.org
to a
> subdirectory of itself.
>
> Good news is apache cms supports .htaccess and the following snippet can.
>
> RewriteEngine *on*
>
> Redirect /
>
http://streams.incubator.apache.org
/site/latest/streams-master/index.html
> Feel free to confirm by visiting streams.staging.incubator.org (you'll be
> redirected to the output of the maven site-deploy)
>
> *I ask that everyone on the list review and send me any concerns or
> feedback they want incorporated before this goes live on
> streams.incubator.apache.org <
http://streams.incubator.apache.org
>.*
>
> Procedurally, the way I expect that to happen is:
> - I'll submit a PR to streams-master/master
> - That PR will merge
> - Update the streams-master jenkins job to run site-deploy, and do the
> needful to get it the necessary credentials
> - Going forward all builds of streams-master/master will publish to
> site/${version}/streams-master
> - Add a few more instructions to jenkins to maintain the symlinks and
> .htaccess
> - If there are any manual steps required to oversee, document that on the
> site itself.
>
> Then, we'll move on to hooking streams-project and streams-examples up to
> this new site (as children with an integrated breadcrumb) and to jenkins CD.
>
> Steve Blackmon
>
mailto:sblackmon@apache.org
>
> On Mon, Feb 29, 2016 at 5:19 PM, Steve Blackmon <
mailto:sblackmon@apache.org
>
> wrote:
>
>> One thing which I think can 'spice up' the overview page a bit more would
>>> be a
>>> diagram providing some very high-level interaction between data sources
>>> and
>>> destinations and the streams role between them.
>>
>>
>> I agree - a diagram depicting streams mediating between sources and
>> destination on the landing page is appropriate. I'll work on this soon.
>>
>> And before replacing the current site several existing pages need to be
>>> moved to
>>> the new one. But once those are done I think this is great improvement.
>>
>>
>> I've pushed a new version (to the source and site links from my prior
>> message) that brings most of the existing content along, using the
>> reporting-info plugin to generate the HTML where possible.
>>
>> On a few of the pages - SCM, CI, and Dependency Info - the output of the
>> maven plugin simply isn't adequate for a multi-repository project such as
>> this, so those are now managed within src/site/markdown
>>
>> Release Setup and Release Process need some cleanup but all of the content
>> has been migrated.
>>
>> Other report-info pages - such as Dependencies, Dependency Management,
>> Dependency Convergence, JavaDocs, Test JavaDocs, - don't make sense at the
>> streams-master level but do exist and are useful within streams-project,
>> streams-examples, and many of their respective sub-modules.
>>
>> In theory those additional report items will show up when navigating into
>> the site hierarchy of the sub-projects (links to each sub-project are
>> present along the header), which will inherit the structure, look-and-feel,
>> and breadcrumb of this site revision. Naturally these changes must be
>> deployed to streams.incubator.apache.org and sub-project pom.xml and
>> site_en.xml will need some tweaking but I'm optimistic it will all fit
>> together nicely.
>>
>> Steve Blackmon
>>
mailto:sblackmon@apache.org
>>
>> On Sat, Feb 27, 2016 at 2:27 AM, Ate Douma <
mailto:ate@douma.nu
> wrote:
>>
>>> On 2016-02-25 23:50, Steve Blackmon wrote:
>>>
>>>> Hello streamers,
>>>>
>>>> I finally consolidated some materials I've been working on into a
>>>> nice-looking thing that is a viable candidate to replace the current
>>>> streams web page. Please take a look and let me know what you think.
>>>> Sometime
>>>> soon I expect to submit a vote to take down the previous site and replace
>>>> it with something derived from this version.
>>>>
>>>> The proposed new site is temporarily being hosted at:
>>>>
>>>>
http://people.apache.org/~sblackmon/0.3-incubating-SNAPSHOT/streams-master/index.html
>>>>
>>>> I placed all of the content into markdown files which are checked into a
>>>> branch of streams-master. This approach allows us to author the site
>>>> using
>>>> markdown, package and deploy it with maven-site, and manage changes using
>>>> git.
>>>>
>>>
>>> This looks cool!
>>>
>>> I like the new overview and faq pages, definitely more concrete about
>>> Streams.
>>>
>>> One thing which I think can 'spice up' the overview page a bit more would
>>> be a
>>> diagram providing some very high-level interaction between data sources
>>> and
>>> destinations and the streams role between them.
>>>
>>> And before replacing the current site several existing pages need to be
>>> moved to
>>> the new one. But once those are done I think this is great improvement.
>>>
>>>
>>>> If you'd like to submit specific changes feel free to do so as
>>>> pull-requests to:
>>>>
https://github.com/steveblackmon/incubator-streams-master.git
>>>>
>>>> Referring back to the original list of goals, some are addressed in whole
>>>> or in part with this revision, others not just yet.
>>>>
>>>> ADDRESSED IN WHOLE:
>>>> - the website isn't providing any practical guidance *why* or *how* to
>>>> use
>>>> Streams
>>>> - there is no public javadoc or other technical documentation
>>>> linked/hosted
>>>> - there is a 'wiki (coming soon)' link which never has been activated
>>>>
>>>> ADDRESSED IN PART:
>>>> - its missing concrete examples and recognizable use-cases for which
>>>> Streams might
>>>> be used, nor comparison against other solutions, and why Streams would
>>>> be a
>>>> good/better solution
>>>> - the Architecture page only has some basic wording but provides no
>>>> insight
>>>> whatsoever about the concrete implementation and certainly not its
>>>> architecture
>>>>
>>>> This is just a start. Communication on the list has been dead lately,
>>>> but
>>>> I'm hoping to change that beginning with this effort to explain what is
>>>> unique and valuable about Streams, and why. Hopefully this take on the
>>>> project will inspire others to get on-board and help out in the coming
>>>> months.
>>>>
>>>
>>> +1, good work!
>>>
>>>
>>>
>>>> Steve Blackmon
>>>>
mailto:sblackmon@apache.org
>>>>
>>>> On Thu, Jan 7, 2016 at 10:49 AM, Ate Douma <
mailto:ate@douma.nu
> wrote:
>>>>
>>>> Hi Steve,
>>>>>
>>>>> Cool. I don't have much extra input or ideas right now, but looking
>>>>> forward to the steps taken you proposed below.
>>>>> And about getting out a next release first soon, if it is not
>>>>> side-tracking these other steps much, sure +1
>>>>>
>>>>> Ate
>>>>>
>>>>> On 2016-01-05 15:35, Steve Blackmon wrote:
>>>>>
>>>>> All,
>>>>>>
>>>>>> Ate gave us a reality check during our board report in December and
>>>>>> enumerated some of the most prominent ways our documentation currently
>>>>>> falls short - of Apache community standards and of any reasonable
>>>>>> useful-ness test. I've responded in-line below - please kick in any
>>>>>> ideas
>>>>>> you have.
>>>>>>
>>>>>>
>>>>>> - the website isn't providing any practical guidance *why* or *how* to
>>>>>> use
>>>>>> Streams
>>>>>>
>>>>>>
>>>>>> I agree. This is the content that belongs on the streams website
>>>>>> landing
>>>>>> page (high-level) or linked from the body (the details). I will take a
>>>>>> first shot at this and circulate for feedback.
>>>>>>
>>>>>> - its missing concrete examples and recognizable use-cases for which
>>>>>> Streams might be used, nor comparison against other solutions, and why
>>>>>> Streams would be a good/better solution
>>>>>>
>>>>>>
>>>>>> The examples documentation site [1] are meant to serve as both concrete
>>>>>> examples and recognizable use-cases, but there are a few problems with
>>>>>> the
>>>>>> current setup.
>>>>>>
>>>>>> 1. The examples site lacks plain-language READMEs for the root
>>>>>> module
>>>>>> and the first children of the root module (the runtime-specific
>>>>>> aggregators).
>>>>>> 2. The examples site isn't linked from the landing page of the
>>>>>> website.
>>>>>> 3. There is no permanent url for latest/current.
>>>>>> 4. There hasn't been a formal release of this repo.
>>>>>> 5. There are broken links within the markdown of some examples.
>>>>>> 6. The plain-language descriptions of the examples themselves
>>>>>> could be
>>>>>>
>>>>>> improved.
>>>>>>
>>>>>> All of these can be addressed (easily I think) and I'm happy to take
>>>>>> them
>>>>>> on.
>>>>>>
>>>>>> Additionally, a page (or several) comparing streams to frameworks such
>>>>>> as
>>>>>> Storm, Spark, Samza, Flink, etc... and describing how they are
>>>>>> different/complementary would be valuable. These questions come up a
>>>>>> lot.
>>>>>>
>>>>>> - the Architecture page only has some basic wording but provides no
>>>>>> insight
>>>>>> whatsoever about the concrete implementation and certainly not its
>>>>>> architecture
>>>>>>
>>>>>>
>>>>>> I think this page needs either an multi-page outline format heavily
>>>>>> linked
>>>>>> into READMEs and interfaces throughout the streams repos, or a system
>>>>>> diagram depicting a simple streams deployment alongside the vocabulary.
>>>>>> Maybe both as several pages.
>>>>>>
>>>>>> - there is no public javadoc or other technical documentation
>>>>>> linked/hosted
>>>>>>
>>>>>>
>>>>>> We do generate java-docs for releases and snapshots. We should add a
>>>>>> link
>>>>>> to the root page. We've made progress but there are still many classes
>>>>>> lacking any javadoc header. We should reiterate a commitment to fixing
>>>>>> that and a schedule for deprecating and deleting any class that doesn't
>>>>>> still serve a distinct purpose.
>>>>>>
>>>>>> An index of the many resource files (json/xml schemas, conversion
>>>>>> rules,
>>>>>> shell scripts, etc...) published to the project and examples release
>>>>>> should
>>>>>> also placed on the main website. Ideally comprehensive, but if not at
>>>>>> least enough to show a developer that they exist and can be
>>>>>> hard-linked to
>>>>>> within their own project resources.
>>>>>>
>>>>>>
>>>>>> - there is a 'wiki (coming soon)' link which never has been activated
>>>>>>
>>>>>>
>>>>>> This link (actually most of streams.incubator.apache.org) predates the
>>>>>> regular publication of the streams-project maven site where READMEs and
>>>>>> other resources are hosted. Personally I'd prefer to see documentation
>>>>>> improving within the code-base rather than in a separate wiki in the
>>>>>> near-term and suggest we just delete this link.
>>>>>>
>>>>>> - the mailing list is NOT used for any form of discussion/information
>>>>>> other
>>>>>> than JIRA and Git(hub) notifications
>>>>>> - especially the latter doesn't give a newbie any indication what is
>>>>>> going
>>>>>> on, nor how to (start) interact
>>>>>>
>>>>>> This is unfortunate and must change for the community to grow. I know
>>>>>> I'm
>>>>>> guilty of having ad-hoc off-list discussions with my team that lead to
>>>>>> new
>>>>>> stories and pull requests. Mea culpa, I resolve to stop doing this in
>>>>>> 2016.
>>>>>>
>>>>>> - there are no blog posts (that I am aware of) around using/trying out
>>>>>> Streams either
>>>>>>
>>>>>>
>>>>>> This is a critical missing piece of the puzzle. Static HTML via apache
>>>>>>
httpd
is not the ideal platform for hosting a blog but some other
>>>>>> projects
>>>>>> have made it work. I'd like us to replace 'Latest News' with a
>>>>>> navigable,
>>>>>> indexable set of blog posts going forward, where release summaries and
>>>>>> new
>>>>>> examples can be published, and 'in-the-wild' mentions of the project
>>>>>> can
>>>>>> be
>>>>>> aggregated.
>>>>>>
>>>>>>
>>>>>> And I personally would prefer the dev@ list to be much more /
>>>>>> primarily
>>>>>> used for actual human interaction, not just these notifications from
>>>>>> JIRA
>>>>>> and Github.
>>>>>> Maybe change the configuration to delegate (some of) these to the
>>>>>> commit@
>>>>>> list instead?
>>>>>>
>>>>>>
>>>>>> I agree. AFAICT though
mailto:commit@streams.incubator.apache.org
does not
>>>>>> exist. Per
http://apache.org/dev/committers.html#mail
we should hold
>>>>>> a
>>>>>> vote to create it. I'll open that vote today.
>>>>>>
>>>>>> What I suggest is to for a while stop (or largely postpone) working on
>>>>>> code
>>>>>> but instead working on any/all of the above points first...
>>>>>>
>>>>>>
>>>>>> I agree that making the website helpful is the top project priority.
>>>>>>
>>>>>> Even working on a next release IMO is far less important than first
>>>>>> building up some explanation *why* a release of Stream is needed.
>>>>>>
>>>>>>
>>>>>> I agree that these concerns should be addressed before the next
>>>>>> release.
>>>>>> The main reason for releasing soon is that per [4] there are 36 issues
>>>>>> resolved since the 0.2 release, including quite a few bugs and
>>>>>> improvements
>>>>>> related to stability, deployment flexibility, or likely to impact the
>>>>>> experience of a new developer.
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>>
>>>>>>
http://streams.incubator.apache.org
/site/0.2-incubating-SNAPSHOT/streams-examples/
>>>>>> [2]
>>>>>>
>>>>>>
>>>>>>
http://streams.incubator.apache.org
/site/0.2-incubating/streams-project/apidocs/index.html
>>>>>> [3]
>>>>>>
>>>>>>
>>>>>>
http://streams.incubator.apache.org
/site/0.2-incubating/streams-project/index.html
>>>>>> [4]
>>>>>>
>>>>>>
>>>>>>
https://issues.apache.org/jira/issues/?jql=project%20%3D%20STREAMS%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20in%20unreleasedVersions()
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Streams Web Site, Maven Sites, misc Documentation - proposed next steps
Posted by Steve Blackmon <sb...@apache.org>.
Thanks Matt, Renato, Ate!
I did squeeze two simple diagrams on the architecture page -
http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-master/architecture.html
Most of the modules have a diagram depicting the components they contain and the types of data each support:
http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-project/streams-contrib/streams-provider-instagram/index.html
And each of the examples has an end-to-end data flow diagram:
http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-examples/streams-examples-local/twitter-userstream-elasticsearch/index.html
Always open to ideas for improvement, happy to jump on any typos or other problems that turn up.
Cheers
Steve Blackmon
sblackmon@apache.org
On Mon, May 02, 2016 at 10:43 AM Matt Franklin
<
mailto:Matt Franklin <m....@gmail.com>
> wrote:
On Fri, Apr 29, 2016 at 6:22 PM Steve Blackmon <
mailto:sblackmon@apache.org
> wrote:
The new website is now up at
http://streams.incubator.apache.org
. It’s not perfect but it’s good enough and way better than it’s ever been. It’s got a basic connection to Google Analytics - if you’d like to be added to the organization just ask.
This looks great! I agree with Ate on a diagram or other visually engaging component.
I’m going to put together a blog post about website revamp and everything that there for the project blog at
https://blogs.apache.org/streams/
Hopefully we’ll also be able to tweet it out from @ApacheStreams if the PMC is able to get control of it (WIP)
Steve Blackmon
mailto:sblackmon@apache.org
On Wed, Apr 27, 2016 at 3:28 PM Steve Blackmon
<
mailto:Steve+Blackmon+%3Csblackmon@apache.org%3E
> wrote:
All,
I think we are pretty close to pulling all of the work on STREAMS-401 (new web page) together into a pretty slick package. Thanks for your patience and feedback.
Some new additions include diagrams on the architecture page and within many modules depicting streams components and data flow among them and with third-party systems, a 5-step tutorial to run a useful stream, consistent look-and-feel across all three repos, integrated breadcrumbs, and rudimentary github and twitter integrations.
Feel free to browse these links to see where we’re at.
http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-master/index.html
http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-project/index.html
http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-examples/index.html
All of the menu link now resolve, or will resolve as soon as everything is promoted. If you get any Not Available errors that you CAN’T resolve by substituting ‘latest’ for ‘0.3-incubating-SNAPSHOT’ in the url, please let me know.
Additionally if you identify any major issues or have any last suggestions please let me know before I kickoff the process to merge the STREAMS-401 branches and promote the site later this week.
Steve Blackmon
mailto:sblackmon@apache.org
On Thu, Mar 03, 2016 at 9:48 AM Ate Douma
<
mailto:Ate+Douma+%3Cate@douma.nu%3E
> wrote:
+1 to everything below :)
Good stuff Steve!
On 2016-03-01 23:50, Steve Blackmon wrote:
> Following up:
>
> I took a pass at cleaning up the release markdown files and they now look
> mostly the same as they did under apache cms.
>
> I also confirmed that site-deploy can write to a sub-directory of
>
http://streams.incubator.apache.org
using scm:svn capability.
>
>
http://streams.incubator.apache.org
/site/0.3-incubating-SNAPSHOT/streams-master/index.html
>
> Recall that streams-project and streams-examples each publish schema
> artifacts to their maven sites, so maintaining prior versions is important.
> streams-master doesn't currently, but since it's a parent of those other
> two and prior experience has led to believe we're better off using the same
> scheme and setting the streams-master
in the pom to be a sibling of
> streams-project and streams-example rather that publishing it to a
> different URL entirely. This is why the version an artifactId are being
> included in the above url.
>
> Brings up two issues -
> resolving the latest release and/or snapshot version at any given time
> ensuring a visitor to
http://streams.incubator.apache.org
lands on the
> streams-master site for that version
>
> While symlinks are a viable way to resolving the latest release and/or
> snapshot version while keeping previous release versions around
>
> i.e. forwarding
>
http://streams.incubator.apache.org
/site/latest/streams-master/index.html
> to
>
http://streams.incubator.apache.org
/site/0.3-incubating-SNAPSHOT/streams-master/index.html
>
> they won't help us forward
http://streams.incubator.apache.org
to a
> subdirectory of itself.
>
> Good news is apache cms supports .htaccess and the following snippet can.
>
> RewriteEngine *on*
>
> Redirect /
>
http://streams.incubator.apache.org
/site/latest/streams-master/index.html
> Feel free to confirm by visiting
http://streams.staging.incubator.org
(you'll be
> redirected to the output of the maven site-deploy)
>
> *I ask that everyone on the list review and send me any concerns or
> feedback they want incorporated before this goes live on
>
http://streams.incubator.apache.org
<
http://streams.incubator.apache.org
>.*
>
> Procedurally, the way I expect that to happen is:
> - I'll submit a PR to streams-master/master
> - That PR will merge
> - Update the streams-master jenkins job to run site-deploy, and do the
> needful to get it the necessary credentials
> - Going forward all builds of streams-master/master will publish to
> site/${version}/streams-master
> - Add a few more instructions to jenkins to maintain the symlinks and
> .htaccess
> - If there are any manual steps required to oversee, document that on the
> site itself.
>
> Then, we'll move on to hooking streams-project and streams-examples up to
> this new site (as children with an integrated breadcrumb) and to jenkins CD.
>
> Steve Blackmon
>
mailto:sblackmon@apache.org
>
> On Mon, Feb 29, 2016 at 5:19 PM, Steve Blackmon <
mailto:sblackmon@apache.org
>
> wrote:
>
>> One thing which I think can 'spice up' the overview page a bit more would
>>> be a
>>> diagram providing some very high-level interaction between data sources
>>> and
>>> destinations and the streams role between them.
>>
>>
>> I agree - a diagram depicting streams mediating between sources and
>> destination on the landing page is appropriate. I'll work on this soon.
>>
>> And before replacing the current site several existing pages need to be
>>> moved to
>>> the new one. But once those are done I think this is great improvement.
>>
>>
>> I've pushed a new version (to the source and site links from my prior
>> message) that brings most of the existing content along, using the
>> reporting-info plugin to generate the HTML where possible.
>>
>> On a few of the pages - SCM, CI, and Dependency Info - the output of the
>> maven plugin simply isn't adequate for a multi-repository project such as
>> this, so those are now managed within src/site/markdown
>>
>> Release Setup and Release Process need some cleanup but all of the content
>> has been migrated.
>>
>> Other report-info pages - such as Dependencies, Dependency Management,
>> Dependency Convergence, JavaDocs, Test JavaDocs, - don't make sense at the
>> streams-master level but do exist and are useful within streams-project,
>> streams-examples, and many of their respective sub-modules.
>>
>> In theory those additional report items will show up when navigating into
>> the site hierarchy of the sub-projects (links to each sub-project are
>> present along the header), which will inherit the structure, look-and-feel,
>> and breadcrumb of this site revision. Naturally these changes must be
>> deployed to
http://streams.incubator.apache.org
and sub-project pom.xml and
>> site_en.xml will need some tweaking but I'm optimistic it will all fit
>> together nicely.
>>
>> Steve Blackmon
>>
mailto:sblackmon@apache.org
>>
>> On Sat, Feb 27, 2016 at 2:27 AM, Ate Douma <
mailto:ate@douma.nu
> wrote:
>>
>>> On 2016-02-25 23:50, Steve Blackmon wrote:
>>>
>>>> Hello streamers,
>>>>
>>>> I finally consolidated some materials I've been working on into a
>>>> nice-looking thing that is a viable candidate to replace the current
>>>> streams web page. Please take a look and let me know what you think.
>>>> Sometime
>>>> soon I expect to submit a vote to take down the previous site and replace
>>>> it with something derived from this version.
>>>>
>>>> The proposed new site is temporarily being hosted at:
>>>>
>>>>
http://people.apache.org/~sblackmon/0.3-incubating-SNAPSHOT/streams-master/index.html
>>>>
>>>> I placed all of the content into markdown files which are checked into a
>>>> branch of streams-master. This approach allows us to author the site
>>>> using
>>>> markdown, package and deploy it with maven-site, and manage changes using
>>>> git.
>>>>
>>>
>>> This looks cool!
>>>
>>> I like the new overview and faq pages, definitely more concrete about
>>> Streams.
>>>
>>> One thing which I think can 'spice up' the overview page a bit more would
>>> be a
>>> diagram providing some very high-level interaction between data sources
>>> and
>>> destinations and the streams role between them.
>>>
>>> And before replacing the current site several existing pages need to be
>>> moved to
>>> the new one. But once those are done I think this is great improvement.
>>>
>>>
>>>> If you'd like to submit specific changes feel free to do so as
>>>> pull-requests to:
>>>>
https://github.com/steveblackmon/incubator-streams-master.git
>>>>
>>>> Referring back to the original list of goals, some are addressed in whole
>>>> or in part with this revision, others not just yet.
>>>>
>>>> ADDRESSED IN WHOLE:
>>>> - the website isn't providing any practical guidance *why* or *how* to
>>>> use
>>>> Streams
>>>> - there is no public javadoc or other technical documentation
>>>> linked/hosted
>>>> - there is a 'wiki (coming soon)' link which never has been activated
>>>>
>>>> ADDRESSED IN PART:
>>>> - its missing concrete examples and recognizable use-cases for which
>>>> Streams might
>>>> be used, nor comparison against other solutions, and why Streams would
>>>> be a
>>>> good/better solution
>>>> - the Architecture page only has some basic wording but provides no
>>>> insight
>>>> whatsoever about the concrete implementation and certainly not its
>>>> architecture
>>>>
>>>> This is just a start. Communication on the list has been dead lately,
>>>> but
>>>> I'm hoping to change that beginning with this effort to explain what is
>>>> unique and valuable about Streams, and why. Hopefully this take on the
>>>> project will inspire others to get on-board and help out in the coming
>>>> months.
>>>>
>>>
>>> +1, good work!
>>>
>>>
>>>
>>>> Steve Blackmon
>>>>
mailto:sblackmon@apache.org
>>>>
>>>> On Thu, Jan 7, 2016 at 10:49 AM, Ate Douma <
mailto:ate@douma.nu
> wrote:
>>>>
>>>> Hi Steve,
>>>>>
>>>>> Cool. I don't have much extra input or ideas right now, but looking
>>>>> forward to the steps taken you proposed below.
>>>>> And about getting out a next release first soon, if it is not
>>>>> side-tracking these other steps much, sure +1
>>>>>
>>>>> Ate
>>>>>
>>>>> On 2016-01-05 15:35, Steve Blackmon wrote:
>>>>>
>>>>> All,
>>>>>>
>>>>>> Ate gave us a reality check during our board report in December and
>>>>>> enumerated some of the most prominent ways our documentation currently
>>>>>> falls short - of Apache community standards and of any reasonable
>>>>>> useful-ness test. I've responded in-line below - please kick in any
>>>>>> ideas
>>>>>> you have.
>>>>>>
>>>>>>
>>>>>> - the website isn't providing any practical guidance *why* or *how* to
>>>>>> use
>>>>>> Streams
>>>>>>
>>>>>>
>>>>>> I agree. This is the content that belongs on the streams website
>>>>>> landing
>>>>>> page (high-level) or linked from the body (the details). I will take a
>>>>>> first shot at this and circulate for feedback.
>>>>>>
>>>>>> - its missing concrete examples and recognizable use-cases for which
>>>>>> Streams might be used, nor comparison against other solutions, and why
>>>>>> Streams would be a good/better solution
>>>>>>
>>>>>>
>>>>>> The examples documentation site [1] are meant to serve as both concrete
>>>>>> examples and recognizable use-cases, but there are a few problems with
>>>>>> the
>>>>>> current setup.
>>>>>>
>>>>>> 1. The examples site lacks plain-language READMEs for the root
>>>>>> module
>>>>>> and the first children of the root module (the runtime-specific
>>>>>> aggregators).
>>>>>> 2. The examples site isn't linked from the landing page of the
>>>>>> website.
>>>>>> 3. There is no permanent url for latest/current.
>>>>>> 4. There hasn't been a formal release of this repo.
>>>>>> 5. There are broken links within the markdown of some examples.
>>>>>> 6. The plain-language descriptions of the examples themselves
>>>>>> could be
>>>>>>
>>>>>> improved.
>>>>>>
>>>>>> All of these can be addressed (easily I think) and I'm happy to take
>>>>>> them
>>>>>> on.
>>>>>>
>>>>>> Additionally, a page (or several) comparing streams to frameworks such
>>>>>> as
>>>>>> Storm, Spark, Samza, Flink, etc... and describing how they are
>>>>>> different/complementary would be valuable. These questions come up a
>>>>>> lot.
>>>>>>
>>>>>> - the Architecture page only has some basic wording but provides no
>>>>>> insight
>>>>>> whatsoever about the concrete implementation and certainly not its
>>>>>> architecture
>>>>>>
>>>>>>
>>>>>> I think this page needs either an multi-page outline format heavily
>>>>>> linked
>>>>>> into READMEs and interfaces throughout the streams repos, or a system
>>>>>> diagram depicting a simple streams deployment alongside the vocabulary.
>>>>>> Maybe both as several pages.
>>>>>>
>>>>>> - there is no public javadoc or other technical documentation
>>>>>> linked/hosted
>>>>>>
>>>>>>
>>>>>> We do generate java-docs for releases and snapshots. We should add a
>>>>>> link
>>>>>> to the root page. We've made progress but there are still many classes
>>>>>> lacking any javadoc header. We should reiterate a commitment to fixing
>>>>>> that and a schedule for deprecating and deleting any class that doesn't
>>>>>> still serve a distinct purpose.
>>>>>>
>>>>>> An index of the many resource files (json/xml schemas, conversion
>>>>>> rules,
>>>>>> shell scripts, etc...) published to the project and examples release
>>>>>> should
>>>>>> also placed on the main website. Ideally comprehensive, but if not at
>>>>>> least enough to show a developer that they exist and can be
>>>>>> hard-linked to
>>>>>> within their own project resources.
>>>>>>
>>>>>>
>>>>>> - there is a 'wiki (coming soon)' link which never has been activated
>>>>>>
>>>>>>
>>>>>> This link (actually most of
http://streams.incubator.apache.org
) predates the
>>>>>> regular publication of the streams-project maven site where READMEs and
>>>>>> other resources are hosted. Personally I'd prefer to see documentation
>>>>>> improving within the code-base rather than in a separate wiki in the
>>>>>> near-term and suggest we just delete this link.
>>>>>>
>>>>>> - the mailing list is NOT used for any form of discussion/information
>>>>>> other
>>>>>> than JIRA and Git(hub) notifications
>>>>>> - especially the latter doesn't give a newbie any indication what is
>>>>>> going
>>>>>> on, nor how to (start) interact
>>>>>>
>>>>>> This is unfortunate and must change for the community to grow. I know
>>>>>> I'm
>>>>>> guilty of having ad-hoc off-list discussions with my team that lead to
>>>>>> new
>>>>>> stories and pull requests. Mea culpa, I resolve to stop doing this in
>>>>>> 2016.
>>>>>>
>>>>>> - there are no blog posts (that I am aware of) around using/trying out
>>>>>> Streams either
>>>>>>
>>>>>>
>>>>>> This is a critical missing piece of the puzzle. Static HTML via apache
>>>>>>
http://httpd
is not the ideal platform for hosting a blog but some other
>>>>>> projects
>>>>>> have made it work. I'd like us to replace 'Latest News' with a
>>>>>> navigable,
>>>>>> indexable set of blog posts going forward, where release summaries and
>>>>>> new
>>>>>> examples can be published, and 'in-the-wild' mentions of the project
>>>>>> can
>>>>>> be
>>>>>> aggregated.
>>>>>>
>>>>>>
>>>>>> And I personally would prefer the dev@ list to be much more /
>>>>>> primarily
>>>>>> used for actual human interaction, not just these notifications from
>>>>>> JIRA
>>>>>> and Github.
>>>>>> Maybe change the configuration to delegate (some of) these to the
>>>>>> commit@
>>>>>> list instead?
>>>>>>
>>>>>>
>>>>>> I agree. AFAICT though
mailto:commit@streams.incubator.apache.org
does not
>>>>>> exist. Per
http://apache.org/dev/committers.html#mail
we should hold
>>>>>> a
>>>>>> vote to create it. I'll open that vote today.
>>>>>>
>>>>>> What I suggest is to for a while stop (or largely postpone) working on
>>>>>> code
>>>>>> but instead working on any/all of the above points first...
>>>>>>
>>>>>>
>>>>>> I agree that making the website helpful is the top project priority.
>>>>>>
>>>>>> Even working on a next release IMO is far less important than first
>>>>>> building up some explanation *why* a release of Stream is needed.
>>>>>>
>>>>>>
>>>>>> I agree that these concerns should be addressed before the next
>>>>>> release.
>>>>>> The main reason for releasing soon is that per [4] there are 36 issues
>>>>>> resolved since the 0.2 release, including quite a few bugs and
>>>>>> improvements
>>>>>> related to stability, deployment flexibility, or likely to impact the
>>>>>> experience of a new developer.
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>>
>>>>>>
http://streams.incubator.apache.org
/site/0.2-incubating-SNAPSHOT/streams-examples/
>>>>>> [2]
>>>>>>
>>>>>>
>>>>>>
http://streams.incubator.apache.org
/site/0.2-incubating/streams-project/apidocs/index.html
>>>>>> [3]
>>>>>>
>>>>>>
>>>>>>
http://streams.incubator.apache.org
/site/0.2-incubating/streams-project/index.html
>>>>>> [4]
>>>>>>
>>>>>>
>>>>>>
https://issues.apache.org/jira/issues/?jql=project%20%3D%20STREAMS%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20in%20unreleasedVersions()
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Streams Web Site, Maven Sites, misc Documentation - proposed next steps
Posted by Matt Franklin <m....@gmail.com>.
On Fri, Apr 29, 2016 at 6:22 PM Steve Blackmon <sb...@apache.org> wrote:
> The new website is now up at streams.incubator.apache.org. It’s not
> perfect but it’s good enough and way better than it’s ever been. It’s got
> a basic connection to Google Analytics - if you’d like to be added to the
> organization just ask.
This looks great! I agree with Ate on a diagram or other visually engaging
component.
>
> I’m going to put together a blog post about website revamp and everything
> that there for the project blog at https://blogs.apache.org/streams/
>
> Hopefully we’ll also be able to tweet it out from @ApacheStreams if the
> PMC is able to get control of it (WIP)
>
>
> Steve Blackmon
> sblackmon@apache.org
>
> On Wed, Apr 27, 2016 at 3:28 PM Steve Blackmon <Steve Blackmon
> <Steve+Blackmon+%3Csblackmon@apache.org%3E>> wrote:
>
>> All,
>>
>> I think we are pretty close to pulling all of the work on STREAMS-401
>> (new web page) together into a pretty slick package. Thanks for your
>> patience and feedback.
>>
>> Some new additions include diagrams on the architecture page and within
>> many modules depicting streams components and data flow among them and with
>> third-party systems, a 5-step tutorial to run a useful stream, consistent
>> look-and-feel across all three repos, integrated breadcrumbs, and
>> rudimentary github and twitter integrations.
>>
>> Feel free to browse these links to see where we’re at.
>>
>>
>> http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-master/index.html
>>
>> http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-project/index.html
>>
>> http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-examples/index.html
>>
>> All of the menu link now resolve, or will resolve as soon as everything
>> is promoted. If you get any Not Available errors that you CAN’T resolve by
>> substituting ‘latest’ for ‘0.3-incubating-SNAPSHOT’ in the url, please let
>> me know.
>>
>> Additionally if you identify any major issues or have any last
>> suggestions please let me know before I kickoff the process to merge the
>> STREAMS-401 branches and promote the site later this week.
>>
>> Steve Blackmon
>> sblackmon@apache.org
>>
>> On Thu, Mar 03, 2016 at 9:48 AM Ate Douma <Ate Douma
>> <Ate+Douma+%3Cate@douma.nu%3E>> wrote:
>>
>>> +1 to everything below :)
>>>
>>> Good stuff Steve!
>>>
>>> On 2016-03-01 23:50, Steve Blackmon wrote:
>>> > Following up:
>>> >
>>> > I took a pass at cleaning up the release markdown files and they now
>>> look
>>> > mostly the same as they did under apache cms.
>>> >
>>> > I also confirmed that site-deploy can write to a sub-directory of
>>> > streams.incubator.apache.org using scm:svn capability.
>>> >
>>> > http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-master/index.html
>>>
>>> >
>>> > Recall that streams-project and streams-examples each publish schema
>>> > artifacts to their maven sites, so maintaining prior versions is
>>> important.
>>> > streams-master doesn't currently, but since it's a parent of those
>>> other
>>> > two and prior experience has led to believe we're better off using the
>>> same
>>> > scheme and setting the streams-master in the pom to be a sibling of
>>> > streams-project and streams-example rather that publishing it to a
>>> > different URL entirely. This is why the version an artifactId are
>>> being
>>> > included in the above url.
>>> >
>>> > Brings up two issues -
>>> > resolving the latest release and/or snapshot version at any given time
>>> > ensuring a visitor to streams.incubator.apache.org lands on the
>>> > streams-master site for that version
>>> >
>>> > While symlinks are a viable way to resolving the latest release and/or
>>> > snapshot version while keeping previous release versions around
>>> >
>>> > i.e. forwarding
>>> > http://streams.incubator.apache.org/site/latest/streams-master/index.html
>>>
>>> > to
>>> > http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-master/index.html
>>>
>>> >
>>> > they won't help us forward http://streams.incubator.apache.org to a
>>> > subdirectory of itself.
>>> >
>>> > Good news is apache cms supports .htaccess and the following snippet
>>> can.
>>> >
>>> > RewriteEngine *on*
>>> >
>>> > Redirect /
>>> > http://streams.incubator.apache.org/site/latest/streams-master/index.html
>>>
>>> > Feel free to confirm by visiting streams.staging.incubator.org
>>> (you'll be
>>> > redirected to the output of the maven site-deploy)
>>> >
>>> > *I ask that everyone on the list review and send me any concerns or
>>> > feedback they want incorporated before this goes live on
>>> > streams.incubator.apache.org <http://streams.incubator.apache.org>.*
>>> >
>>> > Procedurally, the way I expect that to happen is:
>>> > - I'll submit a PR to streams-master/master
>>> > - That PR will merge
>>> > - Update the streams-master jenkins job to run site-deploy, and do the
>>> > needful to get it the necessary credentials
>>> > - Going forward all builds of streams-master/master will publish to
>>> > site/${version}/streams-master
>>> > - Add a few more instructions to jenkins to maintain the symlinks and
>>> > .htaccess
>>> > - If there are any manual steps required to oversee, document that on
>>> the
>>> > site itself.
>>> >
>>> > Then, we'll move on to hooking streams-project and streams-examples up
>>> to
>>> > this new site (as children with an integrated breadcrumb) and to
>>> jenkins CD.
>>> >
>>> > Steve Blackmon
>>> > sblackmon@apache.org
>>> >
>>> > On Mon, Feb 29, 2016 at 5:19 PM, Steve Blackmon <sb...@apache.org>
>>>
>>> > wrote:
>>> >
>>> >> One thing which I think can 'spice up' the overview page a bit more
>>> would
>>> >>> be a
>>> >>> diagram providing some very high-level interaction between data
>>> sources
>>> >>> and
>>> >>> destinations and the streams role between them.
>>> >>
>>> >>
>>> >> I agree - a diagram depicting streams mediating between sources and
>>> >> destination on the landing page is appropriate. I'll work on this
>>> soon.
>>> >>
>>> >> And before replacing the current site several existing pages need to
>>> be
>>> >>> moved to
>>> >>> the new one. But once those are done I think this is great
>>> improvement.
>>> >>
>>> >>
>>> >> I've pushed a new version (to the source and site links from my prior
>>> >> message) that brings most of the existing content along, using the
>>> >> reporting-info plugin to generate the HTML where possible.
>>> >>
>>> >> On a few of the pages - SCM, CI, and Dependency Info - the output of
>>> the
>>> >> maven plugin simply isn't adequate for a multi-repository project
>>> such as
>>> >> this, so those are now managed within src/site/markdown
>>> >>
>>> >> Release Setup and Release Process need some cleanup but all of the
>>> content
>>> >> has been migrated.
>>> >>
>>> >> Other report-info pages - such as Dependencies, Dependency
>>> Management,
>>> >> Dependency Convergence, JavaDocs, Test JavaDocs, - don't make sense
>>> at the
>>> >> streams-master level but do exist and are useful within
>>> streams-project,
>>> >> streams-examples, and many of their respective sub-modules.
>>> >>
>>> >> In theory those additional report items will show up when navigating
>>> into
>>> >> the site hierarchy of the sub-projects (links to each sub-project are
>>> >> present along the header), which will inherit the structure,
>>> look-and-feel,
>>> >> and breadcrumb of this site revision. Naturally these changes must be
>>> >> deployed to streams.incubator.apache.org and sub-project pom.xml and
>>> >> site_en.xml will need some tweaking but I'm optimistic it will all
>>> fit
>>> >> together nicely.
>>> >>
>>> >> Steve Blackmon
>>> >> sblackmon@apache.org
>>> >>
>>> >> On Sat, Feb 27, 2016 at 2:27 AM, Ate Douma <at...@douma.nu> wrote:
>>> >>
>>> >>> On 2016-02-25 23:50, Steve Blackmon wrote:
>>> >>>
>>> >>>> Hello streamers,
>>> >>>>
>>> >>>> I finally consolidated some materials I've been working on into a
>>> >>>> nice-looking thing that is a viable candidate to replace the
>>> current
>>> >>>> streams web page. Please take a look and let me know what you
>>> think.
>>> >>>> Sometime
>>> >>>> soon I expect to submit a vote to take down the previous site and
>>> replace
>>> >>>> it with something derived from this version.
>>> >>>>
>>> >>>> The proposed new site is temporarily being hosted at:
>>> >>>>
>>> >>>>
>>> http://people.apache.org/~sblackmon/0.3-incubating-SNAPSHOT/streams-master/index.html
>>> >>>>
>>> >>>> I placed all of the content into markdown files which are checked
>>> into a
>>> >>>> branch of streams-master. This approach allows us to author the
>>> site
>>> >>>> using
>>> >>>> markdown, package and deploy it with maven-site, and manage changes
>>> using
>>> >>>> git.
>>> >>>>
>>> >>>
>>> >>> This looks cool!
>>> >>>
>>> >>> I like the new overview and faq pages, definitely more concrete
>>> about
>>> >>> Streams.
>>> >>>
>>> >>> One thing which I think can 'spice up' the overview page a bit more
>>> would
>>> >>> be a
>>> >>> diagram providing some very high-level interaction between data
>>> sources
>>> >>> and
>>> >>> destinations and the streams role between them.
>>> >>>
>>> >>> And before replacing the current site several existing pages need to
>>> be
>>> >>> moved to
>>> >>> the new one. But once those are done I think this is great
>>> improvement.
>>> >>>
>>> >>>
>>> >>>> If you'd like to submit specific changes feel free to do so as
>>> >>>> pull-requests to:
>>> >>>> https://github.com/steveblackmon/incubator-streams-master.git
>>> >>>>
>>> >>>> Referring back to the original list of goals, some are addressed in
>>> whole
>>> >>>> or in part with this revision, others not just yet.
>>> >>>>
>>> >>>> ADDRESSED IN WHOLE:
>>> >>>> - the website isn't providing any practical guidance *why* or *how*
>>> to
>>> >>>> use
>>> >>>> Streams
>>> >>>> - there is no public javadoc or other technical documentation
>>> >>>> linked/hosted
>>> >>>> - there is a 'wiki (coming soon)' link which never has been
>>> activated
>>> >>>>
>>> >>>> ADDRESSED IN PART:
>>> >>>> - its missing concrete examples and recognizable use-cases for
>>> which
>>> >>>> Streams might
>>> >>>> be used, nor comparison against other solutions, and why Streams
>>> would
>>> >>>> be a
>>> >>>> good/better solution
>>> >>>> - the Architecture page only has some basic wording but provides no
>>> >>>> insight
>>> >>>> whatsoever about the concrete implementation and certainly not its
>>> >>>> architecture
>>> >>>>
>>> >>>> This is just a start. Communication on the list has been dead
>>> lately,
>>> >>>> but
>>> >>>> I'm hoping to change that beginning with this effort to explain
>>> what is
>>> >>>> unique and valuable about Streams, and why. Hopefully this take on
>>> the
>>> >>>> project will inspire others to get on-board and help out in the
>>> coming
>>> >>>> months.
>>> >>>>
>>> >>>
>>> >>> +1, good work!
>>> >>>
>>> >>>
>>> >>>
>>> >>>> Steve Blackmon
>>> >>>> sblackmon@apache.org
>>> >>>>
>>> >>>> On Thu, Jan 7, 2016 at 10:49 AM, Ate Douma <at...@douma.nu> wrote:
>>> >>>>
>>> >>>> Hi Steve,
>>> >>>>>
>>> >>>>> Cool. I don't have much extra input or ideas right now, but
>>> looking
>>> >>>>> forward to the steps taken you proposed below.
>>> >>>>> And about getting out a next release first soon, if it is not
>>> >>>>> side-tracking these other steps much, sure +1
>>> >>>>>
>>> >>>>> Ate
>>> >>>>>
>>> >>>>> On 2016-01-05 15:35, Steve Blackmon wrote:
>>> >>>>>
>>> >>>>> All,
>>> >>>>>>
>>> >>>>>> Ate gave us a reality check during our board report in December
>>> and
>>> >>>>>> enumerated some of the most prominent ways our documentation
>>> currently
>>> >>>>>> falls short - of Apache community standards and of any reasonable
>>> >>>>>> useful-ness test. I've responded in-line below - please kick in
>>> any
>>> >>>>>> ideas
>>> >>>>>> you have.
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> - the website isn't providing any practical guidance *why* or
>>> *how* to
>>> >>>>>> use
>>> >>>>>> Streams
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> I agree. This is the content that belongs on the streams website
>>> >>>>>> landing
>>> >>>>>> page (high-level) or linked from the body (the details). I will
>>> take a
>>> >>>>>> first shot at this and circulate for feedback.
>>> >>>>>>
>>> >>>>>> - its missing concrete examples and recognizable use-cases for
>>> which
>>> >>>>>> Streams might be used, nor comparison against other solutions,
>>> and why
>>> >>>>>> Streams would be a good/better solution
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> The examples documentation site [1] are meant to serve as both
>>> concrete
>>> >>>>>> examples and recognizable use-cases, but there are a few problems
>>> with
>>> >>>>>> the
>>> >>>>>> current setup.
>>> >>>>>>
>>> >>>>>> 1. The examples site lacks plain-language READMEs for the root
>>> >>>>>> module
>>> >>>>>> and the first children of the root module (the runtime-specific
>>> >>>>>> aggregators).
>>> >>>>>> 2. The examples site isn't linked from the landing page of the
>>> >>>>>> website.
>>> >>>>>> 3. There is no permanent url for latest/current.
>>> >>>>>> 4. There hasn't been a formal release of this repo.
>>> >>>>>> 5. There are broken links within the markdown of some examples.
>>> >>>>>> 6. The plain-language descriptions of the examples themselves
>>> >>>>>> could be
>>> >>>>>>
>>> >>>>>> improved.
>>> >>>>>>
>>> >>>>>> All of these can be addressed (easily I think) and I'm happy to
>>> take
>>> >>>>>> them
>>> >>>>>> on.
>>> >>>>>>
>>> >>>>>> Additionally, a page (or several) comparing streams to frameworks
>>> such
>>> >>>>>> as
>>> >>>>>> Storm, Spark, Samza, Flink, etc... and describing how they are
>>> >>>>>> different/complementary would be valuable. These questions come
>>> up a
>>> >>>>>> lot.
>>> >>>>>>
>>> >>>>>> - the Architecture page only has some basic wording but provides
>>> no
>>> >>>>>> insight
>>> >>>>>> whatsoever about the concrete implementation and certainly not
>>> its
>>> >>>>>> architecture
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> I think this page needs either an multi-page outline format
>>> heavily
>>> >>>>>> linked
>>> >>>>>> into READMEs and interfaces throughout the streams repos, or a
>>> system
>>> >>>>>> diagram depicting a simple streams deployment alongside the
>>> vocabulary.
>>> >>>>>> Maybe both as several pages.
>>> >>>>>>
>>> >>>>>> - there is no public javadoc or other technical documentation
>>> >>>>>> linked/hosted
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> We do generate java-docs for releases and snapshots. We should
>>> add a
>>> >>>>>> link
>>> >>>>>> to the root page. We've made progress but there are still many
>>> classes
>>> >>>>>> lacking any javadoc header. We should reiterate a commitment to
>>> fixing
>>> >>>>>> that and a schedule for deprecating and deleting any class that
>>> doesn't
>>> >>>>>> still serve a distinct purpose.
>>> >>>>>>
>>> >>>>>> An index of the many resource files (json/xml schemas, conversion
>>> >>>>>> rules,
>>> >>>>>> shell scripts, etc...) published to the project and examples
>>> release
>>> >>>>>> should
>>> >>>>>> also placed on the main website. Ideally comprehensive, but if
>>> not at
>>> >>>>>> least enough to show a developer that they exist and can be
>>> >>>>>> hard-linked to
>>> >>>>>> within their own project resources.
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> - there is a 'wiki (coming soon)' link which never has been
>>> activated
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> This link (actually most of streams.incubator.apache.org)
>>> predates the
>>> >>>>>> regular publication of the streams-project maven site where
>>> READMEs and
>>> >>>>>> other resources are hosted. Personally I'd prefer to see
>>> documentation
>>> >>>>>> improving within the code-base rather than in a separate wiki in
>>> the
>>> >>>>>> near-term and suggest we just delete this link.
>>> >>>>>>
>>> >>>>>> - the mailing list is NOT used for any form of
>>> discussion/information
>>> >>>>>> other
>>> >>>>>> than JIRA and Git(hub) notifications
>>> >>>>>> - especially the latter doesn't give a newbie any indication what
>>> is
>>> >>>>>> going
>>> >>>>>> on, nor how to (start) interact
>>> >>>>>>
>>> >>>>>> This is unfortunate and must change for the community to grow. I
>>> know
>>> >>>>>> I'm
>>> >>>>>> guilty of having ad-hoc off-list discussions with my team that
>>> lead to
>>> >>>>>> new
>>> >>>>>> stories and pull requests. Mea culpa, I resolve to stop doing
>>> this in
>>> >>>>>> 2016.
>>> >>>>>>
>>> >>>>>> - there are no blog posts (that I am aware of) around
>>> using/trying out
>>> >>>>>> Streams either
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> This is a critical missing piece of the puzzle. Static HTML via
>>> apache
>>> >>>>>> httpd is not the ideal platform for hosting a blog but some
>>> other
>>> >>>>>> projects
>>> >>>>>> have made it work. I'd like us to replace 'Latest News' with a
>>> >>>>>> navigable,
>>> >>>>>> indexable set of blog posts going forward, where release
>>> summaries and
>>> >>>>>> new
>>> >>>>>> examples can be published, and 'in-the-wild' mentions of the
>>> project
>>> >>>>>> can
>>> >>>>>> be
>>> >>>>>> aggregated.
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> And I personally would prefer the dev@ list to be much more /
>>> >>>>>> primarily
>>> >>>>>> used for actual human interaction, not just these notifications
>>> from
>>> >>>>>> JIRA
>>> >>>>>> and Github.
>>> >>>>>> Maybe change the configuration to delegate (some of) these to the
>>> >>>>>> commit@
>>> >>>>>> list instead?
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> I agree. AFAICT though commit@streams.incubator.apache.org does
>>> not
>>> >>>>>> exist. Per http://apache.org/dev/committers.html#mail we should
>>> hold
>>> >>>>>> a
>>> >>>>>> vote to create it. I'll open that vote today.
>>> >>>>>>
>>> >>>>>> What I suggest is to for a while stop (or largely postpone)
>>> working on
>>> >>>>>> code
>>> >>>>>> but instead working on any/all of the above points first...
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> I agree that making the website helpful is the top project
>>> priority.
>>> >>>>>>
>>> >>>>>> Even working on a next release IMO is far less important than
>>> first
>>> >>>>>> building up some explanation *why* a release of Stream is needed.
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> I agree that these concerns should be addressed before the next
>>> >>>>>> release.
>>> >>>>>> The main reason for releasing soon is that per [4] there are 36
>>> issues
>>> >>>>>> resolved since the 0.2 release, including quite a few bugs and
>>> >>>>>> improvements
>>> >>>>>> related to stability, deployment flexibility, or likely to impact
>>> the
>>> >>>>>> experience of a new developer.
>>> >>>>>>
>>> >>>>>> [1]
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> http://streams.incubator.apache.org/site/0.2-incubating-SNAPSHOT/streams-examples/
>>>
>>> >>>>>> [2]
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> http://streams.incubator.apache.org/site/0.2-incubating/streams-project/apidocs/index.html
>>>
>>> >>>>>> [3]
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> http://streams.incubator.apache.org/site/0.2-incubating/streams-project/index.html
>>>
>>> >>>>>> [4]
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20STREAMS%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20in%20unreleasedVersions()
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>
>>> >
>>>
>>>
>>
>
Re: Streams Web Site, Maven Sites, misc Documentation - proposed next steps
Posted by Steve Blackmon <sb...@apache.org>.
The new website is now up at streams.incubator.apache.org. It’s not perfect but it’s good enough and way better than it’s ever been. It’s got a basic connection to Google Analytics - if you’d like to be added to the organization just ask.
I’m going to put together a blog post about website revamp and everything that there for the project blog at https://blogs.apache.org/streams/
Hopefully we’ll also be able to tweet it out from @ApacheStreams if the PMC is able to get control of it (WIP)
Steve Blackmon
sblackmon@apache.org
On Wed, Apr 27, 2016 at 3:28 PM Steve Blackmon
<
mailto:Steve Blackmon <sb...@apache.org>
> wrote:
All,
I think we are pretty close to pulling all of the work on STREAMS-401 (new web page) together into a pretty slick package. Thanks for your patience and feedback.
Some new additions include diagrams on the architecture page and within many modules depicting streams components and data flow among them and with third-party systems, a 5-step tutorial to run a useful stream, consistent look-and-feel across all three repos, integrated breadcrumbs, and rudimentary github and twitter integrations.
Feel free to browse these links to see where we’re at.
http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-master/index.html
http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-project/index.html
http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-examples/index.html
All of the menu link now resolve, or will resolve as soon as everything is promoted. If you get any Not Available errors that you CAN’T resolve by substituting ‘latest’ for ‘0.3-incubating-SNAPSHOT’ in the url, please let me know.
Additionally if you identify any major issues or have any last suggestions please let me know before I kickoff the process to merge the STREAMS-401 branches and promote the site later this week.
Steve Blackmon
sblackmon@apache.org
On Thu, Mar 03, 2016 at 9:48 AM Ate Douma
<
mailto:Ate Douma <at...@douma.nu>
> wrote:
+1 to everything below :)
Good stuff Steve!
On 2016-03-01 23:50, Steve Blackmon wrote:
> Following up:
>
> I took a pass at cleaning up the release markdown files and they now look
> mostly the same as they did under apache cms.
>
> I also confirmed that site-deploy can write to a sub-directory of
> streams.incubator.apache.org using scm:svn capability.
>
>
http://streams.incubator.apache.org
/site/0.3-incubating-SNAPSHOT/streams-master/index.html
>
> Recall that streams-project and streams-examples each publish schema
> artifacts to their maven sites, so maintaining prior versions is important.
> streams-master doesn't currently, but since it's a parent of those other
> two and prior experience has led to believe we're better off using the same
> scheme and setting the streams-master
in the pom to be a sibling of
> streams-project and streams-example rather that publishing it to a
> different URL entirely. This is why the version an artifactId are being
> included in the above url.
>
> Brings up two issues -
> resolving the latest release and/or snapshot version at any given time
> ensuring a visitor to streams.incubator.apache.org lands on the
> streams-master site for that version
>
> While symlinks are a viable way to resolving the latest release and/or
> snapshot version while keeping previous release versions around
>
> i.e. forwarding
>
http://streams.incubator.apache.org
/site/latest/streams-master/index.html
> to
>
http://streams.incubator.apache.org
/site/0.3-incubating-SNAPSHOT/streams-master/index.html
>
> they won't help us forward
http://streams.incubator.apache.org
to a
> subdirectory of itself.
>
> Good news is apache cms supports .htaccess and the following snippet can.
>
> RewriteEngine *on*
>
> Redirect /
>
http://streams.incubator.apache.org
/site/latest/streams-master/index.html
> Feel free to confirm by visiting streams.staging.incubator.org (you'll be
> redirected to the output of the maven site-deploy)
>
> *I ask that everyone on the list review and send me any concerns or
> feedback they want incorporated before this goes live on
> streams.incubator.apache.org <
http://streams.incubator.apache.org
>.*
>
> Procedurally, the way I expect that to happen is:
> - I'll submit a PR to streams-master/master
> - That PR will merge
> - Update the streams-master jenkins job to run site-deploy, and do the
> needful to get it the necessary credentials
> - Going forward all builds of streams-master/master will publish to
> site/${version}/streams-master
> - Add a few more instructions to jenkins to maintain the symlinks and
> .htaccess
> - If there are any manual steps required to oversee, document that on the
> site itself.
>
> Then, we'll move on to hooking streams-project and streams-examples up to
> this new site (as children with an integrated breadcrumb) and to jenkins CD.
>
> Steve Blackmon
>
mailto:sblackmon@apache.org
>
> On Mon, Feb 29, 2016 at 5:19 PM, Steve Blackmon <
mailto:sblackmon@apache.org
>
> wrote:
>
>> One thing which I think can 'spice up' the overview page a bit more would
>>> be a
>>> diagram providing some very high-level interaction between data sources
>>> and
>>> destinations and the streams role between them.
>>
>>
>> I agree - a diagram depicting streams mediating between sources and
>> destination on the landing page is appropriate. I'll work on this soon.
>>
>> And before replacing the current site several existing pages need to be
>>> moved to
>>> the new one. But once those are done I think this is great improvement.
>>
>>
>> I've pushed a new version (to the source and site links from my prior
>> message) that brings most of the existing content along, using the
>> reporting-info plugin to generate the HTML where possible.
>>
>> On a few of the pages - SCM, CI, and Dependency Info - the output of the
>> maven plugin simply isn't adequate for a multi-repository project such as
>> this, so those are now managed within src/site/markdown
>>
>> Release Setup and Release Process need some cleanup but all of the content
>> has been migrated.
>>
>> Other report-info pages - such as Dependencies, Dependency Management,
>> Dependency Convergence, JavaDocs, Test JavaDocs, - don't make sense at the
>> streams-master level but do exist and are useful within streams-project,
>> streams-examples, and many of their respective sub-modules.
>>
>> In theory those additional report items will show up when navigating into
>> the site hierarchy of the sub-projects (links to each sub-project are
>> present along the header), which will inherit the structure, look-and-feel,
>> and breadcrumb of this site revision. Naturally these changes must be
>> deployed to streams.incubator.apache.org and sub-project pom.xml and
>> site_en.xml will need some tweaking but I'm optimistic it will all fit
>> together nicely.
>>
>> Steve Blackmon
>>
mailto:sblackmon@apache.org
>>
>> On Sat, Feb 27, 2016 at 2:27 AM, Ate Douma <
mailto:ate@douma.nu
> wrote:
>>
>>> On 2016-02-25 23:50, Steve Blackmon wrote:
>>>
>>>> Hello streamers,
>>>>
>>>> I finally consolidated some materials I've been working on into a
>>>> nice-looking thing that is a viable candidate to replace the current
>>>> streams web page. Please take a look and let me know what you think.
>>>> Sometime
>>>> soon I expect to submit a vote to take down the previous site and replace
>>>> it with something derived from this version.
>>>>
>>>> The proposed new site is temporarily being hosted at:
>>>>
>>>>
http://people.apache.org/~sblackmon/0.3-incubating-SNAPSHOT/streams-master/index.html
>>>>
>>>> I placed all of the content into markdown files which are checked into a
>>>> branch of streams-master. This approach allows us to author the site
>>>> using
>>>> markdown, package and deploy it with maven-site, and manage changes using
>>>> git.
>>>>
>>>
>>> This looks cool!
>>>
>>> I like the new overview and faq pages, definitely more concrete about
>>> Streams.
>>>
>>> One thing which I think can 'spice up' the overview page a bit more would
>>> be a
>>> diagram providing some very high-level interaction between data sources
>>> and
>>> destinations and the streams role between them.
>>>
>>> And before replacing the current site several existing pages need to be
>>> moved to
>>> the new one. But once those are done I think this is great improvement.
>>>
>>>
>>>> If you'd like to submit specific changes feel free to do so as
>>>> pull-requests to:
>>>>
https://github.com/steveblackmon/incubator-streams-master.git
>>>>
>>>> Referring back to the original list of goals, some are addressed in whole
>>>> or in part with this revision, others not just yet.
>>>>
>>>> ADDRESSED IN WHOLE:
>>>> - the website isn't providing any practical guidance *why* or *how* to
>>>> use
>>>> Streams
>>>> - there is no public javadoc or other technical documentation
>>>> linked/hosted
>>>> - there is a 'wiki (coming soon)' link which never has been activated
>>>>
>>>> ADDRESSED IN PART:
>>>> - its missing concrete examples and recognizable use-cases for which
>>>> Streams might
>>>> be used, nor comparison against other solutions, and why Streams would
>>>> be a
>>>> good/better solution
>>>> - the Architecture page only has some basic wording but provides no
>>>> insight
>>>> whatsoever about the concrete implementation and certainly not its
>>>> architecture
>>>>
>>>> This is just a start. Communication on the list has been dead lately,
>>>> but
>>>> I'm hoping to change that beginning with this effort to explain what is
>>>> unique and valuable about Streams, and why. Hopefully this take on the
>>>> project will inspire others to get on-board and help out in the coming
>>>> months.
>>>>
>>>
>>> +1, good work!
>>>
>>>
>>>
>>>> Steve Blackmon
>>>>
mailto:sblackmon@apache.org
>>>>
>>>> On Thu, Jan 7, 2016 at 10:49 AM, Ate Douma <
mailto:ate@douma.nu
> wrote:
>>>>
>>>> Hi Steve,
>>>>>
>>>>> Cool. I don't have much extra input or ideas right now, but looking
>>>>> forward to the steps taken you proposed below.
>>>>> And about getting out a next release first soon, if it is not
>>>>> side-tracking these other steps much, sure +1
>>>>>
>>>>> Ate
>>>>>
>>>>> On 2016-01-05 15:35, Steve Blackmon wrote:
>>>>>
>>>>> All,
>>>>>>
>>>>>> Ate gave us a reality check during our board report in December and
>>>>>> enumerated some of the most prominent ways our documentation currently
>>>>>> falls short - of Apache community standards and of any reasonable
>>>>>> useful-ness test. I've responded in-line below - please kick in any
>>>>>> ideas
>>>>>> you have.
>>>>>>
>>>>>>
>>>>>> - the website isn't providing any practical guidance *why* or *how* to
>>>>>> use
>>>>>> Streams
>>>>>>
>>>>>>
>>>>>> I agree. This is the content that belongs on the streams website
>>>>>> landing
>>>>>> page (high-level) or linked from the body (the details). I will take a
>>>>>> first shot at this and circulate for feedback.
>>>>>>
>>>>>> - its missing concrete examples and recognizable use-cases for which
>>>>>> Streams might be used, nor comparison against other solutions, and why
>>>>>> Streams would be a good/better solution
>>>>>>
>>>>>>
>>>>>> The examples documentation site [1] are meant to serve as both concrete
>>>>>> examples and recognizable use-cases, but there are a few problems with
>>>>>> the
>>>>>> current setup.
>>>>>>
>>>>>> 1. The examples site lacks plain-language READMEs for the root
>>>>>> module
>>>>>> and the first children of the root module (the runtime-specific
>>>>>> aggregators).
>>>>>> 2. The examples site isn't linked from the landing page of the
>>>>>> website.
>>>>>> 3. There is no permanent url for latest/current.
>>>>>> 4. There hasn't been a formal release of this repo.
>>>>>> 5. There are broken links within the markdown of some examples.
>>>>>> 6. The plain-language descriptions of the examples themselves
>>>>>> could be
>>>>>>
>>>>>> improved.
>>>>>>
>>>>>> All of these can be addressed (easily I think) and I'm happy to take
>>>>>> them
>>>>>> on.
>>>>>>
>>>>>> Additionally, a page (or several) comparing streams to frameworks such
>>>>>> as
>>>>>> Storm, Spark, Samza, Flink, etc... and describing how they are
>>>>>> different/complementary would be valuable. These questions come up a
>>>>>> lot.
>>>>>>
>>>>>> - the Architecture page only has some basic wording but provides no
>>>>>> insight
>>>>>> whatsoever about the concrete implementation and certainly not its
>>>>>> architecture
>>>>>>
>>>>>>
>>>>>> I think this page needs either an multi-page outline format heavily
>>>>>> linked
>>>>>> into READMEs and interfaces throughout the streams repos, or a system
>>>>>> diagram depicting a simple streams deployment alongside the vocabulary.
>>>>>> Maybe both as several pages.
>>>>>>
>>>>>> - there is no public javadoc or other technical documentation
>>>>>> linked/hosted
>>>>>>
>>>>>>
>>>>>> We do generate java-docs for releases and snapshots. We should add a
>>>>>> link
>>>>>> to the root page. We've made progress but there are still many classes
>>>>>> lacking any javadoc header. We should reiterate a commitment to fixing
>>>>>> that and a schedule for deprecating and deleting any class that doesn't
>>>>>> still serve a distinct purpose.
>>>>>>
>>>>>> An index of the many resource files (json/xml schemas, conversion
>>>>>> rules,
>>>>>> shell scripts, etc...) published to the project and examples release
>>>>>> should
>>>>>> also placed on the main website. Ideally comprehensive, but if not at
>>>>>> least enough to show a developer that they exist and can be
>>>>>> hard-linked to
>>>>>> within their own project resources.
>>>>>>
>>>>>>
>>>>>> - there is a 'wiki (coming soon)' link which never has been activated
>>>>>>
>>>>>>
>>>>>> This link (actually most of streams.incubator.apache.org) predates the
>>>>>> regular publication of the streams-project maven site where READMEs and
>>>>>> other resources are hosted. Personally I'd prefer to see documentation
>>>>>> improving within the code-base rather than in a separate wiki in the
>>>>>> near-term and suggest we just delete this link.
>>>>>>
>>>>>> - the mailing list is NOT used for any form of discussion/information
>>>>>> other
>>>>>> than JIRA and Git(hub) notifications
>>>>>> - especially the latter doesn't give a newbie any indication what is
>>>>>> going
>>>>>> on, nor how to (start) interact
>>>>>>
>>>>>> This is unfortunate and must change for the community to grow. I know
>>>>>> I'm
>>>>>> guilty of having ad-hoc off-list discussions with my team that lead to
>>>>>> new
>>>>>> stories and pull requests. Mea culpa, I resolve to stop doing this in
>>>>>> 2016.
>>>>>>
>>>>>> - there are no blog posts (that I am aware of) around using/trying out
>>>>>> Streams either
>>>>>>
>>>>>>
>>>>>> This is a critical missing piece of the puzzle. Static HTML via apache
>>>>>>
httpd
is not the ideal platform for hosting a blog but some other
>>>>>> projects
>>>>>> have made it work. I'd like us to replace 'Latest News' with a
>>>>>> navigable,
>>>>>> indexable set of blog posts going forward, where release summaries and
>>>>>> new
>>>>>> examples can be published, and 'in-the-wild' mentions of the project
>>>>>> can
>>>>>> be
>>>>>> aggregated.
>>>>>>
>>>>>>
>>>>>> And I personally would prefer the dev@ list to be much more /
>>>>>> primarily
>>>>>> used for actual human interaction, not just these notifications from
>>>>>> JIRA
>>>>>> and Github.
>>>>>> Maybe change the configuration to delegate (some of) these to the
>>>>>> commit@
>>>>>> list instead?
>>>>>>
>>>>>>
>>>>>> I agree. AFAICT though
mailto:commit@streams.incubator.apache.org
does not
>>>>>> exist. Per
http://apache.org/dev/committers.html#mail
we should hold
>>>>>> a
>>>>>> vote to create it. I'll open that vote today.
>>>>>>
>>>>>> What I suggest is to for a while stop (or largely postpone) working on
>>>>>> code
>>>>>> but instead working on any/all of the above points first...
>>>>>>
>>>>>>
>>>>>> I agree that making the website helpful is the top project priority.
>>>>>>
>>>>>> Even working on a next release IMO is far less important than first
>>>>>> building up some explanation *why* a release of Stream is needed.
>>>>>>
>>>>>>
>>>>>> I agree that these concerns should be addressed before the next
>>>>>> release.
>>>>>> The main reason for releasing soon is that per [4] there are 36 issues
>>>>>> resolved since the 0.2 release, including quite a few bugs and
>>>>>> improvements
>>>>>> related to stability, deployment flexibility, or likely to impact the
>>>>>> experience of a new developer.
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>>
>>>>>>
http://streams.incubator.apache.org
/site/0.2-incubating-SNAPSHOT/streams-examples/
>>>>>> [2]
>>>>>>
>>>>>>
>>>>>>
http://streams.incubator.apache.org
/site/0.2-incubating/streams-project/apidocs/index.html
>>>>>> [3]
>>>>>>
>>>>>>
>>>>>>
http://streams.incubator.apache.org
/site/0.2-incubating/streams-project/index.html
>>>>>> [4]
>>>>>>
>>>>>>
>>>>>>
https://issues.apache.org/jira/issues/?jql=project%20%3D%20STREAMS%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20in%20unreleasedVersions()
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Streams Web Site, Maven Sites, misc Documentation - proposed next steps
Posted by Renato Marroquín Mogrovejo <re...@gmail.com>.
Cool stuff Steve! Thanks!
Renato M.
2016-04-27 22:28 GMT+02:00 Steve Blackmon <sb...@apache.org>:
> All,
>
> I think we are pretty close to pulling all of the work on STREAMS-401 (new
> web page) together into a pretty slick package. Thanks for your patience
> and feedback.
>
> Some new additions include diagrams on the architecture page and within
> many modules depicting streams components and data flow among them and with
> third-party systems, a 5-step tutorial to run a useful stream, consistent
> look-and-feel across all three repos, integrated breadcrumbs, and
> rudimentary github and twitter integrations.
>
> Feel free to browse these links to see where we’re at.
>
>
> http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-master/index.html
>
> http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-project/index.html
>
> http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-examples/index.html
>
> All of the menu link now resolve, or will resolve as soon as everything is
> promoted. If you get any Not Available errors that you CAN’T resolve by
> substituting ‘latest’ for ‘0.3-incubating-SNAPSHOT’ in the url, please let
> me know.
>
> Additionally if you identify any major issues or have any last suggestions
> please let me know before I kickoff the process to merge the STREAMS-401
> branches and promote the site later this week.
>
> Steve Blackmon
> sblackmon@apache.org
>
> On Thu, Mar 03, 2016 at 9:48 AM Ate Douma <Ate Douma
> <Ate+Douma+%3Cate@douma.nu%3E>> wrote:
>
>> +1 to everything below :)
>>
>> Good stuff Steve!
>>
>> On 2016-03-01 23:50, Steve Blackmon wrote:
>> > Following up:
>> >
>> > I took a pass at cleaning up the release markdown files and they now
>> look
>> > mostly the same as they did under apache cms.
>> >
>> > I also confirmed that site-deploy can write to a sub-directory of
>> > streams.incubator.apache.org using scm:svn capability.
>> >
>> > http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-master/index.html
>>
>> >
>> > Recall that streams-project and streams-examples each publish schema
>> > artifacts to their maven sites, so maintaining prior versions is
>> important.
>> > streams-master doesn't currently, but since it's a parent of those
>> other
>> > two and prior experience has led to believe we're better off using the
>> same
>> > scheme and setting the streams-master in the pom to be a sibling of
>> > streams-project and streams-example rather that publishing it to a
>> > different URL entirely. This is why the version an artifactId are being
>> > included in the above url.
>> >
>> > Brings up two issues -
>> > resolving the latest release and/or snapshot version at any given time
>> > ensuring a visitor to streams.incubator.apache.org lands on the
>> > streams-master site for that version
>> >
>> > While symlinks are a viable way to resolving the latest release and/or
>> > snapshot version while keeping previous release versions around
>> >
>> > i.e. forwarding
>> > http://streams.incubator.apache.org/site/latest/streams-master/index.html
>>
>> > to
>> > http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-master/index.html
>>
>> >
>> > they won't help us forward http://streams.incubator.apache.org to a
>> > subdirectory of itself.
>> >
>> > Good news is apache cms supports .htaccess and the following snippet
>> can.
>> >
>> > RewriteEngine *on*
>> >
>> > Redirect /
>> > http://streams.incubator.apache.org/site/latest/streams-master/index.html
>>
>> > Feel free to confirm by visiting streams.staging.incubator.org (you'll
>> be
>> > redirected to the output of the maven site-deploy)
>> >
>> > *I ask that everyone on the list review and send me any concerns or
>> > feedback they want incorporated before this goes live on
>> > streams.incubator.apache.org <http://streams.incubator.apache.org>.*
>> >
>> > Procedurally, the way I expect that to happen is:
>> > - I'll submit a PR to streams-master/master
>> > - That PR will merge
>> > - Update the streams-master jenkins job to run site-deploy, and do the
>> > needful to get it the necessary credentials
>> > - Going forward all builds of streams-master/master will publish to
>> > site/${version}/streams-master
>> > - Add a few more instructions to jenkins to maintain the symlinks and
>> > .htaccess
>> > - If there are any manual steps required to oversee, document that on
>> the
>> > site itself.
>> >
>> > Then, we'll move on to hooking streams-project and streams-examples up
>> to
>> > this new site (as children with an integrated breadcrumb) and to
>> jenkins CD.
>> >
>> > Steve Blackmon
>> > sblackmon@apache.org
>> >
>> > On Mon, Feb 29, 2016 at 5:19 PM, Steve Blackmon <sb...@apache.org>
>> > wrote:
>> >
>> >> One thing which I think can 'spice up' the overview page a bit more
>> would
>> >>> be a
>> >>> diagram providing some very high-level interaction between data
>> sources
>> >>> and
>> >>> destinations and the streams role between them.
>> >>
>> >>
>> >> I agree - a diagram depicting streams mediating between sources and
>> >> destination on the landing page is appropriate. I'll work on this
>> soon.
>> >>
>> >> And before replacing the current site several existing pages need to
>> be
>> >>> moved to
>> >>> the new one. But once those are done I think this is great
>> improvement.
>> >>
>> >>
>> >> I've pushed a new version (to the source and site links from my prior
>> >> message) that brings most of the existing content along, using the
>> >> reporting-info plugin to generate the HTML where possible.
>> >>
>> >> On a few of the pages - SCM, CI, and Dependency Info - the output of
>> the
>> >> maven plugin simply isn't adequate for a multi-repository project such
>> as
>> >> this, so those are now managed within src/site/markdown
>> >>
>> >> Release Setup and Release Process need some cleanup but all of the
>> content
>> >> has been migrated.
>> >>
>> >> Other report-info pages - such as Dependencies, Dependency Management,
>> >> Dependency Convergence, JavaDocs, Test JavaDocs, - don't make sense at
>> the
>> >> streams-master level but do exist and are useful within
>> streams-project,
>> >> streams-examples, and many of their respective sub-modules.
>> >>
>> >> In theory those additional report items will show up when navigating
>> into
>> >> the site hierarchy of the sub-projects (links to each sub-project are
>> >> present along the header), which will inherit the structure,
>> look-and-feel,
>> >> and breadcrumb of this site revision. Naturally these changes must be
>> >> deployed to streams.incubator.apache.org and sub-project pom.xml and
>> >> site_en.xml will need some tweaking but I'm optimistic it will all fit
>> >> together nicely.
>> >>
>> >> Steve Blackmon
>> >> sblackmon@apache.org
>> >>
>> >> On Sat, Feb 27, 2016 at 2:27 AM, Ate Douma <at...@douma.nu> wrote:
>> >>
>> >>> On 2016-02-25 23:50, Steve Blackmon wrote:
>> >>>
>> >>>> Hello streamers,
>> >>>>
>> >>>> I finally consolidated some materials I've been working on into a
>> >>>> nice-looking thing that is a viable candidate to replace the current
>> >>>> streams web page. Please take a look and let me know what you think.
>> >>>> Sometime
>> >>>> soon I expect to submit a vote to take down the previous site and
>> replace
>> >>>> it with something derived from this version.
>> >>>>
>> >>>> The proposed new site is temporarily being hosted at:
>> >>>>
>> >>>>
>> http://people.apache.org/~sblackmon/0.3-incubating-SNAPSHOT/streams-master/index.html
>> >>>>
>> >>>> I placed all of the content into markdown files which are checked
>> into a
>> >>>> branch of streams-master. This approach allows us to author the site
>> >>>> using
>> >>>> markdown, package and deploy it with maven-site, and manage changes
>> using
>> >>>> git.
>> >>>>
>> >>>
>> >>> This looks cool!
>> >>>
>> >>> I like the new overview and faq pages, definitely more concrete about
>> >>> Streams.
>> >>>
>> >>> One thing which I think can 'spice up' the overview page a bit more
>> would
>> >>> be a
>> >>> diagram providing some very high-level interaction between data
>> sources
>> >>> and
>> >>> destinations and the streams role between them.
>> >>>
>> >>> And before replacing the current site several existing pages need to
>> be
>> >>> moved to
>> >>> the new one. But once those are done I think this is great
>> improvement.
>> >>>
>> >>>
>> >>>> If you'd like to submit specific changes feel free to do so as
>> >>>> pull-requests to:
>> >>>> https://github.com/steveblackmon/incubator-streams-master.git
>> >>>>
>> >>>> Referring back to the original list of goals, some are addressed in
>> whole
>> >>>> or in part with this revision, others not just yet.
>> >>>>
>> >>>> ADDRESSED IN WHOLE:
>> >>>> - the website isn't providing any practical guidance *why* or *how*
>> to
>> >>>> use
>> >>>> Streams
>> >>>> - there is no public javadoc or other technical documentation
>> >>>> linked/hosted
>> >>>> - there is a 'wiki (coming soon)' link which never has been
>> activated
>> >>>>
>> >>>> ADDRESSED IN PART:
>> >>>> - its missing concrete examples and recognizable use-cases for which
>> >>>> Streams might
>> >>>> be used, nor comparison against other solutions, and why Streams
>> would
>> >>>> be a
>> >>>> good/better solution
>> >>>> - the Architecture page only has some basic wording but provides no
>> >>>> insight
>> >>>> whatsoever about the concrete implementation and certainly not its
>> >>>> architecture
>> >>>>
>> >>>> This is just a start. Communication on the list has been dead
>> lately,
>> >>>> but
>> >>>> I'm hoping to change that beginning with this effort to explain what
>> is
>> >>>> unique and valuable about Streams, and why. Hopefully this take on
>> the
>> >>>> project will inspire others to get on-board and help out in the
>> coming
>> >>>> months.
>> >>>>
>> >>>
>> >>> +1, good work!
>> >>>
>> >>>
>> >>>
>> >>>> Steve Blackmon
>> >>>> sblackmon@apache.org
>> >>>>
>> >>>> On Thu, Jan 7, 2016 at 10:49 AM, Ate Douma <at...@douma.nu> wrote:
>> >>>>
>> >>>> Hi Steve,
>> >>>>>
>> >>>>> Cool. I don't have much extra input or ideas right now, but looking
>> >>>>> forward to the steps taken you proposed below.
>> >>>>> And about getting out a next release first soon, if it is not
>> >>>>> side-tracking these other steps much, sure +1
>> >>>>>
>> >>>>> Ate
>> >>>>>
>> >>>>> On 2016-01-05 15:35, Steve Blackmon wrote:
>> >>>>>
>> >>>>> All,
>> >>>>>>
>> >>>>>> Ate gave us a reality check during our board report in December
>> and
>> >>>>>> enumerated some of the most prominent ways our documentation
>> currently
>> >>>>>> falls short - of Apache community standards and of any reasonable
>> >>>>>> useful-ness test. I've responded in-line below - please kick in
>> any
>> >>>>>> ideas
>> >>>>>> you have.
>> >>>>>>
>> >>>>>>
>> >>>>>> - the website isn't providing any practical guidance *why* or
>> *how* to
>> >>>>>> use
>> >>>>>> Streams
>> >>>>>>
>> >>>>>>
>> >>>>>> I agree. This is the content that belongs on the streams website
>> >>>>>> landing
>> >>>>>> page (high-level) or linked from the body (the details). I will
>> take a
>> >>>>>> first shot at this and circulate for feedback.
>> >>>>>>
>> >>>>>> - its missing concrete examples and recognizable use-cases for
>> which
>> >>>>>> Streams might be used, nor comparison against other solutions, and
>> why
>> >>>>>> Streams would be a good/better solution
>> >>>>>>
>> >>>>>>
>> >>>>>> The examples documentation site [1] are meant to serve as both
>> concrete
>> >>>>>> examples and recognizable use-cases, but there are a few problems
>> with
>> >>>>>> the
>> >>>>>> current setup.
>> >>>>>>
>> >>>>>> 1. The examples site lacks plain-language READMEs for the root
>> >>>>>> module
>> >>>>>> and the first children of the root module (the runtime-specific
>> >>>>>> aggregators).
>> >>>>>> 2. The examples site isn't linked from the landing page of the
>> >>>>>> website.
>> >>>>>> 3. There is no permanent url for latest/current.
>> >>>>>> 4. There hasn't been a formal release of this repo.
>> >>>>>> 5. There are broken links within the markdown of some examples.
>> >>>>>> 6. The plain-language descriptions of the examples themselves
>> >>>>>> could be
>> >>>>>>
>> >>>>>> improved.
>> >>>>>>
>> >>>>>> All of these can be addressed (easily I think) and I'm happy to
>> take
>> >>>>>> them
>> >>>>>> on.
>> >>>>>>
>> >>>>>> Additionally, a page (or several) comparing streams to frameworks
>> such
>> >>>>>> as
>> >>>>>> Storm, Spark, Samza, Flink, etc... and describing how they are
>> >>>>>> different/complementary would be valuable. These questions come up
>> a
>> >>>>>> lot.
>> >>>>>>
>> >>>>>> - the Architecture page only has some basic wording but provides
>> no
>> >>>>>> insight
>> >>>>>> whatsoever about the concrete implementation and certainly not its
>> >>>>>> architecture
>> >>>>>>
>> >>>>>>
>> >>>>>> I think this page needs either an multi-page outline format
>> heavily
>> >>>>>> linked
>> >>>>>> into READMEs and interfaces throughout the streams repos, or a
>> system
>> >>>>>> diagram depicting a simple streams deployment alongside the
>> vocabulary.
>> >>>>>> Maybe both as several pages.
>> >>>>>>
>> >>>>>> - there is no public javadoc or other technical documentation
>> >>>>>> linked/hosted
>> >>>>>>
>> >>>>>>
>> >>>>>> We do generate java-docs for releases and snapshots. We should add
>> a
>> >>>>>> link
>> >>>>>> to the root page. We've made progress but there are still many
>> classes
>> >>>>>> lacking any javadoc header. We should reiterate a commitment to
>> fixing
>> >>>>>> that and a schedule for deprecating and deleting any class that
>> doesn't
>> >>>>>> still serve a distinct purpose.
>> >>>>>>
>> >>>>>> An index of the many resource files (json/xml schemas, conversion
>> >>>>>> rules,
>> >>>>>> shell scripts, etc...) published to the project and examples
>> release
>> >>>>>> should
>> >>>>>> also placed on the main website. Ideally comprehensive, but if not
>> at
>> >>>>>> least enough to show a developer that they exist and can be
>> >>>>>> hard-linked to
>> >>>>>> within their own project resources.
>> >>>>>>
>> >>>>>>
>> >>>>>> - there is a 'wiki (coming soon)' link which never has been
>> activated
>> >>>>>>
>> >>>>>>
>> >>>>>> This link (actually most of streams.incubator.apache.org)
>> predates the
>> >>>>>> regular publication of the streams-project maven site where
>> READMEs and
>> >>>>>> other resources are hosted. Personally I'd prefer to see
>> documentation
>> >>>>>> improving within the code-base rather than in a separate wiki in
>> the
>> >>>>>> near-term and suggest we just delete this link.
>> >>>>>>
>> >>>>>> - the mailing list is NOT used for any form of
>> discussion/information
>> >>>>>> other
>> >>>>>> than JIRA and Git(hub) notifications
>> >>>>>> - especially the latter doesn't give a newbie any indication what
>> is
>> >>>>>> going
>> >>>>>> on, nor how to (start) interact
>> >>>>>>
>> >>>>>> This is unfortunate and must change for the community to grow. I
>> know
>> >>>>>> I'm
>> >>>>>> guilty of having ad-hoc off-list discussions with my team that
>> lead to
>> >>>>>> new
>> >>>>>> stories and pull requests. Mea culpa, I resolve to stop doing this
>> in
>> >>>>>> 2016.
>> >>>>>>
>> >>>>>> - there are no blog posts (that I am aware of) around using/trying
>> out
>> >>>>>> Streams either
>> >>>>>>
>> >>>>>>
>> >>>>>> This is a critical missing piece of the puzzle. Static HTML via
>> apache
>> >>>>>> httpd is not the ideal platform for hosting a blog but some other
>> >>>>>> projects
>> >>>>>> have made it work. I'd like us to replace 'Latest News' with a
>> >>>>>> navigable,
>> >>>>>> indexable set of blog posts going forward, where release summaries
>> and
>> >>>>>> new
>> >>>>>> examples can be published, and 'in-the-wild' mentions of the
>> project
>> >>>>>> can
>> >>>>>> be
>> >>>>>> aggregated.
>> >>>>>>
>> >>>>>>
>> >>>>>> And I personally would prefer the dev@ list to be much more /
>> >>>>>> primarily
>> >>>>>> used for actual human interaction, not just these notifications
>> from
>> >>>>>> JIRA
>> >>>>>> and Github.
>> >>>>>> Maybe change the configuration to delegate (some of) these to the
>> >>>>>> commit@
>> >>>>>> list instead?
>> >>>>>>
>> >>>>>>
>> >>>>>> I agree. AFAICT though commit@streams.incubator.apache.org does
>> not
>> >>>>>> exist. Per http://apache.org/dev/committers.html#mail we should
>> hold
>> >>>>>> a
>> >>>>>> vote to create it. I'll open that vote today.
>> >>>>>>
>> >>>>>> What I suggest is to for a while stop (or largely postpone)
>> working on
>> >>>>>> code
>> >>>>>> but instead working on any/all of the above points first...
>> >>>>>>
>> >>>>>>
>> >>>>>> I agree that making the website helpful is the top project
>> priority.
>> >>>>>>
>> >>>>>> Even working on a next release IMO is far less important than
>> first
>> >>>>>> building up some explanation *why* a release of Stream is needed.
>> >>>>>>
>> >>>>>>
>> >>>>>> I agree that these concerns should be addressed before the next
>> >>>>>> release.
>> >>>>>> The main reason for releasing soon is that per [4] there are 36
>> issues
>> >>>>>> resolved since the 0.2 release, including quite a few bugs and
>> >>>>>> improvements
>> >>>>>> related to stability, deployment flexibility, or likely to impact
>> the
>> >>>>>> experience of a new developer.
>> >>>>>>
>> >>>>>> [1]
>> >>>>>>
>> >>>>>>
>> >>>>>> http://streams.incubator.apache.org/site/0.2-incubating-SNAPSHOT/streams-examples/
>>
>> >>>>>> [2]
>> >>>>>>
>> >>>>>>
>> >>>>>> http://streams.incubator.apache.org/site/0.2-incubating/streams-project/apidocs/index.html
>>
>> >>>>>> [3]
>> >>>>>>
>> >>>>>>
>> >>>>>> http://streams.incubator.apache.org/site/0.2-incubating/streams-project/index.html
>>
>> >>>>>> [4]
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20STREAMS%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20in%20unreleasedVersions()
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>>
>