Project Visualization Tool: Linkurious + incubator-streams

[Steve Blackmon <sb...@apache.org>]

re
> - Visualizations.  There's lots of different stuff to do here, and I
> think it'd be super helpful if everyone just did something they want,
> and then show us the code.

I spent some time on this and have something to share:

Commercial graph browser linkurio.us seeded with contents of people.json.

http://72.182.111.65:3000/workspace/3

(Export attached)

Everyone and every project in that file is connected and searchable,
but just the handful already on this thread are pre-rendered in this
workspace.

I'll keep this online for a few days for review by this group, but
will probably take it down shortly or put it behind auth. If you get
an auth dialog, email me directly.

Source code used:

https://github.com/steveblackmon/streams-apache

The streams-persist-graph functionality of incubator-streams (which
this uses) allows for graphs with arbitrary vertex types, edge types,
and vertex/edge properties to be generated (in batch or real-time)
from activitystrea.ms documents.  Basically an actor or object is
written as a vertex, and verbs become edges connecting two vertices.
Multiple writes can be used against the same graph, and so long as the
identifiers used are consistent, the vertices, edges, and properties
will 'stack'.

This is just one aspect of the larger discussion, but I think having
an searchable, exploratory view of the ecosystem of people and
projects that comprise Apache similar to this would be useful, and
didn't mind spending an evening taking a shot.

Steve Blackmon
sblackmon@apache.org

On Sat, Apr 18, 2015 at 9:55 AM, Shane Curcuru <as...@shanecurcuru.org> wrote:
> LOL, below.
>
> I highly recommend separating the model from the views, so that we can
> efficiently enable our volunteer's energy here to actually accomplish
> something valuable.
>
> So let's work on stuff to do that excites us, but remember to keep the
> technical problems focused on what this PMC believes we can truly create
> and maintain going forward.
>
> Don't worry about everything at once.  Just focus on separate bits:
>
> - Method to scrape source data from our various definitive or even not
> completely definitive but very close places (txt files, websites, LDAP)
>
> - Model and data source that actually holds info about committer lists
> and project metadata.  I'm betting Daniels' projects-new does this very
> well already.
>
> ----------
> - Stable API to get at that model.  Would be really nice if we did this
> just once, so that people working above here don't interfere with people
> working below here.
> ----------
>
> - Visualizations.  There's lots of different stuff to do here, and I
> think it'd be super helpful if everyone just did something they want,
> and then show us the code.
>
> Sure, there's lots of "what is important" to focus on, but I for one
> would love to see real examples of all the cool visualization libraries
> out there, and I know a couple folks already use some of them.
>
> - UI additions for the projects-new/projects websites, which are
> featured at the top level of a.o.  I.e., this is our "projects
> directory", how can we better lead people who arrive there at what they
> want to know?
>
> - (future) UI additions for *other* places.  It would be awesome, for
> example, to provide a tiny scriptlet that any project could inject in
> their website that displays a "see also" menu.  That would link to a
> specific URL on projects.a.o that would say "hey, you came from
> Cassandra, here are: -other big data projects, -other projects in Java,
> -other projects with the same committers... etc." as a service.
>
> - Shane
>
>
> On 4/18/15 5:44 AM, jan i wrote:
>> On Saturday, April 18, 2015, <he...@free.fr> wrote:
>>
>>> It was told the new site would use native json, instead of doap
>>> But I'm not convinced at all, since Doap is an invaluable source of info,
>>> documented, and so on
>>
>> json is also a documented standard, that in general is more known, and I
>> believe has more tools supporting it.
>>
>>
>>>
>>> then imho it would be better to generate json from doap
>>>
>>> I disabled the json edit feature recently since it will cause problems
>>
>> which problems?
>>
>> with a defined json it is simple to generate the doap file.
>>
>> I highly recommend staying at json and using that as base for all our
>> central data.
>>
>> rgds
>> jan i
>>
>>
>>
>>>
>>> regards
>>>
>>> Hervé
>>> ----- Mail d'origine -----
>>> De: Shane Curcuru <asf@shanecurcuru.org <javascript:;>>
>>> À: dev@community.apache.org <javascript:;>
>>> Envoyé: Sat, 18 Apr 2015 06:43:37 +0200 (CEST)
>>> Objet: Re: Project Visualization Tool...
>>>
>>> We had a great session, and a lot of energy, hopefully we can make some
>>> progress. One note: this needs to be a comdev PMC project, and we need
>>> to really plan the data part out if we want to be successful.
>>>
>>> Note that projects-new.a.o is the planned future replacement for
>>> projects.a.o - there are *significant* differences, so you need to look
>>> at the About page and the source repo. In particular, the new site uses
>>> it's own new JSON generated sources which (I think) will no longer use
>>> the DOAPs.
>>>
>>> In particular, Infra currently does *not* consider either the data
>>> gathering (i.e. populating the JSON behind the projects-new site) nor
>>> the visualizations (current or ones we want to build) as core supported
>>> services. So whatever we build needs to be maintained by this PMC to
>>> start with.
>>>
>>> Also, Link dump of useful related bits: ----------------
>>>
>>> Old service, based on crappy cron jobs and DOAP files from projects:
>>> https://projects.apache.org/
>>>
>>> New service, soon to be infra supported, relying on JSON data generated
>>> by infra on a regular schedule:
>>> https://projects-new.apache.org/
>>>
>>> Useful PMC chair report helper, that surfaces a number of different
>>> statistics about your PMC(s), including mailing list stats,
>>> PMC/committer changes, some software releases, etc. etc. (Members have
>>> visibility to all PMCs):
>>> https://reporter.apache.org
>>>
>>> Rob Weir (AOO, Member) used to do some visualization stuff and might
>>> have code ideas:
>>> http://www.robweir.com/blog/2013/05/mapping-apache.html
>>>
>>> Ken Coar's old mailing list stats page:
>>>
>>> https://people.apache.org/~coar/mlists.html
>>>
>>> The AOO project wrote a mailing list visualizer for who talks to whom:
>>> https://blogs.apache.org/OOo/entry/visualizing_the_aoo_dev_list
>>>
>>> Some outside statistics FLOSSmole generated about Apache communities and
>>> lists:
>>> http://flossmole.org/category/tags/apache
>>>
>>> Random other interesting analytics:
>>> The Subversion project has the "contribulyzer"
>>>
>>>
>>>
>>> - Shane
>>>
>>>
>>
>