You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@allura.apache.org by Alvaro del Castillo <ac...@bitergia.com> on 2012/09/11 21:37:21 UTC
Re: Presentation and Metrics work in Allura
Hi guys,
Time to update Allura community about our progress with Metrics Tools:
> Ok. So next steps:
>
> - We are expanding the kind of graphs we support and also the metrics.
> We will integrate them in our Allura module.
You can see our M0 (Milestone0 reports at) GNOME Shell sample:
* SCM and ITS analysis integrated
http://bitergia.com/public/reports/gnomeshell/M0.html
* Different isolates graphs showing also SCM and ITS activity.
http://bitergia.com/public/reports/gnomeshell/milestone0.html
We are working in some other visualizations, like comparing in the same
graph issues and commits:
http://bitergia.com/public/reports/gnomeshell/commits-issues.html
>
> - Control metrics tools execution from taskd.
I am not sure if the tools execution should be programmed using taskd.
What do you think?
>
> - Allura project configuration to enable graphs and auto configure the
> metrics tools to use the Allura backends.
We have improved Bicho tool, the ticket/issues gathering engine, so now
it supports Allura Tickets completely and using incremental queries, so
after the first analysis, it is really quick to reanalyze the tickets
for a project and create the JSON files with the data needed to graph
the tickets activity.
Right now we are working in the configuration of tools inside Allura.
Our plan is to continue working in a Allura for at Github and once the
Metrics Module is in good state, where is the best place to coordinate
the future development?
Kind regards
>
> - Study Zarkov to understand the kind of information it has and how you
> are using it to create graphs.
>
> Kind regards!
>
> >
> > -Dave
> >
> > --
> > Dave Brondsema
> > Principal Software Engineer - sf.net
> > Geeknet
> >
>
>
--
|\_____/| Alvaro del Castillo San Félix
[o] [o] acs@bitergia.com - Chief Technical Officer (CTO)
| V | http://www.bitergia.com
| | "Bridging the gap between developers and stakeholders"
-ooo-ooo-
Re: Presentation and Metrics work in Allura
Posted by Rich Bowen <rb...@rcbowen.com>.
On Sep 11, 2012, at 3:37 PM, Alvaro del Castillo wrote:
> You can see our M0 (Milestone0 reports at) GNOME Shell sample:
>
> * SCM and ITS analysis integrated
> http://bitergia.com/public/reports/gnomeshell/M0.html
>
> * Different isolates graphs showing also SCM and ITS activity.
> http://bitergia.com/public/reports/gnomeshell/milestone0.html
>
> We are working in some other visualizations, like comparing in the same
> graph issues and commits:
>
> http://bitergia.com/public/reports/gnomeshell/commits-issues.html
This stuff is really cool. I'm really looking forward to seeing this stuff available to everyone.
--
Rich Bowen
rbowen@rcbowen.com :: @rbowen
rbowen@apache.org
Re: Presentation and Metrics work in Allura
Posted by Alvaro del Castillo <ac...@bitergia.com>.
El jue, 13-09-2012 a las 19:16 -0400, Dave Brondsema escribió:
> I found http://bitergia.com/public/reports/gnomeshell/README which looks like it
> outlines a lot of the commands and dependencies you use.
Yes, one of our goals is to give the data we use to build our reports.
Transparency :) And also, in some cases like this basic reports, the
methodology so you can also reproduce it.
Cheers!
>
> On 9/13/12 4:01 PM, Dave Brondsema wrote:
> > On 9/11/12 3:37 PM, Alvaro del Castillo wrote:
> >> Hi guys,
> >>
> >> Time to update Allura community about our progress with Metrics Tools:
> >>
> >>> Ok. So next steps:
> >>>
> >>> - We are expanding the kind of graphs we support and also the metrics.
> >>> We will integrate them in our Allura module.
> >>
> >> You can see our M0 (Milestone0 reports at) GNOME Shell sample:
> >>
> >> * SCM and ITS analysis integrated
> >> http://bitergia.com/public/reports/gnomeshell/M0.html
> >>
> >> * Different isolates graphs showing also SCM and ITS activity.
> >> http://bitergia.com/public/reports/gnomeshell/milestone0.html
> >>
> >> We are working in some other visualizations, like comparing in the same
> >> graph issues and commits:
> >>
> >> http://bitergia.com/public/reports/gnomeshell/commits-issues.html
> >
> > Nice graphs. Previously your examples had been shown as graphs within Allura,
> > right? These look like static HTML. Is something different here?
> >
> > Gnome shell is a git repo, right? Does the reporting work with other repo types?
> >
> >>>
> >>> - Control metrics tools execution from taskd.
> >>
> >> I am not sure if the tools execution should be programmed using taskd.
> >> What do you think?
> >
> > How do you run them now? If taskd doesn't run them automatically after each
> > commit, maybe a cron job would be a good way to deploy it. I think end-users
> > could be okay with reports that are generated daily.
> >
> >>
> >>
> >>>
> >>> - Allura project configuration to enable graphs and auto configure the
> >>> metrics tools to use the Allura backends.
> >>
> >> We have improved Bicho tool, the ticket/issues gathering engine, so now
> >> it supports Allura Tickets completely and using incremental queries, so
> >> after the first analysis, it is really quick to reanalyze the tickets
> >> for a project and create the JSON files with the data needed to graph
> >> the tickets activity.
> >
> > Cool
> >
> >>
> >> Right now we are working in the configuration of tools inside Allura.
> >>
> >> Our plan is to continue working in a Allura for at Github and once the
> >> Metrics Module is in good state, where is the best place to coordinate
> >> the future development?
> >>
> >
> > Is https://github.com/Bitergia/AlluraBitergiaMetrics the right repo for me to be
> > looking at?
> >
> > An earlier email mentioned R is used for some of the statistics. Could you
> > point me to where that is used in the code? Or a brief explanation of
> > dependencies in email is cool too.
> >
> > This mailing list is a good place to talk about development, just like you're
> > doing :) Out of curiosity, are you interested in this for general purpose
> > availability, or specifically interested in SourceForge being able to adopt
> > this? I'm sure we'd be interested in getting it up and running, which is why
> > I'm asking about supported SCMs, dependencies, scalability, etc :)
> >
> >
> >
> >
>
>
>
--
|\_____/| Alvaro del Castillo San Félix
[o] [o] acs@bitergia.com - Chief Technical Officer (CTO)
| V | http://www.bitergia.com
| | "Bridging the gap between developers and stakeholders"
-ooo-ooo-
Re: Presentation and Metrics work in Allura
Posted by Dave Brondsema <da...@brondsema.net>.
I found http://bitergia.com/public/reports/gnomeshell/README which looks like it
outlines a lot of the commands and dependencies you use.
On 9/13/12 4:01 PM, Dave Brondsema wrote:
> On 9/11/12 3:37 PM, Alvaro del Castillo wrote:
>> Hi guys,
>>
>> Time to update Allura community about our progress with Metrics Tools:
>>
>>> Ok. So next steps:
>>>
>>> - We are expanding the kind of graphs we support and also the metrics.
>>> We will integrate them in our Allura module.
>>
>> You can see our M0 (Milestone0 reports at) GNOME Shell sample:
>>
>> * SCM and ITS analysis integrated
>> http://bitergia.com/public/reports/gnomeshell/M0.html
>>
>> * Different isolates graphs showing also SCM and ITS activity.
>> http://bitergia.com/public/reports/gnomeshell/milestone0.html
>>
>> We are working in some other visualizations, like comparing in the same
>> graph issues and commits:
>>
>> http://bitergia.com/public/reports/gnomeshell/commits-issues.html
>
> Nice graphs. Previously your examples had been shown as graphs within Allura,
> right? These look like static HTML. Is something different here?
>
> Gnome shell is a git repo, right? Does the reporting work with other repo types?
>
>>>
>>> - Control metrics tools execution from taskd.
>>
>> I am not sure if the tools execution should be programmed using taskd.
>> What do you think?
>
> How do you run them now? If taskd doesn't run them automatically after each
> commit, maybe a cron job would be a good way to deploy it. I think end-users
> could be okay with reports that are generated daily.
>
>>
>>
>>>
>>> - Allura project configuration to enable graphs and auto configure the
>>> metrics tools to use the Allura backends.
>>
>> We have improved Bicho tool, the ticket/issues gathering engine, so now
>> it supports Allura Tickets completely and using incremental queries, so
>> after the first analysis, it is really quick to reanalyze the tickets
>> for a project and create the JSON files with the data needed to graph
>> the tickets activity.
>
> Cool
>
>>
>> Right now we are working in the configuration of tools inside Allura.
>>
>> Our plan is to continue working in a Allura for at Github and once the
>> Metrics Module is in good state, where is the best place to coordinate
>> the future development?
>>
>
> Is https://github.com/Bitergia/AlluraBitergiaMetrics the right repo for me to be
> looking at?
>
> An earlier email mentioned R is used for some of the statistics. Could you
> point me to where that is used in the code? Or a brief explanation of
> dependencies in email is cool too.
>
> This mailing list is a good place to talk about development, just like you're
> doing :) Out of curiosity, are you interested in this for general purpose
> availability, or specifically interested in SourceForge being able to adopt
> this? I'm sure we'd be interested in getting it up and running, which is why
> I'm asking about supported SCMs, dependencies, scalability, etc :)
>
>
>
>
--
Dave Brondsema : dave@brondsema.net
http://www.brondsema.net : personal
http://www.splike.com : programming
<><
Re: Presentation and Metrics work in Allura
Posted by Alvaro del Castillo <ac...@bitergia.com>.
Hi!
El jue, 13-09-2012 a las 16:01 -0400, Dave Brondsema escribió:
> On 9/11/12 3:37 PM, Alvaro del Castillo wrote:
> > Hi guys,
> >
> > Time to update Allura community about our progress with Metrics Tools:
> >
> >> Ok. So next steps:
> >>
> >> - We are expanding the kind of graphs we support and also the metrics.
> >> We will integrate them in our Allura module.
> >
> > You can see our M0 (Milestone0 reports at) GNOME Shell sample:
> >
> > * SCM and ITS analysis integrated
> > http://bitergia.com/public/reports/gnomeshell/M0.html
> >
> > * Different isolates graphs showing also SCM and ITS activity.
> > http://bitergia.com/public/reports/gnomeshell/milestone0.html
> >
> > We are working in some other visualizations, like comparing in the same
> > graph issues and commits:
> >
> > http://bitergia.com/public/reports/gnomeshell/commits-issues.html
>
> Nice graphs. Previously your examples had been shown as graphs within Allura,
> right? These look like static HTML. Is something different here?
No, the idea behind is the same: generate JSON with the metrics so you
can plot them. The JSON files are like the juice from a fruit, the data
sources. In order to squeeze the juice we gather the data, convert it to
a common format, analyze it with SQL, R and other techniques and
generate the results, normally metrics, in JSON.
And all of this work inside Allura web pages also.
>
> Gnome shell is a git repo, right? Does the reporting work with other repo types?
yes, GNOME Shell is a git repo. Currently we depend in our CVSAnalY tool
(http://metricsgrimoire.github.com/CVSAnalY/) that supports CVS,
Subversion and git.
>
> >>
> >> - Control metrics tools execution from taskd.
> >
> > I am not sure if the tools execution should be programmed using taskd.
> > What do you think?
>
> How do you run them now? If taskd doesn't run them automatically after each
> commit, maybe a cron job would be a good way to deploy it. I think end-users
> could be okay with reports that are generated daily.
Right now we execute if manually from command line, and in some cases we
have update scripts that are executed automatically. For small/mid
projects with incremental support for its data sources, taskd could be
an option an execute them after each commit or ticket change. But in
general, a daily report could be created and we can give the user a
"button" to execute the update if she is in a hurry to have the data
updated in this moment.
>
> >
> >
> >>
> >> - Allura project configuration to enable graphs and auto configure the
> >> metrics tools to use the Allura backends.
> >
> > We have improved Bicho tool, the ticket/issues gathering engine, so now
> > it supports Allura Tickets completely and using incremental queries, so
> > after the first analysis, it is really quick to reanalyze the tickets
> > for a project and create the JSON files with the data needed to graph
> > the tickets activity.
>
> Cool
>
> >
> > Right now we are working in the configuration of tools inside Allura.
> >
> > Our plan is to continue working in a Allura for at Github and once the
> > Metrics Module is in good state, where is the best place to coordinate
> > the future development?
> >
>
> Is https://github.com/Bitergia/AlluraBitergiaMetrics the right repo for me to be
> looking at?
Yes!
>
> An earlier email mentioned R is used for some of the statistics. Could you
> point me to where that is used in the code? Or a brief explanation of
> dependencies in email is cool too.
Sure. The R scripts are in:
http://bitergia.com/public/reports/gnomeshell/scripts/
For git: scm-analysis.R
For Allura Tickets: its-analysis.R
But we are working now in https://github.com/VizGrimoire/VizGrimoireR in
order to build a R library for the reports.
>
> This mailing list is a good place to talk about development, just like you're
> doing :)
Great!
> Out of curiosity, are you interested in this for general purpose
> availability, or specifically interested in SourceForge being able to adopt
> this?
I am interested in improving Allura project metrics, and in general,
improving project metrics in developers tools. I am also interested in
Allura project because it is a good open source product, with a solid
and extensible architecture, for deploying forges.
In Bitergia we are also interested in Sourceforge adopting this kind of
metrics, of course. It is a wonderful use case and a good way to reach
millions of users.
> I'm sure we'd be interested in getting it up and running, which is why
> I'm asking about supported SCMs, dependencies, scalability, etc :)
Great!
* SCMs: git, svn, cvs
* Dependencies:
- CVSAnaly: http://metricsgrimoire.github.com/CVSAnalY/
- http://metricsgrimoire.github.com/Bicho/
* Scalability: The visualization process is totally done client side. In
the server the hard work is to generate the JSON. JSON static files
could be distributed easily in an scalable way. For generating the JSON
we need:
+ Allura project data sources with an efficient way to access them
+ Bicho and CVSAnalY do basic download, parsing stuff and write
operations in MySQL. We should measure it for thousands of projects, but
it should be doable.
+ R scripts for the current basic reports are really fast. We can also
do the work using just Python in some cases.
Dave, if you want to play directly with the tools, I can support you if
you find probs. For example, we can analyze Allura project directly.
All the visualization graphs are done with flotr2 and envision, which
are JS framework agnostic and use jquery, underscore and bonzo
libraries. I have integrated all of this inside Allura generated pages
and works nicely.
Kind regards!
--
|\_____/| Alvaro del Castillo San Félix
[o] [o] acs@bitergia.com - Chief Technical Officer (CTO)
| V | http://www.bitergia.com
| | "Bridging the gap between developers and stakeholders"
-ooo-ooo-
Re: Presentation and Metrics work in Allura
Posted by Dave Brondsema <da...@brondsema.net>.
On 9/11/12 3:37 PM, Alvaro del Castillo wrote:
> Hi guys,
>
> Time to update Allura community about our progress with Metrics Tools:
>
>> Ok. So next steps:
>>
>> - We are expanding the kind of graphs we support and also the metrics.
>> We will integrate them in our Allura module.
>
> You can see our M0 (Milestone0 reports at) GNOME Shell sample:
>
> * SCM and ITS analysis integrated
> http://bitergia.com/public/reports/gnomeshell/M0.html
>
> * Different isolates graphs showing also SCM and ITS activity.
> http://bitergia.com/public/reports/gnomeshell/milestone0.html
>
> We are working in some other visualizations, like comparing in the same
> graph issues and commits:
>
> http://bitergia.com/public/reports/gnomeshell/commits-issues.html
Nice graphs. Previously your examples had been shown as graphs within Allura,
right? These look like static HTML. Is something different here?
Gnome shell is a git repo, right? Does the reporting work with other repo types?
>>
>> - Control metrics tools execution from taskd.
>
> I am not sure if the tools execution should be programmed using taskd.
> What do you think?
How do you run them now? If taskd doesn't run them automatically after each
commit, maybe a cron job would be a good way to deploy it. I think end-users
could be okay with reports that are generated daily.
>
>
>>
>> - Allura project configuration to enable graphs and auto configure the
>> metrics tools to use the Allura backends.
>
> We have improved Bicho tool, the ticket/issues gathering engine, so now
> it supports Allura Tickets completely and using incremental queries, so
> after the first analysis, it is really quick to reanalyze the tickets
> for a project and create the JSON files with the data needed to graph
> the tickets activity.
Cool
>
> Right now we are working in the configuration of tools inside Allura.
>
> Our plan is to continue working in a Allura for at Github and once the
> Metrics Module is in good state, where is the best place to coordinate
> the future development?
>
Is https://github.com/Bitergia/AlluraBitergiaMetrics the right repo for me to be
looking at?
An earlier email mentioned R is used for some of the statistics. Could you
point me to where that is used in the code? Or a brief explanation of
dependencies in email is cool too.
This mailing list is a good place to talk about development, just like you're
doing :) Out of curiosity, are you interested in this for general purpose
availability, or specifically interested in SourceForge being able to adopt
this? I'm sure we'd be interested in getting it up and running, which is why
I'm asking about supported SCMs, dependencies, scalability, etc :)
--
Dave Brondsema : dave@brondsema.net
http://www.brondsema.net : personal
http://www.splike.com : programming
<><