You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@creadur.apache.org by Gavin <ga...@16degrees.com.au> on 2009/06/12 13:28:08 UTC
RAT with BuildBot at Apache (was: RE: Report Time)
> -----Original Message-----
> From: ross.gardler@googlemail.com [mailto:ross.gardler@googlemail.com] On
> Behalf Of Ross Gardler
> Sent: Tuesday, 9 June 2009 5:54 AM
> To: rat-dev@incubator.apache.org
> Subject: Re: Report Time
>
<snip>
> I would also suggest that the project should focus on encouraging
> Apache projects to use the tool, perhaps a Gump style build and report
> mechanism should be a focus for the current team. This will raise
> awareness of the utility of RAT.
Some of you may be aware that I am working on the Buildbot at Apache
instance. RAT itself is tested on there every time a commit happens, along
with a few other projects. Still early days yet but there are plenty of
projects out there not using a CI tool, or not using one at Apache, so I'll
be contacting them, and/or blogging about it whatever, to get more projects
to take advantage of its many abilities.
One such ability, which I have implemented over the last few days, is to
make use of RAT. As RAT is already there being built, I decided to automate
the use of the jar produced from the build. Now, projects can utilise
Buildbot to include RAT testing on their source/binaries/etc. I'd also like
to extend that out further - even projects that don't need Buildbot for
anything else, can use it purely for the convenience of automatically having
RAT reports produced for them, and all are welcome to do so.
RAT reports can be displayed (currently) as txt files on the ASF Buildbot
website, they can be emailed (though not currently enabled as a fine balance
needs to be achieved considering per commit triggers). They can also be
produced on demand via IRC if they want.
This is both useful to RAT as well as the projects using it - as RAT project
can implement new features or fixes based on the results/feedback from the
projects using it - one commit later to svn and within a minute all projects
are immediately using the new version of RAT.
Buildbot itself has many other features and capabilities not yet
implemented/realised (Nexus is next on my hit list, as well as a place to
put release candidates (as most folk usually upload that stuff to
people.apache.org/~username)).
My hope here is with integration of RAT is that it can raise the profile of
both RAT and Buildbot within the Apache community. I'm not sure Ross what
you had in mind for increased awareness but I hope this is an option we can
explore.
We can talk about ideas on best/different implementation ideas if folks
agree it is a good idea.
So, I'll show you what it does so far.
http://ci.apache.org/buildbot.html
Above is the main Buildbot index page, linking to amongst other things, RAT
reports page from projects configured to produce them
http://ci.apache.org/waterfall
Above page shows a list of projects currently using Buildbot (a few are
there purely for my testing)
A few of those projects I have configured the builds to upload
websites/documentation/javadoc/Apis - and also RAT reports for a few.
http://ci.apache.org/projects/
This page is generated once per hour. It scans the master Buildbot server
for any uploaded website/docs/rat-reports and creates links to them. I did
it this way so now the page is self-maintaining.
Two of the projects there, Buildr and Forrest, don't actually use Buildbot
for anything else, just showing that it is quite alright for projects to use
it for RAT reports only. In these 2 cases, from (commit to) checkout of
source (which is clobbered every time) to the RAT report being available on
the web; was about 30 seconds.
The content of all those RAT reports I'd like to verify, and sometime soon
ensure that I am indeed producing these reports correctly. Some tweaking
will need to be done.
One major thing I'd to work on soon -- and I'd like help with if anyone has
time (if not it will just take longer I guess) is to make use of the XML
output and so create an xslt to produce nice readable HTML reports to go on
the site. Do we have a DTD? Do we need one? Is the current XML output
suitable or will it need tweaking? Has any of this been done already? This
stuff I can find out soon.
One last thing I though of, I'm sure we can gather info from these reports
and send it to wherever Clutch needs it in order to implement that extra
line on clutch you mentioned Ross.
Anyway, what a long email, I'll go now.
Thoughts, ideas for expansion, ideas on current implementation?
Thanks all.
Gav...
>
> On one of my non-ASF projects I use RAT in my CI server to tell me
> when someone commits a file that breaks the RAT checks. This is really
> useful.
>
> Ross
>
> --
> Ross Gardler
>
RE: RAT with BuildBot at Apache
Posted by Gavin <ga...@16degrees.com.au>.
> -----Original Message-----
> From: David Crossley [mailto:crossley@apache.org]
> Sent: Monday, 20 July 2009 2:08 PM
> To: rat-dev@incubator.apache.org
> Subject: Re: RAT with BuildBot at Apache
>
> Gavin wrote:
> > Robert Burrell Donkin wrote:
> > > Ross Gardler wrote:
> > > > Gavin wrote:
> > > >> One last thing I though of, I'm sure we can gather info from these
> > > reports
> > > >> and send it to wherever Clutch needs it in order to implement that
> > > extra
> > > >> line on clutch you mentioned Ross.
> > > >
> > > > That would be really cool. Perhaps the simplest thing to do would be
> > > > to have your buildbot dump the files in a location that the existing
> > > > Clutch code can access.
> > >
> > > +1
> >
> > FYI I'm going to email incubator list - I guess aimed at David (no idea
> if
> > anyone else works on it)
>
> I do subscribe to rat-dev and read occasionally.
> However, recent email volumes on members@ swamped me.
>
> Gavin sent private email today. I would rather discuss
> on list, so i will bring it back here. See below.
sorry about that, grabbed your attention though ;)
>
> > and see if he'll find this useful and if so where I
> > can put the resulting html/xml reports. If accepted I'll get Buildbot to
> > build RAT reports for all incubating projects. I'll also see about doing
> > that summary page to try and make it useful as feedback for RAT project
> > also.
> >
> > ----------------
> > Just wanted to get a feel for whether or not Buildbot could do something
> for
> > Clutch.
> >
> > On the rat-dev list, amongst some ideas thrown around, Ross suggested
> that
> > perhaps Buildbot could test all incubator projects for RAT and produce
> > individual plus an overall combined report.
> >
> > Ross' suggestion went on to say that perhaps Clutch could be extended by
> an
> > extra column or two and state whether a project passes RAT and if not
> say
> > how many files fail the RAT validation.
> >
> > Would this be a useful addition?
>
> Yes definitely.
>
> > If so I can get to work on it Buildbot end
> > - all I need from you is where you want the results putting so that
> Clutch
> > can use it, how you want it represented (.xml , .txt, .html) etc. I
> could
> > for instance put it in svn somewhere, or perhaps Clutch could gather the
> > results from a ci.apache.org/projects/rat-output/ location?
>
> As long as it is publicly accessible and in a structured format,
> then Clutch can handle it. Even just a CSV file would work.
>
> Perhaps a summary file at ci.apache.org/projects/rat-output/summary.txt
>
> #projectId,numFailed
> foo,12
> bar,0
>
> So, for each podling, Clutch would see if it is in your list.
> If "numFailed gt 0" then alert it.
>
> The table cell would link directly to each project's html RAT report.
>
> Prod me when you have something.
Ok great, will take me a few days to get it sorted I'd imagine.
Gav...
>
> -David
Re: RAT Summary Page (was: RE: RAT with BuildBot at Apache)
Posted by "Hyrum K. Wright" <hy...@mail.utexas.edu>.
On Dec 12, 2009, at 4:59 AM, Gavin wrote:
> OK, so back to this RAT Summary I promised.
>
> I have a working version at:
>
> http://ci.apache.org/projects/rat-master-summary.html
This looks great!
> Still a basic interface currently but before I went too much further it
> would be great if I could get feedback (from David mainly I guess in regards
> to Clutch)
>
> What extra information does Clutch need to make it useful for it?
>
> There are (as of writing) 3 projects up there, these are the only 3 projects
> using RAT at Buildbot that is configured to output xml, the others all being
> txt. I guess I should leave .txt outputs as an alternative but change the
> default so that all participating projects output .xml instead. This way I
> get to convert the output to a better html presentation and at the same time
> get to grab the xml content for the master rat summary page.
Perhaps in the html-ized version, you could have links directly to the offending files' content, rather than embedding it in the report.
> One main hurdle was (and still is to a point) matching the output with the
> project it belongs to. As separate reports it was not a problem that in fact
> no reports contain the name of the project. As a combined summary however
> this information is most vital. For now, as part of as xsl that grabs the
> content from each report, it then wraps it in a new <project name="blah">
> element so then that can be used to get the project name into the output in
> the correct column.
>
> A tiny bit of a workaround is in place in that the project names currently
> are hard coded in the aforementioned wrapper, but that will soon be replaced
> by a fancy looking grep/find/sed/whatever that will insert the project name
> from the file path.
>
> I will continue to improve and work on this over the next few days.
Thanks!
>
> Gav...
>
>> -----Original Message-----
>> From: David Crossley [mailto:crossley@apache.org]
>> Sent: Tuesday, 21 July 2009 10:03 AM
>> To: rat-dev@incubator.apache.org
>> Subject: Re: RAT with BuildBot at Apache
>>
>> David Crossley wrote:
>>> Gavin wrote:
>>>>
>>>> Ross' suggestion went on to say that perhaps Clutch could be extended
>> by an
>>>> extra column or two and state whether a project passes RAT and if not
>> say
>>>> how many files fail the RAT validation.
>>>>
>>>> Would this be a useful addition?
>>>
>>> Yes definitely.
>>>
>>>> If so I can get to work on it Buildbot end
>>>> - all I need from you is where you want the results putting so that
>> Clutch
>>>> can use it, how you want it represented (.xml , .txt, .html) etc. I
>> could
>>>> for instance put it in svn somewhere, or perhaps Clutch could gather
>> the
>>>> results from a ci.apache.org/projects/rat-output/ location?
>>>
>>> As long as it is publicly accessible and in a structured format,
>>> then Clutch can handle it. Even just a CSV file would work.
>>>
>>> Perhaps a summary file at ci.apache.org/projects/rat-output/summary.txt
>>>
>>> #projectId,numFailed
>>> foo,12
>>> bar,0
>>>
>>> So, for each podling, Clutch would see if it is in your list.
>>> If "numFailed gt 0" then alert it.
>>>
>>> The table cell would link directly to each project's html RAT report.
>>>
>>> Prod me when you have something.
>>
>> By the way, no need to wait until you have RAT reports for
>> all incubator projects. If one is not in the buildbot list,
>> then Clutch will just give it a "?" and orange alert colour.
>>
>> -David
>> No virus found in this incoming message.
>> Checked by AVG - www.avg.com
>> Version: 8.5.392 / Virus Database: 270.13.20/2249 - Release Date: 07/21/09
>> 05:58:00
>
Re: RAT Summary Page (was: RE: RAT with BuildBot at Apache)
Posted by David Crossley <cr...@apache.org>.
David Crossley wrote:
> Gavin wrote:
> >
> > http://ci.apache.org/projects/rat-master-summary.html
> >
> > Still a basic interface currently but before I went too much further it
> > would be great if I could get feedback (from David mainly I guess in regards
> > to Clutch)
> >
> > What extra information does Clutch need to make it useful for it?
>
> That RAT page seems to be decent html. So we can just grab the
> relevant column data. That is sufficient.
I decided not to add such a column to the Clutch table.
Afraid that people will glance at Clutch, rather than
properly investigate a project's complete RAT report.
-David
Re: RAT Summary Page (was: RE: RAT with BuildBot at Apache)
Posted by David Crossley <cr...@apache.org>.
Gavin wrote:
>
> http://ci.apache.org/projects/rat-master-summary.html
>
> Still a basic interface currently but before I went too much further it
> would be great if I could get feedback (from David mainly I guess in regards
> to Clutch)
>
> What extra information does Clutch need to make it useful for it?
That RAT page seems to be decent html. So we can just grab the
relevant column data. That is sufficient.
-David
RAT Summary Page (was: RE: RAT with BuildBot at Apache)
Posted by Gavin <ga...@16degrees.com.au>.
OK, so back to this RAT Summary I promised.
I have a working version at:
http://ci.apache.org/projects/rat-master-summary.html
Still a basic interface currently but before I went too much further it
would be great if I could get feedback (from David mainly I guess in regards
to Clutch)
What extra information does Clutch need to make it useful for it?
There are (as of writing) 3 projects up there, these are the only 3 projects
using RAT at Buildbot that is configured to output xml, the others all being
txt. I guess I should leave .txt outputs as an alternative but change the
default so that all participating projects output .xml instead. This way I
get to convert the output to a better html presentation and at the same time
get to grab the xml content for the master rat summary page.
One main hurdle was (and still is to a point) matching the output with the
project it belongs to. As separate reports it was not a problem that in fact
no reports contain the name of the project. As a combined summary however
this information is most vital. For now, as part of as xsl that grabs the
content from each report, it then wraps it in a new <project name="blah">
element so then that can be used to get the project name into the output in
the correct column.
A tiny bit of a workaround is in place in that the project names currently
are hard coded in the aforementioned wrapper, but that will soon be replaced
by a fancy looking grep/find/sed/whatever that will insert the project name
from the file path.
I will continue to improve and work on this over the next few days.
Gav...
> -----Original Message-----
> From: David Crossley [mailto:crossley@apache.org]
> Sent: Tuesday, 21 July 2009 10:03 AM
> To: rat-dev@incubator.apache.org
> Subject: Re: RAT with BuildBot at Apache
>
> David Crossley wrote:
> > Gavin wrote:
> > >
> > > Ross' suggestion went on to say that perhaps Clutch could be extended
> by an
> > > extra column or two and state whether a project passes RAT and if not
> say
> > > how many files fail the RAT validation.
> > >
> > > Would this be a useful addition?
> >
> > Yes definitely.
> >
> > > If so I can get to work on it Buildbot end
> > > - all I need from you is where you want the results putting so that
> Clutch
> > > can use it, how you want it represented (.xml , .txt, .html) etc. I
> could
> > > for instance put it in svn somewhere, or perhaps Clutch could gather
> the
> > > results from a ci.apache.org/projects/rat-output/ location?
> >
> > As long as it is publicly accessible and in a structured format,
> > then Clutch can handle it. Even just a CSV file would work.
> >
> > Perhaps a summary file at ci.apache.org/projects/rat-output/summary.txt
> >
> > #projectId,numFailed
> > foo,12
> > bar,0
> >
> > So, for each podling, Clutch would see if it is in your list.
> > If "numFailed gt 0" then alert it.
> >
> > The table cell would link directly to each project's html RAT report.
> >
> > Prod me when you have something.
>
> By the way, no need to wait until you have RAT reports for
> all incubator projects. If one is not in the buildbot list,
> then Clutch will just give it a "?" and orange alert colour.
>
> -David
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.392 / Virus Database: 270.13.20/2249 - Release Date: 07/21/09
> 05:58:00
Re: RAT with BuildBot at Apache
Posted by David Crossley <cr...@apache.org>.
David Crossley wrote:
> Gavin wrote:
> >
> > Ross' suggestion went on to say that perhaps Clutch could be extended by an
> > extra column or two and state whether a project passes RAT and if not say
> > how many files fail the RAT validation.
> >
> > Would this be a useful addition?
>
> Yes definitely.
>
> > If so I can get to work on it Buildbot end
> > - all I need from you is where you want the results putting so that Clutch
> > can use it, how you want it represented (.xml , .txt, .html) etc. I could
> > for instance put it in svn somewhere, or perhaps Clutch could gather the
> > results from a ci.apache.org/projects/rat-output/ location?
>
> As long as it is publicly accessible and in a structured format,
> then Clutch can handle it. Even just a CSV file would work.
>
> Perhaps a summary file at ci.apache.org/projects/rat-output/summary.txt
>
> #projectId,numFailed
> foo,12
> bar,0
>
> So, for each podling, Clutch would see if it is in your list.
> If "numFailed gt 0" then alert it.
>
> The table cell would link directly to each project's html RAT report.
>
> Prod me when you have something.
By the way, no need to wait until you have RAT reports for
all incubator projects. If one is not in the buildbot list,
then Clutch will just give it a "?" and orange alert colour.
-David
Re: RAT with BuildBot at Apache
Posted by David Crossley <cr...@apache.org>.
Gavin wrote:
> Robert Burrell Donkin wrote:
> > Ross Gardler wrote:
> > > Gavin wrote:
> > >> One last thing I though of, I'm sure we can gather info from these
> > reports
> > >> and send it to wherever Clutch needs it in order to implement that
> > extra
> > >> line on clutch you mentioned Ross.
> > >
> > > That would be really cool. Perhaps the simplest thing to do would be
> > > to have your buildbot dump the files in a location that the existing
> > > Clutch code can access.
> >
> > +1
>
> FYI I'm going to email incubator list - I guess aimed at David (no idea if
> anyone else works on it)
I do subscribe to rat-dev and read occasionally.
However, recent email volumes on members@ swamped me.
Gavin sent private email today. I would rather discuss
on list, so i will bring it back here. See below.
> and see if he'll find this useful and if so where I
> can put the resulting html/xml reports. If accepted I'll get Buildbot to
> build RAT reports for all incubating projects. I'll also see about doing
> that summary page to try and make it useful as feedback for RAT project
> also.
>
> ----------------
> Just wanted to get a feel for whether or not Buildbot could do something for
> Clutch.
>
> On the rat-dev list, amongst some ideas thrown around, Ross suggested that
> perhaps Buildbot could test all incubator projects for RAT and produce
> individual plus an overall combined report.
>
> Ross' suggestion went on to say that perhaps Clutch could be extended by an
> extra column or two and state whether a project passes RAT and if not say
> how many files fail the RAT validation.
>
> Would this be a useful addition?
Yes definitely.
> If so I can get to work on it Buildbot end
> - all I need from you is where you want the results putting so that Clutch
> can use it, how you want it represented (.xml , .txt, .html) etc. I could
> for instance put it in svn somewhere, or perhaps Clutch could gather the
> results from a ci.apache.org/projects/rat-output/ location?
As long as it is publicly accessible and in a structured format,
then Clutch can handle it. Even just a CSV file would work.
Perhaps a summary file at ci.apache.org/projects/rat-output/summary.txt
#projectId,numFailed
foo,12
bar,0
So, for each podling, Clutch would see if it is in your list.
If "numFailed gt 0" then alert it.
The table cell would link directly to each project's html RAT report.
Prod me when you have something.
-David
RE: RAT with BuildBot at Apache
Posted by Gavin <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Robert Burrell Donkin [mailto:robertburrelldonkin@blueyonder.co.uk]
> Sent: Monday, 15 June 2009 7:57 AM
> To: rat-dev@incubator.apache.org
> Subject: Re: RAT with BuildBot at Apache
>
> Ross Gardler wrote:
> > 2009/6/12 Gavin <ga...@16degrees.com.au>:
> >>
> >>> -----Original Message-----
> >>> From: ross.gardler@googlemail.com [mailto:ross.gardler@googlemail.com]
> On
> >>> Behalf Of Ross Gardler
> >>> Sent: Tuesday, 9 June 2009 5:54 AM
> >>> To: rat-dev@incubator.apache.org
> >>> Subject: Re: Report Time
> >>>
> >> <snip>
>
> >> One last thing I though of, I'm sure we can gather info from these
> reports
> >> and send it to wherever Clutch needs it in order to implement that
> extra
> >> line on clutch you mentioned Ross.
> >
> > That would be really cool. Perhaps the simplest thing to do would be
> > to have your buildbot dump the files in a location that the existing
> > Clutch code can access.
>
> +1
>
> - robert
>
FYI I'm going to email incubator list - I guess aimed at David (no idea if
anyone else works on it) and see if he'll find this useful and if so where I
can put the resulting html/xml reports. If accepted I'll get Buildbot to
build RAT reports for all incubating projects. I'll also see about doing
that summary page to try and make it useful as feedback for RAT project
also.
Cheers
Gav...
]
RE: [SOLVED] - RE: RAT with BuildBot at Apache
Posted by Gavin <ga...@16degrees.com.au>.
> -----Original Message-----
> From: David Crossley [mailto:crossley@apache.org]
> Sent: Friday, 16 October 2009 4:59 PM
> To: rat-dev@incubator.apache.org
> Subject: Re: [SOLVED] - RE: RAT with BuildBot at Apache
>
> Gav... wrote:
> > David Crossley wrote:
> > >
> > > Okay, the initial etc/rat-avoid.txt is ready now.
> > > Forrest devs can refine it.
> >
> > Ok, changed and done.
> >
> > Results at
> > http://ci.apache.org/projects/forrest/rat-output.txt
>
> That improves it, thanks.
>
> I tweaked our config some more, but do not yet see
> the results updated. How often does Buildbot run,
> or how is it triggered?
For some projects, whilst testing I had been triggering them manually from
the IRC bot. 'forrest-bot' on the #asftest channel in this case. As Forrest
does not have its own channel currently, along with a few others, that's
where it lives.
(A discussion for Forrest list as to whether we keep it or disable it.)
I have now however enabled per commit builds of the RAT report for Forrest,
so it will recheck and update after every commit.
Gav...
>
> -David
Re: [SOLVED] - RE: RAT with BuildBot at Apache
Posted by David Crossley <cr...@apache.org>.
Gav... wrote:
> David Crossley wrote:
> >
> > Okay, the initial etc/rat-avoid.txt is ready now.
> > Forrest devs can refine it.
>
> Ok, changed and done.
>
> Results at
> http://ci.apache.org/projects/forrest/rat-output.txt
That improves it, thanks.
I tweaked our config some more, but do not yet see
the results updated. How often does Buildbot run,
or how is it triggered?
-David
Re: [SOLVED] - RE: RAT with BuildBot at Apache
Posted by "Gav..." <ga...@16degrees.com.au>.
On Fri, 16 Oct 2009 16:37:27 +1100, David Crossley <cr...@apache.org>
wrote:
> Gav... wrote:
>> David Crossley wrote:
>> >
>> > So perhaps the RAT file should be "rat-avoid.txt" or "rat-exclude.txt"
>> > and we just have duplicate similar files for each purpose.
>> >
>> > Should i go ahead and create that file for Forrest?
>>
>> Yep sure, I'll switch to that when done.
>
> Okay, the initial etc/rat-avoid.txt is ready now.
> Forrest devs can refine it.
Ok, changed and done.
Results at
http://ci.apache.org/projects/forrest/rat-output.txt
>
> By the way, could you also add the following (or similar)
> to the ant build file. This assists until RAT-67.
>
> ---------
> <tstamp/>
> <echo level="info">
> Timestamp: ${DSTAMP} ${TSTAMP}
> </echo>
> ---------
Done also.
Gav...
>
>> On a case by case basis, I dont expect many projects to provide such a
>> file,
>> in which case the normal exclude pattern list will do.
>
> Don't bet on it.
>
> -David
--
Gav...
16 degrees complete web solutions
Re: [SOLVED] - RE: RAT with BuildBot at Apache
Posted by David Crossley <cr...@apache.org>.
Gav... wrote:
> David Crossley wrote:
> >
> > So perhaps the RAT file should be "rat-avoid.txt" or "rat-exclude.txt"
> > and we just have duplicate similar files for each purpose.
> >
> > Should i go ahead and create that file for Forrest?
>
> Yep sure, I'll switch to that when done.
Okay, the initial etc/rat-avoid.txt is ready now.
Forrest devs can refine it.
By the way, could you also add the following (or similar)
to the ant build file. This assists until RAT-67.
---------
<tstamp/>
<echo level="info">
Timestamp: ${DSTAMP} ${TSTAMP}
</echo>
---------
> On a case by case basis, I dont expect many projects to provide such a
> file,
> in which case the normal exclude pattern list will do.
Don't bet on it.
-David
Re: [SOLVED] - RE: RAT with BuildBot at Apache
Posted by "Gav..." <ga...@16degrees.com.au>.
On Fri, 16 Oct 2009 13:52:57 +1100, David Crossley <cr...@apache.org>
wrote:
> Gavin wrote:
>> >
>> > Ok, I have it working now.
>> >
>> > http://ci.apache.org/projects/forrest/rat-output.txt
>> >
>> > That is upto date using the ant task and a forrest.xml build file,
>> > which
>> > contains excludes for all the files/dirs from the forrest license-
>> > avoid.txt
>> > file. So we are down to 111, but I do see more than can be excluded.
>> >
>> > I think it will be easy for me to convert that list
(license-avoid.txt)
>> > into
>> > a list of <exclude name="xxx"/> and have that in an external file,
that
>> > way
>> > it will dynamically update whenever that file changes.
>> >
>> > So, my last question for the list in this particular saga.
>> >
>> > How would I include that external file list into the build file into
th
>> > right place?
>> >
>> > So from:
>> >
>> > <fileset dir="${src-dir}">
>> > <exclude name=".buildbot-sourcedata"/>
>> > <exclude name="admin/"/>
>> > <exclude name="lib/"/>
>> > ...
>> > </fileset>
>> >
>> > to something like:
>> >
>> > <fileset dir="${src-dir}">
>> > -external file list of excludes goes here-
>> > </fileset>
>> >
>> > I'm looking at loadfile but not sure if that is appropriate.
>>
>> For the archives etc, 'excludesfile' is my friend, so I now have:
>>
>> <fileset dir="${src-dir}">
>> <excludesfile name="${src-dir}/etc/relicense-avoid.txt"/>
>> <exclude name=".buildbot-sourcedata"/>
>> <exclude name="**.htaccess"/>
>> </fileset>
>>
>> and it reads the .txt as is, brilliant.
>
> Beauty.
>
> One hitch in Forrest's case. That etc/relicense-avoid.txt
> is not a list of Ant-style patterns, it is a crude list
> of exact text patterns for my old relicense.pl script.
>
> So perhaps the RAT file should be "rat-avoid.txt" or "rat-exclude.txt"
> and we just have duplicate similar files for each purpose.
>
> Should i go ahead and create that file for Forrest?
Yep sure, I'll switch to that when done.
On a case by case basis, I dont expect many projects to provide such a
file,
in which case the normal exclude pattern list will do.
Thanks
Gav...
>
>> Apologies David for taking so long,
>
> No worries. Glad that we could get RAT processing improved.
>
>> and so long to realise that actually,
>> the ant task way of doing it, is actually better in this scenario now.
>>
>> RAT and Buildbot combo are now once again, much more useful.
>>
>> Gav...
>
> Yep, fantastic ... thanks too to Ant.
>
> -David
--
Gav...
16 degrees complete web solutions
Re: [SOLVED] - RE: RAT with BuildBot at Apache
Posted by David Crossley <cr...@apache.org>.
Gavin wrote:
> >
> > Ok, I have it working now.
> >
> > http://ci.apache.org/projects/forrest/rat-output.txt
> >
> > That is upto date using the ant task and a forrest.xml build file, which
> > contains excludes for all the files/dirs from the forrest license-
> > avoid.txt
> > file. So we are down to 111, but I do see more than can be excluded.
> >
> > I think it will be easy for me to convert that list (license-avoid.txt)
> > into
> > a list of <exclude name="xxx"/> and have that in an external file, that
> > way
> > it will dynamically update whenever that file changes.
> >
> > So, my last question for the list in this particular saga.
> >
> > How would I include that external file list into the build file into th
> > right place?
> >
> > So from:
> >
> > <fileset dir="${src-dir}">
> > <exclude name=".buildbot-sourcedata"/>
> > <exclude name="admin/"/>
> > <exclude name="lib/"/>
> > ...
> > </fileset>
> >
> > to something like:
> >
> > <fileset dir="${src-dir}">
> > -external file list of excludes goes here-
> > </fileset>
> >
> > I'm looking at loadfile but not sure if that is appropriate.
>
> For the archives etc, 'excludesfile' is my friend, so I now have:
>
> <fileset dir="${src-dir}">
> <excludesfile name="${src-dir}/etc/relicense-avoid.txt"/>
> <exclude name=".buildbot-sourcedata"/>
> <exclude name="**.htaccess"/>
> </fileset>
>
> and it reads the .txt as is, brilliant.
Beauty.
One hitch in Forrest's case. That etc/relicense-avoid.txt
is not a list of Ant-style patterns, it is a crude list
of exact text patterns for my old relicense.pl script.
So perhaps the RAT file should be "rat-avoid.txt" or "rat-exclude.txt"
and we just have duplicate similar files for each purpose.
Should i go ahead and create that file for Forrest?
> Apologies David for taking so long,
No worries. Glad that we could get RAT processing improved.
> and so long to realise that actually,
> the ant task way of doing it, is actually better in this scenario now.
>
> RAT and Buildbot combo are now once again, much more useful.
>
> Gav...
Yep, fantastic ... thanks too to Ant.
-David
[SOLVED] - RE: RAT with BuildBot at Apache
Posted by Gavin <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Gavin [mailto:gavin@16degrees.com.au]
> Sent: Thursday, 15 October 2009 11:12 PM
> To: rat-dev@incubator.apache.org
> Subject: RE: RAT with BuildBot at Apache
>
>
>
> > -----Original Message-----
> > From: Gavin [mailto:gavin@16degrees.com.au]
> > Sent: Thursday, 15 October 2009 8:16 PM
> > To: rat-dev@incubator.apache.org
> > Subject: RE: RAT with BuildBot at Apache
> >
> >
> >
> > > -----Original Message-----
> > > From: David Crossley [mailto:crossley@apache.org]
> > > Sent: Thursday, 15 October 2009 10:25 AM
> > > To: rat-dev@incubator.apache.org
> > > Subject: Re: RAT with BuildBot at Apache
> > >
> > > Gav... wrote:
> > > > David Crossley wrote:
>
> <snip>
>
> > >
> > > I was trying to help you with the Ant task, not
> > > the commandline.
> > >
> > > It seems to me that the Ant task would have better
> > > abilities to "exclude" stuff.
> >
> > Ok, I give in, I'll do it that way. I would have preferred to have
> > specified
> > the excludes in a command line argument from within the python build
> > script.
> >
> > This way I have to create separate build.xml files on the filesystem,
> > messy
> > in my opinion, as all the build directories for the builds are wiped
> clean
> > each time and a fresh svn checkout provided, so I can not place the ant
> > tasks file in the same place as the rest of the build.
> >
> > Working on it now, thanks for the hints.
> >
>
> Ok, I have it working now.
>
> http://ci.apache.org/projects/forrest/rat-output.txt
>
> That is upto date using the ant task and a forrest.xml build file, which
> contains excludes for all the files/dirs from the forrest license-
> avoid.txt
> file. So we are down to 111, but I do see more than can be excluded.
>
> I think it will be easy for me to convert that list (license-avoid.txt)
> into
> a list of <exclude name="xxx"/> and have that in an external file, that
> way
> it will dynamically update whenever that file changes.
>
> So, my last question for the list in this particular saga.
>
> How would I include that external file list into the build file into th
> right place?
>
> So from:
>
> <fileset dir="${src-dir}">
> <exclude name=".buildbot-sourcedata"/>
> <exclude name="admin/"/>
> <exclude name="lib/"/>
> ...
> </fileset>
>
> to something like:
>
> <fileset dir="${src-dir}">
> -external file list of excludes goes here-
> </fileset>
>
> I'm looking at loadfile but not sure if that is appropriate.
For the archives etc, 'excludesfile' is my friend, so I now have:
<fileset dir="${src-dir}">
<excludesfile name="${src-dir}/etc/relicense-avoid.txt"/>
<exclude name=".buildbot-sourcedata"/>
<exclude name="**.htaccess"/>
</fileset>
and it reads the .txt as is, brilliant.
Apologies David for taking so long, and so long to realise that actually,
the ant task way of doing it, is actually better in this scenario now.
RAT and Buildbot combo are now once again, much more useful.
Gav...
>
> Thanks
>
> Gav...
>
> > Gav...
> >
> >
> > >
> > > -David
> > >
> > > > RAT63 will already be applied as Buildbot always compiles the latest
> > > > snapshot from RAT trunk, so buildbot is using the latest, including
> > > > the above fix.
> > > >
> > > > so,
> > > >
> > > > --exlcude somefile.name somefirstLevelDir/ someotherdir/
> > > >
> > > > does work
> > > >
> > > > but,
> > > >
> > > > --exclude firstLevel/secondlevel/file.name firstLevel/secondLevel/**
> > > >
> > > > does not.
> > > >
> > > > I can not use the maven way I dont think.
> > > >
> > > > > I am keen to help because Buildbot/RAT is giving
> > > > > Forrest a bad rap due to known exclusions not being
> > > > > sufficiently configured.
> > > >
> > > > Yes, and now having mentioned it on the infra list again this is my
> > > > number one priority, I apologise,but I have mentioned the problem
> > above
> > > > on this list before and not had a specific reply to cure it. I hope
> > > > someone can offer ideas, but i will try again to get it working
> > myself.
> > > No virus found in this incoming message.
> > > Checked by AVG - www.avg.com
> > > Version: 8.5.421 / Virus Database: 270.14.17/2436 - Release Date:
> > 10/14/09
> > > 18:32:00
> >
> > No virus found in this incoming message.
> > Checked by AVG - www.avg.com
> > Version: 8.5.421 / Virus Database: 270.14.17/2436 - Release Date:
> 10/15/09
> > 03:57:00
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.421 / Virus Database: 270.14.17/2436 - Release Date: 10/15/09
> 03:57:00
RE: RAT with BuildBot at Apache
Posted by Gavin <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Gavin [mailto:gavin@16degrees.com.au]
> Sent: Thursday, 15 October 2009 8:16 PM
> To: rat-dev@incubator.apache.org
> Subject: RE: RAT with BuildBot at Apache
>
>
>
> > -----Original Message-----
> > From: David Crossley [mailto:crossley@apache.org]
> > Sent: Thursday, 15 October 2009 10:25 AM
> > To: rat-dev@incubator.apache.org
> > Subject: Re: RAT with BuildBot at Apache
> >
> > Gav... wrote:
> > > David Crossley wrote:
<snip>
> >
> > I was trying to help you with the Ant task, not
> > the commandline.
> >
> > It seems to me that the Ant task would have better
> > abilities to "exclude" stuff.
>
> Ok, I give in, I'll do it that way. I would have preferred to have
> specified
> the excludes in a command line argument from within the python build
> script.
>
> This way I have to create separate build.xml files on the filesystem,
> messy
> in my opinion, as all the build directories for the builds are wiped clean
> each time and a fresh svn checkout provided, so I can not place the ant
> tasks file in the same place as the rest of the build.
>
> Working on it now, thanks for the hints.
>
Ok, I have it working now.
http://ci.apache.org/projects/forrest/rat-output.txt
That is upto date using the ant task and a forrest.xml build file, which
contains excludes for all the files/dirs from the forrest license-avoid.txt
file. So we are down to 111, but I do see more than can be excluded.
I think it will be easy for me to convert that list (license-avoid.txt) into
a list of <exclude name="xxx"/> and have that in an external file, that way
it will dynamically update whenever that file changes.
So, my last question for the list in this particular saga.
How would I include that external file list into the build file into th
right place?
So from:
<fileset dir="${src-dir}">
<exclude name=".buildbot-sourcedata"/>
<exclude name="admin/"/>
<exclude name="lib/"/>
...
</fileset>
to something like:
<fileset dir="${src-dir}">
-external file list of excludes goes here-
</fileset>
I'm looking at loadfile but not sure if that is appropriate.
Thanks
Gav...
> Gav...
>
>
> >
> > -David
> >
> > > RAT63 will already be applied as Buildbot always compiles the latest
> > > snapshot from RAT trunk, so buildbot is using the latest, including
> > > the above fix.
> > >
> > > so,
> > >
> > > --exlcude somefile.name somefirstLevelDir/ someotherdir/
> > >
> > > does work
> > >
> > > but,
> > >
> > > --exclude firstLevel/secondlevel/file.name firstLevel/secondLevel/**
> > >
> > > does not.
> > >
> > > I can not use the maven way I dont think.
> > >
> > > > I am keen to help because Buildbot/RAT is giving
> > > > Forrest a bad rap due to known exclusions not being
> > > > sufficiently configured.
> > >
> > > Yes, and now having mentioned it on the infra list again this is my
> > > number one priority, I apologise,but I have mentioned the problem
> above
> > > on this list before and not had a specific reply to cure it. I hope
> > > someone can offer ideas, but i will try again to get it working
> myself.
> > No virus found in this incoming message.
> > Checked by AVG - www.avg.com
> > Version: 8.5.421 / Virus Database: 270.14.17/2436 - Release Date:
> 10/14/09
> > 18:32:00
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.421 / Virus Database: 270.14.17/2436 - Release Date: 10/15/09
> 03:57:00
RE: RAT with BuildBot at Apache
Posted by Gavin <ga...@16degrees.com.au>.
> -----Original Message-----
> From: David Crossley [mailto:crossley@apache.org]
> Sent: Thursday, 15 October 2009 10:25 AM
> To: rat-dev@incubator.apache.org
> Subject: Re: RAT with BuildBot at Apache
>
> Gav... wrote:
> > David Crossley wrote:
> > > David Crossley wrote:
> > >> Gavin wrote:
> > >> >
> > >> > While I'm here, I'm now just playing with ant-task-examples.xml but
> I
> > >> > can't
> > >> > get it working. I renamed it to build.xml and tried to use it as
> is.
> > >> >
> > >> > The error I get is :-
> > >> >
> > >> > D:\Apache2\rat-trunk>ant javasources
> > >> > Buildfile: build.xml
> > >> >
> > >> > -taskdef:
> > >> >
> > >> > javasources:
> > >> >
> > >> > BUILD FAILED
> > >> > D:\Apache2\rat-trunk\build.xml:33: Problem: failed to create task
> or
> > >> > type
> > >> > antlib
> > >> > :org.apache.rat.anttasks:report
> > >> > Cause: The name is undefined.
> > >> > Action: Check the spelling.
> > >> > Action: Check that any custom tasks/types have been declared.
> > >> > Action: Check that any <presetdef>/<macrodef> declarations have
> taken
> > >> > place.
> > >> > No types or tasks have been defined in this namespace yet
> > >> >
> > >> > This appears to be an antlib declaration.
> > >> > Action: Check that the implementing library exists in one of:
> > >> > -D:\Apache2\forrest\tools\ant\lib
> > >> > -C:\Documents and Settings\Gavin\.ant\lib
> > >> > -a directory added on the command line with the -lib
> argument
> > >> >
> > >> > I'm assuming user error, any clues?
> > >>
> > >> Looking at that file (i don't have Maven or RAT locally
> > >> so cannot really help) ...
> > >>
> > >> In the "-taskdef" target, do you have the specifed jars installed?
> > >>
> > >> <include name="**/apache-rat*.jar"/>
> > >> <include name="**/commons-*.jar"/>
> > >>
> > >> Look at the top-level rat/main/trunk/pom.xml
> > >> to get hints about those dependencies.
> > >
> > > Ping. Gavin, did my comments here (and Giridharan's at RAT-63)
> > > assist you with this?
> >
> > I dont think this is related any more. The problem is that I can not
> > specify excludes from the command line below the first level.
>
> I was trying to help you with the Ant task, not
> the commandline.
>
> It seems to me that the Ant task would have better
> abilities to "exclude" stuff.
Ok, I give in, I'll do it that way. I would have preferred to have specified
the excludes in a command line argument from within the python build script.
This way I have to create separate build.xml files on the filesystem, messy
in my opinion, as all the build directories for the builds are wiped clean
each time and a fresh svn checkout provided, so I can not place the ant
tasks file in the same place as the rest of the build.
Working on it now, thanks for the hints.
Gav...
>
> -David
>
> > RAT63 will already be applied as Buildbot always compiles the latest
> > snapshot from RAT trunk, so buildbot is using the latest, including
> > the above fix.
> >
> > so,
> >
> > --exlcude somefile.name somefirstLevelDir/ someotherdir/
> >
> > does work
> >
> > but,
> >
> > --exclude firstLevel/secondlevel/file.name firstLevel/secondLevel/**
> >
> > does not.
> >
> > I can not use the maven way I dont think.
> >
> > > I am keen to help because Buildbot/RAT is giving
> > > Forrest a bad rap due to known exclusions not being
> > > sufficiently configured.
> >
> > Yes, and now having mentioned it on the infra list again this is my
> > number one priority, I apologise,but I have mentioned the problem above
> > on this list before and not had a specific reply to cure it. I hope
> > someone can offer ideas, but i will try again to get it working myself.
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.421 / Virus Database: 270.14.17/2436 - Release Date: 10/14/09
> 18:32:00
Re: RAT with BuildBot at Apache
Posted by David Crossley <cr...@apache.org>.
Gav... wrote:
> David Crossley wrote:
> > David Crossley wrote:
> >> Gavin wrote:
> >> >
> >> > While I'm here, I'm now just playing with ant-task-examples.xml but I
> >> > can't
> >> > get it working. I renamed it to build.xml and tried to use it as is.
> >> >
> >> > The error I get is :-
> >> >
> >> > D:\Apache2\rat-trunk>ant javasources
> >> > Buildfile: build.xml
> >> >
> >> > -taskdef:
> >> >
> >> > javasources:
> >> >
> >> > BUILD FAILED
> >> > D:\Apache2\rat-trunk\build.xml:33: Problem: failed to create task or
> >> > type
> >> > antlib
> >> > :org.apache.rat.anttasks:report
> >> > Cause: The name is undefined.
> >> > Action: Check the spelling.
> >> > Action: Check that any custom tasks/types have been declared.
> >> > Action: Check that any <presetdef>/<macrodef> declarations have taken
> >> > place.
> >> > No types or tasks have been defined in this namespace yet
> >> >
> >> > This appears to be an antlib declaration.
> >> > Action: Check that the implementing library exists in one of:
> >> > -D:\Apache2\forrest\tools\ant\lib
> >> > -C:\Documents and Settings\Gavin\.ant\lib
> >> > -a directory added on the command line with the -lib argument
> >> >
> >> > I'm assuming user error, any clues?
> >>
> >> Looking at that file (i don't have Maven or RAT locally
> >> so cannot really help) ...
> >>
> >> In the "-taskdef" target, do you have the specifed jars installed?
> >>
> >> <include name="**/apache-rat*.jar"/>
> >> <include name="**/commons-*.jar"/>
> >>
> >> Look at the top-level rat/main/trunk/pom.xml
> >> to get hints about those dependencies.
> >
> > Ping. Gavin, did my comments here (and Giridharan's at RAT-63)
> > assist you with this?
>
> I dont think this is related any more. The problem is that I can not
> specify excludes from the command line below the first level.
I was trying to help you with the Ant task, not
the commandline.
It seems to me that the Ant task would have better
abilities to "exclude" stuff.
-David
> RAT63 will already be applied as Buildbot always compiles the latest
> snapshot from RAT trunk, so buildbot is using the latest, including
> the above fix.
>
> so,
>
> --exlcude somefile.name somefirstLevelDir/ someotherdir/
>
> does work
>
> but,
>
> --exclude firstLevel/secondlevel/file.name firstLevel/secondLevel/**
>
> does not.
>
> I can not use the maven way I dont think.
>
> > I am keen to help because Buildbot/RAT is giving
> > Forrest a bad rap due to known exclusions not being
> > sufficiently configured.
>
> Yes, and now having mentioned it on the infra list again this is my
> number one priority, I apologise,but I have mentioned the problem above
> on this list before and not had a specific reply to cure it. I hope
> someone can offer ideas, but i will try again to get it working myself.
Re: RAT with BuildBot at Apache
Posted by "Gav..." <m0...@16degrees.com.au>.
On Wed, 14 Oct 2009 12:08:35 +1100, David Crossley <cr...@apache.org>
wrote:
> David Crossley wrote:
>> Gavin wrote:
>> >
>> > While I'm here, I'm now just playing with ant-task-examples.xml but I
>> > can't
>> > get it working. I renamed it to build.xml and tried to use it as is.
>> >
>> > The error I get is :-
>> >
>> > D:\Apache2\rat-trunk>ant javasources
>> > Buildfile: build.xml
>> >
>> > -taskdef:
>> >
>> > javasources:
>> >
>> > BUILD FAILED
>> > D:\Apache2\rat-trunk\build.xml:33: Problem: failed to create task or
>> > type
>> > antlib
>> > :org.apache.rat.anttasks:report
>> > Cause: The name is undefined.
>> > Action: Check the spelling.
>> > Action: Check that any custom tasks/types have been declared.
>> > Action: Check that any <presetdef>/<macrodef> declarations have taken
>> > place.
>> > No types or tasks have been defined in this namespace yet
>> >
>> > This appears to be an antlib declaration.
>> > Action: Check that the implementing library exists in one of:
>> > -D:\Apache2\forrest\tools\ant\lib
>> > -C:\Documents and Settings\Gavin\.ant\lib
>> > -a directory added on the command line with the -lib argument
>> >
>> > I'm assuming user error, any clues?
>>
>> Looking at that file (i don't have Maven or RAT locally
>> so cannot really help) ...
>>
>> In the "-taskdef" target, do you have the specifed jars installed?
>>
>> <include name="**/apache-rat*.jar"/>
>> <include name="**/commons-*.jar"/>
>>
>> Look at the top-level rat/main/trunk/pom.xml
>> to get hints about those dependencies.
>
> Ping. Gavin, did my comments here (and Giridharan's at RAT-63)
> assist you with this?
I dont think this is related any more. The problem is that I can not
specify excludes from the command line below the first level.
RAT63 will already be applied as Buildbot always compiles the latest
snapshot from RAT trunk, so buildbot is using the latest, including
the above fix.
so,
--exlcude somefile.name somefirstLevelDir/ someotherdir/
does work
but,
--exclude firstLevel/secondlevel/file.name firstLevel/secondLevel/**
does not.
I can not use the maven way I dont think.
>
> I am keen to help because Buildbot/RAT is giving
> Forrest a bad rap due to known exclusions not being
> sufficiently configured.
>
Yes, and now having mentioned it on the infra list again this is my
number one priority, I apologise,but I have mentioned the problem above
on this list before and not had a specific reply to cure it. I hope
someone can offer ideas, but i will try again to get it working myself.
> -David
--
Gav...
Re: RAT with BuildBot at Apache
Posted by David Crossley <cr...@apache.org>.
David Crossley wrote:
> Gavin wrote:
> >
> > While I'm here, I'm now just playing with ant-task-examples.xml but I can't
> > get it working. I renamed it to build.xml and tried to use it as is.
> >
> > The error I get is :-
> >
> > D:\Apache2\rat-trunk>ant javasources
> > Buildfile: build.xml
> >
> > -taskdef:
> >
> > javasources:
> >
> > BUILD FAILED
> > D:\Apache2\rat-trunk\build.xml:33: Problem: failed to create task or type
> > antlib
> > :org.apache.rat.anttasks:report
> > Cause: The name is undefined.
> > Action: Check the spelling.
> > Action: Check that any custom tasks/types have been declared.
> > Action: Check that any <presetdef>/<macrodef> declarations have taken place.
> > No types or tasks have been defined in this namespace yet
> >
> > This appears to be an antlib declaration.
> > Action: Check that the implementing library exists in one of:
> > -D:\Apache2\forrest\tools\ant\lib
> > -C:\Documents and Settings\Gavin\.ant\lib
> > -a directory added on the command line with the -lib argument
> >
> > I'm assuming user error, any clues?
>
> Looking at that file (i don't have Maven or RAT locally
> so cannot really help) ...
>
> In the "-taskdef" target, do you have the specifed jars installed?
>
> <include name="**/apache-rat*.jar"/>
> <include name="**/commons-*.jar"/>
>
> Look at the top-level rat/main/trunk/pom.xml
> to get hints about those dependencies.
Ping. Gavin, did my comments here (and Giridharan's at RAT-63)
assist you with this?
I am keen to help because Buildbot/RAT is giving
Forrest a bad rap due to known exclusions not being
sufficiently configured.
-David
Re: RAT with BuildBot at Apache
Posted by David Crossley <cr...@apache.org>.
Gavin wrote:
>
> While I'm here, I'm now just playing with ant-task-examples.xml but I can't
> get it working. I renamed it to build.xml and tried to use it as is.
>
> The error I get is :-
>
> D:\Apache2\rat-trunk>ant javasources
> Buildfile: build.xml
>
> -taskdef:
>
> javasources:
>
> BUILD FAILED
> D:\Apache2\rat-trunk\build.xml:33: Problem: failed to create task or type
> antlib
> :org.apache.rat.anttasks:report
> Cause: The name is undefined.
> Action: Check the spelling.
> Action: Check that any custom tasks/types have been declared.
> Action: Check that any <presetdef>/<macrodef> declarations have taken place.
> No types or tasks have been defined in this namespace yet
>
> This appears to be an antlib declaration.
> Action: Check that the implementing library exists in one of:
> -D:\Apache2\forrest\tools\ant\lib
> -C:\Documents and Settings\Gavin\.ant\lib
> -a directory added on the command line with the -lib argument
>
> I'm assuming user error, any clues?
Looking at that file (i don't have Maven or RAT locally
so cannot really help) ...
In the "-taskdef" target, do you have the specifed jars installed?
<include name="**/apache-rat*.jar"/>
<include name="**/commons-*.jar"/>
Look at the top-level rat/main/trunk/pom.xml
to get hints about those dependencies.
-David
Unknown Licenses [WAS Re: RAT with BuildBot at Apache]
Posted by Robert Burrell Donkin <ro...@blueyonder.co.uk>.
Gavin wrote:
<snip>
> Using the RAT report above, we have 9 unknown licenses.
> Unknown licenses includes no license at all so may need to re-word that,
> anyway ..
RAT uses heuristic reasoning. unapproved licenses are quite rare (mainly
because RAT understands far too few headers ATM). so, most issues
reported are when RAT cannot deduce a license for a document, hence
'unknown license'. almost all of the time this is caused by an 'unknown
header'.
the best solution is to detached license information: for documents
which can't embed a header, a detached description of the license should
be used.
exclusions are ok for directories like target and local files, but for
documents in source control really license information is required for
all. so, probably as well as exclusions we need to be able to specify
license information for files.
on the command line perhaps something like:
-LApacheLicense20="src/test/SpecialTestFile.txt"
--license ApacheLicense20="src/test/SpecialTestFile.txt"
- robert
RE: RAT with BuildBot at Apache
Posted by Gavin <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Robert Burrell Donkin [mailto:robertburrelldonkin@blueyonder.co.uk]
> Sent: Wednesday, 17 June 2009 8:20 PM
> To: rat-dev@incubator.apache.org
> Subject: Re: RAT with BuildBot at Apache
>
> Gavin wrote:
> >
> >> -----Original Message-----
> >> From: Robert Burrell Donkin
> [mailto:robertburrelldonkin@blueyonder.co.uk]
> >> Sent: Wednesday, 17 June 2009 6:34 AM
> >> To: rat-dev@incubator.apache.org
> >> Subject: Re: RAT with BuildBot at Apache
> >
> > <snip>
> >
> > Of those 9 licenses
> >
> > Empty.txt, notjavadoc.html, bad.txt, src.txt -- are all intentionally
> not
> > licensed and are there as examples.
> >
> > So that leaves us with 5 possible 'real' contenders in RAT for license
> > problems.
> >
> > BUILD.txt - I don't see why we can't add a licence header here, others
> do.
> >
> > src/main/java/org/apache/rat/report/claim/ClaimStatistic.java
> >
> >
> src/main/java/org/apache/rat/report/claim/impl/AbstractClaimReporter.java
> >
> > src/main/java/org/apache/rat/report/claim/impl/ClaimAggregator.java
> >
> > src/test/resources/elements/Source.java
> >
> > All those look like genuine contenders to have licences added.
> >
> > Shall I attach a patch somewhere for those 5 ?
>
> +1
>
> <snip>
Done, patch attached to https://issues.apache.org/jira/browse/RAT-62 sorry
for delay.
Gav...
>
> - robert
>
Re: RAT with BuildBot at Apache
Posted by Robert Burrell Donkin <ro...@blueyonder.co.uk>.
Gavin wrote:
>
>> -----Original Message-----
>> From: Robert Burrell Donkin [mailto:robertburrelldonkin@blueyonder.co.uk]
>> Sent: Wednesday, 17 June 2009 6:34 AM
>> To: rat-dev@incubator.apache.org
>> Subject: Re: RAT with BuildBot at Apache
>
> <snip>
>
>> i've refactored the code and added a simple example (--exclude which
>> uses Commons IO wildcard file filters).
>
> Woohoo, excellent thanks!
>
> And it works just great. Buildbot picked up the new code and deployed the
> new Jar into operation just a minute or so after you made the commit :)
cool
> All I needed to do then was add my excludes to the rat.conf build file, then
> wait some seconds whilst Saxon converted the resultant xml report into html
> and then deployed to the website for viewing. I love it when it all works.
>
> http://ci.apache.org/projects/rat/rat-report.html
>
> Is the up to date HTML report -- as mentioned the HTML could do with a bit
> of tweaking to pretty it up and make it worthwhile looking at over the txt
> version. Which I'll do soon to get it out of the way.
>
> The important thing here is we now have useful and accurate reports.
great
<snip>
> Of those 9 licenses
>
> Empty.txt, notjavadoc.html, bad.txt, src.txt -- are all intentionally not
> licensed and are there as examples.
>
> So that leaves us with 5 possible 'real' contenders in RAT for license
> problems.
>
> BUILD.txt - I don't see why we can't add a licence header here, others do.
>
> src/main/java/org/apache/rat/report/claim/ClaimStatistic.java
>
> src/main/java/org/apache/rat/report/claim/impl/AbstractClaimReporter.java
>
> src/main/java/org/apache/rat/report/claim/impl/ClaimAggregator.java
>
> src/test/resources/elements/Source.java
>
> All those look like genuine contenders to have licences added.
>
> Shall I attach a patch somewhere for those 5 ?
+1
<snip>
> Yeah I figured something like that, Ant's site is a nightmare and I can
> never find what jar I need that is required for 'antlib', I'll keep looking,
> thanks.
antlibs are an pluggable extension mechanism
open http://ant.apache.org/manual/ then 'Concepts and Types' from the
left then scroll down to antlibs
but maybe you'd be better with an example. scan
(http://svn.apache.org/repos/asf/incubator/rat/scan/trunk/) audits
distributions on www.apache.org and
http://svn.apache.org/repos/asf/incubator/rat/scan/trunk/build.xml is an
example of the rat antlib in action.
- robert
Re: RAT with BuildBot at Apache
Posted by Ross Gardler <rg...@apache.org>.
2009/6/17 Gavin <ga...@16degrees.com.au>:
>
>
>> -----Original Message-----
>> From: Robert Burrell Donkin [mailto:robertburrelldonkin@blueyonder.co.uk]
>> Sent: Wednesday, 17 June 2009 6:34 AM
>> To: rat-dev@incubator.apache.org
>> Subject: Re: RAT with BuildBot at Apache
>
> <snip>
>
>>
>> i've refactored the code and added a simple example (--exclude which
>> uses Commons IO wildcard file filters).
>
> Woohoo, excellent thanks!
>
> And it works just great. Buildbot picked up the new code and deployed the
> new Jar into operation just a minute or so after you made the commit :)
>
> All I needed to do then was add my excludes to the rat.conf build file, then
> wait some seconds whilst Saxon converted the resultant xml report into html
> and then deployed to the website for viewing. I love it when it all works.
>
> http://ci.apache.org/projects/rat/rat-report.html
Beuatiful!
Thanks guys.
Ross
RE: RAT with BuildBot at Apache
Posted by Gavin <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Robert Burrell Donkin [mailto:robertburrelldonkin@blueyonder.co.uk]
> Sent: Wednesday, 17 June 2009 6:34 AM
> To: rat-dev@incubator.apache.org
> Subject: Re: RAT with BuildBot at Apache
<snip>
>
> i've refactored the code and added a simple example (--exclude which
> uses Commons IO wildcard file filters).
Woohoo, excellent thanks!
And it works just great. Buildbot picked up the new code and deployed the
new Jar into operation just a minute or so after you made the commit :)
All I needed to do then was add my excludes to the rat.conf build file, then
wait some seconds whilst Saxon converted the resultant xml report into html
and then deployed to the website for viewing. I love it when it all works.
http://ci.apache.org/projects/rat/rat-report.html
Is the up to date HTML report -- as mentioned the HTML could do with a bit
of tweaking to pretty it up and make it worthwhile looking at over the txt
version. Which I'll do soon to get it out of the way.
The important thing here is we now have useful and accurate reports.
Using the RAT report above, we have 9 unknown licenses.
Unknown licenses includes no license at all so may need to re-word that,
anyway ..
Of those 9 licenses
Empty.txt, notjavadoc.html, bad.txt, src.txt -- are all intentionally not
licensed and are there as examples.
So that leaves us with 5 possible 'real' contenders in RAT for license
problems.
BUILD.txt - I don't see why we can't add a licence header here, others do.
src/main/java/org/apache/rat/report/claim/ClaimStatistic.java
src/main/java/org/apache/rat/report/claim/impl/AbstractClaimReporter.java
src/main/java/org/apache/rat/report/claim/impl/ClaimAggregator.java
src/test/resources/elements/Source.java
All those look like genuine contenders to have licences added.
Shall I attach a patch somewhere for those 5 ?
Then I believe we are 'clean' and ready to act as a good example :)
> i'd be happy to review a patch
> for something along the lines of your suggestion (i've refactored the
> code so it should be easier to do now - just take a look at the
> excludesimplementation in
> http://svn.apache.org/repos/asf/incubator/rat/main/trunk/apache-rat-
> core/src/main/java/org/apache/rat/Report.java)
Great, I'll take a look at that soon, thanks.
>
>
> > While I'm here, I'm now just playing with ant-task-examples.xml but I
> can't
> > get it working. I renamed it to build.xml and tried to use it as is.
> >
> > The error I get is :-
> >
> > D:\Apache2\rat-trunk>ant javasources
> > Buildfile: build.xml
> >
> > -taskdef:
> >
> > javasources:
> >
> > BUILD FAILED
> > D:\Apache2\rat-trunk\build.xml:33: Problem: failed to create task or
> type
> > antlib
> > :org.apache.rat.anttasks:report
> > Cause: The name is undefined.
> > Action: Check the spelling.
> > Action: Check that any custom tasks/types have been declared.
> > Action: Check that any <presetdef>/<macrodef> declarations have taken
> place.
> > No types or tasks have been defined in this namespace yet
> >
> > This appears to be an antlib declaration.
> > Action: Check that the implementing library exists in one of:
> > -D:\Apache2\forrest\tools\ant\lib
> > -C:\Documents and Settings\Gavin\.ant\lib
> > -a directory added on the command line with the -lib argument
> >
> > I'm assuming user error, any clues?
>
> i think that you either need to install the rat antlib locally (see ant
> manual since it's fiddly) or add a classpath containing the jars required
Yeah I figured something like that, Ant's site is a nightmare and I can
never find what jar I need that is required for 'antlib', I'll keep looking,
thanks.
Gav...
>
> - robert
>
Re: RAT with BuildBot at Apache
Posted by Robert Burrell Donkin <ro...@blueyonder.co.uk>.
Gavin wrote:
>
>> -----Original Message-----
>> From: Robert Burrell Donkin [mailto:robertburrelldonkin@blueyonder.co.uk]
>> Sent: Tuesday, 16 June 2009 7:33 AM
>> To: rat-dev@incubator.apache.org
>> Subject: Re: RAT with BuildBot at Apache
>>
>> Ross Gardler wrote:
>>> 2009/6/15 Gavin <ga...@16degrees.com.au>:
>>>>> -----Original Message-----
>>>>> From: ross.gardler@googlemail.com [mailto:ross.gardler@googlemail.com]
>> On
>>>>> Behalf Of Ross Gardler
>>>>> Sent: Monday, 15 June 2009 10:09 AM
>>>>> To: rat-dev@incubator.apache.org
>>>>> Subject: Re: RAT with BuildBot at Apache
>>>>>
>>>>> 2009/6/15 Gavin <ga...@16degrees.com.au>:
>>>>>
>>>>>> Is that possible now, excluding sub-dirs ? If so how as I'd love to
>>>>>> implement it so we can have more accurate results to work from for
>>>>> refining.
>>>>>
>>>>> Here's an example from the Simal project (using Maven):
>>>>>
>>>>> <plugin>
>>>>> <groupId>org.codehaus.mojo</groupId>
>>>>> <artifactId>rat-maven-plugin</artifactId>
>>>>> <configuration>
>>>>> <excludes>
>>>>> <exclude>.pmd</exclude>
>>>>> <exclude>derby.log</exclude>
>>>>> <exclude>reports/**/*</exclude>
>>>>> <exclude>simal/**/*</exclude>
>>>>> <exclude>build/**/*</exclude>
>>>>> <exclude>simalDOAPFilestore/**/*</exclude>
>>>>> <exclude>simalRepository/**/*</exclude>
>>>>> <exclude>local.simal.properties</exclude>
>>>>> <exclude>local.simal.test.properties</exclude>
>>>>> </excludes>
>>>>> </configuration>
>>>>> <executions>
>>>>> <execution>
>>>>> <phase>verify</phase>
>>>>> <goals>
>>>>> <goal>check</goal>
>>>>> </goals>
>>>>> </execution>
>>>>> </executions>
>>>>> </plugin>
>>>> Ah, ok great thanks. My implementations so far have just been command
>> line
>>>> usage, so I'll have to move over to the Maven or Ant plugin ways then I
>>>> guess.
>>> It must be in the code, is it not implemented in the CLI? Sounds like
>>> a feature if not.
>>
>> there's quite a few features in the code that just need some CLI work to
>> support on the command line
>>
>> would you want it to be something like comma separated list eg
>>
>> %rat --excludes="target, build/**/*, *.svn"
>>
>
> Hi Robert, yes that would be great, thanks!
>
> I commented on RAT-2 yesterday saying just that.
> I also added that perhaps there could be an
>
> --exclude-file="file-of-excludes"
>
> but that's not essential, your idea will be just fine.
i've refactored the code and added a simple example (--exclude which
uses Commons IO wildcard file filters). i'd be happy to review a patch
for something along the lines of your suggestion (i've refactored the
code so it should be easier to do now - just take a look at the
excludesimplementation in
http://svn.apache.org/repos/asf/incubator/rat/main/trunk/apache-rat-core/src/main/java/org/apache/rat/Report.java)
> While I'm here, I'm now just playing with ant-task-examples.xml but I can't
> get it working. I renamed it to build.xml and tried to use it as is.
>
> The error I get is :-
>
> D:\Apache2\rat-trunk>ant javasources
> Buildfile: build.xml
>
> -taskdef:
>
> javasources:
>
> BUILD FAILED
> D:\Apache2\rat-trunk\build.xml:33: Problem: failed to create task or type
> antlib
> :org.apache.rat.anttasks:report
> Cause: The name is undefined.
> Action: Check the spelling.
> Action: Check that any custom tasks/types have been declared.
> Action: Check that any <presetdef>/<macrodef> declarations have taken place.
> No types or tasks have been defined in this namespace yet
>
> This appears to be an antlib declaration.
> Action: Check that the implementing library exists in one of:
> -D:\Apache2\forrest\tools\ant\lib
> -C:\Documents and Settings\Gavin\.ant\lib
> -a directory added on the command line with the -lib argument
>
> I'm assuming user error, any clues?
i think that you either need to install the rat antlib locally (see ant
manual since it's fiddly) or add a classpath containing the jars required
- robert
RE: RAT with BuildBot at Apache
Posted by Gavin <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Robert Burrell Donkin [mailto:robertburrelldonkin@blueyonder.co.uk]
> Sent: Tuesday, 16 June 2009 7:33 AM
> To: rat-dev@incubator.apache.org
> Subject: Re: RAT with BuildBot at Apache
>
> Ross Gardler wrote:
> > 2009/6/15 Gavin <ga...@16degrees.com.au>:
> >>
> >>> -----Original Message-----
> >>> From: ross.gardler@googlemail.com [mailto:ross.gardler@googlemail.com]
> On
> >>> Behalf Of Ross Gardler
> >>> Sent: Monday, 15 June 2009 10:09 AM
> >>> To: rat-dev@incubator.apache.org
> >>> Subject: Re: RAT with BuildBot at Apache
> >>>
> >>> 2009/6/15 Gavin <ga...@16degrees.com.au>:
> >>>
> >>>> Is that possible now, excluding sub-dirs ? If so how as I'd love to
> >>>> implement it so we can have more accurate results to work from for
> >>> refining.
> >>>
> >>> Here's an example from the Simal project (using Maven):
> >>>
> >>> <plugin>
> >>> <groupId>org.codehaus.mojo</groupId>
> >>> <artifactId>rat-maven-plugin</artifactId>
> >>> <configuration>
> >>> <excludes>
> >>> <exclude>.pmd</exclude>
> >>> <exclude>derby.log</exclude>
> >>> <exclude>reports/**/*</exclude>
> >>> <exclude>simal/**/*</exclude>
> >>> <exclude>build/**/*</exclude>
> >>> <exclude>simalDOAPFilestore/**/*</exclude>
> >>> <exclude>simalRepository/**/*</exclude>
> >>> <exclude>local.simal.properties</exclude>
> >>> <exclude>local.simal.test.properties</exclude>
> >>> </excludes>
> >>> </configuration>
> >>> <executions>
> >>> <execution>
> >>> <phase>verify</phase>
> >>> <goals>
> >>> <goal>check</goal>
> >>> </goals>
> >>> </execution>
> >>> </executions>
> >>> </plugin>
> >> Ah, ok great thanks. My implementations so far have just been command
> line
> >> usage, so I'll have to move over to the Maven or Ant plugin ways then I
> >> guess.
> >
> > It must be in the code, is it not implemented in the CLI? Sounds like
> > a feature if not.
>
>
> there's quite a few features in the code that just need some CLI work to
> support on the command line
>
> would you want it to be something like comma separated list eg
>
> %rat --excludes="target, build/**/*, *.svn"
>
Hi Robert, yes that would be great, thanks!
I commented on RAT-2 yesterday saying just that.
I also added that perhaps there could be an
--exclude-file="file-of-excludes"
but that's not essential, your idea will be just fine.
While I'm here, I'm now just playing with ant-task-examples.xml but I can't
get it working. I renamed it to build.xml and tried to use it as is.
The error I get is :-
D:\Apache2\rat-trunk>ant javasources
Buildfile: build.xml
-taskdef:
javasources:
BUILD FAILED
D:\Apache2\rat-trunk\build.xml:33: Problem: failed to create task or type
antlib
:org.apache.rat.anttasks:report
Cause: The name is undefined.
Action: Check the spelling.
Action: Check that any custom tasks/types have been declared.
Action: Check that any <presetdef>/<macrodef> declarations have taken place.
No types or tasks have been defined in this namespace yet
This appears to be an antlib declaration.
Action: Check that the implementing library exists in one of:
-D:\Apache2\forrest\tools\ant\lib
-C:\Documents and Settings\Gavin\.ant\lib
-a directory added on the command line with the -lib argument
I'm assuming user error, any clues?
Thanks
Gav...
> - robert
RE: RAT with BuildBot at Apache
Posted by Gavin <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Robert Burrell Donkin [mailto:robertburrelldonkin@blueyonder.co.uk]
> Sent: Tuesday, 16 June 2009 7:33 AM
> To: rat-dev@incubator.apache.org
> Subject: Re: RAT with BuildBot at Apache
>
<snip>
>
>
> there's quite a few features in the code that just need some CLI work to
> support on the command line
>
> would you want it to be something like comma separated list eg
>
> %rat --excludes="target, build/**/*, *.svn"
>
> - robert
Hi Robert,
Just come to do some configs that require sub-dirs and I can't get it to
work at all.
I tries your above example and that doesn't work for me.
I find anything with a / in it gets ignored.
So things like var/main/etc/blah or dir/dir2 or dir/dir2/** all don't work.
Only thing that do work are one word scenarios, such as .buildbot-sourcedata
or just mentioning the top level dir such as target or etc or dir.
I cant see what I've done wrong, I've looked at the code but doesn't help me
much.
Any ideas appreciated.
Gav...
Re: RAT with BuildBot at Apache
Posted by Robert Burrell Donkin <ro...@blueyonder.co.uk>.
Ross Gardler wrote:
> 2009/6/15 Gavin <ga...@16degrees.com.au>:
>>
>>> -----Original Message-----
>>> From: ross.gardler@googlemail.com [mailto:ross.gardler@googlemail.com] On
>>> Behalf Of Ross Gardler
>>> Sent: Monday, 15 June 2009 10:09 AM
>>> To: rat-dev@incubator.apache.org
>>> Subject: Re: RAT with BuildBot at Apache
>>>
>>> 2009/6/15 Gavin <ga...@16degrees.com.au>:
>>>
>>>> Is that possible now, excluding sub-dirs ? If so how as I'd love to
>>>> implement it so we can have more accurate results to work from for
>>> refining.
>>>
>>> Here's an example from the Simal project (using Maven):
>>>
>>> <plugin>
>>> <groupId>org.codehaus.mojo</groupId>
>>> <artifactId>rat-maven-plugin</artifactId>
>>> <configuration>
>>> <excludes>
>>> <exclude>.pmd</exclude>
>>> <exclude>derby.log</exclude>
>>> <exclude>reports/**/*</exclude>
>>> <exclude>simal/**/*</exclude>
>>> <exclude>build/**/*</exclude>
>>> <exclude>simalDOAPFilestore/**/*</exclude>
>>> <exclude>simalRepository/**/*</exclude>
>>> <exclude>local.simal.properties</exclude>
>>> <exclude>local.simal.test.properties</exclude>
>>> </excludes>
>>> </configuration>
>>> <executions>
>>> <execution>
>>> <phase>verify</phase>
>>> <goals>
>>> <goal>check</goal>
>>> </goals>
>>> </execution>
>>> </executions>
>>> </plugin>
>> Ah, ok great thanks. My implementations so far have just been command line
>> usage, so I'll have to move over to the Maven or Ant plugin ways then I
>> guess.
>
> It must be in the code, is it not implemented in the CLI? Sounds like
> a feature if not.
there's quite a few features in the code that just need some CLI work to
support on the command line
would you want it to be something like comma separated list eg
%rat --excludes="target, build/**/*, *.svn"
- robert
Re: RAT with BuildBot at Apache
Posted by Ross Gardler <rg...@apache.org>.
2009/6/15 Gavin <ga...@16degrees.com.au>:
>
>
>> -----Original Message-----
>> From: ross.gardler@googlemail.com [mailto:ross.gardler@googlemail.com] On
>> Behalf Of Ross Gardler
>> Sent: Monday, 15 June 2009 10:09 AM
>> To: rat-dev@incubator.apache.org
>> Subject: Re: RAT with BuildBot at Apache
>>
>> 2009/6/15 Gavin <ga...@16degrees.com.au>:
>>
>> > Is that possible now, excluding sub-dirs ? If so how as I'd love to
>> > implement it so we can have more accurate results to work from for
>> refining.
>>
>> Here's an example from the Simal project (using Maven):
>>
>> <plugin>
>> <groupId>org.codehaus.mojo</groupId>
>> <artifactId>rat-maven-plugin</artifactId>
>> <configuration>
>> <excludes>
>> <exclude>.pmd</exclude>
>> <exclude>derby.log</exclude>
>> <exclude>reports/**/*</exclude>
>> <exclude>simal/**/*</exclude>
>> <exclude>build/**/*</exclude>
>> <exclude>simalDOAPFilestore/**/*</exclude>
>> <exclude>simalRepository/**/*</exclude>
>> <exclude>local.simal.properties</exclude>
>> <exclude>local.simal.test.properties</exclude>
>> </excludes>
>> </configuration>
>> <executions>
>> <execution>
>> <phase>verify</phase>
>> <goals>
>> <goal>check</goal>
>> </goals>
>> </execution>
>> </executions>
>> </plugin>
>
> Ah, ok great thanks. My implementations so far have just been command line
> usage, so I'll have to move over to the Maven or Ant plugin ways then I
> guess.
It must be in the code, is it not implemented in the CLI? Sounds like
a feature if not.
Ross
RE: RAT with BuildBot at Apache
Posted by Gavin <ga...@16degrees.com.au>.
> -----Original Message-----
> From: ross.gardler@googlemail.com [mailto:ross.gardler@googlemail.com] On
> Behalf Of Ross Gardler
> Sent: Monday, 15 June 2009 10:09 AM
> To: rat-dev@incubator.apache.org
> Subject: Re: RAT with BuildBot at Apache
>
> 2009/6/15 Gavin <ga...@16degrees.com.au>:
>
> > Is that possible now, excluding sub-dirs ? If so how as I'd love to
> > implement it so we can have more accurate results to work from for
> refining.
>
> Here's an example from the Simal project (using Maven):
>
> <plugin>
> <groupId>org.codehaus.mojo</groupId>
> <artifactId>rat-maven-plugin</artifactId>
> <configuration>
> <excludes>
> <exclude>.pmd</exclude>
> <exclude>derby.log</exclude>
> <exclude>reports/**/*</exclude>
> <exclude>simal/**/*</exclude>
> <exclude>build/**/*</exclude>
> <exclude>simalDOAPFilestore/**/*</exclude>
> <exclude>simalRepository/**/*</exclude>
> <exclude>local.simal.properties</exclude>
> <exclude>local.simal.test.properties</exclude>
> </excludes>
> </configuration>
> <executions>
> <execution>
> <phase>verify</phase>
> <goals>
> <goal>check</goal>
> </goals>
> </execution>
> </executions>
> </plugin>
Ah, ok great thanks. My implementations so far have just been command line
usage, so I'll have to move over to the Maven or Ant plugin ways then I
guess.
Thanks
Gav...
>
> Ross
Re: RAT with BuildBot at Apache
Posted by Ross Gardler <rg...@apache.org>.
2009/6/15 Gavin <ga...@16degrees.com.au>:
> Is that possible now, excluding sub-dirs ? If so how as I'd love to
> implement it so we can have more accurate results to work from for refining.
Here's an example from the Simal project (using Maven):
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>rat-maven-plugin</artifactId>
<configuration>
<excludes>
<exclude>.pmd</exclude>
<exclude>derby.log</exclude>
<exclude>reports/**/*</exclude>
<exclude>simal/**/*</exclude>
<exclude>build/**/*</exclude>
<exclude>simalDOAPFilestore/**/*</exclude>
<exclude>simalRepository/**/*</exclude>
<exclude>local.simal.properties</exclude>
<exclude>local.simal.test.properties</exclude>
</excludes>
</configuration>
<executions>
<execution>
<phase>verify</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
Ross
RE: RAT with BuildBot at Apache
Posted by Gavin <ga...@16degrees.com.au>.
> -----Original Message-----
> From: ross.gardler@googlemail.com [mailto:ross.gardler@googlemail.com] On
> Behalf Of Ross Gardler
> Sent: Monday, 15 June 2009 9:39 AM
> To: rat-dev@incubator.apache.org
> Subject: Re: RAT with BuildBot at Apache
>
> 2009/6/14 Robert Burrell Donkin <ro...@blueyonder.co.uk>:
> > Ross Gardler wrote:
> >> 2009/6/12 Gavin <ga...@16degrees.com.au>:
> ...
>
> >>> http://ci.apache.org/projects/
> >>
> >> This shows that RAT is not all that useful right now unless properly
> >> configured. I was intrigued to see that RAT itself has 85 unapproved
> >> licences - http://ci.apache.org/projects/rat/rat-report.txt
> >>
> >> I'd say the first thing the RAT team need to do is get that down to 0.
> >
> > the problem is that a number of these are false positives (everything in
> > target sub directories are just consequences of the build) or
> > intentional (some of the tests for missing headers need to have no
> headers)
>
> The includes/excludes needs setting up correctly to bring it down to
> 0. False positives will always be false positives to the developer
> should add them to the excludes list. So, for example, the whole
> target directory should be excluded.
Is that possible now, excluding sub-dirs ? If so how as I'd love to
implement it so we can have more accurate results to work from for refining.
I've also written a xslt (based on the txt output currently) and use Saxon9B
to convert it to HTML from the command line on the server. I'll have these
simple HTML reports online today sometime. It would be useful to have this
integrated into RAT I guess so that a -H option or similar can produce HTML
outputs for us. The end user projects and buildbot need then only alter a
basic supplied CSS stylesheet for their needs.
Gav...
>
> Ross
Re: RAT with BuildBot at Apache
Posted by Ross Gardler <rg...@apache.org>.
2009/6/14 Robert Burrell Donkin <ro...@blueyonder.co.uk>:
> Ross Gardler wrote:
>> 2009/6/12 Gavin <ga...@16degrees.com.au>:
...
>>> http://ci.apache.org/projects/
>>
>> This shows that RAT is not all that useful right now unless properly
>> configured. I was intrigued to see that RAT itself has 85 unapproved
>> licences - http://ci.apache.org/projects/rat/rat-report.txt
>>
>> I'd say the first thing the RAT team need to do is get that down to 0.
>
> the problem is that a number of these are false positives (everything in
> target sub directories are just consequences of the build) or
> intentional (some of the tests for missing headers need to have no headers)
The includes/excludes needs setting up correctly to bring it down to
0. False positives will always be false positives to the developer
should add them to the excludes list. So, for example, the whole
target directory should be excluded.
Ross
Re: RAT with BuildBot at Apache
Posted by Robert Burrell Donkin <ro...@blueyonder.co.uk>.
Ross Gardler wrote:
> 2009/6/12 Gavin <ga...@16degrees.com.au>:
>>
>>> -----Original Message-----
>>> From: ross.gardler@googlemail.com [mailto:ross.gardler@googlemail.com] On
>>> Behalf Of Ross Gardler
>>> Sent: Tuesday, 9 June 2009 5:54 AM
>>> To: rat-dev@incubator.apache.org
>>> Subject: Re: Report Time
>>>
>> <snip>
>>
>>> I would also suggest that the project should focus on encouraging
>>> Apache projects to use the tool, perhaps a Gump style build and report
>>> mechanism should be a focus for the current team. This will raise
>>> awareness of the utility of RAT.
>
> ...
>
>> One such ability, which I have implemented over the last few days, is to
>> make use of RAT. As RAT is already there being built, I decided to automate
>> the use of the jar produced from the build. Now, projects can utilise
>> Buildbot to include RAT testing on their source/binaries/etc. I
>
> Cool!
+1
>> RAT reports can be displayed (currently) as txt files on the ASF Buildbot
>> website, they can be emailed (though not currently enabled as a fine balance
>> needs to be achieved considering per commit triggers). They can also be
>> produced on demand via IRC if they want.
>
> I'd suggest being able to configure the mail to only be sent if there
> is a problem. It's realy useful to have RAT say "hey you forgot to add
> a licence header to this new file"
+1
the feedback from scan is that a message once a month or so is useful
even if everything's cool but otherwise only mail when stuff goes wrong
>> This is both useful to RAT as well as the projects using it - as RAT project
>> can implement new features or fixes based on the results/feedback from the
>> projects using it - one commit later to svn and within a minute all projects
>> are immediately using the new version of RAT.
>
> Is there any kind of summary report screen that would help RAT
> developers see unintended side effects that cause multiple errors
> across projects?
that would be useful :-)
>> Buildbot itself has many other features and capabilities not yet
>> implemented/realised (Nexus is next on my hit list, as well as a place to
>> put release candidates (as most folk usually upload that stuff to
>> people.apache.org/~username)).
>
> I'd suggest putting it into the lab (no releases, but the code is what
> matters at this stage).
+1
>> My hope here is with integration of RAT is that it can raise the profile of
>> both RAT and Buildbot within the Apache community. I'm not sure Ross what
>> you had in mind for increased awareness but I hope this is an option we can
>> explore.
>
> This is *exactly* what I had in mind. The RAT web site should have a
> section saying, "look RAT is used in the Apache Buildbot" etc.
+1
the web site is in need of some TLC
>> We can talk about ideas on best/different implementation ideas if folks
>> agree it is a good idea.
>
> +1
+1
>> http://ci.apache.org/buildbot.html
>
> ...
>
>> A few of those projects I have configured the builds to upload
>> websites/documentation/javadoc/Apis - and also RAT reports for a few.
>>
>> http://ci.apache.org/projects/
>
> This shows that RAT is not all that useful right now unless properly
> configured. I was intrigued to see that RAT itself has 85 unapproved
> licences - http://ci.apache.org/projects/rat/rat-report.txt
>
> I'd say the first thing the RAT team need to do is get that down to 0.
the problem is that a number of these are false positives (everything in
target sub directories are just consequences of the build) or
intentional (some of the tests for missing headers need to have no headers)
detached license information seems like the right way to solve this. so
that sounds like an implementation priority.
>> One major thing I'd to work on soon -- and I'd like help with if anyone has
>> time (if not it will just take longer I guess) is to make use of the XML
>> output and so create an xslt to produce nice readable HTML reports to go on
>> the site. Do we have a DTD? Do we need one? Is the current XML output
>> suitable or will it need tweaking? Has any of this been done already? This
>> stuff I can find out soon.
>
> I'll have to pass on al those questions I'm afraid.
there isn't a DTD or schema ATM since the current format is a bit rubbish
i'd prefer to move to RDF or xhtml+rdf but that might be a barrier to
entry and may be difficult to transform
so, maybe it'd be best to start from the other direction: brainstorm
about the design of a really good HTML report and then just work backwards.
ideas?
>> One last thing I though of, I'm sure we can gather info from these reports
>> and send it to wherever Clutch needs it in order to implement that extra
>> line on clutch you mentioned Ross.
>
> That would be really cool. Perhaps the simplest thing to do would be
> to have your buildbot dump the files in a location that the existing
> Clutch code can access.
+1
- robert
Re: RAT with BuildBot at Apache (was: RE: Report Time)
Posted by Ross Gardler <rg...@apache.org>.
2009/6/12 Gavin <ga...@16degrees.com.au>:
>
>
>> -----Original Message-----
>> From: ross.gardler@googlemail.com [mailto:ross.gardler@googlemail.com] On
>> Behalf Of Ross Gardler
>> Sent: Tuesday, 9 June 2009 5:54 AM
>> To: rat-dev@incubator.apache.org
>> Subject: Re: Report Time
>>
>
> <snip>
>
>> I would also suggest that the project should focus on encouraging
>> Apache projects to use the tool, perhaps a Gump style build and report
>> mechanism should be a focus for the current team. This will raise
>> awareness of the utility of RAT.
>
...
> One such ability, which I have implemented over the last few days, is to
> make use of RAT. As RAT is already there being built, I decided to automate
> the use of the jar produced from the build. Now, projects can utilise
> Buildbot to include RAT testing on their source/binaries/etc. I
Cool!
> RAT reports can be displayed (currently) as txt files on the ASF Buildbot
> website, they can be emailed (though not currently enabled as a fine balance
> needs to be achieved considering per commit triggers). They can also be
> produced on demand via IRC if they want.
I'd suggest being able to configure the mail to only be sent if there
is a problem. It's realy useful to have RAT say "hey you forgot to add
a licence header to this new file"
> This is both useful to RAT as well as the projects using it - as RAT project
> can implement new features or fixes based on the results/feedback from the
> projects using it - one commit later to svn and within a minute all projects
> are immediately using the new version of RAT.
Is there any kind of summary report screen that would help RAT
developers see unintended side effects that cause multiple errors
across projects?
> Buildbot itself has many other features and capabilities not yet
> implemented/realised (Nexus is next on my hit list, as well as a place to
> put release candidates (as most folk usually upload that stuff to
> people.apache.org/~username)).
I'd suggest putting it into the lab (no releases, but the code is what
matters at this stage).
> My hope here is with integration of RAT is that it can raise the profile of
> both RAT and Buildbot within the Apache community. I'm not sure Ross what
> you had in mind for increased awareness but I hope this is an option we can
> explore.
This is *exactly* what I had in mind. The RAT web site should have a
section saying, "look RAT is used in the Apache Buildbot" etc.
> We can talk about ideas on best/different implementation ideas if folks
> agree it is a good idea.
+1
> http://ci.apache.org/buildbot.html
...
> A few of those projects I have configured the builds to upload
> websites/documentation/javadoc/Apis - and also RAT reports for a few.
>
> http://ci.apache.org/projects/
This shows that RAT is not all that useful right now unless properly
configured. I was intrigued to see that RAT itself has 85 unapproved
licences - http://ci.apache.org/projects/rat/rat-report.txt
I'd say the first thing the RAT team need to do is get that down to 0.
> One major thing I'd to work on soon -- and I'd like help with if anyone has
> time (if not it will just take longer I guess) is to make use of the XML
> output and so create an xslt to produce nice readable HTML reports to go on
> the site. Do we have a DTD? Do we need one? Is the current XML output
> suitable or will it need tweaking? Has any of this been done already? This
> stuff I can find out soon.
I'll have to pass on al those questions I'm afraid.
> One last thing I though of, I'm sure we can gather info from these reports
> and send it to wherever Clutch needs it in order to implement that extra
> line on clutch you mentioned Ross.
That would be really cool. Perhaps the simplest thing to do would be
to have your buildbot dump the files in a location that the existing
Clutch code can access.
Ross