Getting a list of ASF projects

[Alan Cabrera <ad...@toolazydogs.com>]

Is there a way to programatically get a list of ASF projects without having to scrape HTML?


Regards,
Alan

Re: Getting a list of ASF projects

[Nick Burch <ap...@gagravarr.org>]

On Sun, 9 Jun 2013, Alan Cabrera wrote:
> Is there a way to programatically get a list of ASF projects without 
> having to scrape HTML?

You probably don't want to be scraping the HTML from the homepage. Quite a 
few recently graduated projects seemed to have skipped that part of the 
graduation task list, and weren't listed there. I had a go at fixing the 
list up last week, but may have still missed one or two, and given how 
many hadn't done it you can probably assume there will be more making the 
same mistake in future...

(I only noticed because someone queried their project being missing from a 
talk I gave. I pointed out that they were missing from www.a.o, and 
pointed out the section in the graduation page that said to add it, so it 
was nothing personal!)

Nick

Re: Getting a list of ASF projects

[Sam Ruby <ru...@intertwingly.net>]

On Sun, Jun 9, 2013 at 2:11 PM, Alan Cabrera <li...@toolazydogs.com> wrote:
>
> On Jun 9, 2013, at 11:07 AM, Sam Ruby <ru...@intertwingly.net> wrote:
>
>> On Sun, Jun 9, 2013 at 1:52 PM, Alan Cabrera <ad...@toolazydogs.com> wrote:
>>> Is there a way to programatically get a list of ASF projects without having to scrape HTML?
>>
>> For example, here is one solution:
>>
>> ldapsearch -x -LLL -b ou=pmc,ou=committees,ou=groups,dc=apache,dc=org dn
>
> This might be interesting to add to whimsy?

Sure!  Got a patch?  :-)

By the way, that's actually the underlying query that the following page uses:

https://whimsy.apache.org/roster/committee/

(more precisely, that's the command line equivalent to the underlying
ldap apis that are used by that page)

Adding a json output for that page is easy.  At the moment, that page
is password protected.  Or a separate page containing only public
information could be added.

> Regards,
> Alan

P.S.  If this is likely a one time thing, describe what you would like
to see further, and I'll be happy to whip something up.  Otherwise, I
would much rather "teach you to fish" :-)

Re: Getting a list of ASF projects

[Alan Cabrera <li...@toolazydogs.com>]

On Jun 9, 2013, at 11:07 AM, Sam Ruby <ru...@intertwingly.net> wrote:

> On Sun, Jun 9, 2013 at 1:52 PM, Alan Cabrera <ad...@toolazydogs.com> wrote:
>> Is there a way to programatically get a list of ASF projects without having to scrape HTML?
> 
> For example, here is one solution:
> 
> ldapsearch -x -LLL -b ou=pmc,ou=committees,ou=groups,dc=apache,dc=org dn

This might be interesting to add to whimsy?


Regards,
Alan

Re: Getting a list of ASF projects

[sebb <se...@gmail.com>]

On 10 June 2013 09:21, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> Sam Ruby wrote on Sun, Jun 09, 2013 at 14:07:50 -0400:
>> On Sun, Jun 9, 2013 at 1:52 PM, Alan Cabrera <ad...@toolazydogs.com> wrote:
>> > Is there a way to programatically get a list of ASF projects without having to scrape HTML?
>>
>> You probably need to be a bit more explicit about your constraints. :-)
>
> s/constraints/question/
>
> For starters, projects.apache.org lists some projects which are not
> top-level projects

For example almost all the Commons components.

It also does not list all TLPs; the information there is currently
provided by projects, not all have done so yet.

Re: Getting a list of ASF projects

[Daniel Shahaf <d....@daniel.shahaf.name>]

Sam Ruby wrote on Sun, Jun 09, 2013 at 14:07:50 -0400:
> On Sun, Jun 9, 2013 at 1:52 PM, Alan Cabrera <ad...@toolazydogs.com> wrote:
> > Is there a way to programatically get a list of ASF projects without having to scrape HTML?
> 
> You probably need to be a bit more explicit about your constraints. :-)

s/constraints/question/

For starters, projects.apache.org lists some projects which are not
top-level projects

Re: Getting a list of ASF projects

[Sam Ruby <ru...@intertwingly.net>]

On Sun, Jun 9, 2013 at 1:52 PM, Alan Cabrera <ad...@toolazydogs.com> wrote:
> Is there a way to programatically get a list of ASF projects without having to scrape HTML?

You probably need to be a bit more explicit about your constraints. :-)

For example, here is one solution:

ldapsearch -x -LLL -b ou=pmc,ou=committees,ou=groups,dc=apache,dc=org dn

Another solution that is relatively easy to parse:

https://svn.apache.org/repos/private/committers/board/committee-info.txt

> Regards,
> Alan

- Sam Ruby