You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@community.apache.org by Daniel Gruno <hu...@apache.org> on 2015/09/21 14:45:16 UTC
Re: svn commit: r1703934 - in /comdev/projects.apache.org:
scripts/README.txt scripts/import/parsecommittees.py
site/json/foundation/.htaccess site/json/foundation/HEADER.html
site/json/foundation/pmcs.json
You just broke reporter.apache.org by doing this, please revert.
With regards,
Daniel.
On 09/19/2015 03:03 AM, sebb@apache.org wrote:
> Author: sebb
> Date: Sat Sep 19 01:03:02 2015
> New Revision: 1703934
>
> URL: http://svn.apache.org/viewvc?rev=1703934&view=rev
> Log:
> pmcs.json no longer used, so drop it
>
> Removed:
> comdev/projects.apache.org/site/json/foundation/pmcs.json
> Modified:
> comdev/projects.apache.org/scripts/README.txt
> comdev/projects.apache.org/scripts/import/parsecommittees.py
> comdev/projects.apache.org/site/json/foundation/.htaccess
> comdev/projects.apache.org/site/json/foundation/HEADER.html
>
> Modified: comdev/projects.apache.org/scripts/README.txt
> URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/scripts/README.txt?rev=1703934&r1=1703933&r2=1703934&view=diff
> ==============================================================================
> --- comdev/projects.apache.org/scripts/README.txt (original)
> +++ comdev/projects.apache.org/scripts/README.txt Sat Sep 19 01:03:02 2015
> @@ -10,7 +10,8 @@ various sources:
> - parsecommitters.py: Fetches and parses the committer (LDAP) list via
> people.apache.org.
> in: http://people.apache.org/committer-index.html
> - out: site/json/foundation/people.json + site/json/foundation/groups.json
> + out: site/json/foundation/people.json
> + + site/json/foundation/groups.json
> List of committers with reference to groups (people.json) and groups with corresponding committers (groups.json)
>
> - podlings.py: Reads podlings.xml from the incubator site and creates a JSON
> @@ -29,16 +30,21 @@ various sources:
>
> - parsecommittees.py: Parses committee-info.txt to detect new and retired committees and imports PMC data (RDF) from
> PMC data files
> - in: site/json/foundation/committees.json + site/json/foundation/committees-retired.json
> - + data/board/committee-info.txt (https://svn.apache.org/repos/private/committers/board/committee-info.txt)
> - + data/committees.xml + PMC data data/committees/*.rdf
> - out: site/json/foundation/committees.json + site/json/foundation/committees-retired.json + site/json/foundation/pmcs.json
> - + site/doap/{committeeId}/pmc-doap.rdf + site/doap/{committeeId}/pmc.rdf
> + in: site/json/foundation/committees.json
> + + site/json/foundation/committees-retired.json
> + + data/board/committee-info.txt (https://svn.apache.org/repos/private/committers/board/committee-info.txt)
> + + data/committees.xml - list of where to find PMC description RDF files
> + + data/committees/*.rdf - local PMC description RDF files
> + out: site/json/foundation/committees.json (updated)
> + + site/json/foundation/committees-retired.json (updated)
> + + site/doap/{committeeId}/pmc-doap.rdf - this is an exact copy of the file listed in data/committees.xml
> + + site/doap/{committeeId}/pmc.rdf
>
> - parseprojects.py: Parses existing projects RDF(DOAP) files and turns them into JSON objects.
> in: data/projects.xml + projects' DOAP files
> - out: site/json/projects/*.json + site/json/foundation/projects.json
> - + site/doap/{committeeId}/{project}.rdf
> + out: site/json/projects/*.json
> + + site/json/foundation/projects.json
> + + site/doap/{committeeId}/{project}.rdf - these are exact copies of the RDFs listed in data/projects.xml
>
> NOTICE: what prevents import scripts to be added to cron?
> 1. parse committees.py requires committee-info.txt, which is not available on project-vm (require authentication)
>
> Modified: comdev/projects.apache.org/scripts/import/parsecommittees.py
> URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/scripts/import/parsecommittees.py?rev=1703934&r1=1703933&r2=1703934&view=diff
> ==============================================================================
> --- comdev/projects.apache.org/scripts/import/parsecommittees.py (original)
> +++ comdev/projects.apache.org/scripts/import/parsecommittees.py Sat Sep 19 01:03:02 2015
> @@ -21,7 +21,7 @@ Updates:
> ../../site/json/foundation/committees-retired.json
>
> Writes:
> -../../site/json/foundation/pmcs.json
> +../../site/doap/<committeeId>/pmc.rdf
> """
>
> # Committee names from committees-info.txt that do not match committees-evolution.json
> @@ -85,7 +85,6 @@ def handleChild(el):
>
> # get PMC Data from /data/committees.xml
> print("reading PMC Data (/data/committees.xml)")
> -pmcs = {}
> pmcDataUrls = {} # id -> url
> with open("../../data/committees.xml", "r") as f:
> xmldoc = minidom.parseString(f.read())
> @@ -123,8 +122,6 @@ for s in itemlist :
> else:
> pmcjson[k] = v
>
> - pmcs[committeeId] = pmcjson
> -
> # copy PMC RDF data to /doap/{committeeId}/pmc-doap.rdf
> if type(rdf) is str:
> mode = "w"
> @@ -249,7 +246,6 @@ for pmc in re.findall(r"\* .+?\s+\(est\.
> committers.sort()
> pmc.sort()
> # don't store committers and PMC members arrays in committee: it's easy to get from groups.json
> - #committee['pmcs'] = pmc
> #committee['committers'] = committers
> if len(pmc) == 0:
> print('WARN: %s (%s established in %s) has no PMC members LDAP group (id=%s)' % (committeeId, committeeName, committee['established'], pmcgroup))
> @@ -336,8 +332,3 @@ print("Writing json/foundation/committee
> with open("../../site/json/foundation/committees-retired.json", "w") as f:
> f.write(json.dumps(committeesRetired, sort_keys=True, indent=0))
> f.close()
> -
> -print("Writing json/foundation/pmcs.json...")
> -with open ("../../site/json/foundation/pmcs.json", "w") as f:
> - f.write(json.dumps(pmcs, sort_keys=True, indent=0))
> - f.close()
>
> Modified: comdev/projects.apache.org/site/json/foundation/.htaccess
> URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/site/json/foundation/.htaccess?rev=1703934&r1=1703933&r2=1703934&view=diff
> ==============================================================================
> --- comdev/projects.apache.org/site/json/foundation/.htaccess (original)
> +++ comdev/projects.apache.org/site/json/foundation/.htaccess Sat Sep 19 01:03:02 2015
> @@ -4,8 +4,6 @@ AddDescription "imported with <code>impo
> AddDescription "generated by <code>cronjobs/parsecommitters.py</code>" groups.json
> AddDescription "generated by <code>cronjobs/parsecommitters.py</code>" people.json
> AddDescription "generated by <code>cronjobs/parsecommitters.py</code>" people_name.json
> -AddDescription "imported with <code>import/parsepmcs.py</code> from <a href='https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/projects/data_files/'>PMC DOAP files</a>" pmcs.json
> -AddDescription "imported with <code>import/parsecommittees.py</code>" pmcs.json
> AddDescription "generated by <code>cronjobs/podlings.py</code>" podlings.json
> AddDescription "generated by <code>cronjobs/podlings.py</code>" podlings-history.json
> AddDescription "imported with <code>import/rdfparse.py</code> from <a href='https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/projects/files.xml'>projects' DOAP files</a>" projects.json
>
> Modified: comdev/projects.apache.org/site/json/foundation/HEADER.html
> URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/site/json/foundation/HEADER.html?rev=1703934&r1=1703933&r2=1703934&view=diff
> ==============================================================================
> --- comdev/projects.apache.org/site/json/foundation/HEADER.html (original)
> +++ comdev/projects.apache.org/site/json/foundation/HEADER.html Sat Sep 19 01:03:02 2015
> @@ -9,6 +9,5 @@ for more information.
> <li><a href="projects.json"><code>projects.json</code></a> imported with <code>import/rdfparse.py</code> from <a href='https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/projects/files.xml'>projects' DOAP files</a></li>
> <li><a href="releases.json"><code>releases.json</code></a> generated by <code>cronjobs/parsereleases.py</code> from <a href="http://www.apache.org/dist/">http://www.apache.org/dist/</a></li>
> <li><a href="committees.json"><code>committees.json</code></a> and <a href="committees-retired.json"><code>committees-retired.json</code></a> imported with <code>import/parsecommittees.py</code> from <code>https://svn.apache.org/repos/private/committers/board/committee-info.txt</code></li>
> -<li><i><a href="pmcs.json"><code>pmcs.json</code></a> imported with <code>import/parsecommittees.py</code> from <a href='https://svn.apache.org/repos/asf/comdev/projects.apache.org/data/committees.xml'>PMC DOAP files</a> (not used any more, replaced by committees.json)</i></li>
> </ul>
> <code><a href="http://svn.apache.org/viewvc/comdev/projects.apache.org/site/json/foundation/">site/json/foundation</a></code>
> \ No newline at end of file
>
>
Re: svn commit: r1703934 - in /comdev/projects.apache.org:
scripts/README.txt scripts/import/parsecommittees.py site/json/foundation/.htaccess
site/json/foundation/HEADER.html site/json/foundation/pmcs.json
Posted by sebb <se...@gmail.com>.
Sorry about that; I have reverted the change.
I notice that reporter.a.o also uses chairs.json which was previously deleted.
However that was broken as it was derived from the listing on the
foundation website.
The info is available elsewhere, so I will fix reporter.a.o shortly to
use that rather than re-instating the outdated data and code.
On 21 September 2015 at 13:45, Daniel Gruno <hu...@apache.org> wrote:
> You just broke reporter.apache.org by doing this, please revert.
>
> With regards,
> Daniel.
> On 09/19/2015 03:03 AM, sebb@apache.org wrote:
>> Author: sebb
>> Date: Sat Sep 19 01:03:02 2015
>> New Revision: 1703934
>>
>> URL: http://svn.apache.org/viewvc?rev=1703934&view=rev
>> Log:
>> pmcs.json no longer used, so drop it
>>
>> Removed:
>> comdev/projects.apache.org/site/json/foundation/pmcs.json
>> Modified:
>> comdev/projects.apache.org/scripts/README.txt
>> comdev/projects.apache.org/scripts/import/parsecommittees.py
>> comdev/projects.apache.org/site/json/foundation/.htaccess
>> comdev/projects.apache.org/site/json/foundation/HEADER.html
>>
>> Modified: comdev/projects.apache.org/scripts/README.txt
>> URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/scripts/README.txt?rev=1703934&r1=1703933&r2=1703934&view=diff
>> ==============================================================================
>> --- comdev/projects.apache.org/scripts/README.txt (original)
>> +++ comdev/projects.apache.org/scripts/README.txt Sat Sep 19 01:03:02 2015
>> @@ -10,7 +10,8 @@ various sources:
>> - parsecommitters.py: Fetches and parses the committer (LDAP) list via
>> people.apache.org.
>> in: http://people.apache.org/committer-index.html
>> - out: site/json/foundation/people.json + site/json/foundation/groups.json
>> + out: site/json/foundation/people.json
>> + + site/json/foundation/groups.json
>> List of committers with reference to groups (people.json) and groups with corresponding committers (groups.json)
>>
>> - podlings.py: Reads podlings.xml from the incubator site and creates a JSON
>> @@ -29,16 +30,21 @@ various sources:
>>
>> - parsecommittees.py: Parses committee-info.txt to detect new and retired committees and imports PMC data (RDF) from
>> PMC data files
>> - in: site/json/foundation/committees.json + site/json/foundation/committees-retired.json
>> - + data/board/committee-info.txt (https://svn.apache.org/repos/private/committers/board/committee-info.txt)
>> - + data/committees.xml + PMC data data/committees/*.rdf
>> - out: site/json/foundation/committees.json + site/json/foundation/committees-retired.json + site/json/foundation/pmcs.json
>> - + site/doap/{committeeId}/pmc-doap.rdf + site/doap/{committeeId}/pmc.rdf
>> + in: site/json/foundation/committees.json
>> + + site/json/foundation/committees-retired.json
>> + + data/board/committee-info.txt (https://svn.apache.org/repos/private/committers/board/committee-info.txt)
>> + + data/committees.xml - list of where to find PMC description RDF files
>> + + data/committees/*.rdf - local PMC description RDF files
>> + out: site/json/foundation/committees.json (updated)
>> + + site/json/foundation/committees-retired.json (updated)
>> + + site/doap/{committeeId}/pmc-doap.rdf - this is an exact copy of the file listed in data/committees.xml
>> + + site/doap/{committeeId}/pmc.rdf
>>
>> - parseprojects.py: Parses existing projects RDF(DOAP) files and turns them into JSON objects.
>> in: data/projects.xml + projects' DOAP files
>> - out: site/json/projects/*.json + site/json/foundation/projects.json
>> - + site/doap/{committeeId}/{project}.rdf
>> + out: site/json/projects/*.json
>> + + site/json/foundation/projects.json
>> + + site/doap/{committeeId}/{project}.rdf - these are exact copies of the RDFs listed in data/projects.xml
>>
>> NOTICE: what prevents import scripts to be added to cron?
>> 1. parse committees.py requires committee-info.txt, which is not available on project-vm (require authentication)
>>
>> Modified: comdev/projects.apache.org/scripts/import/parsecommittees.py
>> URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/scripts/import/parsecommittees.py?rev=1703934&r1=1703933&r2=1703934&view=diff
>> ==============================================================================
>> --- comdev/projects.apache.org/scripts/import/parsecommittees.py (original)
>> +++ comdev/projects.apache.org/scripts/import/parsecommittees.py Sat Sep 19 01:03:02 2015
>> @@ -21,7 +21,7 @@ Updates:
>> ../../site/json/foundation/committees-retired.json
>>
>> Writes:
>> -../../site/json/foundation/pmcs.json
>> +../../site/doap/<committeeId>/pmc.rdf
>> """
>>
>> # Committee names from committees-info.txt that do not match committees-evolution.json
>> @@ -85,7 +85,6 @@ def handleChild(el):
>>
>> # get PMC Data from /data/committees.xml
>> print("reading PMC Data (/data/committees.xml)")
>> -pmcs = {}
>> pmcDataUrls = {} # id -> url
>> with open("../../data/committees.xml", "r") as f:
>> xmldoc = minidom.parseString(f.read())
>> @@ -123,8 +122,6 @@ for s in itemlist :
>> else:
>> pmcjson[k] = v
>>
>> - pmcs[committeeId] = pmcjson
>> -
>> # copy PMC RDF data to /doap/{committeeId}/pmc-doap.rdf
>> if type(rdf) is str:
>> mode = "w"
>> @@ -249,7 +246,6 @@ for pmc in re.findall(r"\* .+?\s+\(est\.
>> committers.sort()
>> pmc.sort()
>> # don't store committers and PMC members arrays in committee: it's easy to get from groups.json
>> - #committee['pmcs'] = pmc
>> #committee['committers'] = committers
>> if len(pmc) == 0:
>> print('WARN: %s (%s established in %s) has no PMC members LDAP group (id=%s)' % (committeeId, committeeName, committee['established'], pmcgroup))
>> @@ -336,8 +332,3 @@ print("Writing json/foundation/committee
>> with open("../../site/json/foundation/committees-retired.json", "w") as f:
>> f.write(json.dumps(committeesRetired, sort_keys=True, indent=0))
>> f.close()
>> -
>> -print("Writing json/foundation/pmcs.json...")
>> -with open ("../../site/json/foundation/pmcs.json", "w") as f:
>> - f.write(json.dumps(pmcs, sort_keys=True, indent=0))
>> - f.close()
>>
>> Modified: comdev/projects.apache.org/site/json/foundation/.htaccess
>> URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/site/json/foundation/.htaccess?rev=1703934&r1=1703933&r2=1703934&view=diff
>> ==============================================================================
>> --- comdev/projects.apache.org/site/json/foundation/.htaccess (original)
>> +++ comdev/projects.apache.org/site/json/foundation/.htaccess Sat Sep 19 01:03:02 2015
>> @@ -4,8 +4,6 @@ AddDescription "imported with <code>impo
>> AddDescription "generated by <code>cronjobs/parsecommitters.py</code>" groups.json
>> AddDescription "generated by <code>cronjobs/parsecommitters.py</code>" people.json
>> AddDescription "generated by <code>cronjobs/parsecommitters.py</code>" people_name.json
>> -AddDescription "imported with <code>import/parsepmcs.py</code> from <a href='https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/projects/data_files/'>PMC DOAP files</a>" pmcs.json
>> -AddDescription "imported with <code>import/parsecommittees.py</code>" pmcs.json
>> AddDescription "generated by <code>cronjobs/podlings.py</code>" podlings.json
>> AddDescription "generated by <code>cronjobs/podlings.py</code>" podlings-history.json
>> AddDescription "imported with <code>import/rdfparse.py</code> from <a href='https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/projects/files.xml'>projects' DOAP files</a>" projects.json
>>
>> Modified: comdev/projects.apache.org/site/json/foundation/HEADER.html
>> URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/site/json/foundation/HEADER.html?rev=1703934&r1=1703933&r2=1703934&view=diff
>> ==============================================================================
>> --- comdev/projects.apache.org/site/json/foundation/HEADER.html (original)
>> +++ comdev/projects.apache.org/site/json/foundation/HEADER.html Sat Sep 19 01:03:02 2015
>> @@ -9,6 +9,5 @@ for more information.
>> <li><a href="projects.json"><code>projects.json</code></a> imported with <code>import/rdfparse.py</code> from <a href='https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/projects/files.xml'>projects' DOAP files</a></li>
>> <li><a href="releases.json"><code>releases.json</code></a> generated by <code>cronjobs/parsereleases.py</code> from <a href="http://www.apache.org/dist/">http://www.apache.org/dist/</a></li>
>> <li><a href="committees.json"><code>committees.json</code></a> and <a href="committees-retired.json"><code>committees-retired.json</code></a> imported with <code>import/parsecommittees.py</code> from <code>https://svn.apache.org/repos/private/committers/board/committee-info.txt</code></li>
>> -<li><i><a href="pmcs.json"><code>pmcs.json</code></a> imported with <code>import/parsecommittees.py</code> from <a href='https://svn.apache.org/repos/asf/comdev/projects.apache.org/data/committees.xml'>PMC DOAP files</a> (not used any more, replaced by committees.json)</i></li>
>> </ul>
>> <code><a href="http://svn.apache.org/viewvc/comdev/projects.apache.org/site/json/foundation/">site/json/foundation</a></code>
>> \ No newline at end of file
>>
>>
>