You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sling.apache.org by "Bertrand Delacretaz (JIRA)" <ji...@apache.org> on 2018/01/31 10:27:00 UTC
[jira] [Comment Edited] (SLING-7262) Generate reusable lists of Git
repositories for machine consumption
[ https://issues.apache.org/jira/browse/SLING-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346554#comment-16346554 ]
Bertrand Delacretaz edited comment on SLING-7262 at 1/31/18 10:26 AM:
----------------------------------------------------------------------
Trying to summarize the various initiatives going on, as well as our requirements, please correct/complete as needed.
*Inputs*
* Either the GitHub API to list all Sling repositories as currently used at [https://github.com/apache/sling-aggregator]
* Or the Gitbox OPML at [https://gitbox.apache.org/repos/asf?a=opml] as currently used by [https://github.com/oliverlietz/apache-sling-aggregator]
*Desired Outputs*
* 1) A manifest for the Google Repo tool, which requires just the repository names and their paths.
* 2) Our Jenkins jobs, currently generated manually by running scripts from https://github.com/apache/sling-tooling-jenkins
* 3) A list suitable for updating [http://sling.apache.org/repolist.html] (or a similar canonical Web page)
* 4) A Maven reactor pom that allows for building everything or sets of modules, probably based on Maven profiles
For 1) I haven't checked if the Repo manifest format officially allows us to add extra information, currently we have an additional "groups" attribute, if that's supported we might add "tags", "badges" and "descriptions" as needed.
For 3) the repositories need to be grouped by functional areas (already partially done) and should also have tags or badges for contrib/core/legacy and similar classifications. It would be nice to have a brief description of each repository as well in that list, maybe the first few lines of their README (excluding "belongs to Apache Sling" of course)
*Environment*
The generation of these files needs to run as a Jenkins build so that they can be regenerated automatically.
Commiting the output of Jenkins jobs to Git is easy if the target is the {{asf-site}} branch, for that we just need to tie the jobs to nodes having the {{git-websites}} label. Committing the output to the {{asf-site}} branch of the [https://github.com/apache/sling-site] repository seems to be a good solution to me and other scripts can pick it up there at a stable URL, even if the raw result is not visible on the website.
*Here's what we currently have "in production"*
* [https://github.com/apache/sling-aggregator] generates a Repo manifest, needs to be updated manually by running a script and pushing the result when new Git repos are created
* [http://sling.apache.org/repolist.html] is generated from that Repo manifest. It's missing tags/badges and descriptions
* Our Jenkins jobs are generated from [https://github.com/apache/sling-tooling-jenkins] and optionally use information from a {{.sling-module.xml}} file described at [https://cwiki.apache.org/confluence/display/SLING/Sling+module+descriptor]
* I don't think we have an official Maven aggregator pom at this point
* [~olli] has been working on [https://github.com/oliverlietz/apache-sling-aggregator] with useful things in various branches of that repository, which generate a Repo manifest, a Maven pom and a GitHub page.
See [https://github.com/apache/sling-org-apache-sling-launchpad-contrib-testing] for an example of a sling:contrib badge that can easily be parsed from the README and is displayed there, as opposed to {{.sling-module.xml}} which has no impact on the README page on GitHub:
{code:java}
curl -s https://raw.githubusercontent.com/apache/sling-org-apache-sling-launchpad-contrib-testing/master/README.md | grep sling:badge
{code}
We should consolidate all that and ideally generate all the outputs from a single master list that's automatically updated at regular intervals.
was (Author: bdelacretaz):
Trying to summarize the various initiatives going on, as well as our requirements, please correct/complete as needed.
*Inputs*
* Either the GitHub API to list all Sling repositories as currently used at https://github.com/apache/sling-aggregator
* Or the Gitbox OPML at https://gitbox.apache.org/repos/asf?a=opml as currently used by https://github.com/oliverlietz/apache-sling-aggregator
*Desired Outputs*
* 1) A manifest for the Google Repo tool, which requires just the repository names and their paths.
* 2) A list suitable for updating http://sling.apache.org/repolist.html (or a similar canonical Web page)
* 3) A Maven reactor pom that allows for building everything or sets of modules, probably based on Maven profiles
For 1) I haven't checked if the Repo manifest format officially allows us to add extra information, currently we have an additional "groups" attribute, if that's supported we might add "tags", "badges" and "descriptions" as needed.
For 2) the repositories need to be grouped by functional areas (already partially done) and should also have tags or badges for contrib/core/legacy and similar classifications. It would be nice to have a brief description of each repository as well in that list, maybe the first few lines of their README (excluding "belongs to Apache Sling" of course)
*Environment*
The generation of these files needs to run as a Jenkins build so that they can be regenerated automatically.
Commiting the output of Jenkins jobs to Git is easy if the target is the {{asf-site}} branch, for that we just need to tie the jobs to nodes having the {{git-websites}} label. Committing the output to the {{asf-site}} branch of the https://github.com/apache/sling-site repository seems to be a good solution to me and other scripts can pick it up there at a stable URL, even if the raw result is not visible on the website.
*Here's what we currently have "in production"*
* https://github.com/apache/sling-aggregator generates a Repo manifest, needs to be updated manually by running a script and pushing the result when new Git repos are created
* http://sling.apache.org/repolist.html is generated from that Repo manifest. It's missing tags/badges and descriptions
* Our Jenkins jobs are generated from https://github.com/apache/sling-tooling-jenkins and optionally use information from a {{.sling-module.xml}} file described at https://cwiki.apache.org/confluence/display/SLING/Sling+module+descriptor
* I don't think we have an official Maven aggregator pom at this point
* [~olli] has been working on https://github.com/oliverlietz/apache-sling-aggregator with useful things in various branches of that repository, which generate a Repo manifest, a Maven pom and a GitHub page.
See https://github.com/apache/sling-org-apache-sling-launchpad-contrib-testing for an example of a sling:contrib badge that can easily be parsed from the README and is displayed there, as opposed to {{.sling-module.xml}} which has no impact on the README page on GitHub:
{code}
curl -s https://raw.githubusercontent.com/apache/sling-org-apache-sling-launchpad-contrib-testing/master/README.md | grep sling:badge
{code}
We should consolidate all that and ideally generate all the outputs from a single master list that's automatically updated at regular intervals.
> Generate reusable lists of Git repositories for machine consumption
> -------------------------------------------------------------------
>
> Key: SLING-7262
> URL: https://issues.apache.org/jira/browse/SLING-7262
> Project: Sling
> Issue Type: Task
> Components: Build and Source Control
> Reporter: Robert Munteanu
> Priority: Major
>
> We currently have two 'clients' that query the github repos and the .sling-module.xml descriptors:
> * the jenkins job creation DSL script
> * the repo manifest creation script
> We've also discussed in SLING-7161 the possibility of listing all repositories on the website.
> To minimize the effort of talking to github in each location and reduce the number of calls to Github ( for rate limiting purposes ) we should generate a JSON/XML file that contains all information about the git repos. This file is in turns read by various "clients" that need the information.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)