You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@community.apache.org by Rich Bowen <rb...@rcbowen.com> on 2025/10/02 14:21:27 UTC
Weekly project growth reports
Hi, folks,
For the last couple of months I’ve been producing these - https://boxofclue.com/apache-highlights/
The process is that I have a checkout of every repo under https://github.com/apache (just metadata, not actual files. 2.9Gb total) and I grind through them to generate some metrics like:
First time commit (ie, had a commit in a merged PR the first time)
10th/100th/1000th/etc commit
There are some false positives, which I think come from, for example, X makes a first-time commit to iceberg-fortran but they’ve contributed to iceberg-rust before. But for the most part, it gives a really great weekly snapshot of who the new people in your project are. I’ve gotten a couple positive comments from a handful of projects that are using this data to welcome new contributors, which was the intent of the thing. (I post it to Mastodon every week.)
I’d like to run this on our VM, rather than running it on my laptop every Monday morning. I’d also like to link to the reports from a couple of places on our website (and don’t really want to link to boxofclue.com <http://boxofclue.com/> from there!) But I wanted to run it by you folks first, before taking the liberty to do that. Does anybody have any objections to me doing this?
—
Rich Bowen
rbowen@rcbowen.com
Re: Weekly project growth reports
Posted by Kanchana Welagedara <ka...@gmail.com>.
Hi Rich ,
Super useful data, I can clearly see how we can extend this to extract
more data/metrics around committer/ PMC growth as well.
My knowledge is limited . But in this area of software development and
compliance and launching global software at the enterprise level I can
explain related information, Here's what I understand: GDPR (General Data
Protection Regulation) is a regulatory framework created by the European
Union to protect the privacy and personal data of EU citizens. It applies
to any organization worldwide that collects, processes, or stores data of
EU residents. Key requirements include explicit user consent for data
collection, the right to access, correct, and delete personal data,
mandatory data breach notifications, data minimization, and purpose
limitation, with heavy penalties for non-compliance (up to 4% of global
revenue - we are non-profile so don't know :). ).
This creates a conflict for ASF. Contributors voluntarily submit their
names/emails, but GDPR requires explicit consent and clarity on how data
will be stored or published. If an EU contributor requests deletion, ASF
faces a tension: bylaws and open source principles require permanent
archives for accountability, while GDPR enforces the ‘Right to be
Forgotten.’
*To resolve this, ASF needs to clearly explain how public data is used,
stored, and why it cannot be fully deleted, setting expectations before
participation/publishing data.*
Cheers,
Kanchana
On Thu, Oct 2, 2025 at 12:20 PM sebb <se...@gmail.com> wrote:
> On Thu, 2 Oct 2025 at 17:20, Rich Bowen <rb...@rcbowen.com> wrote:
> >
> > On Oct 2, 2025, at 11:36 AM, sebb <se...@gmail.com> wrote:
> > >
> > >> Also, I suppose a related question is, do you think anyone would have
> any objection to their name being listed on such a document on an Apache
> website? I cannot personally think why they would (and this is all
> already-public data) but I suppose it is possible that someone might, and I
> want to be sensitive to that.
> > >
> > > AIUI, just because a particular item of PII is published in one
> > > location does not mean it can be published elsewhere.
> >
> > Yeah, that’s what I was a little concerned about. The legalities (and,
> indeed, just individual preferences or sensitivities) around aggregating
> metrics remains a bit fuzzy to me. Do you think that this is better kept to
> myself, then?
> >
> > > Does the data have to be fully public?
> > > Indeed would it mean anything to the general public?
> >
> >
> > I think it’s most valuable to other contributors on the same project -
> who are not necessarily committers or PMC members. What I’m specifically
> trying to encourage with this data is for individuals on projects to
> welcome and celebrate new community participants, and milestones of
> existing participants, since that kind of recognition tends to lead to
> higher retention rates, according to research that I’ve seen at several
> recent conferences. And that is, after all, the mandate of this PMC.
> >
> > But I do want to do this in a way that is respectful to those same
> contributors.
>
> I were a new joiner, and I did not want to appear in the listings, I'm
> not sure I would be happy to have to ask for my data to be omitted.
>
> > —
> > Rich Bowen
> > rbowen@rcbowen.com
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@community.apache.org
> For additional commands, e-mail: dev-help@community.apache.org
>
>
Re: Weekly project growth reports
Posted by Jarek Potiuk <ja...@potiuk.com>.
I am not.a lawyer, but I don't think we publish any "new" information -
this information is fully available to anyone who is reading (public)
github API so it's already published. It's as easy as:
```
git clone URL (might be our public gitbox)
git --no-pager log --pretty=format:"%h %an <%ae>" | sort | uniq
```
We are just providing an aggregation of this information. And we already
did for years in not-so-distant-past - for many years our newsletter
contained the most information about top 5 (I think) contributors. I don't
think those reports provide "more" information than the one that is already
public (and with explicitl consent of the user who sets "name" and "email"
in their commits to public repository. It's also already available in
countless aggregations summaries and statistics that various people do with
that data - say https://insights.linuxfoundation.org/ for example.
if someone does not want to, they are completely free to set their
name/email to **anything they want** when they make commit. That's a very
good privacy protection when they want to contirbute things and keep their
privacy - but once you committed to a repo with your real name / email,
it's **done**. you committed to it, and it's public. And telling that you
committed "x" commits does not make it any more public than counting `git
--no-pager log --pretty=format:"%h %an <%ae>"` that include your name/email.
There might be some concerns whether you want to be "compared" with others
publcly, but that's a different story. IMHO it does not change the PII
status - since you are just aggregating things and your PII data (for the
same things - commits to a specific repo) is alrady published.
But maybe I am ignorant here - what's the source of your assumption that it
might be different Sebb? Any prior art or specific chapters of GDPR
regulation that made you think differently ? I am curious, as it's quite
counterintuitive, but I am sure you have a reason to believe it is
problematic?
J.
On Sat, Oct 4, 2025 at 10:27 AM Travis Wright <tr...@gmail.com>
wrote:
> What the fuk
>
> On Sat, Oct 4, 2025, 11:05 AM sebb <se...@gmail.com> wrote:
>
> > On Sat, 4 Oct 2025 at 02:44, Craig Russell <ap...@gmail.com> wrote:
> > >
> > >
> > >
> > > > On Oct 2, 2025, at 09:58, sebb <se...@gmail.com> wrote:
> > > >
> > > > On Thu, 2 Oct 2025 at 17:20, Rich Bowen <rb...@rcbowen.com> wrote:
> > > >>
> > > >> On Oct 2, 2025, at 11:36 AM, sebb <se...@gmail.com> wrote:
> > > >>>
> > > >>>> Also, I suppose a related question is, do you think anyone would
> > have any objection to their name being listed on such a document on an
> > Apache website? I cannot personally think why they would (and this is all
> > already-public data) but I suppose it is possible that someone might,
> and I
> > want to be sensitive to that.
> > > >>>
> > > >>> AIUI, just because a particular item of PII is published in one
> > > >>> location does not mean it can be published elsewhere.
> > > >>
> > > >> Yeah, that’s what I was a little concerned about. The legalities
> > (and, indeed, just individual preferences or sensitivities) around
> > aggregating metrics remains a bit fuzzy to me. Do you think that this is
> > better kept to myself, then?
> > > >>
> > > >>> Does the data have to be fully public?
> > > >>> Indeed would it mean anything to the general public?
> > > >>
> > > >>
> > > >> I think it’s most valuable to other contributors on the same project
> > - who are not necessarily committers or PMC members. What I’m
> specifically
> > trying to encourage with this data is for individuals on projects to
> > welcome and celebrate new community participants, and milestones of
> > existing participants, since that kind of recognition tends to lead to
> > higher retention rates, according to research that I’ve seen at several
> > recent conferences. And that is, after all, the mandate of this PMC.
> > > >>
> > > >> But I do want to do this in a way that is respectful to those same
> > contributors.
> > > >
> > > > I were a new joiner, and I did not want to appear in the listings,
> I'm
> > > > not sure I would be happy to have to ask for my data to be omitted.
> > >
> > > What we do know is that the contributors' github id and email address
> > are public, assuming they appear in communications to the project's
> public
> > mail lists. So I have no privacy concerns "publishing" these bits of PII
> > after they are already public.
> >
> > I thought once, but AIUI now, publication in one arena does not give
> > carte blanche to publication anywhere else.
> >
> > > In any event, at the time when they accept an invitation for committer,
> > these bits will necessarily become public via announcement that they have
> > been voted and accepted for committer.
> >
> > There's also big a difference between being published as one name in
> > hundreds or thousands of others, and as one name in a handful.
> >
> > > Craig
> > >
> > > >
> > > >> —
> > > >> Rich Bowen
> > > >> rbowen@rcbowen.com
> > > >>
> > > >>
> > > >>
> > > >>
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@community.apache.org
> > > > For additional commands, e-mail: dev-help@community.apache.org
> > > >
> > >
> > > Craig L Russell
> > > clr@apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@community.apache.org
> > For additional commands, e-mail: dev-help@community.apache.org
> >
> >
>
Re: Weekly project growth reports
Posted by Travis Wright <tr...@gmail.com>.
What the fuk
On Sat, Oct 4, 2025, 11:05 AM sebb <se...@gmail.com> wrote:
> On Sat, 4 Oct 2025 at 02:44, Craig Russell <ap...@gmail.com> wrote:
> >
> >
> >
> > > On Oct 2, 2025, at 09:58, sebb <se...@gmail.com> wrote:
> > >
> > > On Thu, 2 Oct 2025 at 17:20, Rich Bowen <rb...@rcbowen.com> wrote:
> > >>
> > >> On Oct 2, 2025, at 11:36 AM, sebb <se...@gmail.com> wrote:
> > >>>
> > >>>> Also, I suppose a related question is, do you think anyone would
> have any objection to their name being listed on such a document on an
> Apache website? I cannot personally think why they would (and this is all
> already-public data) but I suppose it is possible that someone might, and I
> want to be sensitive to that.
> > >>>
> > >>> AIUI, just because a particular item of PII is published in one
> > >>> location does not mean it can be published elsewhere.
> > >>
> > >> Yeah, that’s what I was a little concerned about. The legalities
> (and, indeed, just individual preferences or sensitivities) around
> aggregating metrics remains a bit fuzzy to me. Do you think that this is
> better kept to myself, then?
> > >>
> > >>> Does the data have to be fully public?
> > >>> Indeed would it mean anything to the general public?
> > >>
> > >>
> > >> I think it’s most valuable to other contributors on the same project
> - who are not necessarily committers or PMC members. What I’m specifically
> trying to encourage with this data is for individuals on projects to
> welcome and celebrate new community participants, and milestones of
> existing participants, since that kind of recognition tends to lead to
> higher retention rates, according to research that I’ve seen at several
> recent conferences. And that is, after all, the mandate of this PMC.
> > >>
> > >> But I do want to do this in a way that is respectful to those same
> contributors.
> > >
> > > I were a new joiner, and I did not want to appear in the listings, I'm
> > > not sure I would be happy to have to ask for my data to be omitted.
> >
> > What we do know is that the contributors' github id and email address
> are public, assuming they appear in communications to the project's public
> mail lists. So I have no privacy concerns "publishing" these bits of PII
> after they are already public.
>
> I thought once, but AIUI now, publication in one arena does not give
> carte blanche to publication anywhere else.
>
> > In any event, at the time when they accept an invitation for committer,
> these bits will necessarily become public via announcement that they have
> been voted and accepted for committer.
>
> There's also big a difference between being published as one name in
> hundreds or thousands of others, and as one name in a handful.
>
> > Craig
> >
> > >
> > >> —
> > >> Rich Bowen
> > >> rbowen@rcbowen.com
> > >>
> > >>
> > >>
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@community.apache.org
> > > For additional commands, e-mail: dev-help@community.apache.org
> > >
> >
> > Craig L Russell
> > clr@apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@community.apache.org
> For additional commands, e-mail: dev-help@community.apache.org
>
>
Re: Weekly project growth reports
Posted by sebb <se...@gmail.com>.
On Sat, 4 Oct 2025 at 02:44, Craig Russell <ap...@gmail.com> wrote:
>
>
>
> > On Oct 2, 2025, at 09:58, sebb <se...@gmail.com> wrote:
> >
> > On Thu, 2 Oct 2025 at 17:20, Rich Bowen <rb...@rcbowen.com> wrote:
> >>
> >> On Oct 2, 2025, at 11:36 AM, sebb <se...@gmail.com> wrote:
> >>>
> >>>> Also, I suppose a related question is, do you think anyone would have any objection to their name being listed on such a document on an Apache website? I cannot personally think why they would (and this is all already-public data) but I suppose it is possible that someone might, and I want to be sensitive to that.
> >>>
> >>> AIUI, just because a particular item of PII is published in one
> >>> location does not mean it can be published elsewhere.
> >>
> >> Yeah, that’s what I was a little concerned about. The legalities (and, indeed, just individual preferences or sensitivities) around aggregating metrics remains a bit fuzzy to me. Do you think that this is better kept to myself, then?
> >>
> >>> Does the data have to be fully public?
> >>> Indeed would it mean anything to the general public?
> >>
> >>
> >> I think it’s most valuable to other contributors on the same project - who are not necessarily committers or PMC members. What I’m specifically trying to encourage with this data is for individuals on projects to welcome and celebrate new community participants, and milestones of existing participants, since that kind of recognition tends to lead to higher retention rates, according to research that I’ve seen at several recent conferences. And that is, after all, the mandate of this PMC.
> >>
> >> But I do want to do this in a way that is respectful to those same contributors.
> >
> > I were a new joiner, and I did not want to appear in the listings, I'm
> > not sure I would be happy to have to ask for my data to be omitted.
>
> What we do know is that the contributors' github id and email address are public, assuming they appear in communications to the project's public mail lists. So I have no privacy concerns "publishing" these bits of PII after they are already public.
I thought once, but AIUI now, publication in one arena does not give
carte blanche to publication anywhere else.
> In any event, at the time when they accept an invitation for committer, these bits will necessarily become public via announcement that they have been voted and accepted for committer.
There's also big a difference between being published as one name in
hundreds or thousands of others, and as one name in a handful.
> Craig
>
> >
> >> —
> >> Rich Bowen
> >> rbowen@rcbowen.com
> >>
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@community.apache.org
> > For additional commands, e-mail: dev-help@community.apache.org
> >
>
> Craig L Russell
> clr@apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@community.apache.org
For additional commands, e-mail: dev-help@community.apache.org
Re: Weekly project growth reports
Posted by Craig Russell <ap...@gmail.com>.
> On Oct 2, 2025, at 09:58, sebb <se...@gmail.com> wrote:
>
> On Thu, 2 Oct 2025 at 17:20, Rich Bowen <rb...@rcbowen.com> wrote:
>>
>> On Oct 2, 2025, at 11:36 AM, sebb <se...@gmail.com> wrote:
>>>
>>>> Also, I suppose a related question is, do you think anyone would have any objection to their name being listed on such a document on an Apache website? I cannot personally think why they would (and this is all already-public data) but I suppose it is possible that someone might, and I want to be sensitive to that.
>>>
>>> AIUI, just because a particular item of PII is published in one
>>> location does not mean it can be published elsewhere.
>>
>> Yeah, that’s what I was a little concerned about. The legalities (and, indeed, just individual preferences or sensitivities) around aggregating metrics remains a bit fuzzy to me. Do you think that this is better kept to myself, then?
>>
>>> Does the data have to be fully public?
>>> Indeed would it mean anything to the general public?
>>
>>
>> I think it’s most valuable to other contributors on the same project - who are not necessarily committers or PMC members. What I’m specifically trying to encourage with this data is for individuals on projects to welcome and celebrate new community participants, and milestones of existing participants, since that kind of recognition tends to lead to higher retention rates, according to research that I’ve seen at several recent conferences. And that is, after all, the mandate of this PMC.
>>
>> But I do want to do this in a way that is respectful to those same contributors.
>
> I were a new joiner, and I did not want to appear in the listings, I'm
> not sure I would be happy to have to ask for my data to be omitted.
What we do know is that the contributors' github id and email address are public, assuming they appear in communications to the project's public mail lists. So I have no privacy concerns "publishing" these bits of PII after they are already public.
In any event, at the time when they accept an invitation for committer, these bits will necessarily become public via announcement that they have been voted and accepted for committer.
Craig
>
>> —
>> Rich Bowen
>> rbowen@rcbowen.com
>>
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@community.apache.org
> For additional commands, e-mail: dev-help@community.apache.org
>
Craig L Russell
clr@apache.org
Re: Weekly project growth reports
Posted by sebb <se...@gmail.com>.
On Thu, 2 Oct 2025 at 17:20, Rich Bowen <rb...@rcbowen.com> wrote:
>
> On Oct 2, 2025, at 11:36 AM, sebb <se...@gmail.com> wrote:
> >
> >> Also, I suppose a related question is, do you think anyone would have any objection to their name being listed on such a document on an Apache website? I cannot personally think why they would (and this is all already-public data) but I suppose it is possible that someone might, and I want to be sensitive to that.
> >
> > AIUI, just because a particular item of PII is published in one
> > location does not mean it can be published elsewhere.
>
> Yeah, that’s what I was a little concerned about. The legalities (and, indeed, just individual preferences or sensitivities) around aggregating metrics remains a bit fuzzy to me. Do you think that this is better kept to myself, then?
>
> > Does the data have to be fully public?
> > Indeed would it mean anything to the general public?
>
>
> I think it’s most valuable to other contributors on the same project - who are not necessarily committers or PMC members. What I’m specifically trying to encourage with this data is for individuals on projects to welcome and celebrate new community participants, and milestones of existing participants, since that kind of recognition tends to lead to higher retention rates, according to research that I’ve seen at several recent conferences. And that is, after all, the mandate of this PMC.
>
> But I do want to do this in a way that is respectful to those same contributors.
I were a new joiner, and I did not want to appear in the listings, I'm
not sure I would be happy to have to ask for my data to be omitted.
> —
> Rich Bowen
> rbowen@rcbowen.com
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@community.apache.org
For additional commands, e-mail: dev-help@community.apache.org
Re: Weekly project growth reports
Posted by Rich Bowen <rb...@rcbowen.com>.
On Oct 2, 2025, at 11:36 AM, sebb <se...@gmail.com> wrote:
>
>> Also, I suppose a related question is, do you think anyone would have any objection to their name being listed on such a document on an Apache website? I cannot personally think why they would (and this is all already-public data) but I suppose it is possible that someone might, and I want to be sensitive to that.
>
> AIUI, just because a particular item of PII is published in one
> location does not mean it can be published elsewhere.
Yeah, that’s what I was a little concerned about. The legalities (and, indeed, just individual preferences or sensitivities) around aggregating metrics remains a bit fuzzy to me. Do you think that this is better kept to myself, then?
> Does the data have to be fully public?
> Indeed would it mean anything to the general public?
I think it’s most valuable to other contributors on the same project - who are not necessarily committers or PMC members. What I’m specifically trying to encourage with this data is for individuals on projects to welcome and celebrate new community participants, and milestones of existing participants, since that kind of recognition tends to lead to higher retention rates, according to research that I’ve seen at several recent conferences. And that is, after all, the mandate of this PMC.
But I do want to do this in a way that is respectful to those same contributors.
—
Rich Bowen
rbowen@rcbowen.com
Re: Weekly project growth reports
Posted by sebb <se...@gmail.com>.
On Thu, 2 Oct 2025 at 16:26, Rich Bowen <rb...@rcbowen.com> wrote:
>
>
>
> > On Oct 2, 2025, at 10:45 AM, sebb <se...@gmail.com> wrote:
> >
> > On Thu, 2 Oct 2025 at 15:30, Rich Bowen <rbowen@rcbowen.com <ma...@rcbowen.com>> wrote:
> >>
> >> Hi, folks,
> >>
> >> For the last couple of months I’ve been producing these - https://boxofclue.com/apache-highlights/
> >>
> >> The process is that I have a checkout of every repo under https://github.com/apache (just metadata, not actual files. 2.9Gb total) and I grind through them to generate some metrics like:
> >>
> >> First time commit (ie, had a commit in a merged PR the first time)
> >> 10th/100th/1000th/etc commit
> >>
> >> There are some false positives, which I think come from, for example, X makes a first-time commit to iceberg-fortran but they’ve contributed to iceberg-rust before. But for the most part, it gives a really great weekly snapshot of who the new people in your project are. I’ve gotten a couple positive comments from a handful of projects that are using this data to welcome new contributors, which was the intent of the thing. (I post it to Mastodon every week.)
> >>
> >> I’d like to run this on our VM, rather than running it on my laptop every Monday morning. I’d also like to link to the reports from a couple of places on our website (and don’t really want to link to boxofclue.com <http://boxofclue.com/> <http://boxofclue.com/> from there!) But I wanted to run it by you folks first, before taking the liberty to do that. Does anybody have any objections to me doing this?
> >
> > Which VM did you have in mind?
>
> projects.apache.org <http://projects.apache.org/> seems to make the most sense, since this is project metrics.
>
> Also, I suppose a related question is, do you think anyone would have any objection to their name being listed on such a document on an Apache website? I cannot personally think why they would (and this is all already-public data) but I suppose it is possible that someone might, and I want to be sensitive to that.
AIUI, just because a particular item of PII is published in one
location does not mean it can be published elsewhere.
Does the data have to be fully public?
Indeed would it mean anything to the general public?
It could be restricted to committers or perhaps members (ASF or (P)PMC)
> —
> Rich Bowen
> rbowen@rcbowen.com
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@community.apache.org
For additional commands, e-mail: dev-help@community.apache.org
Re: Weekly project growth reports
Posted by Rich Bowen <rb...@rcbowen.com>.
> On Oct 2, 2025, at 10:45 AM, sebb <se...@gmail.com> wrote:
>
> On Thu, 2 Oct 2025 at 15:30, Rich Bowen <rbowen@rcbowen.com <ma...@rcbowen.com>> wrote:
>>
>> Hi, folks,
>>
>> For the last couple of months I’ve been producing these - https://boxofclue.com/apache-highlights/
>>
>> The process is that I have a checkout of every repo under https://github.com/apache (just metadata, not actual files. 2.9Gb total) and I grind through them to generate some metrics like:
>>
>> First time commit (ie, had a commit in a merged PR the first time)
>> 10th/100th/1000th/etc commit
>>
>> There are some false positives, which I think come from, for example, X makes a first-time commit to iceberg-fortran but they’ve contributed to iceberg-rust before. But for the most part, it gives a really great weekly snapshot of who the new people in your project are. I’ve gotten a couple positive comments from a handful of projects that are using this data to welcome new contributors, which was the intent of the thing. (I post it to Mastodon every week.)
>>
>> I’d like to run this on our VM, rather than running it on my laptop every Monday morning. I’d also like to link to the reports from a couple of places on our website (and don’t really want to link to boxofclue.com <http://boxofclue.com/> <http://boxofclue.com/> from there!) But I wanted to run it by you folks first, before taking the liberty to do that. Does anybody have any objections to me doing this?
>
> Which VM did you have in mind?
projects.apache.org <http://projects.apache.org/> seems to make the most sense, since this is project metrics.
Also, I suppose a related question is, do you think anyone would have any objection to their name being listed on such a document on an Apache website? I cannot personally think why they would (and this is all already-public data) but I suppose it is possible that someone might, and I want to be sensitive to that.
—
Rich Bowen
rbowen@rcbowen.com
Re: Weekly project growth reports
Posted by Rich Bowen <rb...@rcbowen.com>.
> On Oct 2, 2025, at 10:45 AM, sebb <se...@gmail.com> wrote:
>
> On Thu, 2 Oct 2025 at 15:30, Rich Bowen <rbowen@rcbowen.com <ma...@rcbowen.com>> wrote:
>>
>> Hi, folks,
>>
>> For the last couple of months I’ve been producing these - https://boxofclue.com/apache-highlights/
>>
>> The process is that I have a checkout of every repo under https://github.com/apache (just metadata, not actual files. 2.9Gb total) and I grind through them to generate some metrics like:
>>
>> First time commit (ie, had a commit in a merged PR the first time)
>> 10th/100th/1000th/etc commit
>>
>> There are some false positives, which I think come from, for example, X makes a first-time commit to iceberg-fortran but they’ve contributed to iceberg-rust before. But for the most part, it gives a really great weekly snapshot of who the new people in your project are. I’ve gotten a couple positive comments from a handful of projects that are using this data to welcome new contributors, which was the intent of the thing. (I post it to Mastodon every week.)
>>
>> I’d like to run this on our VM, rather than running it on my laptop every Monday morning. I’d also like to link to the reports from a couple of places on our website (and don’t really want to link to boxofclue.com <http://boxofclue.com/> <http://boxofclue.com/> from there!) But I wanted to run it by you folks first, before taking the liberty to do that. Does anybody have any objections to me doing this?
>
> Which VM did you have in mind?
projects.apache.org <http://projects.apache.org/> seems to make the most sense, since this is project metrics.
Also, I suppose a related question is, do you think anyone would have any objection to their name being listed on such a document on an Apache website? I cannot personally think why they would (and this is all already-public data) but I suppose it is possible that someone might, and I want to be sensitive to that.
—
Rich Bowen
rbowen@rcbowen.com
Re: Weekly project growth reports
Posted by sebb <se...@gmail.com>.
On Thu, 2 Oct 2025 at 15:30, Rich Bowen <rb...@rcbowen.com> wrote:
>
> Hi, folks,
>
> For the last couple of months I’ve been producing these - https://boxofclue.com/apache-highlights/
>
> The process is that I have a checkout of every repo under https://github.com/apache (just metadata, not actual files. 2.9Gb total) and I grind through them to generate some metrics like:
>
> First time commit (ie, had a commit in a merged PR the first time)
> 10th/100th/1000th/etc commit
>
> There are some false positives, which I think come from, for example, X makes a first-time commit to iceberg-fortran but they’ve contributed to iceberg-rust before. But for the most part, it gives a really great weekly snapshot of who the new people in your project are. I’ve gotten a couple positive comments from a handful of projects that are using this data to welcome new contributors, which was the intent of the thing. (I post it to Mastodon every week.)
>
> I’d like to run this on our VM, rather than running it on my laptop every Monday morning. I’d also like to link to the reports from a couple of places on our website (and don’t really want to link to boxofclue.com <http://boxofclue.com/> from there!) But I wanted to run it by you folks first, before taking the liberty to do that. Does anybody have any objections to me doing this?
Which VM did you have in mind?
> —
> Rich Bowen
> rbowen@rcbowen.com
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@community.apache.org
For additional commands, e-mail: dev-help@community.apache.org
Re: Weekly project growth reports
Posted by Rich Bowen <rb...@rcbowen.com>.
> On Oct 2, 2025, at 11:21 AM, giovanni@paclan.it wrote:
>
> On 10/2/25 4:21 PM, Rich Bowen wrote:
>> Hi, folks,
>> For the last couple of months I’ve been producing these - https://boxofclue.com/apache-highlights/
> interesting project
>
>> The process is that I have a checkout of every repo under https://github.com/apache (just metadata, not actual files. 2.9Gb total) and I grind through them to generate some metrics like:
> this doesn't seem to work for svn mirrored repositories (httpd and spamassassin at least), is this a known issue ?
Yes. And hopefully one of the advantages of putting the code upstream is that folks could improve it to encompass cases like that. But at the moment, it’s just a little hobby project for myself, and I haven’t had a chance to tinker with svn repos.
And there are also other problems with the data, I’m sure, which more eyes will help uncover.
—
Rich Bowen
rbowen@rcbowen.com
Re: Weekly project growth reports
Posted by gi...@paclan.it.
On 10/2/25 4:21 PM, Rich Bowen wrote:
> Hi, folks,
>
> For the last couple of months I’ve been producing these - https://boxofclue.com/apache-highlights/
>
interesting project
> The process is that I have a checkout of every repo under https://github.com/apache (just metadata, not actual files. 2.9Gb total) and I grind through them to generate some metrics like:
>
this doesn't seem to work for svn mirrored repositories (httpd and spamassassin at least), is this a known issue ?
Thanks
Giovanni
> First time commit (ie, had a commit in a merged PR the first time)
> 10th/100th/1000th/etc commit
>
> There are some false positives, which I think come from, for example, X makes a first-time commit to iceberg-fortran but they’ve contributed to iceberg-rust before. But for the most part, it gives a really great weekly snapshot of who the new people in your project are. I’ve gotten a couple positive comments from a handful of projects that are using this data to welcome new contributors, which was the intent of the thing. (I post it to Mastodon every week.)
>
> I’d like to run this on our VM, rather than running it on my laptop every Monday morning. I’d also like to link to the reports from a couple of places on our website (and don’t really want to link to boxofclue.com <http://boxofclue.com/> from there!) But I wanted to run it by you folks first, before taking the liberty to do that. Does anybody have any objections to me doing this?
>
> —
> Rich Bowen
> rbowen@rcbowen.com
>
>
>
>
>