You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Sean Owen <so...@cloudera.com> on 2014/11/06 13:13:01 UTC
JIRA + PR backlog
(Different topic, indulge me one more reply --)
Yes the number of JIRAs/PRs closed is unprecedented too and that
deserves big praise. The project has stuck to making all changes and
discussion in this public process, which is so powerful. Adjusted for
the sheer inbound volume, Spark is doing a much better job than other
projects; I would not hold them up as a benchmark of 'good enough', to
be honest.
JIRA is usually under-managed and it's a pet issue of mine. My motive
is that core contributor / committer time is very valuable and in
short supply. On the one hand we could use lots more of it to shepherd
changes and fix bugs in the core that only the very experienced can.
On the other hand, you all deserve time to work on your own changes,
build a business, etc.
So I harp on JIRA management as a way to save time:
- Merging PRs sooner means less rebasing / retesting
- Bouncing back bad PRs/JIRAs early teaches everyone what's acceptable
as a good PR/JIRA and prevents the noise in the first place
- Resolving issues soon prevents duplicates from being filed
- Recording 'WontFix' resolutions early heads off repeated
discussion/work on out of scope topics
I have more concrete ideas about managing this but it's not for now.
For now, thanks for zapping some old JIRAs this morning and for
endorsing the idea of staying on top of the issue list in general. As
a long-time fan I hope I can help from the sidelines by also closing
JIRAs I'm all but certain are stale, and review minor PRs to clear the
way for maintainers to take on the more important work.
On Thu, Nov 6, 2014 at 7:21 AM, Matei Zaharia <ma...@gmail.com> wrote:
> Several people asked about having maintainers review the PR queue for their modules regularly, and I like that idea. We have a new tool now to help with that in https://spark-prs.appspot.com.
>
> In terms of the set of open PRs itself, it is large but note that there are also 2800 *closed* PRs, which means we close the majority of PRs (and I don't know the exact stats but I'd guess that 90% of those are accepted and merged). I think one problem is that with GitHub, people often develop something as a PR and have a lot of discussion on there (including whether we even want the feature). I recently updated our "how to contribute" page to encourage opening a JIRA and having discussions on the dev list first, but I do think we need to be faster with closing ones that we don't have a plan to merge. Note that Hadoop, Hive, HBase, etc also have about 300 issues each in the "patch available" state, so this is some kind of universal constant :P.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: JIRA + PR backlog
Posted by Josh Rosen <ro...@gmail.com>.
If possible, it would be great if we could set up a DNS alias, such as prs.spark.apache.org, to point to this app. I think the appspot domain is being blocked by some users’ firewalls, preventing them from accessing the site.
On November 11, 2014 at 1:34:36 PM, Nicholas Chammas (nicholas.chammas@gmail.com) wrote:
That's a good idea.
To encourage and leverage the community to review PRs and JIRA issues, we
should link to both the PR dashboard <https://spark-prs.appspot.com/> and
the stale JIRA filter
<https://issues.apache.org/jira/browse/SPARK-560?filter=12329614> (or
whatever JIRA filter or dashboard we want) from someplace prominent,
perhaps from the CONTRIBUTING
<https://github.com/apache/spark/blob/master/CONTRIBUTING.md> file.
Nick
On Tue, Nov 11, 2014 at 4:12 PM, Patrick Wendell <pw...@gmail.com> wrote:
> I wonder if we should be linking to that dashboard somewhere from our
> official docs or the wiki...
>
> On Tue, Nov 11, 2014 at 12:23 PM, Nicholas Chammas
> <ni...@gmail.com> wrote:
> > Yeah, kudos to Josh for putting that together.
> >
> > On Tue, Nov 11, 2014 at 3:26 AM, Yu Ishikawa <
> yuu.ishikawa+spark@gmail.com>
> > wrote:
> >
> >> Great jobs!
> >> I didn't know "Spark PR Dashboard."
> >>
> >> Thanks
> >> Yu Ishikawa
> >>
> >>
> >>
> >> -----
> >> -- Yu Ishikawa
> >> --
> >> View this message in context:
> >>
> http://apache-spark-developers-list.1001551.n3.nabble.com/JIRA-PR-backlog-tp9157p9282.html
> >> Sent from the Apache Spark Developers List mailing list archive at
> >> Nabble.com.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >> For additional commands, e-mail: dev-help@spark.apache.org
> >>
> >>
>
Re: JIRA + PR backlog
Posted by Nicholas Chammas <ni...@gmail.com>.
That's a good idea.
To encourage and leverage the community to review PRs and JIRA issues, we
should link to both the PR dashboard <https://spark-prs.appspot.com/> and
the stale JIRA filter
<https://issues.apache.org/jira/browse/SPARK-560?filter=12329614> (or
whatever JIRA filter or dashboard we want) from someplace prominent,
perhaps from the CONTRIBUTING
<https://github.com/apache/spark/blob/master/CONTRIBUTING.md> file.
Nick
On Tue, Nov 11, 2014 at 4:12 PM, Patrick Wendell <pw...@gmail.com> wrote:
> I wonder if we should be linking to that dashboard somewhere from our
> official docs or the wiki...
>
> On Tue, Nov 11, 2014 at 12:23 PM, Nicholas Chammas
> <ni...@gmail.com> wrote:
> > Yeah, kudos to Josh for putting that together.
> >
> > On Tue, Nov 11, 2014 at 3:26 AM, Yu Ishikawa <
> yuu.ishikawa+spark@gmail.com>
> > wrote:
> >
> >> Great jobs!
> >> I didn't know "Spark PR Dashboard."
> >>
> >> Thanks
> >> Yu Ishikawa
> >>
> >>
> >>
> >> -----
> >> -- Yu Ishikawa
> >> --
> >> View this message in context:
> >>
> http://apache-spark-developers-list.1001551.n3.nabble.com/JIRA-PR-backlog-tp9157p9282.html
> >> Sent from the Apache Spark Developers List mailing list archive at
> >> Nabble.com.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >> For additional commands, e-mail: dev-help@spark.apache.org
> >>
> >>
>
Re: JIRA + PR backlog
Posted by Patrick Wendell <pw...@gmail.com>.
I wonder if we should be linking to that dashboard somewhere from our
official docs or the wiki...
On Tue, Nov 11, 2014 at 12:23 PM, Nicholas Chammas
<ni...@gmail.com> wrote:
> Yeah, kudos to Josh for putting that together.
>
> On Tue, Nov 11, 2014 at 3:26 AM, Yu Ishikawa <yu...@gmail.com>
> wrote:
>
>> Great jobs!
>> I didn't know "Spark PR Dashboard."
>>
>> Thanks
>> Yu Ishikawa
>>
>>
>>
>> -----
>> -- Yu Ishikawa
>> --
>> View this message in context:
>> http://apache-spark-developers-list.1001551.n3.nabble.com/JIRA-PR-backlog-tp9157p9282.html
>> Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: JIRA + PR backlog
Posted by Nicholas Chammas <ni...@gmail.com>.
Yeah, kudos to Josh for putting that together.
On Tue, Nov 11, 2014 at 3:26 AM, Yu Ishikawa <yu...@gmail.com>
wrote:
> Great jobs!
> I didn't know "Spark PR Dashboard."
>
> Thanks
> Yu Ishikawa
>
>
>
> -----
> -- Yu Ishikawa
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/JIRA-PR-backlog-tp9157p9282.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: JIRA + PR backlog
Posted by Yu Ishikawa <yu...@gmail.com>.
Great jobs!
I didn't know "Spark PR Dashboard."
Thanks
Yu Ishikawa
-----
-- Yu Ishikawa
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/JIRA-PR-backlog-tp9157p9282.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: JIRA + PR backlog
Posted by Nicholas Chammas <ni...@gmail.com>.
I think better tooling will make it much easier for committers to trim the
list of stale JIRA issues and PRs. Convenience enables action.
- Spark PR Dashboard <https://spark-prs.appspot.com/>: Additional
filters for stale PRs
<https://github.com/databricks/spark-pr-dashboard/issues/1> or PRs
waiting on committer response would be great.
- Stale Spark JIRA issues
<https://issues.apache.org/jira/issues/?filter=12329614&jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20updated%20%3C%3D%20-90d%20ORDER%20BY%20updated%20ASC>:
This filter is sorted by the least recently updated issues first and can be
filtered additionally by component. There are many, many easy wins in this
filter.
Nick
On Thu, Nov 6, 2014 at 7:13 AM, Sean Owen <so...@cloudera.com> wrote:
> (Different topic, indulge me one more reply --)
>
> Yes the number of JIRAs/PRs closed is unprecedented too and that
> deserves big praise. The project has stuck to making all changes and
> discussion in this public process, which is so powerful. Adjusted for
> the sheer inbound volume, Spark is doing a much better job than other
> projects; I would not hold them up as a benchmark of 'good enough', to
> be honest.
>
> JIRA is usually under-managed and it's a pet issue of mine. My motive
> is that core contributor / committer time is very valuable and in
> short supply. On the one hand we could use lots more of it to shepherd
> changes and fix bugs in the core that only the very experienced can.
> On the other hand, you all deserve time to work on your own changes,
> build a business, etc.
>
> So I harp on JIRA management as a way to save time:
> - Merging PRs sooner means less rebasing / retesting
> - Bouncing back bad PRs/JIRAs early teaches everyone what's acceptable
> as a good PR/JIRA and prevents the noise in the first place
> - Resolving issues soon prevents duplicates from being filed
> - Recording 'WontFix' resolutions early heads off repeated
> discussion/work on out of scope topics
>
> I have more concrete ideas about managing this but it's not for now.
> For now, thanks for zapping some old JIRAs this morning and for
> endorsing the idea of staying on top of the issue list in general. As
> a long-time fan I hope I can help from the sidelines by also closing
> JIRAs I'm all but certain are stale, and review minor PRs to clear the
> way for maintainers to take on the more important work.
>
>
> On Thu, Nov 6, 2014 at 7:21 AM, Matei Zaharia <ma...@gmail.com>
> wrote:
> > Several people asked about having maintainers review the PR queue for
> their modules regularly, and I like that idea. We have a new tool now to
> help with that in https://spark-prs.appspot.com.
> >
> > In terms of the set of open PRs itself, it is large but note that there
> are also 2800 *closed* PRs, which means we close the majority of PRs (and I
> don't know the exact stats but I'd guess that 90% of those are accepted and
> merged). I think one problem is that with GitHub, people often develop
> something as a PR and have a lot of discussion on there (including whether
> we even want the feature). I recently updated our "how to contribute" page
> to encourage opening a JIRA and having discussions on the dev list first,
> but I do think we need to be faster with closing ones that we don't have a
> plan to merge. Note that Hadoop, Hive, HBase, etc also have about 300
> issues each in the "patch available" state, so this is some kind of
> universal constant :P.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>