You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Julian Foad <ju...@btopenworld.com> on 2009/07/19 18:26:20 UTC
Re: Proposal for sponsored development of "Obliterate"
I now have a fairly detailed proposal for developing "Obliterate".
(Attached.)
In order to make "Obliterate" a reality, and to do it in the best way
for the Subversion community, I want to find sponsorship for doing it as
part of the normal community development process.
Do you (anybody) think I should or should not...
- E-mail this to individuals who have expressed an interest in
"obliterate" in the past (with a few sentences saying why I'm emailing
it to them)?
- E-mail this to the svn "dev" and "users" lists? And Tortoise? Any
others?
- Post it on some on-line "proposals looking for sponsorship" web
sites such as <www.guru.com>?
Do you think it's OK to leave time and money estimates till after
someone shows an interest?
Appreciating any advice,
and review of the proposal itself.
- Julian
On Thu, 2009-05-21, I (Julian Foad) wrote:
> After being out of the Subversion world since January I now have time on
> my hands and I want to design and implement the much talked-about
> "Obliterate" feature. I'm looking for help in wording a proposal for
> doing it as a "sponsored" development, which I'll advertise in public
> and send to anybody I think might respond. The development itself will,
> of course, be done in full participation and only with the full
> agreement of the community, and the aim will be to integrate the results
> into trunk, not to create a special version.
>
> With this in mind, I've started drafting a proposal, which is attached.
> I broke it down into tasks that are roughly sequential, and which could
> possibly be charged for separately. Do you think this proposal contains
> the right sort of information? Too much? Something missing?
>
> One point I wonder about is whether I should try to get sponsorship for
> the whole thing or whether it would be better to do task 1 (requirements
> analysis) and maybe 2 (high-level design proposals) as a volunteer in
> order to give a potential sponsor better confidence and less risk.
>
> Thoughts please?
>
> One thought of mine is this would work best if I come up with at least
> one other proposal for some different work, and then a sponsor can feel
> that their money will really lead to their preferred feature being
> developed first.
>
> I also expect I could advertise this as a "distributed funding" project
> on one of the web sites that help organise that.
>
> - Julian
>
> ------------------------------------------------------
> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2352672
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2372448
Re: Proposal for sponsored development of "Obliterate"
Posted by Jack Repenning <jr...@collab.net>.
On Jul 27, 2009, at 11:22 AM, Bill Tutt wrote:
> Software development teams that frequently check in large binary
> files that are produced during the build process (or indeed checked
> in as part of the development process) are candidates for desiring #3.
Oh, yes, you're right. And, indeed, IBM Rational ClearCase provides
this form.
-==-
Jack Repenning
Chief Technology Officer
CollabNet, Inc.
8000 Marina Boulevard, Suite 600
Brisbane, California 94005
office: +1 650.228.2562
twitter: http://twitter.com/jrep
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2376101
Re: Proposal for sponsored development of "Obliterate"
Posted by Bill Tutt <bi...@gmail.com>.
On Mon, Jul 27, 2009 at 1:51 PM, Jack Repenning <jr...@collab.net>wrote:
> So we would have these Obliterate cases:
>
> 1. Utterly removing all traces that a certain file or collection of
> files ever existed in the repository. (My restatement of your first
> case.)
>
> 2. Removing obsolete data from space motives. (Your second.)
>
> 3. Selective excision of history from file or files whose prior and
> subsequent history remains intact. (Your "notable case.")
>
> Not the least of my reasons for wanting to call this a separate item
> is this: at CollabNet, we've had many customers asking for #1 and #2,
> but to the best of my knowledge not a single word has ever been spoken
> in favor of #3.
>
Software development teams that frequently check in large binary files that
are produced during the build process (or indeed checked in as part of the
development process) are candidates for desiring #3.
I don't how many development shops run into these kinds of issues let alone
with Subversion.
I do know significant space was saved in the Windows and Visual Studio
Source Depot servers was saved by obliterating these large binary files
between major publicly released milestones.
Both of these very large version control trees have a public\ directory that
contains (among other things) checked in DLL export .lib files for DLLs that
are used across component boundaries and are updated in the daily build
process. This allows parital branch enlistments for developers.
#3 may just be a hidden use case that people don't immediately think of when
thinking about obliterate functionality.
Bill
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2376091
Re: Proposal for sponsored development of "Obliterate"
Posted by Julian Foad <ju...@btopenworld.com>.
On Tue, 2009-07-28 at 09:36 -0700, Jack Repenning wrote:
> On Jul 28, 2009, at 4:55 AM, Julian Foad wrote:
>
> > Can you give me a link to somewhere I can read about this [Eclipse]
> > requirement
> > for removal? I'm interested in discerning the extent to which they
> > want to trace the banned code to WCs, mirror repositories, backups,
> > and the like, and also
> > what kind of audit trail they would prefer to see in (or outside) the
> > repository. This is an aspect of "obliterate" that I have hardly gone
> > into.
>
> Some useful links. Look for "Parallel IP Process" in each:
>
> http://wiki.eclipse.org/Development_Resources/New_Commmitter_Handbook#Licenses.2C_Intellectual_Property_Due_Diligence.2C_and_other_Legal_Stuff
>
> (or http://bit.ly/16dWHw)
For the record...
The reference to "removing" in this document reads, "a contribution can
be committed into a project's source code repository while the due
diligence process is undertaken. If, at the end of the due diligence
process, the contribution is rejected, it must be removed from the
source code repository."
> http://eclipse.org/legal/EclipseLegalProcessPoster.pdf
>
> http://eclipse.org/org/documents/Eclipse_Policy_and_Procedure_for_3rd_Party_Dependencies_Final.pdf
>
> (or http://bit.ly/seyVp)
>
> http://wiki.eclipse.org/Development_Resources/HOWTO/Parallel_IP_Process
> (or http://bit.ly/5LIve)
Thanks, Jack.
For the record (for anyone else reading this), this is what I found in
those docs. The first document says "If [...] the contribution is
rejected, it must be removed from the source code repository." The last
document has a link to
<http://www.eclipse.org/org/documents/Eclipse_IP_Policy.pdf?page=2>
which says: "reasonable steps should be taken to ensure that any Content
which fails the Required Due Diligence is no longer made available as
Distributed Content." "Distributed Content" means content "distributed
by the Eclipse Foundation via its Repository or other means".
> In the Eclipse case, I believe the answer to all your "how concerned
> are they..." questions is "not in the slightest." They are, for
> example, supremely happy with what CVS gives them, the ability to
> centrally delete a tree of *,v files and let the dust fall where it
> may.
So, yes, that's how I understand it too.
- Julian
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2376646
Re: Proposal for sponsored development of "Obliterate"
Posted by Jack Repenning <jr...@collab.net>.
On Jul 28, 2009, at 4:55 AM, Julian Foad wrote:
> Can you give me a link to somewhere I can read about this [Eclipse]
> requirement
> for removal? I'm interested in discerning the extent to which they
> want to trace the banned code to WCs, mirror repositories, backups,
> and the like, and also
> what kind of audit trail they would prefer to see in (or outside) the
> repository. This is an aspect of "obliterate" that I have hardly gone
> into.
Some useful links. Look for "Parallel IP Process" in each:
http://wiki.eclipse.org/Development_Resources/New_Commmitter_Handbook#Licenses.2C_Intellectual_Property_Due_Diligence.2C_and_other_Legal_Stuff
(or http://bit.ly/16dWHw)
http://eclipse.org/legal/EclipseLegalProcessPoster.pdf
http://eclipse.org/org/documents/Eclipse_Policy_and_Procedure_for_3rd_Party_Dependencies_Final.pdf
(or http://bit.ly/seyVp)
http://wiki.eclipse.org/Development_Resources/HOWTO/Parallel_IP_Process
(or http://bit.ly/5LIve)
In the Eclipse case, I believe the answer to all your "how concerned
are they..." questions is "not in the slightest." They are, for
example, supremely happy with what CVS gives them, the ability to
centrally delete a tree of *,v files and let the dust fall where it
may.
-==-
Jack Repenning
Chief Technology Officer
CollabNet, Inc.
8000 Marina Boulevard, Suite 600
Brisbane, California 94005
office: +1 650.228.2562
twitter: http://twitter.com/jrep
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2376332
Re: Proposal for sponsored development of "Obliterate"
Posted by Julian Foad <ju...@btopenworld.com>.
On Mon, 2009-07-27, Jack Repenning wrote:
> Going back to my restatement of your first case: one change I made in
> the wording was to remove the word "accidentally." There's some point
> to this: the thing added was often no accident, but rather further
> review, or changing circumstances, compel the removal. A real-world
> example that may be familiar to at least some readers: in the Eclipse
> community, they have fairly detailed constraints on the licensing of
> externally sourced modules added to the Eclipse site. But because
> review is expensive, they also have a procedure called something like
> "Deferred Intellectual Property," where an Eclipse project is allowed
> to add previously un-reviewed stuff speculatively, in parallel with
> launching the license review. If the review should turn out badly,
> then the component must be removed--utterly removed: the lawyers have
> determined that the mere presence of this module in the build tree is
> a license violation.
Can you give me a link to somewhere I can read about this requirement
for removal?
I'm interested in discerning the extent to which they want to trace the
banned code to WCs, mirror repositories, backups, and the like, and also
what kind of audit trail they would prefer to see in (or outside) the
repository. This is an aspect of "obliterate" that I have hardly gone
into.
- Julian
> They're not concerned with preserving working
> copy health for any working copy that includes the banned module: any
> such working copy is a liability; if anything, it *should* begin
> failing. They're not concerned with excising a nibble of history, they
> want the whole dang thing gone, as if it had never been. And it may
> well be that quite a number of development and interim builds have
> been done, quite a number of dependent changes have been made, with
> the now-banned module in place: it is, if anything, *desirable* that
> such builds now become un-buildable, such changes fail to compile, and
> whatever work it takes now be performed to replace or do without the
> now-banned module. I removed "accidentally" in order to open the door
> to this possibility of nontrivial amounts of accumulated history, and
> nontrivial history breakage.
- Julian
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2376267
Re: Proposal for sponsored development of "Obliterate"
Posted by Julian Foad <ju...@btopenworld.com>.
Jack,
For now, just a quick thank-you for this response. I appreciate your
points. I'll look for my "notable case" (your no. 3) among the use cases
I've seen, and work out some amendments.
- Julian
Jack Repenning wrote:
> On Jul 19, 2009, at 11:26 AM, Julian Foad wrote:
>
> > I now have a fairly detailed proposal for developing "Obliterate".
> > (Attached.)
>
>
> In your Intro, you call out the two distinct forms I've know of for
> Obliterate, then add what you call a "notable case of the former":
> removal of certain deltas of a file, leaving prior and subsequent
> history intact. I think this case deserves visibility as yet a third
> case, rather than being treated as a subcase of either of the others.
> So we would have these Obliterate cases:
>
> 1. Utterly removing all traces that a certain file or collection of
> files ever existed in the repository. (My restatement of your first
> case.)
>
> 2. Removing obsolete data from space motives. (Your second.)
>
> 3. Selective excision of history from file or files whose prior and
> subsequent history remains intact. (Your "notable case.")
>
> Not the least of my reasons for wanting to call this a separate item
> is this: at CollabNet, we've had many customers asking for #1 and #2,
> but to the best of my knowledge not a single word has ever been spoken
> in favor of #3.
>
> Given the frequent and emphatic demands for numbers 1 and 2 we've had,
> I am candidly baffled why CollabNet has not managed to fund this work
> to date (and I hope you'll include us in your list of people to whom
> to submit the proposal). But I would be surprised if CollabNet were
> willing to extend any deadlines or provide any incremental funding for
> the third form.
>
> Similarly, the commercial input we've had on this feature is almost
> totally unconcerned about the impact on working copies. While no doubt
> everyone would agree that preserving WC health where possible is good,
> rather than bad, the people who have offered us money for anything in
> this domain were perfectly willing to sacrifice this in order to get
> the primary, server-side benefits.
>
> Going back to my restatement of your first case: one change I made in
> the wording was to remove the word "accidentally." There's some point
> to this: the thing added was often no accident, but rather further
> review, or changing circumstances, compel the removal. A real-world
> example that may be familiar to at least some readers: in the Eclipse
> community, they have fairly detailed constraints on the licensing of
> externally sourced modules added to the Eclipse site. But because
> review is expensive, they also have a procedure called something like
> "Deferred Intellectual Property," where an Eclipse project is allowed
> to add previously un-reviewed stuff speculatively, in parallel with
> launching the license review. If the review should turn out badly,
> then the component must be removed--utterly removed: the lawyers have
> determined that the mere presence of this module in the build tree is
> a license violation. They're not concerned with preserving working
> copy health for any working copy that includes the banned module: any
> such working copy is a liability; if anything, it *should* begin
> failing. They're not concerned with excising a nibble of history, they
> want the whole dang thing gone, as if it had never been. And it may
> well be that quite a number of development and interim builds have
> been done, quite a number of dependent changes have been made, with
> the now-banned module in place: it is, if anything, *desirable* that
> such builds now become un-buildable, such changes fail to compile, and
> whatever work it takes now be performed to replace or do without the
> now-banned module. I removed "accidentally" in order to open the door
> to this possibility of nontrivial amounts of accumulated history, and
> nontrivial history breakage.
>
> You asked:
>
> > Do you think it's OK to leave time and money estimates till after
> > someone shows an interest?
>
> It's perfectly reasonable to leave out time and money from the
> proposal you provide here, since it doesn't even commit itself to what
> will actually be done! ;-) But I would think anyone considering
> sponsorship would want to know that, and would certainly need a cost
> write-up before committing. I dunno, though, how you best dance your
> way into something more concrete (cf. "baffled," above).
>
> -==-
> Jack Repenning
> Chief Technology Officer
> CollabNet, Inc.
> 8000 Marina Boulevard, Suite 600
> Brisbane, California 94005
> office: +1 650.228.2562
> twitter: http://twitter.com/jrep
>
>
>
>
>
>
>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2376100
Re: Proposal for sponsored development of "Obliterate"
Posted by Jack Repenning <jr...@collab.net>.
On Jul 19, 2009, at 11:26 AM, Julian Foad wrote:
> I now have a fairly detailed proposal for developing "Obliterate".
> (Attached.)
In your Intro, you call out the two distinct forms I've know of for
Obliterate, then add what you call a "notable case of the former":
removal of certain deltas of a file, leaving prior and subsequent
history intact. I think this case deserves visibility as yet a third
case, rather than being treated as a subcase of either of the others.
So we would have these Obliterate cases:
1. Utterly removing all traces that a certain file or collection of
files ever existed in the repository. (My restatement of your first
case.)
2. Removing obsolete data from space motives. (Your second.)
3. Selective excision of history from file or files whose prior and
subsequent history remains intact. (Your "notable case.")
Not the least of my reasons for wanting to call this a separate item
is this: at CollabNet, we've had many customers asking for #1 and #2,
but to the best of my knowledge not a single word has ever been spoken
in favor of #3.
Given the frequent and emphatic demands for numbers 1 and 2 we've had,
I am candidly baffled why CollabNet has not managed to fund this work
to date (and I hope you'll include us in your list of people to whom
to submit the proposal). But I would be surprised if CollabNet were
willing to extend any deadlines or provide any incremental funding for
the third form.
Similarly, the commercial input we've had on this feature is almost
totally unconcerned about the impact on working copies. While no doubt
everyone would agree that preserving WC health where possible is good,
rather than bad, the people who have offered us money for anything in
this domain were perfectly willing to sacrifice this in order to get
the primary, server-side benefits.
Going back to my restatement of your first case: one change I made in
the wording was to remove the word "accidentally." There's some point
to this: the thing added was often no accident, but rather further
review, or changing circumstances, compel the removal. A real-world
example that may be familiar to at least some readers: in the Eclipse
community, they have fairly detailed constraints on the licensing of
externally sourced modules added to the Eclipse site. But because
review is expensive, they also have a procedure called something like
"Deferred Intellectual Property," where an Eclipse project is allowed
to add previously un-reviewed stuff speculatively, in parallel with
launching the license review. If the review should turn out badly,
then the component must be removed--utterly removed: the lawyers have
determined that the mere presence of this module in the build tree is
a license violation. They're not concerned with preserving working
copy health for any working copy that includes the banned module: any
such working copy is a liability; if anything, it *should* begin
failing. They're not concerned with excising a nibble of history, they
want the whole dang thing gone, as if it had never been. And it may
well be that quite a number of development and interim builds have
been done, quite a number of dependent changes have been made, with
the now-banned module in place: it is, if anything, *desirable* that
such builds now become un-buildable, such changes fail to compile, and
whatever work it takes now be performed to replace or do without the
now-banned module. I removed "accidentally" in order to open the door
to this possibility of nontrivial amounts of accumulated history, and
nontrivial history breakage.
You asked:
> Do you think it's OK to leave time and money estimates till after
> someone shows an interest?
It's perfectly reasonable to leave out time and money from the
proposal you provide here, since it doesn't even commit itself to what
will actually be done! ;-) But I would think anyone considering
sponsorship would want to know that, and would certainly need a cost
write-up before committing. I dunno, though, how you best dance your
way into something more concrete (cf. "baffled," above).
-==-
Jack Repenning
Chief Technology Officer
CollabNet, Inc.
8000 Marina Boulevard, Suite 600
Brisbane, California 94005
office: +1 650.228.2562
twitter: http://twitter.com/jrep
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2376079
Re: Proposal for sponsored development of "Obliterate"
Posted by Julian Foad <ju...@btopenworld.com>.
On Mon, 2009-07-20 at 12:46 +0200, Philipp Marek wrote:
> On Sonntag, 19. Juli 2009, Julian Foad wrote:
> > I now have a fairly detailed proposal for developing "Obliterate".
> > (Attached.)
> ...
> > Appreciating any advice,
> >
> > and review of the proposal itself.
>
>
> > Integration
> > If working on a branch, work shall be merged into trunk at least
> > at every defined milestone.
> I'm not so sure that merging to trunk would be possible/wise at your
> "singularly" defined points in time.
> At the least it might interfere with some other things.
Yes, that's a good point. What I meant was that the development must not
be allowed to remain isolated on a branch for a very long time.
> > Sponsor and contractor will respond to queries from the other
> > by end of next working day.
> Good luck with that.
> I don't think that the paying end wants to agree to this.
> (But maybe that's just my bad experience.)
Heh. Yes. Instead, I should say something less rigid and more
encouraging, such as, "The sponsor and the contractor should keep in
close contact throughout the project."
> The links at the end could be clickable.
>
>
> There's not much news in there, is it?
Thanks for the review.
- Julian
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2376043
Re: Proposal for sponsored development of "Obliterate"
Posted by Philipp Marek <ph...@emerion.com>.
On Sonntag, 19. Juli 2009, Julian Foad wrote:
> I now have a fairly detailed proposal for developing "Obliterate".
> (Attached.)
...
> Appreciating any advice,
>
> and review of the proposal itself.
> Integration
> If working on a branch, work shall be merged into trunk at least
> at every defined milestone.
I'm not so sure that merging to trunk would be possible/wise at your
"singularly" defined points in time.
At the least it might interfere with some other things.
> Sponsor and contractor will respond to queries from the other
> by end of next working day.
Good luck with that.
I don't think that the paying end wants to agree to this.
(But maybe that's just my bad experience.)
The links at the end could be clickable.
There's not much news in there, is it?
Regards,
Phil