You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Paul Hammant <pa...@hammant.org> on 2017/09/13 02:22:36 UTC

Sparse checkouts suggestion

Compared to Perforce's client-spec, Subversion's sparse checkouts are quite
cumbersome:

svn checkout http://svn.apache.org/repos/asf/subversion --depth=immediates
cd subversion/trunk
svn update --set-depth infinity
cd ../tags
svn update --set-depth immediates
cd 1.7.7
svn update --set-depth infinity

.. and similar.

Could Subversion follow Perforce and allow an *alternate* mechanism that
leveraged include and exclude globbing paths?

Maybe in the root of working copy, the contents of a *special* file could
be honored:

.sparse_mappings.txt

Sample contents:

   exclude **/*
   include trunk/**/*
   include tags/*
   include tags/1.7.7/**/*

Where the end user to do svn-up from root (after that file changed), and
assuming 'svn st' were 'clean', the working copy would reshape itself.
Specifically directories would appear and disappear, and that NOT
necessarily be subversion adds or deletes - it'd feel the same as
permission changes within the directory tree.  Of course different
teammates with the same checkout may see entirely different things
(depending on the lines within their .sparse_mappings.txt

When I say special file, I mean Subversion isn't going to mark it as
candidate for svn-add when it sees it.  Meaning, it is set on the client
side by tooling and left there.  Tooling may include expand/contract
scripts similar to what Google have for their monorepo.

Elsewhere i have documented Google's fu and the need for it for scaled
Trunk-Based Development in a few places:

1. https://paulhammant.com/2014/01/06/googlers-subset-their-trunk/ (based
on 2009 memories). Yes I know they are in Piper now and out of Perforce.

2. Previously I forked a medium-sized Monorepo on Github, and did the the
complete expand/contract work for it - https://github.com/paul-
hammant-fork/jooby-monorepo-experiment - in Python.

3. I have generally written about monorepos here: https://
trunkbaseddevelopment.com/monorepos/

4. And the expanding contracting kind here: https://
trunkbaseddevelopment.com/expanding-contracting-monorepos/

So, in the tradition of the Subversion project, I'd like to discuss this
here, then go on to raise a JIRA ticket.

- Paul

Re: Sparse checkouts suggestion

Posted by Johan Corveleyn <jc...@gmail.com>.
On Thu, Sep 21, 2017 at 12:16 PM, Paul Hammant <pa...@hammant.org> wrote:
>>
>>
>> 2. Previously I forked a medium-sized Monorepo on Github, and did the the
>> complete expand/contract work for it -
>> https://github.com/paul-hammant-fork/jooby-monorepo-experiment - in Python.
>>
>
> LOGIBALL's Tim Krüger has just written about a deployment of an
> expanding/contracting monorepo for Git and Maven based on the above proof of
> concept: https://timkrueger.me/a-maven-git-monorepo/ (I'd helped him with
> the article).

FWIW, at my company we're actually working on something similar for
SVN + Ivy (we have a monorepo too, in svn, with all of our projects
together under /trunk, and sparse working copies for developers and on
our build servers).

We want to have automatically expanding/contracting sparse working
copies, based on the ivy dependencies between modules (starting from a
given "main module"). So far I've written a written a hackish python
script that finds all "ivy.xml" files in our tree as quickly as
possible (up to a certain directory level), 'svn cat's them, and
invokes an ant-ivy task to determine a list of the needed modules for
the given main module (in the form of relative paths in svn where
those ivy.xml's reside).

What's still left is "applying" that list of subdirs to a given
working copy (or checking out a new one with these subdirs). So here
too, I'd like to have a view(spec) feature that can handle the output
from our "ivy dependency tree" script as input, and adapt the wc
sparsity to it (or checkout a new one with the given sparsity).

For now I've had to put my work on that auto-scoping functionality on
hold (other priorities), but I hope to get back to it someday.

-- 
Johan

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
>
>
>
> 2. Previously I forked a medium-sized Monorepo on Github, and did the the
> complete expand/contract work for it - https://github.com/paul-hamm
> ant-fork/jooby-monorepo-experiment - in Python.
>
>
LOGIBALL's Tim Krüger has just written about a deployment of an
expanding/contracting monorepo for Git and Maven based on the above proof
of concept: https://timkrueger.me/a-maven-git-monorepo/ (I'd helped him
with the article).

There's are two Git realities that have to be closely monitored for teams
wanting to do trunk-based development in a monorepo:

1) the clone-size has an upper limit (accounts differ as to where that is)
2) there's a push-pull bottleneck when the team starts to attempt to push
at the same time (Git makes you pull stuff you don't have before you push)

Subversion doesn't have to consider those year-on-year. It does of course
have merge snafus that Git doesn't have. Then again Git has merge snafus
that Svn doesn't have - at least for creative branch use, IMO.

- Paul

Re: Sparse checkouts suggestion

Posted by Branko Čibej <br...@apache.org>.
On 14.09.2017 15:49, Paul Hammant wrote:
> Would I retain the previous value of os.system and set it back after
> any assertions (pass or fail) so that following tests would not affected?
> Or does it und itself naturally when Pytest recurses in the next test
> in its suite ?

It wouldn't undo itself unless PyTest runs a separate instance of the
interpreter for each test case, which I don't believe it does. The best
way for restoring the original function in the module would be by using
an appropriately crafted context manager in the scope of your test case.
Either that, or add the mock/revert to the per-testcase setup and
teardown methods.

> As I hinted I couldn't find such an example online, with an emphatic
> 'best practice' claim attached.

There's no reason to expect to find any kind of best practice for
anything online. :)

In the case of Python, the semantics of how module imports work is the
clue to the obvious way to mock functions (and classes, etc) from
modules you don't otherwise control.

> What do you intend to convey with "Well well well"?

That I just made an emphatic best practice available online. :D

-- Brane

Re: Sparse checkouts suggestion

Posted by Branko Čibej <br...@apache.org>.
On 16.09.2017 20:06, Paul Hammant wrote:
>
>
>     foo.py:
>
>     import os
>     def my_mocked_os_system(*args, **kwargs):
>         print('a mockery')
>     os.system = my_mocked_os_system
>
>
>     bar.py:
>
>     import os
>     os.system("echo bar")
>
>     import foo
>     os.system("echo foo")
>
>
>     Then:
>
>     $ PYTHONPATH=. python bar.py
>     bar
>     a mockery
>
>
> I've spent the last 5 hours looking at mock decorators for Python (and
> MagicMock and stackoverflow and blogs). I can't get it working on my
> own - I'm not experienced enough with Python.  Specifically, I can't
> do it _less_ lines of code and have it the correct Pythonic way. I
> need help from someone that's prepared to remote pair on it.

Why are you complicating your life with trying to implement complex
decorators? Python decorators are _hard_.  Just replace the default
functions in the module in the testcase setup method and restore them in
the teardown method. Or write a simple context manager and use it in
your test case, e.g., like this:

    def test_foo():
        with mock_my_stuff():
            # code for testing foo goes here
            pass


Now all you have to do is write the mock_my_stuff() function, which
should return a context manager object that injects the mock functions
into the module in its __enter__() method an restores them in the
__exit__() method. This standard module will help make this even
simpler: https://docs.python.org/2/library/contextlib.html

-- Brane

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
>
>
> foo.py:
>
> import os
> def my_mocked_os_system(*args, **kwargs):
>     print('a mockery')
> os.system = my_mocked_os_system
>
>
> bar.py:
>
> import os
> os.system("echo bar")
>
> import foo
> os.system("echo foo")
>
>
> Then:
>
> $ PYTHONPATH=. python bar.py
> bar
> a mockery
>
>
I've spent the last 5 hours looking at mock decorators for Python (and
MagicMock and stackoverflow and blogs). I can't get it working on my own -
I'm not experienced enough with Python.  Specifically, I can't do it *less*
lines of code and have it the correct Pythonic way. I need help from
someone that's prepared to remote pair on it.

- Paul

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
Would I retain the previous value of os.system and set it back after any
assertions (pass or fail) so that following tests would not affected?
Or does it und itself naturally when Pytest recurses in the next test in
its suite ?

As I hinted I couldn't find such an example online, with an emphatic 'best
practice' claim attached.

What do you intend to convey with "Well well well"?

- Paul

Re: Sparse checkouts suggestion

Posted by Branko Čibej <br...@apache.org>.
On 14.09.2017 15:29, Paul Hammant wrote:
>
>     Yes, that one's pretty nice, but as a rule we try not to burden our
>     users with dependencies outside the standard library. The script uses
>     os.system, what, twice? Hardly worth replacing.
>
>     On the other hand, we have wrappers around subprocess.Popen in
>     subversion/tests/cmdline/svntest/main.py; you should be able to
>     recycle
>     those.
>
>
> I wasn't suggesting any change to os.system for this script. I'm just
> adding tests. I couldn't find decent examples of os.system mocking
> (and stderr/out) so I did it old-school style. Mocking in Python is a
> mess. No cohesive 'winning' approach/framework, and how-to information
> not consolidated anywhere.

Well, well, well.

foo.py:

import os
def my_mocked_os_system(*args, **kwargs):
    print('a mockery')
os.system = my_mocked_os_system


bar.py:

import os
os.system("echo bar")

import foo
os.system("echo foo")


Then:

$ PYTHONPATH=. python bar.py 
bar
a mockery


:)

-- Brane

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
>
> Yes, that one's pretty nice, but as a rule we try not to burden our
> users with dependencies outside the standard library. The script uses
> os.system, what, twice? Hardly worth replacing.
>
> On the other hand, we have wrappers around subprocess.Popen in
> subversion/tests/cmdline/svntest/main.py; you should be able to recycle
> those.
>

I wasn't suggesting any change to os.system for this script. I'm just
adding tests. I couldn't find decent examples of os.system mocking (and
stderr/out) so I did it old-school style. Mocking in Python is a mess. No
cohesive 'winning' approach/framework, and how-to information not
consolidated anywhere.

As much as I like Mockito for Java, some of those OS level things are as
ugly to mock there too -
http://adambien.blog/roller/abien/entry/testing_system_out_println_outputs

- Paul

Re: Sparse checkouts suggestion

Posted by Branko Čibej <br...@apache.org>.
On 14.09.2017 15:04, Paul Hammant wrote:
> My aim was to minimally impact the prod source. The context object is
> a good idea.
>
> I've been using https://amoffat.github.io/sh/ and quite enjoying it
> for my other python/svn project.

Yes, that one's pretty nice, but as a rule we try not to burden our
users with dependencies outside the standard library. The script uses
os.system, what, twice? Hardly worth replacing.

On the other hand, we have wrappers around subprocess.Popen in
subversion/tests/cmdline/svntest/main.py; you should be able to recycle
those.

-- Brane

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
My aim was to minimally impact the prod source. The context object is a
good idea.

I've been using https://amoffat.github.io/sh/ and quite enjoying it for my
other python/svn project.

Re: Sparse checkouts suggestion

Posted by Branko Čibej <br...@apache.org>.
On 14.09.2017 14:50, Paul Hammant wrote:
> I ran pep8 over the sources and thought I'd remediated all the
> training spaces already. I'll finish the job.
>
> My alternative to the four globals is to pass the function pointers as
> parameters through one or two intermediates to the functions that
> would use them.  Would that be acceptable?  I can't just delete the
> globals (as is) because the places that need then would break on
> invocation.  You saw why I was doing that, right?

Function parameters are better. Why not create a context object and pass
just one extra arg around?

Not sure though why you need os.system on that list ... these days it'd
be a lot better to use the subprocess module anyway.

-- Brane

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
I ran pep8 over the sources and thought I'd remediated all the training
spaces already. I'll finish the job.

My alternative to the four globals is to pass the function pointers as
parameters through one or two intermediates to the functions that would use
them.  Would that be acceptable?  I can't just delete the globals (as is)
because the places that need then would break on invocation.  You saw why I
was doing that, right?

- Paul

Re: Sparse checkouts suggestion

Posted by Branko Čibej <br...@apache.org>.
On 14.09.2017 12:26, Paul Hammant wrote:
> Thanks. 
>
> Here's what I expect to land - https://github.com/apache/subversion/pull/5

Why o why did you go to the trouble of adding the global variables in
svn-viewspec.py? They add exactly zero value and cause nothing but code
churn. -1 to that change ... the rest there look cosmetic and OK.

BTW, you left trailing spaces on some of the lines (github diff shows
them); please fix that.

The tests should be in tools/client-side. subversion/tests/cmdline are
for testing the main command-line tools, and have their own test
infrastructure (which predates PyTest, FWIW). We do not install, nor
officially support, the bits in the tools/ directory.

Also please read the community guide, especially the part about log
messages, before committing.
http://subversion.apache.org/docs/community-guide/

> Ten tests - ten passing in less than 0.1s
>
> I don't know if there is a clean workflow from Git commits back into
> Subversion so, I'll make the same commit through the Svn HTTPS
> interface, if code review comments are approving.

There isn't currently, as far as I know.

-- Brane

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
Thanks.

Here's what I expect to land - https://github.com/apache/subversion/pull/5
Ten tests - ten passing in less than 0.1s

I don't know if there is a clean workflow from Git commits back into
Subversion so, I'll make the same commit through the Svn HTTPS interface,
if code review comments are approving.

- Paul

Re: Sparse checkouts suggestion

Posted by Branko Čibej <br...@apache.org>.
On 13.09.2017 19:47, Paul Hammant wrote:
> OK, so I am up at 93% coverage with a PyTest script. As an Apache
> Member I'm already CLA'd. I could start to write tests that break the
> script in ways that it should handle more gracefully, but I need
> assurance that I'm not wasting my time.
>
> I'm not expecting committership (especially since my last published
> last C was Xmas os 91), but am expecting to be told there is interest
> in merging my pull-request. *Cough* applying my patches, I mean :-P

The Subversion project has the universal commit bit. If you're already a
committer anywhere, you can commit here. Of course it's nice to ask
first. :)

-- Brane


Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
OK, so I am up at 93% coverage with a PyTest script. As an Apache Member
I'm already CLA'd. I could start to write tests that break the script in
ways that it should handle more gracefully, but I need assurance that I'm
not wasting my time.

I'm not expecting committership (especially since my last published last C
was Xmas os 91), but am expecting to be told there is interest in merging
my pull-request. *Cough* applying my patches, I mean :-P

- Paul

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
T'was line endings and redundant spaces, fixed with a strip(), but as I
say, Java's asserts are better to noobs.

Here's the test:
https://gist.github.com/paul-hammant/058161485a227299e5d7c34cc6a33264

I had to refactor the actual script too. A tiny backwards-compatible bit.

-ph

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
I have a test for the inlined example. Well it would pass, but PyTest's
assert function is about 10x inferior to JUnit's or Hamcrest's
assertEquals() and isn't telling me where I've futzed up a CR somewhere.
This even *with* optional args (that should be mandatory -s and -vv). Gotta
go to work now - more later.

Re: Sparse checkouts suggestion

Posted by Stefan Sperling <st...@apache.org>.
On Wed, Sep 13, 2017 at 06:53:14AM -0400, Paul Hammant wrote:
> Oh tools/client-side/svn-viewspec.py doesn't have any tests :-(
> 
> The good news is that it can be refactored to have test quite easily. I
> think I'll be able to donate some code.

I would also be happy to see further improvements to this script.

A script is always a useful prototype for built-in features.
See svnmerge.py (quirks of the built-in implementation nonwithstanding :)

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
Oh tools/client-side/svn-viewspec.py doesn't have any tests :-(

The good news is that it can be refactored to have test quite easily. I
think I'll be able to donate some code.

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
>
>
> I'm not sure if you agree or not, but I think the fine-grained globbing
> include/exclude capability as Perforce has it. is required.
>
>
> "Required" implies "working copy format change". That's always hard.
>
> "Optional" means "get some of the features sooner."
>

I'm a big advocate for "ship-it now, add features later".

I'll try again:

"I'm not sure if you agree or not, but I think the fine-grained globbing
include/exclude capability as Perforce has it, is *ultimately* required."

Re: Sparse checkouts suggestion

Posted by Branko Čibej <br...@apache.org>.
On 21.09.2017 12:08, Paul Hammant wrote:
>
>     [...]
>
>     if /foo does not exist at checkout time -- and if we'd want /* to mean
>     "include anything new that appears in the repository", not "include
>     everything else that's in the repository at the time of the checkout".
>
>
> I'm not sure if you agree or not, but I think the fine-grained
> globbing include/exclude capability as Perforce has it. is required.

"Required" implies "working copy format change". That's always hard.

"Optional" means "get some of the features sooner."

Take your pick. :)

-- Brane

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
>
> [...]
>
> if /foo does not exist at checkout time -- and if we'd want /* to mean
> "include anything new that appears in the repository", not "include
> everything else that's in the repository at the time of the checkout".
>

I'm not sure if you agree or not, but I think the fine-grained globbing
include/exclude capability as Perforce has it. is required.

- Paul

Re: Sparse checkouts suggestion

Posted by Branko Čibej <br...@apache.org>.
On 20.09.2017 08:48, Branko Čibej wrote:
> On 19.09.2017 18:05, Branko Čibej wrote:
>> On 19.09.2017 13:24, Daniel Shahaf wrote:
>>> Branko Čibej wrote on Tue, 19 Sep 2017 06:03 +0200:
>>>> On 19.09.2017 00:59, Paul Hammant wrote:
>>>>> I like the `svn co svn://vcs/trunk --view foo`. Well maybe if in a
>>>>> following `svn up` it would *remember the current view*, it would be good.
>>>> I don't think there's any need to remember the "current view": a sparse
>>>> working copy already maintains its topology, so the view spec is only
>>>> needed on checkout to construct the sparse WC (and then update to
>>>> explicitly change it, etc.)
>>> What if the view excludes A/D, and then the wc is updated from a
>>> revision where A/D doesn't exist, to a revision where it does?
>> I still say we shouldn't remember the view spec; the WC should remember
>> the topology. Your example might involve the WC remembering exclusions
>> for nonexistent paths. Yes, that would be a WC format change.
>
> On consideration ... it might not even require a format change, just a
> constraint change -- I don't know whether that requires a format bump
> for backwards compatibility.
>
> The basic idea is that this would have to work:
>
> $ svnadmin create repo
> $ svn co file://$(pwd)/repo wc
> Checked out revision 0.
> $ cd wc
> $ svn up --set-depth exclude X
> svn: E155010: The node '.../wc/X' was not found.
>
> ^^^ This currently fails, but if it didn't and we recorded the exclusion
> in the WC and later X appeared in the repository, the client would
> silently ignore it.
>
> As I said, I've not a clue what that means for WC backwards
> compatibility. But it should be enough to implement 'future exclusion'
> in views.
>
> Of course, whether 'future exclusion' even makes sense as a feature is a
> different question.

For a moment I thought that our sparse working copy does have one
missing feature; consider a view defined like this:

    exclude: /*
    include: /foo

It turns out that we can model this with our current feature set: a
checkout would essentially do this:

    svn checkout --depth empty $REPO $WC
    svn update --set-depth infinity $WC/foo

However we could't use that for:

    exclude: /foo
    include: /*

if /foo does not exist at checkout time -- and if we'd want /* to mean
"include anything new that appears in the repository", not "include
everything else that's in the repository at the time of the checkout".

-- Brane


Re: Sparse checkouts suggestion

Posted by Branko Čibej <br...@apache.org>.
On 19.09.2017 18:05, Branko Čibej wrote:
> On 19.09.2017 13:24, Daniel Shahaf wrote:
>> Branko Čibej wrote on Tue, 19 Sep 2017 06:03 +0200:
>>> On 19.09.2017 00:59, Paul Hammant wrote:
>>>> I like the `svn co svn://vcs/trunk --view foo`. Well maybe if in a
>>>> following `svn up` it would *remember the current view*, it would be good.
>>> I don't think there's any need to remember the "current view": a sparse
>>> working copy already maintains its topology, so the view spec is only
>>> needed on checkout to construct the sparse WC (and then update to
>>> explicitly change it, etc.)
>> What if the view excludes A/D, and then the wc is updated from a
>> revision where A/D doesn't exist, to a revision where it does?
> I still say we shouldn't remember the view spec; the WC should remember
> the topology. Your example might involve the WC remembering exclusions
> for nonexistent paths. Yes, that would be a WC format change.


On consideration ... it might not even require a format change, just a
constraint change -- I don't know whether that requires a format bump
for backwards compatibility.

The basic idea is that this would have to work:

$ svnadmin create repo
$ svn co file://$(pwd)/repo wc
Checked out revision 0.
$ cd wc
$ svn up --set-depth exclude X
svn: E155010: The node '.../wc/X' was not found.

^^^ This currently fails, but if it didn't and we recorded the exclusion
in the WC and later X appeared in the repository, the client would
silently ignore it.

As I said, I've not a clue what that means for WC backwards
compatibility. But it should be enough to implement 'future exclusion'
in views.

Of course, whether 'future exclusion' even makes sense as a feature is a
different question.

-- Brane

Re: Sparse checkouts suggestion

Posted by Branko Čibej <br...@apache.org>.
On 19.09.2017 13:24, Daniel Shahaf wrote:
> Branko Čibej wrote on Tue, 19 Sep 2017 06:03 +0200:
>> On 19.09.2017 00:59, Paul Hammant wrote:
>>> I like the `svn co svn://vcs/trunk --view foo`. Well maybe if in a
>>> following `svn up` it would *remember the current view*, it would be good.
>>
>> I don't think there's any need to remember the "current view": a sparse
>> working copy already maintains its topology, so the view spec is only
>> needed on checkout to construct the sparse WC (and then update to
>> explicitly change it, etc.)
> What if the view excludes A/D, and then the wc is updated from a
> revision where A/D doesn't exist, to a revision where it does?

I still say we shouldn't remember the view spec; the WC should remember
the topology. Your example might involve the WC remembering exclusions
for nonexistent paths. Yes, that would be a WC format change.

The reason for /not/ remembering the viewspec -- which would also
involve a format change, IMO -- is that, whilst as in your example the
viewspec could exclude an as yet nonexistent A/D, the user may want to
include it in her working copy. If the WC remembers nonexistent
excludes, that translates to a simple 'svn update A/D'. If the WC
remembers the view spec, it would require changing that viewspec (or,
more likely, creating a new one), otherwise the next update would
exclude the item again.

-- Brane

Re: Sparse checkouts suggestion

Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Branko Čibej wrote on Tue, 19 Sep 2017 06:03 +0200:
> On 19.09.2017 00:59, Paul Hammant wrote:
> > I like the `svn co svn://vcs/trunk --view foo`. Well maybe if in a
> > following `svn up` it would *remember the current view*, it would be good.
> 
> 
> I don't think there's any need to remember the "current view": a sparse
> working copy already maintains its topology, so the view spec is only
> needed on checkout to construct the sparse WC (and then update to
> explicitly change it, etc.)

What if the view excludes A/D, and then the wc is updated from a
revision where A/D doesn't exist, to a revision where it does?

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
> On 19.09.2017 00:59, Paul Hammant wrote:
> > I like the `svn co svn://vcs/trunk --view foo`. Well maybe if in a
> > following `svn up` it would *remember the current view*, it would be
> good.
>
>
> I don't think there's any need to remember the "current view": a sparse
> working copy already maintains its topology, so the view spec is only
> needed on checkout to construct the sparse WC (and then update to
> explicitly change it, etc.)


Yes, you're right.

Re: Sparse checkouts suggestion

Posted by Branko Čibej <br...@apache.org>.
On 19.09.2017 00:59, Paul Hammant wrote:
> I like the `svn co svn://vcs/trunk --view foo`. Well maybe if in a
> following `svn up` it would *remember the current view*, it would be good.


I don't think there's any need to remember the "current view": a sparse
working copy already maintains its topology, so the view spec is only
needed on checkout to construct the sparse WC (and then update to
explicitly change it, etc.)

-- Brane

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
I like the `svn co svn://vcs/trunk --view foo`. Well maybe if in a
following `svn up` it would *remember the current view*, it would be good.


> I'm not sure about listing the "available views" in a default
> property, but okay, it's a possibility. I think it would be nice to be
> able to read it from a file, a url or a property (chosen by the user).
> In theory I'd say it would be nice to read it from a pipe, so you can
> 'cat', 'svn cat' or 'svn propget' the viewspec as input.
>

Google's 9-million* source files at HEAD revision: When checking out the
trunk, and going on to subset down to your expected checkout, there's so
many applications and services in the trunk that their `gcheckout`
technology makes a custom include/exclude set for each developer need based
on a traversal of the Blaze BUILD files.  You're mentioning "list" as a sub
command, yet it's not needed at all.

I never saw the source for gcheckout, and it's long gone given Perforce was
replaced by Piper in 2012, and a FUSE took over developer working copy, but
it is still worth understanding to a deeper level.

> gcheckout adsense:tests,adwords:tests,others

The last line ofthe script would invoke

    p4 client -i < "$P4_CLIENTSPEC_TEMP"

And you would do `p4 sync` on the command line following that.

The number of permutations ==  factorial for (num of modules * packages or
namespaces * build targets)

Don't aim for a concise list of view-specs, or "available" views. Leave it
completely open for globbing include and excludes (as Perforce/Google) to
be generated by smart script techs, and only verify the spec for
correctness on ingestion, IMO.

- Paul

Re: Sparse checkouts suggestion

Posted by Johan Corveleyn <jc...@gmail.com>.
On Wed, Sep 13, 2017 at 12:45 PM, Paul Hammant <pa...@hammant.org> wrote:
>>
>>
>> http://svn.apache.org/repos/asf/subversion/trunk/tools/client-side/svn-viewspec.py
>>
>
> Excellent!
>
> I see you posted a StackOverflow answer on that too, Mark -
> https://stackoverflow.com/questions/7481860/create-folder-structure-based-on-file
>
> I'm one of those people that's blind to a lib or framework until I see
> examples of how to use (can't read tutorials, can't read ref docs). I found
> https://prabhugs.wordpress.com/2010/10/11/effective-and-easy-sparse-checkouts-svn-viewspec-py-script/
> too which gets me halfway there.
>
> The svn-viewspec script is implicitly include-centric, right?  Perforce's
> choice back in the 90's was to allow excludes and includes which is more
> powerful even if it is dangerous.

I'd like more powerful "viewspec" features in (core) Subversion too.
The svn-viewspec.py script (and it's viewspec format) is already quite
useful, but I miss a couple of things (I know, this is open source and
patches are welcome, but I'm not able to dig deeper atm, so just
listing some ideas here):

1. Interaction with 'update'. I should be able to 'svn update
--set-depth $viewspec' or something like that, making an existing WC
sparse like in the viewspec.

2. Create a viewspec file out of an existing sparse WC (serialization
of the sparseness, if you will). This also touches on another missing
feature: there is currently no command to view / examine the
sparseness of a working copy.

3. Excludes should be possible in the viewspec format, analogous to
the "--set-depth exclude" working copy depth. So we have parity
between the viewspec format and WC sparseness.

4. With svn-viewspec.py, the "base url" is part of the viewspec file.
It would be nice to be able to give a "base url" by other means,
independent of the viewspec file -- so I can apply the same viewspec
on a branch as on trunk (assuming a multi-project trunk which is
sparsely checked out).

5. Commands handling a viewspec file: should they read/write the
viewspec from a property, a file, a pipe, ...?


I think Brane already suggested a good approach for those last two
points elsethread:

On Thu, Sep 14, 2017 at 3:43 AM, Branko Čibej <br...@apache.org> wrote:
> How about something along the lines of this:
>
> $ svn co svn://vcs/trunk --view foo

+1, this shows the "base url" being independent of the viewspec.

> with:
>
> $ svn propget svn:views svn://vsc/trunk
> [foo]
> bar/**
> baz/qux
>
> [bar]
> baz/**
>
>
> In other words: available views are defined in a svn:views property on
> the directory, and the user selects which (if any) view to use during
> checkout, update or switch. Obviously once the working copy structure is
> set up it'd remain sticky unless explicitly changed.
>
> For user-specific layouts, I can imagine having a --view-file or
> --view-prop option to tell the client where to read the view definitions
> from; but the svn:views property would be the default.

I'm not sure about listing the "available views" in a default
property, but okay, it's a possibility. I think it would be nice to be
able to read it from a file, a url or a property (chosen by the user).
In theory I'd say it would be nice to read it from a pipe, so you can
'cat', 'svn cat' or 'svn propget' the viewspec as input.

-- 
Johan

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
>
>
> http://svn.apache.org/repos/asf/subversion/trunk/tools/
> client-side/svn-viewspec.py
>
>
Excellent!

I see you posted a StackOverflow answer on that too, Mark -
https://stackoverflow.com/questions/7481860/create-folder-structure-based-on-file

I'm one of those people that's blind to a lib or framework until I see
examples of how to use (can't read tutorials, can't read ref docs). I found
https://prabhugs.wordpress.com/2010/10/11/effective-and-easy-sparse-checkouts-svn-viewspec-py-script/
too which gets me halfway there.

The svn-viewspec script is implicitly include-centric, right?  Perforce's
choice back in the 90's was to allow excludes and includes which is more
powerful even if it is dangerous.

Also: no top posting on the subversion lists, Mark. At least not if you
intend to stick around :)

- Paul

Re: Sparse checkouts suggestion

Posted by Mark Phippard <ma...@gmail.com>.
Have you seen:

http://svn.apache.org/repos/asf/subversion/trunk/tools/client-side/svn-viewspec.py



> On Sep 12, 2017, at 10:22 PM, Paul Hammant <pa...@hammant.org> wrote:
> 
> Compared to Perforce's client-spec, Subversion's sparse checkouts are quite cumbersome:
> 
> svn checkout http://svn.apache.org/repos/asf/subversion --depth=immediates
> cd subversion/trunk
> svn update --set-depth infinity
> cd ../tags
> svn update --set-depth immediates
> cd 1.7.7
> svn update --set-depth infinity
> 
> .. and similar.
> 
> Could Subversion follow Perforce and allow an alternate mechanism that leveraged include and exclude globbing paths?
> 
> Maybe in the root of working copy, the contents of a special file could be honored:
> 
> .sparse_mappings.txt
> 
> Sample contents:
> 
>    exclude **/*
>    include trunk/**/*
>    include tags/*
>    include tags/1.7.7/**/*
>    
> Where the end user to do svn-up from root (after that file changed), and assuming 'svn st' were 'clean', the working copy would reshape itself. Specifically directories would appear and disappear, and that NOT necessarily be subversion adds or deletes - it'd feel the same as permission changes within the directory tree.  Of course different teammates with the same checkout may see entirely different things (depending on the lines within their .sparse_mappings.txt
> 
> When I say special file, I mean Subversion isn't going to mark it as candidate for svn-add when it sees it.  Meaning, it is set on the client side by tooling and left there.  Tooling may include expand/contract scripts similar to what Google have for their monorepo.
> 
> Elsewhere i have documented Google's fu and the need for it for scaled Trunk-Based Development in a few places: 
> 
> 1. https://paulhammant.com/2014/01/06/googlers-subset-their-trunk/ (based on 2009 memories). Yes I know they are in Piper now and out of Perforce.
> 
> 2. Previously I forked a medium-sized Monorepo on Github, and did the the complete expand/contract work for it - https://github.com/paul-hammant-fork/jooby-monorepo-experiment - in Python.
> 
> 3. I have generally written about monorepos here: https://trunkbaseddevelopment.com/monorepos/
> 
> 4. And the expanding contracting kind here: https://trunkbaseddevelopment.com/expanding-contracting-monorepos/
> 
> So, in the tradition of the Subversion project, I'd like to discuss this here, then go on to raise a JIRA ticket.
> 
> - Paul

Re: Sparse checkouts suggestion

Posted by Branko Čibej <br...@apache.org>.
On 13.09.2017 12:40, Paul Hammant wrote:
>
>     >     This is an old idea. If you want to implement it, a file is
>     not the
>     >     right place for the view definition; a property on a directory
>     >     might be.
>     >
>     >
>     > A Svn property that sets for the user and workspace only, and
>     retains
>     > no history, right?  Meaning if the user had two checkouts of the
>     same
>     > Svn URL, then they could maintain two different sparse mappings,
>     right?
>
>     That would be sort of hard to do. :)
>
>
> So you're in favor of svn properties for this or against :-P
>  
>
>     Are
>     you suggesting that every user first checks out a working copy,
>     selects
>     the view spec, and updates to make it sparse? Because that, whilst of
>     course possible, is hardly optimal.
>
>
> No.
>
> I would expect, like Googlers do/did, 'checkout --depth none'
> then run a script to populate that hidden file
> then do 'svn up'
>  
>
>     I find it hard to believe that every user would want a different
>     view of
>     the tree. Certainly that's not the way we used the equivalent
>     ClearCase
>     feature in a previous $dayjob.
>
>
> Workflow for noob to the Adsense team (Perforce command changed to
> Subversion as you requested)
>
>     1. svn co svn://vcs/trunk --depth none
>     2. ./gcheckout.sh adsense 
>     3. svn up
>
>
> Workflow for noob to the Adwords team:
>
>     1. svn co svn://vcs/trunk --depth none
>     2. ./gcheckout.sh adwords
>     3. svn up 
>
>
> An eyeball comparison of the two workstation's working copy
> directories would discover that there were many directories/files in
> common, and many different (as you would expect for two different
> applications with different teams, release cadences etc).  Google
> shares code internally at source level.
>
> Workflow for a senior engineer wanting to make a change to the common
> code between adsense and adwords:
>
> 1. cd to existing working copy in 'no pending changes' status
> 2. ./gcheckout.sh adwords+adwords
> 3. svn up

How about something along the lines of this:

$ svn co svn://vcs/trunk --view foo

with:

$ svn propget svn:views svn://vsc/trunk
[foo]
bar/**
baz/qux

[bar]
baz/**


In other words: available views are defined in a svn:views property on
the directory, and the user selects which (if any) view to use during
checkout, update or switch. Obviously once the working copy structure is
set up it'd remain sticky unless explicitly changed.

For user-specific layouts, I can imagine having a --view-file or
--view-prop option to tell the client where to read the view definitions
from; but the svn:views property would be the default.

-- Brane

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
> >     This is an old idea. If you want to implement it, a file is not the
> >     right place for the view definition; a property on a directory
> >     might be.
> >
> >
> > A Svn property that sets for the user and workspace only, and retains
> > no history, right?  Meaning if the user had two checkouts of the same
> > Svn URL, then they could maintain two different sparse mappings, right?
>
> That would be sort of hard to do. :)
>

So you're in favor of svn properties for this or against :-P


> Are
> you suggesting that every user first checks out a working copy, selects
> the view spec, and updates to make it sparse? Because that, whilst of
> course possible, is hardly optimal.
>

No.

I would expect, like Googlers do/did, 'checkout --depth none'
then run a script to populate that hidden file
then do 'svn up'


> I find it hard to believe that every user would want a different view of
> the tree. Certainly that's not the way we used the equivalent ClearCase
> feature in a previous $dayjob.
>

Workflow for noob to the Adsense team (Perforce command changed to
Subversion as you requested)

1. svn co svn://vcs/trunk --depth none
2. ./gcheckout.sh adsense
3. svn up


Workflow for noob to the Adwords team:

1. svn co svn://vcs/trunk --depth none
2. ./gcheckout.sh adwords
3. svn up


An eyeball comparison of the two workstation's working copy directories
would discover that there were many directories/files in common, and many
different (as you would expect for two different applications with
different teams, release cadences etc).  Google shares code internally at
source level.

Workflow for a senior engineer wanting to make a change to the common code
between adsense and adwords:

1. cd to existing working copy in 'no pending changes' status
2. ./gcheckout.sh adwords+adwords
3. svn up

- Paul

Re: Sparse checkouts suggestion

Posted by Branko Čibej <br...@apache.org>.
On 13.09.2017 09:25, Paul Hammant wrote:
>
>     This is an old idea. If you want to implement it, a file is not the
>     right place for the view definition; a property on a directory
>     might be.
>
>
> A Svn property that sets for the user and workspace only, and retains
> no history, right?  Meaning if the user had two checkouts of the same
> Svn URL, then they could maintain two different sparse mappings, right?

That would be sort of hard to do. :)


> That would make it like Perforce's implementation. The perforce
> commands for export a client spec, and import again:
>
>     p4 client -o > .p4_clientspec_mappings.txt
>
>     # modify something
>
>     p4 client -i < .p4_clientspec_mappings.txt
>
> Judicious use of this in Google is part of their economic miracle and
> one of the dynamics of their scaling Trunk-Based Development to 25,000
> developers in one trunk in one repo, with 9 million source files at
> HEAD revision :)

Hmm. What exactly are you proposing anyway? I'd like to see this
described in terms of Subversion workflows, not Perforce workflows. Are
you suggesting that every user first checks out a working copy, selects
the view spec, and updates to make it sparse? Because that, whilst of
course possible, is hardly optimal.

I find it hard to believe that every user would want a different view of
the tree. Certainly that's not the way we used the equivalent ClearCase
feature in a previous $dayjob.

-- Brane

Re: Sparse checkouts suggestion

Posted by Paul Hammant <pa...@hammant.org>.
>
> This is an old idea. If you want to implement it, a file is not the
> right place for the view definition; a property on a directory might be.
>

A Svn property that sets for the user and workspace only, and retains no
history, right?  Meaning if the user had two checkouts of the same Svn URL,
then they could maintain two different sparse mappings, right?

That would make it like Perforce's implementation. The perforce commands
for export a client spec, and import again:

p4 client -o > .p4_clientspec_mappings.txt

# modify something

p4 client -i < .p4_clientspec_mappings.txt

Judicious use of this in Google is part of their economic miracle and one
of the dynamics of their scaling Trunk-Based Development to 25,000
developers in one trunk in one repo, with 9 million source files at HEAD
revision :)

- Paul

Re: Sparse checkouts suggestion

Posted by Branko Čibej <br...@apache.org>.
On 13.09.2017 04:22, Paul Hammant wrote:
> Compared to Perforce's client-spec, Subversion's sparse checkouts are
> quite cumbersome:
>
>     svn checkout http://svn.apache.org/repos/asf/subversion
>     --depth=immediates
>     cd subversion/trunk
>     svn update --set-depth infinity
>     cd ../tags
>     svn update --set-depth immediates
>     cd 1.7.7
>     svn update --set-depth infinity
>
> .. and similar.
>
> Could Subversion follow Perforce and allow an _alternate_ mechanism
> that leveraged include and exclude globbing paths?
>
> Maybe in the root of working copy, the contents of a /special/ file
> could be honored:
>
> .sparse_mappings.txt
>
> Sample contents:
>
>    exclude **/*
>    include trunk/**/*
>    include tags/*
>    include tags/1.7.7/**/*
>    
> Where the end user to do svn-up from root (after that file changed),
> and assuming 'svn st' were 'clean', the working copy would reshape
> itself. Specifically directories would appear and disappear, and that
> NOT necessarily be subversion adds or deletes - it'd feel the same as
> permission changes within the directory tree.  Of course different
> teammates with the same checkout may see entirely different things
> (depending on the lines within their .sparse_mappings.txt


This is an old idea. If you want to implement it, a file is not the
right place for the view definition; a property on a directory might be.

-- Brane