You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whimsical.apache.org by Sam Ruby <ru...@intertwingly.net> on 2019/11/20 17:25:25 UTC
Re: [whimsy] branch master updated: Notes on container layout
On Wed, Nov 20, 2019 at 4:21 AM <se...@apache.org> wrote:
>
> +If any of the configuration files under .subversion etc contain absolute references to
> +files (such as CA certificates), these will need to be fixed somehow (e.g. create links on
> +the container)
> +
> Known not to work (ToDos)
> -------------------------
>
> +* Fix up CA cert handling
@sebb: Does https://github.com/apache/whimsy/pull/75 fix this?
- Sam Ruby
Re: [whimsy] branch master updated: Notes on container layout
Posted by Sam Ruby <ru...@intertwingly.net>.
On Sun, Nov 24, 2019 at 7:35 PM sebb <se...@gmail.com> wrote:
>
> There are lots of places where the code assumes that the root is /srv.
> AFAICT, that assumption only fails in the docker case, and is only needed
> when setting up the container.
You may want to run "rake svn:update" periodically.
But, be that as it may, perhaps it is time to revisit the assumption
that the root may be anything other than /srv.
Meanwhile, the following should help:
https://github.com/apache/whimsy/commit/083860e5197b6a16e0cab3770e76c6e095c60680
- Sam Ruby
Re: [whimsy] branch master updated: Notes on container layout
Posted by Matt Sicker <bo...@gmail.com>.
How about an environment variable override with a default location
matching the old behavior? Then the Docker image can be run with that
environment variable set to the Docker-specific area.
On Sun, 24 Nov 2019 at 18:35, sebb <se...@gmail.com> wrote:
>
> On Sun, 24 Nov 2019 at 20:13, Sam Ruby <ru...@intertwingly.net> wrote:
>
> > On Sun, Nov 24, 2019 at 2:43 PM sebb <se...@gmail.com> wrote:
> > >
> > > My local Whimsy server installation (not via Docker) no longer works
> > > properly, because Config finds the wrong root.
> > >
> > > Whimsy is installed under my home directory; the server runs as a
> > different
> > > user, so has a different home directory.
> > > So when Config walks up the tree, it does not recognise my home copy of
> > > .whimsy as such, and assumes it must be a root marker.
> >
> > I've pushed a possible quick fix.
> >
> >
> That has fixed the immediate problem, but I still have concerns.
>
> There are lots of places where the code assumes that the root is /srv.
> AFAICT, that assumption only fails in the docker case, and is only needed
> when setting up the container.
> However the change affects all installations.
>
>
> > > Maybe search for a different file, such as .whimsy_root, instead?
> > >
> > > Or require root to be defined in the file itself.
> >
> > My concern there is that this directory may be mounted under a different
> > path.
> >
> >
> That would not apply to the .whimsy_root solution.
>
>
> > > On further reflection, the override function is also potentially
> > > problematic where the server is run as a different user.
> >
> > - Sam Ruby
> >
--
Matt Sicker <bo...@gmail.com>
Re: [whimsy] branch master updated: Notes on container layout
Posted by sebb <se...@gmail.com>.
On Sun, 24 Nov 2019 at 20:13, Sam Ruby <ru...@intertwingly.net> wrote:
> On Sun, Nov 24, 2019 at 2:43 PM sebb <se...@gmail.com> wrote:
> >
> > My local Whimsy server installation (not via Docker) no longer works
> > properly, because Config finds the wrong root.
> >
> > Whimsy is installed under my home directory; the server runs as a
> different
> > user, so has a different home directory.
> > So when Config walks up the tree, it does not recognise my home copy of
> > .whimsy as such, and assumes it must be a root marker.
>
> I've pushed a possible quick fix.
>
>
That has fixed the immediate problem, but I still have concerns.
There are lots of places where the code assumes that the root is /srv.
AFAICT, that assumption only fails in the docker case, and is only needed
when setting up the container.
However the change affects all installations.
> > Maybe search for a different file, such as .whimsy_root, instead?
> >
> > Or require root to be defined in the file itself.
>
> My concern there is that this directory may be mounted under a different
> path.
>
>
That would not apply to the .whimsy_root solution.
> > On further reflection, the override function is also potentially
> > problematic where the server is run as a different user.
>
> - Sam Ruby
>
Re: [whimsy] branch master updated: Notes on container layout
Posted by Sam Ruby <ru...@intertwingly.net>.
On Sun, Nov 24, 2019 at 2:43 PM sebb <se...@gmail.com> wrote:
>
> My local Whimsy server installation (not via Docker) no longer works
> properly, because Config finds the wrong root.
>
> Whimsy is installed under my home directory; the server runs as a different
> user, so has a different home directory.
> So when Config walks up the tree, it does not recognise my home copy of
> .whimsy as such, and assumes it must be a root marker.
I've pushed a possible quick fix.
> Maybe search for a different file, such as .whimsy_root, instead?
>
> Or require root to be defined in the file itself.
My concern there is that this directory may be mounted under a different path.
> On further reflection, the override function is also potentially
> problematic where the server is run as a different user.
- Sam Ruby
Re: [whimsy] branch master updated: Notes on container layout
Posted by sebb <se...@gmail.com>.
On Sat, 23 Nov 2019 at 13:37, sebb <se...@gmail.com> wrote:
> On Fri, 22 Nov 2019 at 19:50, Sam Ruby <ru...@intertwingly.net> wrote:
>
>> On Fri, Nov 22, 2019 at 10:39 AM sebb <se...@gmail.com> wrote:
>> >
>> > On Fri, 22 Nov 2019 at 14:26, Sam Ruby <ru...@intertwingly.net> wrote:
>> >
>> > > On Fri, Nov 22, 2019 at 5:47 AM sebb <se...@gmail.com> wrote:
>> > > >
>> > > > On Thu, 21 Nov 2019 at 17:00, Sam Ruby <ru...@intertwingly.net>
>> wrote:
>> > > >
>> > > > > On Thu, Nov 21, 2019 at 10:11 AM sebb <se...@gmail.com> wrote:
>> > > > > >
>> > > > > > That still causes issues for bundler.
>> > > > > > Turns out this is caused by the line
>> > > > > > - $HOME:/root
>> > > > > > which causes bundler to use the wrong path.
>> > > > > > I think it's unnecessary now anyway.
>> > > > >
>> > > > > That was an oversight, I meant to delete it. But if we were to
>> > > > > symlink the home directory, I suspect that would have been useful
>> to
>> > > > > you as I gather you had symlinks in your /srv directory
>> (presumably to
>> > > > > locations within your home directory)?
>> > > > >
>> > > > >
>> > > > Yes, my /srv/svn directory has links to some existing checkouts
>> under my
>> > > > home directory.
>> > > > [That would require mounting $HOME:$HOME to preserve links, not
>> > > $HOME:/root]
>> > > >
>> > > > But I was actually testing a separate docker workspace with its own
>> SVN
>> > > > checkout, as suggested in DOCKER.md
>> > > >
>> > > > > If I remove the line, SVN does now find the CA
>> > > > > > I can interactively update SVN, however I am prompted for the
>> > > password.
>> > > > > > This means that 'rake svn:update' fails when logged in to the
>> > > container
>> > > > > > interactively.
>> > > > > > And of course it does not work from the host as part of 'rake
>> > > > > docker:update'
>> > > > >
>> > > > > Presuming that the subversion file system versions are
>> compatible, you
>> > > > > can do the svn:update and git:pull steps from the host.
>> > > > >
>> > > > >
>> > > > Not really.
>> > > >
>> > > > If you are using a separate workspace for the container, the
>> easiest way
>> > > to
>> > > > set it up is to run rake svn:update in the container
>> > > > Running the command on the host will default to updating the host
>> > > version.
>> > > >
>> > > > Once the checkouts have been set up, it would of course be possible
>> to
>> > > run
>> > > > "svn up" separately on each of them on the host.
>> > > >
>> > > >
>> > > > > > Somehow we need to get the password across to the container.
>> > > > >
>> > > > > Maybe not. Except for the few places where the whimsysvn user is
>> used
>> > > > > to enable access from non ASF-members, the password provided in
>> the
>> > > > > HTTP authentication is used.
>> > > > >
>> > > >
>> > > >
>> > > > > > Note: it's probably not necessary to mount the whole of HOME
>> > > > > > One could mount just .subversion,.git and .ssh:
>> > > > > > - $HOME/.subversion:$HOME/.subversion
>> > > > > > - $HOME/.gitconfig:$HOME/.gitconfig
>> > > > > > - $HOME/.ssh:$HOME/.ssh
>> > > > > >
>> > > > > > Maybe then replace the corresponding /root mounts with links.
>> > > > >
>> > > > > Done.
>> > > > >
>> > > >
>> > > > Thanks!
>> > > > That works for SVN and git when logged into the container.
>> > > >
>> > > > However there is now no rake command to populate the container
>> /srv/svn.
>> > > > Was that intentional?
>> > >
>> > > I'm not fully understanding the behavior you are seeing. I gathered
>> > > from your previous email that svn checkout would require some sort of
>> > > prompting to you, and the container was not set up to prompt you.
>> > > Specifically, you said "Somehow we need to get the password across to
>> > > the container." Is that problem solved?
>> > >
>> > >
>> > No, the problem is not solved.
>> >
>> > I have set up SVN to use the macOS keychain.
>> > So I don't have to enter a password when running SVN on the host (except
>> > sometimes when the keychain seems to forget that I have allowed SVN to
>> use
>> > it).
>> >
>> > However when running in the container, SVN needs the password to be
>> > provided.
>> > This is fine when running 'svn up' etc, but I get an error when running
>> > 'rake svn:update' in the container:
>> >
>> > root@xxx:/srv/whimsy# rake svn:update
>> >
>> > /srv/svn/Bills
>> > #!: failed!
>> > #!: Updating '.':
>> > #!: svn: E170013: Unable to connect to a repository at URL '
>> > https://svn.apache.org/repos/private/financials/Bills'
>> > #!: svn: E215004: No more credentials or we tried too many times.
>> > #!: Authentication failed
>> > #!: will retry in 10 seconds
>> >
>> > The same happens if using docker to pass the command to the container.
>>
>> This appears to be a regression introduced in
>>
>> https://github.com/apache/whimsy/commit/891674d13f7967321bdf507a7cafd11ba5222fef
>> ,
>> which overrides stdin. I've attempted to provide a fix:
>>
>> https://github.com/apache/whimsy/commit/5306a99a1a0ba8eefca9cba2c56ae548e0bef231
>>
>>
> Yes, that fixes the prompt whilst logged in.
>
> However, if I execute 'rake svn:update' via docker, I get a prompt for the
> 'root' password:
>
> /srv/svn/Bills
>
> Authentication realm: <https://svn.apache.org:443> ASF Members
>
> Password for 'root':
>
> Note: svn update uses my correct id whilst logged in; it's only when
> invoked by docker that it uses root.
>
> > > So I intended to go the other way - have you do your svn up on on the
>> > > host. My oversight was that I did not add a step (presumably to
>> > > update ~/.whimsy) to configure where svn and git directories should be
>> > > placed. I can rectify that, but first I want confirmation that I
>> > > understand what you are seeing.
>> > >
>> > > Does svn checkout and svn up work as you would expect in the
>> > > container?
>> >
>> > Individual svn commands do now work; I no longer get the missing CA
>> issue.
>> > But SVN prompts for the password.
>>
>> I gather that svn on ubuntu doesn't support svn passtype keychain. It
>> looks like there are other password stores (e.g. gpg-agent) that may
>> work on both macOS and Ubuntu:
>>
>>
>> https://subgit.com/documentation/auth-book.html#chapter-26-using-svn-credentials-cache
>>
>>
> We are not using SubGit are we?
>
> In any case, we don't want to constrain how users set up SVN on their host
> system.
>
> SVN would need to be configured differently in the container if a
> different auth mech is to be used.
>
> > Also 'rake svn:update' does not work.
>> >
>> > If so, we can go that way. Otherwise I can update the
>> > > code and instructions to have you configure the location of the host
>> > > directory.
>> > >
>> > >
>> > The instructions originally envisaged a separate workspace with separate
>> > SVN/Git checkouts.
>> > I think this relied on being able to map /srv to the workspace when
>> running
>> > in the container.
>> > Thus 'rake svn:update' would use the new workspace when run in the
>> > container.
>> >
>> > If 'rake svn:update' is only run on the host, then somehow it would
>> need to
>> > use two different locations for the git/svn checkouts depending where it
>> > was run from.
>> >
>> > Is that possible?
>> > If possible, would it be useful?
>> > Or is it just more complication?
>>
>> The following commit should do that. Note the need to do a `touch
>> .whimsy` in the home directory. You are, of course, welcome to put
>> additional configuration overrides into this file, but its mere
>> existence is all that is necessary to establish a root directory.
>>
>>
>> https://github.com/apache/whimsy/commit/e150d91a0fddcf16e9368616cffe3d3690142c8a
>>
>>
> Good idea to allow overrides from walking up the path.
>
> However I'm not sure that it's a good idea to use mere existence to change
> the root directory.
> This is opaque and thus likely to cause hard to debug issues.
>
>
Just come across one such issue.
My local Whimsy server installation (not via Docker) no longer works
properly, because Config finds the wrong root.
Whimsy is installed under my home directory; the server runs as a different
user, so has a different home directory.
So when Config walks up the tree, it does not recognise my home copy of
.whimsy as such, and assumes it must be a root marker.
Maybe search for a different file, such as .whimsy_root, instead?
Or require root to be defined in the file itself.
On further reflection, the override function is also potentially
problematic where the server is run as a different user.
> - Sam Ruby
>>
>> > - Sam Ruby
>> > >
>> > > > > - Sam Ruby
>> > > > >
>> > > > > > On Thu, 21 Nov 2019 at 13:49, Sam Ruby <ru...@intertwingly.net>
>> > > wrote:
>> > > > > >
>> > > > > > > On Thu, Nov 21, 2019 at 8:01 AM sebb <se...@gmail.com>
>> wrote:
>> > > > > > > >
>> > > > > > > > Might work better if the home directory was mounted
>> directly as
>> > > well?
>> > > > > > >
>> > > > > > > Give it a try
>> > > > > > >
>> > > > >
>> > >
>> https://github.com/apache/whimsy/commit/b4c7bc2f10a21078e44d0c8e528eab00da296ab2
>> > > > > > >
>> > > > > > > - Sam Ruby
>> > > > > > >
>> > > > >
>> > >
>>
>
Re: [whimsy] branch master updated: Notes on container layout
Posted by sebb <se...@gmail.com>.
On Fri, 22 Nov 2019 at 19:50, Sam Ruby <ru...@intertwingly.net> wrote:
> On Fri, Nov 22, 2019 at 10:39 AM sebb <se...@gmail.com> wrote:
> >
> > On Fri, 22 Nov 2019 at 14:26, Sam Ruby <ru...@intertwingly.net> wrote:
> >
> > > On Fri, Nov 22, 2019 at 5:47 AM sebb <se...@gmail.com> wrote:
> > > >
> > > > On Thu, 21 Nov 2019 at 17:00, Sam Ruby <ru...@intertwingly.net>
> wrote:
> > > >
> > > > > On Thu, Nov 21, 2019 at 10:11 AM sebb <se...@gmail.com> wrote:
> > > > > >
> > > > > > That still causes issues for bundler.
> > > > > > Turns out this is caused by the line
> > > > > > - $HOME:/root
> > > > > > which causes bundler to use the wrong path.
> > > > > > I think it's unnecessary now anyway.
> > > > >
> > > > > That was an oversight, I meant to delete it. But if we were to
> > > > > symlink the home directory, I suspect that would have been useful
> to
> > > > > you as I gather you had symlinks in your /srv directory
> (presumably to
> > > > > locations within your home directory)?
> > > > >
> > > > >
> > > > Yes, my /srv/svn directory has links to some existing checkouts
> under my
> > > > home directory.
> > > > [That would require mounting $HOME:$HOME to preserve links, not
> > > $HOME:/root]
> > > >
> > > > But I was actually testing a separate docker workspace with its own
> SVN
> > > > checkout, as suggested in DOCKER.md
> > > >
> > > > > If I remove the line, SVN does now find the CA
> > > > > > I can interactively update SVN, however I am prompted for the
> > > password.
> > > > > > This means that 'rake svn:update' fails when logged in to the
> > > container
> > > > > > interactively.
> > > > > > And of course it does not work from the host as part of 'rake
> > > > > docker:update'
> > > > >
> > > > > Presuming that the subversion file system versions are compatible,
> you
> > > > > can do the svn:update and git:pull steps from the host.
> > > > >
> > > > >
> > > > Not really.
> > > >
> > > > If you are using a separate workspace for the container, the easiest
> way
> > > to
> > > > set it up is to run rake svn:update in the container
> > > > Running the command on the host will default to updating the host
> > > version.
> > > >
> > > > Once the checkouts have been set up, it would of course be possible
> to
> > > run
> > > > "svn up" separately on each of them on the host.
> > > >
> > > >
> > > > > > Somehow we need to get the password across to the container.
> > > > >
> > > > > Maybe not. Except for the few places where the whimsysvn user is
> used
> > > > > to enable access from non ASF-members, the password provided in the
> > > > > HTTP authentication is used.
> > > > >
> > > >
> > > >
> > > > > > Note: it's probably not necessary to mount the whole of HOME
> > > > > > One could mount just .subversion,.git and .ssh:
> > > > > > - $HOME/.subversion:$HOME/.subversion
> > > > > > - $HOME/.gitconfig:$HOME/.gitconfig
> > > > > > - $HOME/.ssh:$HOME/.ssh
> > > > > >
> > > > > > Maybe then replace the corresponding /root mounts with links.
> > > > >
> > > > > Done.
> > > > >
> > > >
> > > > Thanks!
> > > > That works for SVN and git when logged into the container.
> > > >
> > > > However there is now no rake command to populate the container
> /srv/svn.
> > > > Was that intentional?
> > >
> > > I'm not fully understanding the behavior you are seeing. I gathered
> > > from your previous email that svn checkout would require some sort of
> > > prompting to you, and the container was not set up to prompt you.
> > > Specifically, you said "Somehow we need to get the password across to
> > > the container." Is that problem solved?
> > >
> > >
> > No, the problem is not solved.
> >
> > I have set up SVN to use the macOS keychain.
> > So I don't have to enter a password when running SVN on the host (except
> > sometimes when the keychain seems to forget that I have allowed SVN to
> use
> > it).
> >
> > However when running in the container, SVN needs the password to be
> > provided.
> > This is fine when running 'svn up' etc, but I get an error when running
> > 'rake svn:update' in the container:
> >
> > root@xxx:/srv/whimsy# rake svn:update
> >
> > /srv/svn/Bills
> > #!: failed!
> > #!: Updating '.':
> > #!: svn: E170013: Unable to connect to a repository at URL '
> > https://svn.apache.org/repos/private/financials/Bills'
> > #!: svn: E215004: No more credentials or we tried too many times.
> > #!: Authentication failed
> > #!: will retry in 10 seconds
> >
> > The same happens if using docker to pass the command to the container.
>
> This appears to be a regression introduced in
>
> https://github.com/apache/whimsy/commit/891674d13f7967321bdf507a7cafd11ba5222fef
> ,
> which overrides stdin. I've attempted to provide a fix:
>
> https://github.com/apache/whimsy/commit/5306a99a1a0ba8eefca9cba2c56ae548e0bef231
>
>
Yes, that fixes the prompt whilst logged in.
However, if I execute 'rake svn:update' via docker, I get a prompt for the
'root' password:
/srv/svn/Bills
Authentication realm: <https://svn.apache.org:443> ASF Members
Password for 'root':
Note: svn update uses my correct id whilst logged in; it's only when
invoked by docker that it uses root.
> > So I intended to go the other way - have you do your svn up on on the
> > > host. My oversight was that I did not add a step (presumably to
> > > update ~/.whimsy) to configure where svn and git directories should be
> > > placed. I can rectify that, but first I want confirmation that I
> > > understand what you are seeing.
> > >
> > > Does svn checkout and svn up work as you would expect in the
> > > container?
> >
> > Individual svn commands do now work; I no longer get the missing CA
> issue.
> > But SVN prompts for the password.
>
> I gather that svn on ubuntu doesn't support svn passtype keychain. It
> looks like there are other password stores (e.g. gpg-agent) that may
> work on both macOS and Ubuntu:
>
>
> https://subgit.com/documentation/auth-book.html#chapter-26-using-svn-credentials-cache
>
>
We are not using SubGit are we?
In any case, we don't want to constrain how users set up SVN on their host
system.
SVN would need to be configured differently in the container if a different
auth mech is to be used.
> Also 'rake svn:update' does not work.
> >
> > If so, we can go that way. Otherwise I can update the
> > > code and instructions to have you configure the location of the host
> > > directory.
> > >
> > >
> > The instructions originally envisaged a separate workspace with separate
> > SVN/Git checkouts.
> > I think this relied on being able to map /srv to the workspace when
> running
> > in the container.
> > Thus 'rake svn:update' would use the new workspace when run in the
> > container.
> >
> > If 'rake svn:update' is only run on the host, then somehow it would need
> to
> > use two different locations for the git/svn checkouts depending where it
> > was run from.
> >
> > Is that possible?
> > If possible, would it be useful?
> > Or is it just more complication?
>
> The following commit should do that. Note the need to do a `touch
> .whimsy` in the home directory. You are, of course, welcome to put
> additional configuration overrides into this file, but its mere
> existence is all that is necessary to establish a root directory.
>
>
> https://github.com/apache/whimsy/commit/e150d91a0fddcf16e9368616cffe3d3690142c8a
>
>
Good idea to allow overrides from walking up the path.
However I'm not sure that it's a good idea to use mere existence to change
the root directory.
This is opaque and thus likely to cause hard to debug issues.
- Sam Ruby
>
> > - Sam Ruby
> > >
> > > > > - Sam Ruby
> > > > >
> > > > > > On Thu, 21 Nov 2019 at 13:49, Sam Ruby <ru...@intertwingly.net>
> > > wrote:
> > > > > >
> > > > > > > On Thu, Nov 21, 2019 at 8:01 AM sebb <se...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Might work better if the home directory was mounted directly
> as
> > > well?
> > > > > > >
> > > > > > > Give it a try
> > > > > > >
> > > > >
> > >
> https://github.com/apache/whimsy/commit/b4c7bc2f10a21078e44d0c8e528eab00da296ab2
> > > > > > >
> > > > > > > - Sam Ruby
> > > > > > >
> > > > >
> > >
>
Re: [whimsy] branch master updated: Notes on container layout
Posted by Sam Ruby <ru...@intertwingly.net>.
On Fri, Nov 22, 2019 at 10:39 AM sebb <se...@gmail.com> wrote:
>
> On Fri, 22 Nov 2019 at 14:26, Sam Ruby <ru...@intertwingly.net> wrote:
>
> > On Fri, Nov 22, 2019 at 5:47 AM sebb <se...@gmail.com> wrote:
> > >
> > > On Thu, 21 Nov 2019 at 17:00, Sam Ruby <ru...@intertwingly.net> wrote:
> > >
> > > > On Thu, Nov 21, 2019 at 10:11 AM sebb <se...@gmail.com> wrote:
> > > > >
> > > > > That still causes issues for bundler.
> > > > > Turns out this is caused by the line
> > > > > - $HOME:/root
> > > > > which causes bundler to use the wrong path.
> > > > > I think it's unnecessary now anyway.
> > > >
> > > > That was an oversight, I meant to delete it. But if we were to
> > > > symlink the home directory, I suspect that would have been useful to
> > > > you as I gather you had symlinks in your /srv directory (presumably to
> > > > locations within your home directory)?
> > > >
> > > >
> > > Yes, my /srv/svn directory has links to some existing checkouts under my
> > > home directory.
> > > [That would require mounting $HOME:$HOME to preserve links, not
> > $HOME:/root]
> > >
> > > But I was actually testing a separate docker workspace with its own SVN
> > > checkout, as suggested in DOCKER.md
> > >
> > > > If I remove the line, SVN does now find the CA
> > > > > I can interactively update SVN, however I am prompted for the
> > password.
> > > > > This means that 'rake svn:update' fails when logged in to the
> > container
> > > > > interactively.
> > > > > And of course it does not work from the host as part of 'rake
> > > > docker:update'
> > > >
> > > > Presuming that the subversion file system versions are compatible, you
> > > > can do the svn:update and git:pull steps from the host.
> > > >
> > > >
> > > Not really.
> > >
> > > If you are using a separate workspace for the container, the easiest way
> > to
> > > set it up is to run rake svn:update in the container
> > > Running the command on the host will default to updating the host
> > version.
> > >
> > > Once the checkouts have been set up, it would of course be possible to
> > run
> > > "svn up" separately on each of them on the host.
> > >
> > >
> > > > > Somehow we need to get the password across to the container.
> > > >
> > > > Maybe not. Except for the few places where the whimsysvn user is used
> > > > to enable access from non ASF-members, the password provided in the
> > > > HTTP authentication is used.
> > > >
> > >
> > >
> > > > > Note: it's probably not necessary to mount the whole of HOME
> > > > > One could mount just .subversion,.git and .ssh:
> > > > > - $HOME/.subversion:$HOME/.subversion
> > > > > - $HOME/.gitconfig:$HOME/.gitconfig
> > > > > - $HOME/.ssh:$HOME/.ssh
> > > > >
> > > > > Maybe then replace the corresponding /root mounts with links.
> > > >
> > > > Done.
> > > >
> > >
> > > Thanks!
> > > That works for SVN and git when logged into the container.
> > >
> > > However there is now no rake command to populate the container /srv/svn.
> > > Was that intentional?
> >
> > I'm not fully understanding the behavior you are seeing. I gathered
> > from your previous email that svn checkout would require some sort of
> > prompting to you, and the container was not set up to prompt you.
> > Specifically, you said "Somehow we need to get the password across to
> > the container." Is that problem solved?
> >
> >
> No, the problem is not solved.
>
> I have set up SVN to use the macOS keychain.
> So I don't have to enter a password when running SVN on the host (except
> sometimes when the keychain seems to forget that I have allowed SVN to use
> it).
>
> However when running in the container, SVN needs the password to be
> provided.
> This is fine when running 'svn up' etc, but I get an error when running
> 'rake svn:update' in the container:
>
> root@xxx:/srv/whimsy# rake svn:update
>
> /srv/svn/Bills
> #!: failed!
> #!: Updating '.':
> #!: svn: E170013: Unable to connect to a repository at URL '
> https://svn.apache.org/repos/private/financials/Bills'
> #!: svn: E215004: No more credentials or we tried too many times.
> #!: Authentication failed
> #!: will retry in 10 seconds
>
> The same happens if using docker to pass the command to the container.
This appears to be a regression introduced in
https://github.com/apache/whimsy/commit/891674d13f7967321bdf507a7cafd11ba5222fef,
which overrides stdin. I've attempted to provide a fix:
https://github.com/apache/whimsy/commit/5306a99a1a0ba8eefca9cba2c56ae548e0bef231
> > So I intended to go the other way - have you do your svn up on on the
> > host. My oversight was that I did not add a step (presumably to
> > update ~/.whimsy) to configure where svn and git directories should be
> > placed. I can rectify that, but first I want confirmation that I
> > understand what you are seeing.
> >
> > Does svn checkout and svn up work as you would expect in the
> > container?
>
> Individual svn commands do now work; I no longer get the missing CA issue.
> But SVN prompts for the password.
I gather that svn on ubuntu doesn't support svn passtype keychain. It
looks like there are other password stores (e.g. gpg-agent) that may
work on both macOS and Ubuntu:
https://subgit.com/documentation/auth-book.html#chapter-26-using-svn-credentials-cache
> Also 'rake svn:update' does not work.
>
> If so, we can go that way. Otherwise I can update the
> > code and instructions to have you configure the location of the host
> > directory.
> >
> >
> The instructions originally envisaged a separate workspace with separate
> SVN/Git checkouts.
> I think this relied on being able to map /srv to the workspace when running
> in the container.
> Thus 'rake svn:update' would use the new workspace when run in the
> container.
>
> If 'rake svn:update' is only run on the host, then somehow it would need to
> use two different locations for the git/svn checkouts depending where it
> was run from.
>
> Is that possible?
> If possible, would it be useful?
> Or is it just more complication?
The following commit should do that. Note the need to do a `touch
.whimsy` in the home directory. You are, of course, welcome to put
additional configuration overrides into this file, but its mere
existence is all that is necessary to establish a root directory.
https://github.com/apache/whimsy/commit/e150d91a0fddcf16e9368616cffe3d3690142c8a
- Sam Ruby
> - Sam Ruby
> >
> > > > - Sam Ruby
> > > >
> > > > > On Thu, 21 Nov 2019 at 13:49, Sam Ruby <ru...@intertwingly.net>
> > wrote:
> > > > >
> > > > > > On Thu, Nov 21, 2019 at 8:01 AM sebb <se...@gmail.com> wrote:
> > > > > > >
> > > > > > > Might work better if the home directory was mounted directly as
> > well?
> > > > > >
> > > > > > Give it a try
> > > > > >
> > > >
> > https://github.com/apache/whimsy/commit/b4c7bc2f10a21078e44d0c8e528eab00da296ab2
> > > > > >
> > > > > > - Sam Ruby
> > > > > >
> > > >
> >
Re: [whimsy] branch master updated: Notes on container layout
Posted by sebb <se...@gmail.com>.
On Fri, 22 Nov 2019 at 14:26, Sam Ruby <ru...@intertwingly.net> wrote:
> On Fri, Nov 22, 2019 at 5:47 AM sebb <se...@gmail.com> wrote:
> >
> > On Thu, 21 Nov 2019 at 17:00, Sam Ruby <ru...@intertwingly.net> wrote:
> >
> > > On Thu, Nov 21, 2019 at 10:11 AM sebb <se...@gmail.com> wrote:
> > > >
> > > > That still causes issues for bundler.
> > > > Turns out this is caused by the line
> > > > - $HOME:/root
> > > > which causes bundler to use the wrong path.
> > > > I think it's unnecessary now anyway.
> > >
> > > That was an oversight, I meant to delete it. But if we were to
> > > symlink the home directory, I suspect that would have been useful to
> > > you as I gather you had symlinks in your /srv directory (presumably to
> > > locations within your home directory)?
> > >
> > >
> > Yes, my /srv/svn directory has links to some existing checkouts under my
> > home directory.
> > [That would require mounting $HOME:$HOME to preserve links, not
> $HOME:/root]
> >
> > But I was actually testing a separate docker workspace with its own SVN
> > checkout, as suggested in DOCKER.md
> >
> > > If I remove the line, SVN does now find the CA
> > > > I can interactively update SVN, however I am prompted for the
> password.
> > > > This means that 'rake svn:update' fails when logged in to the
> container
> > > > interactively.
> > > > And of course it does not work from the host as part of 'rake
> > > docker:update'
> > >
> > > Presuming that the subversion file system versions are compatible, you
> > > can do the svn:update and git:pull steps from the host.
> > >
> > >
> > Not really.
> >
> > If you are using a separate workspace for the container, the easiest way
> to
> > set it up is to run rake svn:update in the container
> > Running the command on the host will default to updating the host
> version.
> >
> > Once the checkouts have been set up, it would of course be possible to
> run
> > "svn up" separately on each of them on the host.
> >
> >
> > > > Somehow we need to get the password across to the container.
> > >
> > > Maybe not. Except for the few places where the whimsysvn user is used
> > > to enable access from non ASF-members, the password provided in the
> > > HTTP authentication is used.
> > >
> >
> >
> > > > Note: it's probably not necessary to mount the whole of HOME
> > > > One could mount just .subversion,.git and .ssh:
> > > > - $HOME/.subversion:$HOME/.subversion
> > > > - $HOME/.gitconfig:$HOME/.gitconfig
> > > > - $HOME/.ssh:$HOME/.ssh
> > > >
> > > > Maybe then replace the corresponding /root mounts with links.
> > >
> > > Done.
> > >
> >
> > Thanks!
> > That works for SVN and git when logged into the container.
> >
> > However there is now no rake command to populate the container /srv/svn.
> > Was that intentional?
>
> I'm not fully understanding the behavior you are seeing. I gathered
> from your previous email that svn checkout would require some sort of
> prompting to you, and the container was not set up to prompt you.
> Specifically, you said "Somehow we need to get the password across to
> the container." Is that problem solved?
>
>
No, the problem is not solved.
I have set up SVN to use the macOS keychain.
So I don't have to enter a password when running SVN on the host (except
sometimes when the keychain seems to forget that I have allowed SVN to use
it).
However when running in the container, SVN needs the password to be
provided.
This is fine when running 'svn up' etc, but I get an error when running
'rake svn:update' in the container:
root@xxx:/srv/whimsy# rake svn:update
/srv/svn/Bills
#!: failed!
#!: Updating '.':
#!: svn: E170013: Unable to connect to a repository at URL '
https://svn.apache.org/repos/private/financials/Bills'
#!: svn: E215004: No more credentials or we tried too many times.
#!: Authentication failed
#!: will retry in 10 seconds
The same happens if using docker to pass the command to the container.
> So I intended to go the other way - have you do your svn up on on the
> host. My oversight was that I did not add a step (presumably to
> update ~/.whimsy) to configure where svn and git directories should be
> placed. I can rectify that, but first I want confirmation that I
> understand what you are seeing.
>
> Does svn checkout and svn up work as you would expect in the
> container?
Individual svn commands do now work; I no longer get the missing CA issue.
But SVN prompts for the password.
Also 'rake svn:update' does not work.
If so, we can go that way. Otherwise I can update the
> code and instructions to have you configure the location of the host
> directory.
>
>
The instructions originally envisaged a separate workspace with separate
SVN/Git checkouts.
I think this relied on being able to map /srv to the workspace when running
in the container.
Thus 'rake svn:update' would use the new workspace when run in the
container.
If 'rake svn:update' is only run on the host, then somehow it would need to
use two different locations for the git/svn checkouts depending where it
was run from.
Is that possible?
If possible, would it be useful?
Or is it just more complication?
- Sam Ruby
>
> > > - Sam Ruby
> > >
> > > > On Thu, 21 Nov 2019 at 13:49, Sam Ruby <ru...@intertwingly.net>
> wrote:
> > > >
> > > > > On Thu, Nov 21, 2019 at 8:01 AM sebb <se...@gmail.com> wrote:
> > > > > >
> > > > > > Might work better if the home directory was mounted directly as
> well?
> > > > >
> > > > > Give it a try
> > > > >
> > >
> https://github.com/apache/whimsy/commit/b4c7bc2f10a21078e44d0c8e528eab00da296ab2
> > > > >
> > > > > - Sam Ruby
> > > > >
> > >
>
Re: [whimsy] branch master updated: Notes on container layout
Posted by Sam Ruby <ru...@intertwingly.net>.
On Fri, Nov 22, 2019 at 5:47 AM sebb <se...@gmail.com> wrote:
>
> On Thu, 21 Nov 2019 at 17:00, Sam Ruby <ru...@intertwingly.net> wrote:
>
> > On Thu, Nov 21, 2019 at 10:11 AM sebb <se...@gmail.com> wrote:
> > >
> > > That still causes issues for bundler.
> > > Turns out this is caused by the line
> > > - $HOME:/root
> > > which causes bundler to use the wrong path.
> > > I think it's unnecessary now anyway.
> >
> > That was an oversight, I meant to delete it. But if we were to
> > symlink the home directory, I suspect that would have been useful to
> > you as I gather you had symlinks in your /srv directory (presumably to
> > locations within your home directory)?
> >
> >
> Yes, my /srv/svn directory has links to some existing checkouts under my
> home directory.
> [That would require mounting $HOME:$HOME to preserve links, not $HOME:/root]
>
> But I was actually testing a separate docker workspace with its own SVN
> checkout, as suggested in DOCKER.md
>
> > If I remove the line, SVN does now find the CA
> > > I can interactively update SVN, however I am prompted for the password.
> > > This means that 'rake svn:update' fails when logged in to the container
> > > interactively.
> > > And of course it does not work from the host as part of 'rake
> > docker:update'
> >
> > Presuming that the subversion file system versions are compatible, you
> > can do the svn:update and git:pull steps from the host.
> >
> >
> Not really.
>
> If you are using a separate workspace for the container, the easiest way to
> set it up is to run rake svn:update in the container
> Running the command on the host will default to updating the host version.
>
> Once the checkouts have been set up, it would of course be possible to run
> "svn up" separately on each of them on the host.
>
>
> > > Somehow we need to get the password across to the container.
> >
> > Maybe not. Except for the few places where the whimsysvn user is used
> > to enable access from non ASF-members, the password provided in the
> > HTTP authentication is used.
> >
>
>
> > > Note: it's probably not necessary to mount the whole of HOME
> > > One could mount just .subversion,.git and .ssh:
> > > - $HOME/.subversion:$HOME/.subversion
> > > - $HOME/.gitconfig:$HOME/.gitconfig
> > > - $HOME/.ssh:$HOME/.ssh
> > >
> > > Maybe then replace the corresponding /root mounts with links.
> >
> > Done.
> >
>
> Thanks!
> That works for SVN and git when logged into the container.
>
> However there is now no rake command to populate the container /srv/svn.
> Was that intentional?
I'm not fully understanding the behavior you are seeing. I gathered
from your previous email that svn checkout would require some sort of
prompting to you, and the container was not set up to prompt you.
Specifically, you said "Somehow we need to get the password across to
the container." Is that problem solved?
So I intended to go the other way - have you do your svn up on on the
host. My oversight was that I did not add a step (presumably to
update ~/.whimsy) to configure where svn and git directories should be
placed. I can rectify that, but first I want confirmation that I
understand what you are seeing.
Does svn checkout and svn up work as you would expect in the
container? If so, we can go that way. Otherwise I can update the
code and instructions to have you configure the location of the host
directory.
- Sam Ruby
> > - Sam Ruby
> >
> > > On Thu, 21 Nov 2019 at 13:49, Sam Ruby <ru...@intertwingly.net> wrote:
> > >
> > > > On Thu, Nov 21, 2019 at 8:01 AM sebb <se...@gmail.com> wrote:
> > > > >
> > > > > Might work better if the home directory was mounted directly as well?
> > > >
> > > > Give it a try
> > > >
> > https://github.com/apache/whimsy/commit/b4c7bc2f10a21078e44d0c8e528eab00da296ab2
> > > >
> > > > - Sam Ruby
> > > >
> >
Re: [whimsy] branch master updated: Notes on container layout
Posted by sebb <se...@gmail.com>.
On Thu, 21 Nov 2019 at 17:00, Sam Ruby <ru...@intertwingly.net> wrote:
> On Thu, Nov 21, 2019 at 10:11 AM sebb <se...@gmail.com> wrote:
> >
> > That still causes issues for bundler.
> > Turns out this is caused by the line
> > - $HOME:/root
> > which causes bundler to use the wrong path.
> > I think it's unnecessary now anyway.
>
> That was an oversight, I meant to delete it. But if we were to
> symlink the home directory, I suspect that would have been useful to
> you as I gather you had symlinks in your /srv directory (presumably to
> locations within your home directory)?
>
>
Yes, my /srv/svn directory has links to some existing checkouts under my
home directory.
[That would require mounting $HOME:$HOME to preserve links, not $HOME:/root]
But I was actually testing a separate docker workspace with its own SVN
checkout, as suggested in DOCKER.md
> If I remove the line, SVN does now find the CA
> > I can interactively update SVN, however I am prompted for the password.
> > This means that 'rake svn:update' fails when logged in to the container
> > interactively.
> > And of course it does not work from the host as part of 'rake
> docker:update'
>
> Presuming that the subversion file system versions are compatible, you
> can do the svn:update and git:pull steps from the host.
>
>
Not really.
If you are using a separate workspace for the container, the easiest way to
set it up is to run rake svn:update in the container
Running the command on the host will default to updating the host version.
Once the checkouts have been set up, it would of course be possible to run
"svn up" separately on each of them on the host.
> > Somehow we need to get the password across to the container.
>
> Maybe not. Except for the few places where the whimsysvn user is used
> to enable access from non ASF-members, the password provided in the
> HTTP authentication is used.
>
> > Note: it's probably not necessary to mount the whole of HOME
> > One could mount just .subversion,.git and .ssh:
> > - $HOME/.subversion:$HOME/.subversion
> > - $HOME/.gitconfig:$HOME/.gitconfig
> > - $HOME/.ssh:$HOME/.ssh
> >
> > Maybe then replace the corresponding /root mounts with links.
>
> Done.
>
Thanks!
That works for SVN and git when logged into the container.
However there is now no rake command to populate the container /srv/svn.
Was that intentional?
> - Sam Ruby
>
> > On Thu, 21 Nov 2019 at 13:49, Sam Ruby <ru...@intertwingly.net> wrote:
> >
> > > On Thu, Nov 21, 2019 at 8:01 AM sebb <se...@gmail.com> wrote:
> > > >
> > > > Might work better if the home directory was mounted directly as well?
> > >
> > > Give it a try
> > >
> https://github.com/apache/whimsy/commit/b4c7bc2f10a21078e44d0c8e528eab00da296ab2
> > >
> > > - Sam Ruby
> > >
>
Re: [whimsy] branch master updated: Notes on container layout
Posted by Sam Ruby <ru...@intertwingly.net>.
On Thu, Nov 21, 2019 at 10:11 AM sebb <se...@gmail.com> wrote:
>
> That still causes issues for bundler.
> Turns out this is caused by the line
> - $HOME:/root
> which causes bundler to use the wrong path.
> I think it's unnecessary now anyway.
That was an oversight, I meant to delete it. But if we were to
symlink the home directory, I suspect that would have been useful to
you as I gather you had symlinks in your /srv directory (presumably to
locations within your home directory)?
> If I remove the line, SVN does now find the CA
> I can interactively update SVN, however I am prompted for the password.
> This means that 'rake svn:update' fails when logged in to the container
> interactively.
> And of course it does not work from the host as part of 'rake docker:update'
Presuming that the subversion file system versions are compatible, you
can do the svn:update and git:pull steps from the host.
> Somehow we need to get the password across to the container.
Maybe not. Except for the few places where the whimsysvn user is used
to enable access from non ASF-members, the password provided in the
HTTP authentication is used.
> Note: it's probably not necessary to mount the whole of HOME
> One could mount just .subversion,.git and .ssh:
> - $HOME/.subversion:$HOME/.subversion
> - $HOME/.gitconfig:$HOME/.gitconfig
> - $HOME/.ssh:$HOME/.ssh
>
> Maybe then replace the corresponding /root mounts with links.
Done.
- Sam Ruby
> On Thu, 21 Nov 2019 at 13:49, Sam Ruby <ru...@intertwingly.net> wrote:
>
> > On Thu, Nov 21, 2019 at 8:01 AM sebb <se...@gmail.com> wrote:
> > >
> > > Might work better if the home directory was mounted directly as well?
> >
> > Give it a try
> > https://github.com/apache/whimsy/commit/b4c7bc2f10a21078e44d0c8e528eab00da296ab2
> >
> > - Sam Ruby
> >
Re: [whimsy] branch master updated: Notes on container layout
Posted by sebb <se...@gmail.com>.
That still causes issues for bundler.
Turns out this is caused by the line
- $HOME:/root
which causes bundler to use the wrong path.
I think it's unnecessary now anyway.
If I remove the line, SVN does now find the CA
I can interactively update SVN, however I am prompted for the password.
This means that 'rake svn:update' fails when logged in to the container
interactively.
And of course it does not work from the host as part of 'rake docker:update'
Somehow we need to get the password across to the container.
Note: it's probably not necessary to mount the whole of HOME
One could mount just .subversion,.git and .ssh:
- $HOME/.subversion:$HOME/.subversion
- $HOME/.gitconfig:$HOME/.gitconfig
- $HOME/.ssh:$HOME/.ssh
Maybe then replace the corresponding /root mounts with links.
On Thu, 21 Nov 2019 at 13:49, Sam Ruby <ru...@intertwingly.net> wrote:
> On Thu, Nov 21, 2019 at 8:01 AM sebb <se...@gmail.com> wrote:
> >
> > Might work better if the home directory was mounted directly as well?
>
> Give it a try
> https://github.com/apache/whimsy/commit/b4c7bc2f10a21078e44d0c8e528eab00da296ab2
>
> - Sam Ruby
>
Re: [whimsy] branch master updated: Notes on container layout
Posted by Sam Ruby <ru...@intertwingly.net>.
On Thu, Nov 21, 2019 at 8:01 AM sebb <se...@gmail.com> wrote:
>
> Might work better if the home directory was mounted directly as well?
Give it a try https://github.com/apache/whimsy/commit/b4c7bc2f10a21078e44d0c8e528eab00da296ab2
- Sam Ruby
Re: [whimsy] branch master updated: Notes on container layout
Posted by sebb <se...@gmail.com>.
On Wed, 20 Nov 2019 at 17:25, Sam Ruby <ru...@intertwingly.net> wrote:
> On Wed, Nov 20, 2019 at 4:21 AM <se...@apache.org> wrote:
> >
> > +If any of the configuration files under .subversion etc contain
> absolute references to
> > +files (such as CA certificates), these will need to be fixed somehow
> (e.g. create links on
> > +the container)
> > +
> > Known not to work (ToDos)
> > -------------------------
> >
> > +* Fix up CA cert handling
>
> @sebb: Does https://github.com/apache/whimsy/pull/75 fix this?
>
>
Unfortunately not.
I still get the same error:
svn: E170013: Unable to connect to a repository at URL '
https://svn.apache.org/repos/private/financials/Bills'
svn: E125009: Invalid config: unable to load certificate file
'/Users/sebb/.subversion/Thawte_Premium_Server_CA.pem'
However when I checked on the container, it is there:
# ls -l /Users/sebb/.subversion/Thawte_Premium_Server_CA.pem
-rw-r--r-- 1 root root 1194 Dec 8 2014
/Users/sebb/.subversion/Thawte_Premium_Server_CA.pem
Maybe svn does not handle soft links well?
I also get errors with bundler:
Retrying fetcher due to error (2/4): Bundler::PermissionError There was an
error while trying to write to
`/root/.bundle/cache/compact_index/rubygems.org.443.29b0360b937aa4d161703e6160654e47/info/google-protobuf`.
It is likely that you need to grant write permissions for that path.
Might work better if the home directory was mounted directly as well?
> - Sam Ruby
>