You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@yetus.apache.org by suraj acharya <su...@gmail.com> on 2016/11/19 19:30:22 UTC
Usage of docker mode with yetus.
Hi,
I had a couple of questions about the useage of yetus in docker mode.
I have been working on AVRO-1887 and trying to add yetus pre-commit to AVRO.
I wrote a build plugin since AVRO supports multiple languages.
usage :
/bin/bash ~/yetus/precommit/test-patch.sh --plugins=all,-compile,-javadoc
--user-plugins=/root/avro/share/pre_commit/ --build-tool=build
https://github.com/apache/avro/pull/154 --robot --docker
--dockerfile=share/docker/Dockerfile
Questions :
1. What is the expected user to run the test-patch.sh script?
In my dev environment, I tried to run it as root, and it errored out since
the user root and group root existed. I had to make the following change to
allow the user root to run the commands. If this looks reasonable, I can
submit a patch for the same.
diff --git a/precommit/core.d/docker.sh b/precommit/core.d/docker.sh
index 2dae49d..e933883 100755
--- a/precommit/core.d/docker.sh
+++ b/precommit/core.d/docker.sh
@@ -552,9 +552,9 @@ FROM ${baseimagename}
LABEL org.apache.yetus=""
LABEL org.apache.yetus.testpatch.patch="tp-${DOCKER_ID}"
LABEL org.apache.yetus.testpatch.project=${PROJECT_NAME}
-RUN groupadd --non-unique -g ${GROUP_ID} ${USER_NAME}
-RUN useradd -g ${GROUP_ID} -u ${USER_ID} -m ${USER_NAME}
-RUN chown -R ${USER_NAME} /home/${USER_NAME}
+RUN groupadd --non-unique -g ${GROUP_ID} ${USER_NAME} || true
+RUN useradd -g ${GROUP_ID} -u ${USER_ID} -m ${USER_NAME} || true
+RUN chown -R ${USER_NAME} /home/${USER_NAME} || true
ENV HOME /home/${USER_NAME}
USER ${USER_NAME}
PatchSpecificDocker
2. My understanding of the --user-plugins is that, yetus will copy over the
whole directory and try to run the pre-commit for the build in the
user_plugin directory.
However, when I run it it throws an error saying :
cp: cannot stat ‘/root/avro/share/pre_commit/*’: No such file or directory.
And the whole run fails.
However, if I run it in a non docker environment it runs successfully.
I copied the build_plugin.sh file over to the yetus repository
(yetus/precommit/test-patch.d/build_plugin.sh) and the same command now ran
to finish successfully.
The question is what is the expected behaviour? And where should the files
be present? Am i missing some flags?
I have attached the output of the three runs if anyone has queries.
Thanks
-Suraj Acharya
Re: Usage of docker mode with yetus.
Posted by suraj acharya <su...@gmail.com>.
Thanks a lot.
Filed in YETUS-469 and YETUS-470.
-Suraj Acharya
On Sat, Nov 19, 2016 at 2:16 PM, Allen Wittenauer <aw...@effectivemachines.com>
wrote:
>
> > On Nov 19, 2016, at 11:30 AM, suraj acharya <su...@gmail.com> wrote:
> >
> > Questions :
> > 1. What is the expected user to run the test-patch.sh script?
>
> It's expected to be the same user that is executing the script
> outside of the container. This way when we import volumes the uid, etc
> match.
>
> > In my dev environment, I tried to run it as root,
>
> Oh, that's going to be problematic.
>
> > and it errored out since the user root and group root existed.
>
> Right.
>
> > I had to make the following change to allow the user root to run the
> commands. If this looks reasonable, I can submit a patch for the same.
>
> Yup, I think that's reasonable.
>
> >
> > 2. My understanding of the --user-plugins is that, yetus will copy over
> the whole directory and try to run the pre-commit for the build in the
> user_plugin directory.
> > However, when I run it it throws an error saying :
> > cp: cannot stat ‘/root/avro/share/pre_commit/*’: No such file or
> directory. And the whole run fails.
> > However, if I run it in a non docker environment it runs successfully.
>
> Your understanding is correct. Docker mode basically copies
> everything into a holding area. That holding area is then used to
> re-execute under Docker with everything in a known, fixed location. This
> prevents a lot of work later in having to guess what needs be mounted in
> volumes, changing code underneath us if we're patching ourselves, and
> various other craziness/edge cases.
>
> That said, I have a hunch that the quoting is wrong in copytpbits:
>
> cp -pr "${USER_PLUGIN_DIR}/*" \
> "${PATCH_DIR}/precommit/user-plugins"
>
>
> At a minimum, "${USER_PLUGIN_DIR}/*" should be
> "${USER_PLUGIN_DIR}"/*. But this line should probably be
>
> cp -pr "${USER_PLUGIN_DIR}"/. \
> "${PATCH_DIR}/precommit/user-plugins"
>
> ... to avoid any shell escape issues as well as copy dot-files.
> We could go extra crazy and use cp -pR, but that's probably wildly
> unnecessary.
>
>
>
Re: Usage of docker mode with yetus.
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Nov 19, 2016, at 11:30 AM, suraj acharya <su...@gmail.com> wrote:
>
> Questions :
> 1. What is the expected user to run the test-patch.sh script?
It's expected to be the same user that is executing the script outside of the container. This way when we import volumes the uid, etc match.
> In my dev environment, I tried to run it as root,
Oh, that's going to be problematic.
> and it errored out since the user root and group root existed.
Right.
> I had to make the following change to allow the user root to run the commands. If this looks reasonable, I can submit a patch for the same.
Yup, I think that's reasonable.
>
> 2. My understanding of the --user-plugins is that, yetus will copy over the whole directory and try to run the pre-commit for the build in the user_plugin directory.
> However, when I run it it throws an error saying :
> cp: cannot stat ‘/root/avro/share/pre_commit/*’: No such file or directory. And the whole run fails.
> However, if I run it in a non docker environment it runs successfully.
Your understanding is correct. Docker mode basically copies everything into a holding area. That holding area is then used to re-execute under Docker with everything in a known, fixed location. This prevents a lot of work later in having to guess what needs be mounted in volumes, changing code underneath us if we're patching ourselves, and various other craziness/edge cases.
That said, I have a hunch that the quoting is wrong in copytpbits:
cp -pr "${USER_PLUGIN_DIR}/*" \
"${PATCH_DIR}/precommit/user-plugins"
At a minimum, "${USER_PLUGIN_DIR}/*" should be "${USER_PLUGIN_DIR}"/*. But this line should probably be
cp -pr "${USER_PLUGIN_DIR}"/. \
"${PATCH_DIR}/precommit/user-plugins"
... to avoid any shell escape issues as well as copy dot-files. We could go extra crazy and use cp -pR, but that's probably wildly unnecessary.