You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@yetus.apache.org by Pierre Smits <pi...@apache.org> on 2020/02/15 11:59:38 UTC
ERROR: is not a git repo.
Hi all,
As per suggestion from Sean via a private conversation on Slack I am
reaching ou to the community with an error I am experiencing as I am
getting my feet wet while working with Yetus.
Let me state some facts first:
- using a MacBook
- the Yetus code is from a cloned fork from Github, placed in
~/dev/asf/yetus
- the project to work with is OFBiz with code from a cloned fork from
Github (apache/ofbiz-framework), placed in ~/dev/asf/ofbiz/ofbiz-framework
(the --basedir parameter)
- the test environment is set to ~/dev/test/yetus ( the --patchdir
parameter)
- the patch to work with is a pull request in the OFBiz repo on Github:
https://github.com/apache/ofbiz-framework/pull/12
In a first trial run, in the basedir I executed
'~/dev/asf/yetus/precommit/src/main/shell/test-patch.sh
~/Downloads/OFBIZ-11236_shipmentNotesV3.patch' and it executed perfectly.
In a second trial run I am extending on that, but doing it outside of the
basedir, to run it against a pull request instead of a local patch file. So
I have added/changed:
- the patch dir is set to ~dev/test/yetus (--patch-dir parameter)
- the source to work with is a pull request in the OFBiz repo on Github:
https://github.com/apache/ofbiz-framework/pull/12
All the above is giving me following command to execute:
~/dev/asf/yetus/precommit/src/main/shell/test-patch.sh --plugins=all
--build-tool=gradle --patch-dir=~/dev/test/yetus
--basedir=~/dev/asf/ofbiz/ofbiz-framework --docker
https://github.com/apache/ofbiz-framework/pull/12
But this didn't execute so well, delivering following result:
WARNING: Found both spotbugs and findbugs. Disabling findbugs.
Modes: Docker
Processing: https://github.com/apache/ofbiz-framework/pull/12
GITHUB PR #12 is being downloaded from
https://api.github.com/repos/apache/ofbiz-framework/pulls/12
JSON data at Sat Feb 15 12:55:58 CET 2020
Patch data at Sat Feb 15 12:55:59 CET 2020
Diff data at Sat Feb 15 12:56:00 CET 2020
============================================================================
============================================================================
Confirming git environment
============================================================================
============================================================================
ERROR: is not a git repo.
============================================================================
============================================================================
Finished build.
============================================================================
============================================================================
I am suspecting that Yetus is complaining about the basedir
(~/dev/asf/ofbiz/ofbiz-framework), but that is where my local git clone is.
So, it seems I am doing something wrong. But what...
Could you help me with some pointers?
Met vriendelijke groet,
Pierre Smits
*Proud* *contributor* (but without privileges)* of* Apache OFBiz
<https://ofbiz.apache.org/>, since 2008
*Apache Trafodion <https://trafodion.apache.org>, Vice President*
*Apache Directory <https://directory.apache.org>, PMC Member*
Apache Incubator <https://incubator.apache.org>, committer
Apache Steve <https://steve.apache.org>, committer
Re: ERROR: is not a git repo.
Posted by Pierre Smits <pi...@apache.org>.
Thanks Allen.
I will raise the ticket and look at the code file suggested. As for the
patch-dir location: this just for now, while I get acquainted.
Met vrinedelijke groet,
Pierre Smits
*Proud* *contributor* (but without privileges)* of* Apache OFBiz
<https://ofbiz.apache.org/>, since 2008
*Apache Trafodion <https://trafodion.apache.org>, Vice President*
*Apache Directory <https://directory.apache.org>, PMC Member*
Apache Incubator <https://incubator.apache.org>, committer
Apache Steve <https://steve.apache.org>, committer
On Sat, Feb 15, 2020 at 8:53 PM Allen Wittenauer
<aw...@effectivemachines.com.invalid> wrote:
>
>
> > On Feb 15, 2020, at 10:32 AM, Pierre Smits <pi...@apache.org>
> wrote:
> >
> > Hi Allen, all,
> >
> > Thank you for the prompt reply.
> >
> > My apologies for the lengthy postings, and dragging this thread.
>
> Don't apologize! The questions are great and appreciated! They
> point out short comings, either in the form of bugs or docs or missing
> features or ... !
>
> > I am still
> > learning as I go along in my endeavour to reach something like what HBASE
> > has ( https://issues.apache.org/jira/browse/HBASE-14498).
>
> You can't trust those HBase people. ;)
>
> > Based on my conclusion in a posting here earlier, I adjusted the startup
> > command to
> >
> > ~/dev/asf/yetus/precommit/src/main/shell/test-patch.sh
> > --basedir=/Users/nl04748/dev/asf/ofbiz/ofbiz-framework
> --build-tool=gradle
> > --dirty-workspace --docker --patch-dir=/Users/nl04748/dev/test/yetus
> > --plugins=all --project=ofbiz
> > https://github.com/apache/ofbiz-framework/pull/12
>
>
> Looks reasonable, although you should probably set --patch-dir to
> a directory you don't care about, since precommit really really really
> wants that dir to be scratch space. Also, unless you have multiple build
> tools configured or doing something really funky in your repo, you probably
> don't need to set the build-tool. precommit has gotten pretty good about
> guessing which tool to use.
>
> > executable '' for 'gradle' does not exist.
>
> Hmm. That's... unexpected.
>
> > So in docker mode, misses the gradle package. It seems to me that, when
> > telling the script that the build tools to be used is gradle (parameter
> > --buildtool
> > =gradle) the generated image (yetus/ofbiz in this case) should have
> gradle
> > installed. Correct?
>
> Correct.
>
> > If so, the Dockerfile file in the test-patch-docker folder needs to be
> > adjusted.
> > I can imagine that a specific Gradle version may be required, so that may
> > be an additional parameter, like --gradle-version=5.5.0 (in the case of
> > OFBiz)
>
> Full disclosure: the gradle handling hasn't been well stressed in
> a very long time and even then was primarily built for my internal Kafka
> builds. I know gradle has changed quite a bit since it was written. There's
> a good chance you're going to hit some bugs.
>
> That said, something isn't right here, and it feels like a bug in
> the gradle handling. IIRC, gradlew should be getting used to download
> gradle and not be required to be in the docker image.
>
> *looks at code*
>
> This... doesn't feel correct at all. I'm sort of surprised if it
> worked outside of Docker! But really, gradlew should be getting used to
> install gradle and answer the gradle version question. It also tries to do
> a gradle -b bootstrap.gradle which is in no way guaranteed to work.
>
> We definitely need a JIRA open to fix this mess. :( I see git
> blame isn't going to ease my guilt. Haha.
>
> For the short term, you can either add gradle to the Dockerfile,
> start hacking pieces out of
> precommit/src/main/shell/test-patch.d/gradle.sh, or run with
> --build-tool=nobuild.
>
> For the longer term, we need some code fixes. They shouldn't be
> too painful, luckily.
Re: ERROR: is not a git repo.
Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.
> On Feb 15, 2020, at 10:32 AM, Pierre Smits <pi...@apache.org> wrote:
>
> Hi Allen, all,
>
> Thank you for the prompt reply.
>
> My apologies for the lengthy postings, and dragging this thread.
Don't apologize! The questions are great and appreciated! They point out short comings, either in the form of bugs or docs or missing features or ... !
> I am still
> learning as I go along in my endeavour to reach something like what HBASE
> has ( https://issues.apache.org/jira/browse/HBASE-14498).
You can't trust those HBase people. ;)
> Based on my conclusion in a posting here earlier, I adjusted the startup
> command to
>
> ~/dev/asf/yetus/precommit/src/main/shell/test-patch.sh
> --basedir=/Users/nl04748/dev/asf/ofbiz/ofbiz-framework --build-tool=gradle
> --dirty-workspace --docker --patch-dir=/Users/nl04748/dev/test/yetus
> --plugins=all --project=ofbiz
> https://github.com/apache/ofbiz-framework/pull/12
Looks reasonable, although you should probably set --patch-dir to a directory you don't care about, since precommit really really really wants that dir to be scratch space. Also, unless you have multiple build tools configured or doing something really funky in your repo, you probably don't need to set the build-tool. precommit has gotten pretty good about guessing which tool to use.
> executable '' for 'gradle' does not exist.
Hmm. That's... unexpected.
> So in docker mode, misses the gradle package. It seems to me that, when
> telling the script that the build tools to be used is gradle (parameter
> --buildtool
> =gradle) the generated image (yetus/ofbiz in this case) should have gradle
> installed. Correct?
Correct.
> If so, the Dockerfile file in the test-patch-docker folder needs to be
> adjusted.
> I can imagine that a specific Gradle version may be required, so that may
> be an additional parameter, like --gradle-version=5.5.0 (in the case of
> OFBiz)
Full disclosure: the gradle handling hasn't been well stressed in a very long time and even then was primarily built for my internal Kafka builds. I know gradle has changed quite a bit since it was written. There's a good chance you're going to hit some bugs.
That said, something isn't right here, and it feels like a bug in the gradle handling. IIRC, gradlew should be getting used to download gradle and not be required to be in the docker image.
*looks at code*
This... doesn't feel correct at all. I'm sort of surprised if it worked outside of Docker! But really, gradlew should be getting used to install gradle and answer the gradle version question. It also tries to do a gradle -b bootstrap.gradle which is in no way guaranteed to work.
We definitely need a JIRA open to fix this mess. :( I see git blame isn't going to ease my guilt. Haha.
For the short term, you can either add gradle to the Dockerfile, start hacking pieces out of precommit/src/main/shell/test-patch.d/gradle.sh, or run with --build-tool=nobuild.
For the longer term, we need some code fixes. They shouldn't be too painful, luckily.
Re: ERROR: is not a git repo.
Posted by Pierre Smits <pi...@apache.org>.
Hi Allen, all,
Thank you for the prompt reply.
My apologies for the lengthy postings, and dragging this thread. I am still
learning as I go along in my endeavour to reach something like what HBASE
has ( https://issues.apache.org/jira/browse/HBASE-14498).
Based on my conclusion in a posting here earlier, I adjusted the startup
command to
~/dev/asf/yetus/precommit/src/main/shell/test-patch.sh
--basedir=/Users/nl04748/dev/asf/ofbiz/ofbiz-framework --build-tool=gradle
--dirty-workspace --docker --patch-dir=/Users/nl04748/dev/test/yetus
--plugins=all --project=ofbiz
https://github.com/apache/ofbiz-framework/pull/12
which kicked in/off nicely and ran its course. To finally end with a nice
report:
Total elapsed build time: 40m 15s
WARNING: Setting /usr/lib/jvm/java-8-openjdk-amd64 as the JAVA_HOME.
WARNING: Found both spotbugs and findbugs. Disabling findbugs.
Modes: Docker
Processing: https://github.com/apache/ofbiz-framework/pull/12
GITHUB PR #12 is being downloaded from
https://api.github.com/repos/apache/ofbiz-framework/pulls/12
JSON data at Sat Feb 15 17:23:32 UTC 2020
Patch data at Sat Feb 15 17:23:32 UTC 2020
Diff data at Sat Feb 15 17:23:33 UTC 2020
============================================================================
============================================================================
Confirming git environment
============================================================================
============================================================================
dirty workspace mode; applying against existing branch
Testing patch on develop.
============================================================================
============================================================================
Re-exec mode detected. Continuing.
============================================================================
============================================================================
============================================================================
============================================================================
Determining needed tests
============================================================================
============================================================================
(Depending upon input size and number of plug-ins, this may take a while)
executable '' for 'gradle' does not exist.
============================================================================
============================================================================
Adding comment to Gitlab
============================================================================
============================================================================
Gitlab Plugin: no credentials provided to write a comment.
-1 overall
_____ _ _ _
| ___|_ _(_) |_ _ _ __ ___| |
| |_ / _` | | | | | | '__/ _ \ |
| _| (_| | | | |_| | | | __/_|
|_| \__,_|_|_|\__,_|_| \___(_)
| Vote | Subsystem | Runtime | Comment
============================================================================
| 0 | reexec | 40m 24s | Docker mode activated.
+---------------------------------------------------------------------------
| | | | Prechecks
+---------------------------------------------------------------------------
| -1 | gradle | 0m 35s | ERROR: gradle is not available.
|| Subsystem || Report/Notes ||
============================================================================
| Docker | Client=18.09.2 Server=18.09.2 base:
/Users/nl04748/dev/test/yetus/Dockerfile |
| GITHUB PR | https://github.com/apache/ofbiz-framework/pull/12 |
| Optional Tests | dupname asflicense xml |
| uname | Linux 99e99167b8f4 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | gradle |
| git revision | develop / 969f48b |
| versions | git=2.7.4 |
| Powered by | Apache Yetus 0.12.0-SNAPSHOT https://yetus.apache.org |
Cleaning up docker image used for testing.
============================================================================
============================================================================
Finished build.
============================================================================
============================================================================
So in docker mode, misses the gradle package. It seems to me that, when
telling the script that the build tools to be used is gradle (parameter
--buildtool
=gradle) the generated image (yetus/ofbiz in this case) should have gradle
installed. Correct?
If so, the Dockerfile file in the test-patch-docker folder needs to be
adjusted.
I can imagine that a specific Gradle version may be required, so that may
be an additional parameter, like --gradle-version=5.5.0 (in the case of
OFBiz)
Met vriendelijke groet,
Pierre Smits
*Proud* *contributor* (but without privileges)* of* Apache OFBiz
<https://ofbiz.apache.org/>, since 2008
*Apache Trafodion <https://trafodion.apache.org>, Vice President*
*Apache Directory <https://directory.apache.org>, PMC Member*
Apache Incubator <https://incubator.apache.org>, committer
Apache Steve <https://steve.apache.org>, committer
On Sat, Feb 15, 2020 at 7:00 PM Allen Wittenauer
<aw...@effectivemachines.com.invalid> wrote:
>
>
> > On Feb 15, 2020, at 6:41 AM, Pierre Smits <pi...@apache.org>
> wrote:
> >
> > This indicates to me that for --patch-dir and --basedir parameters their
> > values are treated as literally, instead of relatively (~ indicating the
> > user's home folder).
>
> Correct. Tilde prefix expansion is not currently supported.
>
> First the causes:
>
> - The interactive shell isn't expanding ~ due to the tilde
> not being in front of a word. e.g., ~/ will expand but --foo=~/ will not.
> - precommit isn't expanding it because the code base
> properly quotes it's variables to specifically avoid unexpected shell meta
> character expansions, deal with spaces in paths, etc, etc.
>
> Is this fixable? Maybe?
>
> For those unfamiliar, a quick lesson in ~:
>
> There are basically 7 different forms of ~ that bash
> supports. Everyone has probably seen two of them:
>
> ~/ == $HOME
> ~username/ == username's $HOME
>
> But there are 5 other forms that are fairly rare to see in
> the wild:
>
> ~+/ == ${PWD}
> ~-/ == ${OLDPWD}
>
> ~N/ == dirs +N
> ~+N/ == dirs +N
> ~-N/ == dirs -N
>
> There are a few different ways to make these work. One would be
> to completely rewrite how parameters are passed and processed. :) The
> second would be to eval the variables, but that has a HIGH degree of risk
> of getting it wrong (e.g., what does PWD mean at that point?). The third
> would be to create a special ~ evaluator and do the work itself.
>
> Substituting ~/ for $HOME is trivial.
> ~username/ is almost certainly going to require getent or eval. I
> can't think of any other way off the top of my head that would work for
> places where user's don't come from /etc/passwd.
> ~+/ should be precommit's $STARTINGDIR
> ~-/ would need like a $PRESTARTINGDIR==$OLDPWD or something
> dirs support would mean getting a copy of the pre-existing
> directory stack and referencing it, since precommit uses pushd and popd
> liberally.
>
> It'd probably be worthwhile to at least handle the ~/ form since I
> suspect that's going to be by and far the most common use case.
Re: ERROR: is not a git repo.
Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.
> On Feb 15, 2020, at 6:41 AM, Pierre Smits <pi...@apache.org> wrote:
>
> This indicates to me that for --patch-dir and --basedir parameters their
> values are treated as literally, instead of relatively (~ indicating the
> user's home folder).
Correct. Tilde prefix expansion is not currently supported.
First the causes:
- The interactive shell isn't expanding ~ due to the tilde not being in front of a word. e.g., ~/ will expand but --foo=~/ will not.
- precommit isn't expanding it because the code base properly quotes it's variables to specifically avoid unexpected shell meta character expansions, deal with spaces in paths, etc, etc.
Is this fixable? Maybe?
For those unfamiliar, a quick lesson in ~:
There are basically 7 different forms of ~ that bash supports. Everyone has probably seen two of them:
~/ == $HOME
~username/ == username's $HOME
But there are 5 other forms that are fairly rare to see in the wild:
~+/ == ${PWD}
~-/ == ${OLDPWD}
~N/ == dirs +N
~+N/ == dirs +N
~-N/ == dirs -N
There are a few different ways to make these work. One would be to completely rewrite how parameters are passed and processed. :) The second would be to eval the variables, but that has a HIGH degree of risk of getting it wrong (e.g., what does PWD mean at that point?). The third would be to create a special ~ evaluator and do the work itself.
Substituting ~/ for $HOME is trivial.
~username/ is almost certainly going to require getent or eval. I can't think of any other way off the top of my head that would work for places where user's don't come from /etc/passwd.
~+/ should be precommit's $STARTINGDIR
~-/ would need like a $PRESTARTINGDIR==$OLDPWD or something
dirs support would mean getting a copy of the pre-existing directory stack and referencing it, since precommit uses pushd and popd liberally.
It'd probably be worthwhile to at least handle the ~/ form since I suspect that's going to be by and far the most common use case.
Re: ERROR: is not a git repo.
Posted by Pierre Smits <pi...@apache.org>.
Additionally, what I did find out, when I started '
~/dev/asf/yetus/precommit/src/main/shell/test-patch.sh --plugins=all
--build-tool=gradle --patch-dir=~/dev/test/yetus
--basedir=~/dev/asf/ofbiz/ofbiz-framework
--docker https://github.com/apache/ofbiz-framework/pull/12' in my
~/dev/test folder, was:
1. in ~/dev/test a subfolder tree was created (~/dev/test/yetus)
containing following files:
1. githubub-pull.json;
2. input.diff
3. input.patch
This indicates to me that for --patch-dir and --basedir parameters their
values are treated as literally, instead of relatively (~ indicating the
user's home folder).
Met vriendelijke groet,
Pierre Smits
*Proud* *contributor* (but without privileges)* of* Apache OFBiz
<https://ofbiz.apache.org/>, since 2008
*Apache Trafodion <https://trafodion.apache.org>, Vice President*
*Apache Directory <https://directory.apache.org>, PMC Member*
Apache Incubator <https://incubator.apache.org>, committer
Apache Steve <https://steve.apache.org>, committer
On Sat, Feb 15, 2020 at 12:59 PM Pierre Smits <pi...@apache.org>
wrote:
> Hi all,
> As per suggestion from Sean via a private conversation on Slack I am
> reaching ou to the community with an error I am experiencing as I am
> getting my feet wet while working with Yetus.
>
> Let me state some facts first:
>
> - using a MacBook
> - the Yetus code is from a cloned fork from Github, placed in
> ~/dev/asf/yetus
> - the project to work with is OFBiz with code from a cloned fork from
> Github (apache/ofbiz-framework), placed in ~/dev/asf/ofbiz/ofbiz-framework
> (the --basedir parameter)
> - the test environment is set to ~/dev/test/yetus ( the --patchdir
> parameter)
> - the patch to work with is a pull request in the OFBiz repo on
> Github: https://github.com/apache/ofbiz-framework/pull/12
>
> In a first trial run, in the basedir I executed
> '~/dev/asf/yetus/precommit/src/main/shell/test-patch.sh
> ~/Downloads/OFBIZ-11236_shipmentNotesV3.patch' and it executed perfectly.
>
> In a second trial run I am extending on that, but doing it outside of the
> basedir, to run it against a pull request instead of a local patch file. So
> I have added/changed:
>
> - the patch dir is set to ~dev/test/yetus (--patch-dir parameter)
> - the source to work with is a pull request in the OFBiz repo on Github:
> https://github.com/apache/ofbiz-framework/pull/12
>
> All the above is giving me following command to execute:
>
> ~/dev/asf/yetus/precommit/src/main/shell/test-patch.sh --plugins=all
> --build-tool=gradle --patch-dir=~/dev/test/yetus
> --basedir=~/dev/asf/ofbiz/ofbiz-framework --docker
> https://github.com/apache/ofbiz-framework/pull/12
>
>
> But this didn't execute so well, delivering following result:
>
> WARNING: Found both spotbugs and findbugs. Disabling findbugs.
>
> Modes: Docker
>
> Processing: https://github.com/apache/ofbiz-framework/pull/12
>
> GITHUB PR #12 is being downloaded from
>
> https://api.github.com/repos/apache/ofbiz-framework/pulls/12
>
> JSON data at Sat Feb 15 12:55:58 CET 2020
>
> Patch data at Sat Feb 15 12:55:59 CET 2020
>
> Diff data at Sat Feb 15 12:56:00 CET 2020
>
>
>
>
> ============================================================================
>
>
> ============================================================================
>
> Confirming git environment
>
>
> ============================================================================
>
>
> ============================================================================
>
>
>
> ERROR: is not a git repo.
>
>
>
>
> ============================================================================
>
>
> ============================================================================
>
> Finished build.
>
>
> ============================================================================
>
>
> ============================================================================
>
>
> I am suspecting that Yetus is complaining about the basedir
> (~/dev/asf/ofbiz/ofbiz-framework), but that is where my local git clone is.
>
> So, it seems I am doing something wrong. But what...
>
> Could you help me with some pointers?
>
> Met vriendelijke groet,
>
> Pierre Smits
> *Proud* *contributor* (but without privileges)* of* Apache OFBiz
> <https://ofbiz.apache.org/>, since 2008
>
> *Apache Trafodion <https://trafodion.apache.org>, Vice President*
> *Apache Directory <https://directory.apache.org>, PMC Member*
> Apache Incubator <https://incubator.apache.org>, committer
> Apache Steve <https://steve.apache.org>, committer
>