You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Apekshit Sharma <ap...@cloudera.com> on 2016/01/25 00:44:12 UTC
Trying to understand checkstyle with Yetus
So looking at artifacts of a build (
https://builds.apache.org/job/PreCommit-HBASE-Build/261/artifact/patchprocess/),
I see checkstyle for only 'hbase-server', why so? Is it that we haven't
turned it on for other modules yet?
Also I see following five different files, what is each one for?
- branch-checkstyle-hbase-server.txt
- diff-checkstyle-hbase-server.txt
- maven-branch-checkstyle-hbase-server.txt
- maven-patch-checkstyle-hbase-server.txt
- patch-checkstyle-hbase-server.txt
- Appy
Re: Trying to understand checkstyle with Yetus
Posted by Apekshit Sharma <ap...@cloudera.com>.
Got it.
Thanks a lot Sean and Allen for explaining.
On Mon, Jan 25, 2016 at 7:58 PM, Allen Wittenauer <aw...@altiscale.com> wrote:
> I think you pretty much covered it Sean. haha.
>
> It’s probably worth mentioning that the ability of Yetus to
> shrink/grow the scope of how much work it does is a factor of a few things:
>
> a) the build system in use (esp post-YETUS-70; committed
> today to master)
> b) the personality
> c) the plug-in doing the work
>
> Maven, for example, has been written to primarily focus on
> individual pom.xml’s as dividing points. In contrast, our gradle support
> only covers building everything. In general, personalities know better and
> so can override much of that behavior, even on a per plug-in basis. For
> example, when Hadoop does a compile, it uses base common directories
> (CHANGED_UNION_MODULES in Yetus-jargon) rather than individual modules when
> it can so that maven dependencies are handled better. Some plug-ins such
> as shellcheck will tend to process everything in one go, regardless of the
> personality’s direction though since there is no compile phase and breakups
> by modules are fairly meaniningless.
>
> In the case of checkstyle in particular, probably worth mentioning
> it got a lot smarter in master and the content of those individual files
> and what we report to the user reflects that. We’ve been slowing cleaning
> up that code to more consistent with the rest of Yetus too, since a lot of
> that was written before a lot of the APIs existed.
>
> > On Jan 25, 2016, at 9:07 AM, Sean Busbey <bu...@cloudera.com> wrote:
> >
> > Hi Appy!
> >
> > I'm copying in dev@yetus, since those folks are better positioned to
> answer
> > these questions.
> >
> > The output for checkstyle is only present on that build because the patch
> > that it is testing only changed the hbase-server module. By default Yetus
> > only does correctness checks on modules that have been changed. Currently
> > the HBase personality maintains this behavior.
> >
> > It might help explain the five files to arrange them in terms of the
> phases
> > Yetus Precommit uses when evaluating a patch[1] (I've skipped phases
> > unrelated to checkstyle):
> >
> > - post-compile for branch (e.g. master, branch-1, branch-1.2, etc)
> >
> > after checking out and installing everything, changed modules have
> > checkstyle run against the _current_ state of the repo
> >
> > -- the checkstyle results go into branch-checkstyle-${module_name}.txt
> > -- the console output of running the checkstyle command (since hbase uses
> > maven) goes into maven-branch-checkstyle-${module_name}.txt
> >
> > - post-compile for patch (i.e. after applying the changes to be tested)
> >
> > -- the checkstyle results go into patch-checkstyle-${module_name}.txt
> > -- the console output of running the checkstyle command (since hbase uses
> > maven) goes into maven-branch-checkstyle-${module_name}.txt
> > -- the difference between the branch and patch versions of the checkstyle
> > output is calculated[2] and stored in diff-checkstyle-${module}.txt
> >
> >
> >
> > Sounds like we (Yetus) should build some overview docs on plugins. Thanks
> > for the question!
> >
> > [1]: http://yetus.apache.org/documentation/0.1.0/precommit-architecture/
> > [2]:
> >
> https://github.com/apache/yetus/blob/master/precommit/test-patch.d/checkstyle.sh#L65
> >
> > On Sun, Jan 24, 2016 at 5:44 PM, Apekshit Sharma <ap...@cloudera.com>
> wrote:
> >
> >> So looking at artifacts of a build (
> >>
> >>
> https://builds.apache.org/job/PreCommit-HBASE-Build/261/artifact/patchprocess/
> >> ),
> >> I see checkstyle for only 'hbase-server', why so? Is it that we haven't
> >> turned it on for other modules yet?
> >>
> >> Also I see following five different files, what is each one for?
> >>
> >> - branch-checkstyle-hbase-server.txt
> >> - diff-checkstyle-hbase-server.txt
> >> - maven-branch-checkstyle-hbase-server.txt
> >> - maven-patch-checkstyle-hbase-server.txt
> >> - patch-checkstyle-hbase-server.txt
> >>
> >>
> >> - Appy
> >>
> >
> >
> >
> > --
> > Sean
>
>
--
Regards
Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
650-963-6311
Re: Trying to understand checkstyle with Yetus
Posted by Apekshit Sharma <ap...@cloudera.com>.
Got it.
Thanks a lot Sean and Allen for explaining.
On Mon, Jan 25, 2016 at 7:58 PM, Allen Wittenauer <aw...@altiscale.com> wrote:
> I think you pretty much covered it Sean. haha.
>
> It’s probably worth mentioning that the ability of Yetus to
> shrink/grow the scope of how much work it does is a factor of a few things:
>
> a) the build system in use (esp post-YETUS-70; committed
> today to master)
> b) the personality
> c) the plug-in doing the work
>
> Maven, for example, has been written to primarily focus on
> individual pom.xml’s as dividing points. In contrast, our gradle support
> only covers building everything. In general, personalities know better and
> so can override much of that behavior, even on a per plug-in basis. For
> example, when Hadoop does a compile, it uses base common directories
> (CHANGED_UNION_MODULES in Yetus-jargon) rather than individual modules when
> it can so that maven dependencies are handled better. Some plug-ins such
> as shellcheck will tend to process everything in one go, regardless of the
> personality’s direction though since there is no compile phase and breakups
> by modules are fairly meaniningless.
>
> In the case of checkstyle in particular, probably worth mentioning
> it got a lot smarter in master and the content of those individual files
> and what we report to the user reflects that. We’ve been slowing cleaning
> up that code to more consistent with the rest of Yetus too, since a lot of
> that was written before a lot of the APIs existed.
>
> > On Jan 25, 2016, at 9:07 AM, Sean Busbey <bu...@cloudera.com> wrote:
> >
> > Hi Appy!
> >
> > I'm copying in dev@yetus, since those folks are better positioned to
> answer
> > these questions.
> >
> > The output for checkstyle is only present on that build because the patch
> > that it is testing only changed the hbase-server module. By default Yetus
> > only does correctness checks on modules that have been changed. Currently
> > the HBase personality maintains this behavior.
> >
> > It might help explain the five files to arrange them in terms of the
> phases
> > Yetus Precommit uses when evaluating a patch[1] (I've skipped phases
> > unrelated to checkstyle):
> >
> > - post-compile for branch (e.g. master, branch-1, branch-1.2, etc)
> >
> > after checking out and installing everything, changed modules have
> > checkstyle run against the _current_ state of the repo
> >
> > -- the checkstyle results go into branch-checkstyle-${module_name}.txt
> > -- the console output of running the checkstyle command (since hbase uses
> > maven) goes into maven-branch-checkstyle-${module_name}.txt
> >
> > - post-compile for patch (i.e. after applying the changes to be tested)
> >
> > -- the checkstyle results go into patch-checkstyle-${module_name}.txt
> > -- the console output of running the checkstyle command (since hbase uses
> > maven) goes into maven-branch-checkstyle-${module_name}.txt
> > -- the difference between the branch and patch versions of the checkstyle
> > output is calculated[2] and stored in diff-checkstyle-${module}.txt
> >
> >
> >
> > Sounds like we (Yetus) should build some overview docs on plugins. Thanks
> > for the question!
> >
> > [1]: http://yetus.apache.org/documentation/0.1.0/precommit-architecture/
> > [2]:
> >
> https://github.com/apache/yetus/blob/master/precommit/test-patch.d/checkstyle.sh#L65
> >
> > On Sun, Jan 24, 2016 at 5:44 PM, Apekshit Sharma <ap...@cloudera.com>
> wrote:
> >
> >> So looking at artifacts of a build (
> >>
> >>
> https://builds.apache.org/job/PreCommit-HBASE-Build/261/artifact/patchprocess/
> >> ),
> >> I see checkstyle for only 'hbase-server', why so? Is it that we haven't
> >> turned it on for other modules yet?
> >>
> >> Also I see following five different files, what is each one for?
> >>
> >> - branch-checkstyle-hbase-server.txt
> >> - diff-checkstyle-hbase-server.txt
> >> - maven-branch-checkstyle-hbase-server.txt
> >> - maven-patch-checkstyle-hbase-server.txt
> >> - patch-checkstyle-hbase-server.txt
> >>
> >>
> >> - Appy
> >>
> >
> >
> >
> > --
> > Sean
>
>
--
Regards
Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
650-963-6311
Re: Trying to understand checkstyle with Yetus
Posted by Allen Wittenauer <aw...@altiscale.com>.
I think you pretty much covered it Sean. haha.
It’s probably worth mentioning that the ability of Yetus to shrink/grow the scope of how much work it does is a factor of a few things:
a) the build system in use (esp post-YETUS-70; committed today to master)
b) the personality
c) the plug-in doing the work
Maven, for example, has been written to primarily focus on individual pom.xml’s as dividing points. In contrast, our gradle support only covers building everything. In general, personalities know better and so can override much of that behavior, even on a per plug-in basis. For example, when Hadoop does a compile, it uses base common directories (CHANGED_UNION_MODULES in Yetus-jargon) rather than individual modules when it can so that maven dependencies are handled better. Some plug-ins such as shellcheck will tend to process everything in one go, regardless of the personality’s direction though since there is no compile phase and breakups by modules are fairly meaniningless.
In the case of checkstyle in particular, probably worth mentioning it got a lot smarter in master and the content of those individual files and what we report to the user reflects that. We’ve been slowing cleaning up that code to more consistent with the rest of Yetus too, since a lot of that was written before a lot of the APIs existed.
> On Jan 25, 2016, at 9:07 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
> Hi Appy!
>
> I'm copying in dev@yetus, since those folks are better positioned to answer
> these questions.
>
> The output for checkstyle is only present on that build because the patch
> that it is testing only changed the hbase-server module. By default Yetus
> only does correctness checks on modules that have been changed. Currently
> the HBase personality maintains this behavior.
>
> It might help explain the five files to arrange them in terms of the phases
> Yetus Precommit uses when evaluating a patch[1] (I've skipped phases
> unrelated to checkstyle):
>
> - post-compile for branch (e.g. master, branch-1, branch-1.2, etc)
>
> after checking out and installing everything, changed modules have
> checkstyle run against the _current_ state of the repo
>
> -- the checkstyle results go into branch-checkstyle-${module_name}.txt
> -- the console output of running the checkstyle command (since hbase uses
> maven) goes into maven-branch-checkstyle-${module_name}.txt
>
> - post-compile for patch (i.e. after applying the changes to be tested)
>
> -- the checkstyle results go into patch-checkstyle-${module_name}.txt
> -- the console output of running the checkstyle command (since hbase uses
> maven) goes into maven-branch-checkstyle-${module_name}.txt
> -- the difference between the branch and patch versions of the checkstyle
> output is calculated[2] and stored in diff-checkstyle-${module}.txt
>
>
>
> Sounds like we (Yetus) should build some overview docs on plugins. Thanks
> for the question!
>
> [1]: http://yetus.apache.org/documentation/0.1.0/precommit-architecture/
> [2]:
> https://github.com/apache/yetus/blob/master/precommit/test-patch.d/checkstyle.sh#L65
>
> On Sun, Jan 24, 2016 at 5:44 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
>
>> So looking at artifacts of a build (
>>
>> https://builds.apache.org/job/PreCommit-HBASE-Build/261/artifact/patchprocess/
>> ),
>> I see checkstyle for only 'hbase-server', why so? Is it that we haven't
>> turned it on for other modules yet?
>>
>> Also I see following five different files, what is each one for?
>>
>> - branch-checkstyle-hbase-server.txt
>> - diff-checkstyle-hbase-server.txt
>> - maven-branch-checkstyle-hbase-server.txt
>> - maven-patch-checkstyle-hbase-server.txt
>> - patch-checkstyle-hbase-server.txt
>>
>>
>> - Appy
>>
>
>
>
> --
> Sean
Re: Trying to understand checkstyle with Yetus
Posted by Allen Wittenauer <aw...@altiscale.com>.
I think you pretty much covered it Sean. haha.
It’s probably worth mentioning that the ability of Yetus to shrink/grow the scope of how much work it does is a factor of a few things:
a) the build system in use (esp post-YETUS-70; committed today to master)
b) the personality
c) the plug-in doing the work
Maven, for example, has been written to primarily focus on individual pom.xml’s as dividing points. In contrast, our gradle support only covers building everything. In general, personalities know better and so can override much of that behavior, even on a per plug-in basis. For example, when Hadoop does a compile, it uses base common directories (CHANGED_UNION_MODULES in Yetus-jargon) rather than individual modules when it can so that maven dependencies are handled better. Some plug-ins such as shellcheck will tend to process everything in one go, regardless of the personality’s direction though since there is no compile phase and breakups by modules are fairly meaniningless.
In the case of checkstyle in particular, probably worth mentioning it got a lot smarter in master and the content of those individual files and what we report to the user reflects that. We’ve been slowing cleaning up that code to more consistent with the rest of Yetus too, since a lot of that was written before a lot of the APIs existed.
> On Jan 25, 2016, at 9:07 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
> Hi Appy!
>
> I'm copying in dev@yetus, since those folks are better positioned to answer
> these questions.
>
> The output for checkstyle is only present on that build because the patch
> that it is testing only changed the hbase-server module. By default Yetus
> only does correctness checks on modules that have been changed. Currently
> the HBase personality maintains this behavior.
>
> It might help explain the five files to arrange them in terms of the phases
> Yetus Precommit uses when evaluating a patch[1] (I've skipped phases
> unrelated to checkstyle):
>
> - post-compile for branch (e.g. master, branch-1, branch-1.2, etc)
>
> after checking out and installing everything, changed modules have
> checkstyle run against the _current_ state of the repo
>
> -- the checkstyle results go into branch-checkstyle-${module_name}.txt
> -- the console output of running the checkstyle command (since hbase uses
> maven) goes into maven-branch-checkstyle-${module_name}.txt
>
> - post-compile for patch (i.e. after applying the changes to be tested)
>
> -- the checkstyle results go into patch-checkstyle-${module_name}.txt
> -- the console output of running the checkstyle command (since hbase uses
> maven) goes into maven-branch-checkstyle-${module_name}.txt
> -- the difference between the branch and patch versions of the checkstyle
> output is calculated[2] and stored in diff-checkstyle-${module}.txt
>
>
>
> Sounds like we (Yetus) should build some overview docs on plugins. Thanks
> for the question!
>
> [1]: http://yetus.apache.org/documentation/0.1.0/precommit-architecture/
> [2]:
> https://github.com/apache/yetus/blob/master/precommit/test-patch.d/checkstyle.sh#L65
>
> On Sun, Jan 24, 2016 at 5:44 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
>
>> So looking at artifacts of a build (
>>
>> https://builds.apache.org/job/PreCommit-HBASE-Build/261/artifact/patchprocess/
>> ),
>> I see checkstyle for only 'hbase-server', why so? Is it that we haven't
>> turned it on for other modules yet?
>>
>> Also I see following five different files, what is each one for?
>>
>> - branch-checkstyle-hbase-server.txt
>> - diff-checkstyle-hbase-server.txt
>> - maven-branch-checkstyle-hbase-server.txt
>> - maven-patch-checkstyle-hbase-server.txt
>> - patch-checkstyle-hbase-server.txt
>>
>>
>> - Appy
>>
>
>
>
> --
> Sean
Re: Trying to understand checkstyle with Yetus
Posted by Sean Busbey <bu...@cloudera.com>.
Hi Appy!
I'm copying in dev@yetus, since those folks are better positioned to answer
these questions.
The output for checkstyle is only present on that build because the patch
that it is testing only changed the hbase-server module. By default Yetus
only does correctness checks on modules that have been changed. Currently
the HBase personality maintains this behavior.
It might help explain the five files to arrange them in terms of the phases
Yetus Precommit uses when evaluating a patch[1] (I've skipped phases
unrelated to checkstyle):
- post-compile for branch (e.g. master, branch-1, branch-1.2, etc)
after checking out and installing everything, changed modules have
checkstyle run against the _current_ state of the repo
-- the checkstyle results go into branch-checkstyle-${module_name}.txt
-- the console output of running the checkstyle command (since hbase uses
maven) goes into maven-branch-checkstyle-${module_name}.txt
- post-compile for patch (i.e. after applying the changes to be tested)
-- the checkstyle results go into patch-checkstyle-${module_name}.txt
-- the console output of running the checkstyle command (since hbase uses
maven) goes into maven-branch-checkstyle-${module_name}.txt
-- the difference between the branch and patch versions of the checkstyle
output is calculated[2] and stored in diff-checkstyle-${module}.txt
Sounds like we (Yetus) should build some overview docs on plugins. Thanks
for the question!
[1]: http://yetus.apache.org/documentation/0.1.0/precommit-architecture/
[2]:
https://github.com/apache/yetus/blob/master/precommit/test-patch.d/checkstyle.sh#L65
On Sun, Jan 24, 2016 at 5:44 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
> So looking at artifacts of a build (
>
> https://builds.apache.org/job/PreCommit-HBASE-Build/261/artifact/patchprocess/
> ),
> I see checkstyle for only 'hbase-server', why so? Is it that we haven't
> turned it on for other modules yet?
>
> Also I see following five different files, what is each one for?
>
> - branch-checkstyle-hbase-server.txt
> - diff-checkstyle-hbase-server.txt
> - maven-branch-checkstyle-hbase-server.txt
> - maven-patch-checkstyle-hbase-server.txt
> - patch-checkstyle-hbase-server.txt
>
>
> - Appy
>
--
Sean
Re: Trying to understand checkstyle with Yetus
Posted by Sean Busbey <bu...@cloudera.com>.
Hi Appy!
I'm copying in dev@yetus, since those folks are better positioned to answer
these questions.
The output for checkstyle is only present on that build because the patch
that it is testing only changed the hbase-server module. By default Yetus
only does correctness checks on modules that have been changed. Currently
the HBase personality maintains this behavior.
It might help explain the five files to arrange them in terms of the phases
Yetus Precommit uses when evaluating a patch[1] (I've skipped phases
unrelated to checkstyle):
- post-compile for branch (e.g. master, branch-1, branch-1.2, etc)
after checking out and installing everything, changed modules have
checkstyle run against the _current_ state of the repo
-- the checkstyle results go into branch-checkstyle-${module_name}.txt
-- the console output of running the checkstyle command (since hbase uses
maven) goes into maven-branch-checkstyle-${module_name}.txt
- post-compile for patch (i.e. after applying the changes to be tested)
-- the checkstyle results go into patch-checkstyle-${module_name}.txt
-- the console output of running the checkstyle command (since hbase uses
maven) goes into maven-branch-checkstyle-${module_name}.txt
-- the difference between the branch and patch versions of the checkstyle
output is calculated[2] and stored in diff-checkstyle-${module}.txt
Sounds like we (Yetus) should build some overview docs on plugins. Thanks
for the question!
[1]: http://yetus.apache.org/documentation/0.1.0/precommit-architecture/
[2]:
https://github.com/apache/yetus/blob/master/precommit/test-patch.d/checkstyle.sh#L65
On Sun, Jan 24, 2016 at 5:44 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
> So looking at artifacts of a build (
>
> https://builds.apache.org/job/PreCommit-HBASE-Build/261/artifact/patchprocess/
> ),
> I see checkstyle for only 'hbase-server', why so? Is it that we haven't
> turned it on for other modules yet?
>
> Also I see following five different files, what is each one for?
>
> - branch-checkstyle-hbase-server.txt
> - diff-checkstyle-hbase-server.txt
> - maven-branch-checkstyle-hbase-server.txt
> - maven-patch-checkstyle-hbase-server.txt
> - patch-checkstyle-hbase-server.txt
>
>
> - Appy
>
--
Sean