You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Alex Amato <aj...@google.com> on 2019/02/25 18:05:54 UTC
What quick command to catch common issues before pushing a python PR?
I made a thread about this a while back for java, but I don't think the
same commands like sptoless work for python.
auto fixing lint issues
running and quick checks which would fail the PR (without running the whole
precommit?)
Something like findbugs to detect common issues (i.e. py3 compliance)
FWIW, this is what I have been using for java. It will catch pretty much
everything except presubmit test failures.
./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
compileTestJava && ./gradlew compileJava
Re: What quick command to catch common issues before pushing a python PR?
Posted by Pablo Estrada <pa...@google.com>.
Fancy : )
On Wed, Mar 20, 2019 at 1:25 AM Robert Bradshaw <ro...@google.com> wrote:
> I use tox as well. Actually, I use detox and retox (parallel versions
> of tox, easily installable with pip) which can speed things up quite a
> bit.
>
> On Wed, Mar 20, 2019 at 1:33 AM Pablo Estrada <pa...@google.com> wrote:
> >
> > Correction - the command is now: tox -e py35-gcp,py35-lint
> >
> > And it ran on my machine in 5min 40s. Not blazing fast, but at least
> significantly faster than waiting for Jenkins : )
> > Best
> > -P.
> >
> > On Tue, Mar 19, 2019 at 5:22 PM Pablo Estrada <pa...@google.com>
> wrote:
> >>
> >> I use a selection of tox tasks. Here are the tox tasks that I use the
> most:
> >> - py27-gcp
> >> - py35-gcp
> >> - py27-cython
> >> - py35-cython
> >> - py35-lint
> >> - py27-lint
> >>
> >> Most recently, I'll run `tox -e py3-gcp,py3-lint`, which run fairly
> quickly. You can choose which subset works for you.
> >> My insight is: Lints are pretty fast, so it's fine to add a couple
> different lints. Unittest runs are pretty slow, so I usually go for the one
> with most coverage for my change (x-gcp, or x-cython).
> >> Best
> >> -P.
> >>
> >> On Mon, Feb 25, 2019 at 4:33 PM Ruoyun Huang <ru...@google.com> wrote:
> >>>
> >>> nvm. Don't take my previous non-scientific comparison (only ran it
> once) too seriously. :-)
> >>>
> >>> I tried to repeat each for multiple times and now the difference
> diminishes. likely there was a transient error in caching.
> >>>
> >>> On Mon, Feb 25, 2019 at 3:38 PM Kenneth Knowles <ke...@apache.org>
> wrote:
> >>>>
> >>>> Ah, that is likely caused by us having ill-defined tasks that cannot
> be cached. Or is it that the configuration time is so significant?
> >>>>
> >>>> Kenn
> >>>>
> >>>> On Mon, Feb 25, 2019 at 11:05 AM Ruoyun Huang <ru...@google.com>
> wrote:
> >>>>>
> >>>>> Out of curiosity as a light gradle user, I did a side by side
> comparison, and the readings confirm what Ken and Michael suggests.
> >>>>>
> >>>>> In the same repository, do gradle clean then followed by either of
> the two commands. Measure their runtime respectively. The latter one takes
> 1/3 running time.
> >>>>>
> >>>>> time ./gradlew spotlessApply && ./gradlew checkstyleMain &&
> ./gradlew checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain &&
> ./gradlew compileTestJava && ./gradlew compileJava
> >>>>> real 9m29.330s user 0m11.330s sys 0m1.239s
> >>>>>
> >>>>> time ./gradlew spotlessApply checkstyleMain checkstyleTest javadoc
> findbugsMain compileJava compileTestJava
> >>>>> real 3m35.573s
> >>>>> user 0m2.701s
> >>>>> sys 0m0.327s
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Feb 25, 2019 at 10:47 AM Alex Amato <aj...@google.com>
> wrote:
> >>>>>>
> >>>>>> @Michael, no particular reason. I think Ken's suggestion makes more
> sense.
> >>>>>>
> >>>>>> On Mon, Feb 25, 2019 at 10:36 AM Udi Meiri <eh...@google.com>
> wrote:
> >>>>>>>
> >>>>>>> Talking about Python:
> >>>>>>> I only know of "./gradlew lint", which include style and some py3
> compliance checking.
> >>>>>>> There is no auto-fix like spotlessApply AFAIK.
> >>>>>>>
> >>>>>>> As a side-note, I really dislike our python line continuation
> indent rule, since pycharm can't be configured to adhere to it and I find
> myself manually adjusting whitespace all the time.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon, Feb 25, 2019 at 10:22 AM Kenneth Knowles <ke...@apache.org>
> wrote:
> >>>>>>>>
> >>>>>>>> FWIW gradle is a depgraph-based build system. You can gain a few
> seconds by putting all but spotlessApply in one command.
> >>>>>>>>
> >>>>>>>> ./gradlew spotlessApply && ./gradlew checkstyleMain
> checkstyleTest javadoc findbugsMain compileTestJava compileJava
> >>>>>>>>
> >>>>>>>> It might be clever to define a meta-task. Gradle "base plugin"
> has the notable check (build and run tests), assemble (make artifacts), and
> build (assemble + check, badly named!)
> >>>>>>>>
> >>>>>>>> I think something like "everything except running tests and
> building artifacts" might be helpful.
> >>>>>>>>
> >>>>>>>> Kenn
> >>>>>>>>
> >>>>>>>> On Mon, Feb 25, 2019 at 10:13 AM Alex Amato <aj...@google.com>
> wrote:
> >>>>>>>>>
> >>>>>>>>> I made a thread about this a while back for java, but I don't
> think the same commands like sptoless work for python.
> >>>>>>>>>
> >>>>>>>>> auto fixing lint issues
> >>>>>>>>> running and quick checks which would fail the PR (without
> running the whole precommit?)
> >>>>>>>>> Something like findbugs to detect common issues (i.e. py3
> compliance)
> >>>>>>>>>
> >>>>>>>>> FWIW, this is what I have been using for java. It will catch
> pretty much everything except presubmit test failures.
> >>>>>>>>>
> >>>>>>>>> ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
> checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
> compileTestJava && ./gradlew compileJava
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> ================
> >>>>> Ruoyun Huang
> >>>>>
> >>>
> >>>
> >>> --
> >>> ================
> >>> Ruoyun Huang
> >>>
>
Re: What quick command to catch common issues before pushing a python PR?
Posted by Robert Bradshaw <ro...@google.com>.
I use tox as well. Actually, I use detox and retox (parallel versions
of tox, easily installable with pip) which can speed things up quite a
bit.
On Wed, Mar 20, 2019 at 1:33 AM Pablo Estrada <pa...@google.com> wrote:
>
> Correction - the command is now: tox -e py35-gcp,py35-lint
>
> And it ran on my machine in 5min 40s. Not blazing fast, but at least significantly faster than waiting for Jenkins : )
> Best
> -P.
>
> On Tue, Mar 19, 2019 at 5:22 PM Pablo Estrada <pa...@google.com> wrote:
>>
>> I use a selection of tox tasks. Here are the tox tasks that I use the most:
>> - py27-gcp
>> - py35-gcp
>> - py27-cython
>> - py35-cython
>> - py35-lint
>> - py27-lint
>>
>> Most recently, I'll run `tox -e py3-gcp,py3-lint`, which run fairly quickly. You can choose which subset works for you.
>> My insight is: Lints are pretty fast, so it's fine to add a couple different lints. Unittest runs are pretty slow, so I usually go for the one with most coverage for my change (x-gcp, or x-cython).
>> Best
>> -P.
>>
>> On Mon, Feb 25, 2019 at 4:33 PM Ruoyun Huang <ru...@google.com> wrote:
>>>
>>> nvm. Don't take my previous non-scientific comparison (only ran it once) too seriously. :-)
>>>
>>> I tried to repeat each for multiple times and now the difference diminishes. likely there was a transient error in caching.
>>>
>>> On Mon, Feb 25, 2019 at 3:38 PM Kenneth Knowles <ke...@apache.org> wrote:
>>>>
>>>> Ah, that is likely caused by us having ill-defined tasks that cannot be cached. Or is it that the configuration time is so significant?
>>>>
>>>> Kenn
>>>>
>>>> On Mon, Feb 25, 2019 at 11:05 AM Ruoyun Huang <ru...@google.com> wrote:
>>>>>
>>>>> Out of curiosity as a light gradle user, I did a side by side comparison, and the readings confirm what Ken and Michael suggests.
>>>>>
>>>>> In the same repository, do gradle clean then followed by either of the two commands. Measure their runtime respectively. The latter one takes 1/3 running time.
>>>>>
>>>>> time ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew compileTestJava && ./gradlew compileJava
>>>>> real 9m29.330s user 0m11.330s sys 0m1.239s
>>>>>
>>>>> time ./gradlew spotlessApply checkstyleMain checkstyleTest javadoc findbugsMain compileJava compileTestJava
>>>>> real 3m35.573s
>>>>> user 0m2.701s
>>>>> sys 0m0.327s
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 25, 2019 at 10:47 AM Alex Amato <aj...@google.com> wrote:
>>>>>>
>>>>>> @Michael, no particular reason. I think Ken's suggestion makes more sense.
>>>>>>
>>>>>> On Mon, Feb 25, 2019 at 10:36 AM Udi Meiri <eh...@google.com> wrote:
>>>>>>>
>>>>>>> Talking about Python:
>>>>>>> I only know of "./gradlew lint", which include style and some py3 compliance checking.
>>>>>>> There is no auto-fix like spotlessApply AFAIK.
>>>>>>>
>>>>>>> As a side-note, I really dislike our python line continuation indent rule, since pycharm can't be configured to adhere to it and I find myself manually adjusting whitespace all the time.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 25, 2019 at 10:22 AM Kenneth Knowles <ke...@apache.org> wrote:
>>>>>>>>
>>>>>>>> FWIW gradle is a depgraph-based build system. You can gain a few seconds by putting all but spotlessApply in one command.
>>>>>>>>
>>>>>>>> ./gradlew spotlessApply && ./gradlew checkstyleMain checkstyleTest javadoc findbugsMain compileTestJava compileJava
>>>>>>>>
>>>>>>>> It might be clever to define a meta-task. Gradle "base plugin" has the notable check (build and run tests), assemble (make artifacts), and build (assemble + check, badly named!)
>>>>>>>>
>>>>>>>> I think something like "everything except running tests and building artifacts" might be helpful.
>>>>>>>>
>>>>>>>> Kenn
>>>>>>>>
>>>>>>>> On Mon, Feb 25, 2019 at 10:13 AM Alex Amato <aj...@google.com> wrote:
>>>>>>>>>
>>>>>>>>> I made a thread about this a while back for java, but I don't think the same commands like sptoless work for python.
>>>>>>>>>
>>>>>>>>> auto fixing lint issues
>>>>>>>>> running and quick checks which would fail the PR (without running the whole precommit?)
>>>>>>>>> Something like findbugs to detect common issues (i.e. py3 compliance)
>>>>>>>>>
>>>>>>>>> FWIW, this is what I have been using for java. It will catch pretty much everything except presubmit test failures.
>>>>>>>>>
>>>>>>>>> ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew compileTestJava && ./gradlew compileJava
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> ================
>>>>> Ruoyun Huang
>>>>>
>>>
>>>
>>> --
>>> ================
>>> Ruoyun Huang
>>>
Re: What quick command to catch common issues before pushing a python PR?
Posted by Pablo Estrada <pa...@google.com>.
Correction - the command is now: tox -e py35-gcp,py35-lint
And it ran on my machine in 5min 40s. Not blazing fast, but at least
significantly faster than waiting for Jenkins : )
Best
-P.
On Tue, Mar 19, 2019 at 5:22 PM Pablo Estrada <pa...@google.com> wrote:
> I use a selection of tox tasks. Here are the tox tasks that I use the most:
> - py27-gcp
> - py35-gcp
> - py27-cython
> - py35-cython
> - py35-lint
> - py27-lint
>
> Most recently, I'll run `tox -e py3-gcp,py3-lint`, which run fairly
> quickly. You can choose which subset works for you.
> My insight is: Lints are pretty fast, so it's fine to add a couple
> different lints. Unittest runs are pretty slow, so I usually go for the one
> with most coverage for my change (x-gcp, or x-cython).
> Best
> -P.
>
> On Mon, Feb 25, 2019 at 4:33 PM Ruoyun Huang <ru...@google.com> wrote:
>
>> nvm. Don't take my previous non-scientific comparison (only ran it once)
>> too seriously. :-)
>>
>> I tried to repeat each for multiple times and now the difference
>> diminishes. likely there was a transient error in caching.
>>
>> On Mon, Feb 25, 2019 at 3:38 PM Kenneth Knowles <ke...@apache.org> wrote:
>>
>>> Ah, that is likely caused by us having ill-defined tasks that cannot be
>>> cached. Or is it that the configuration time is so significant?
>>>
>>> Kenn
>>>
>>> On Mon, Feb 25, 2019 at 11:05 AM Ruoyun Huang <ru...@google.com> wrote:
>>>
>>>> Out of curiosity as a light gradle user, I did a side by side
>>>> comparison, and the readings confirm what Ken and Michael suggests.
>>>>
>>>> In the same repository, do gradle clean then followed by either of the
>>>> two commands. Measure their runtime respectively. The latter one takes
>>>> *1/3* running time.
>>>>
>>>> time ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
>>>> checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
>>>> compileTestJava && ./gradlew compileJava
>>>> real 9m29.330s user 0m11.330s sys 0m1.239s
>>>>
>>>> time ./gradlew spotlessApply checkstyleMain checkstyleTest javadoc
>>>> findbugsMain compileJava compileTestJava
>>>> real 3m35.573s
>>>> user 0m2.701s
>>>> sys 0m0.327s
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Feb 25, 2019 at 10:47 AM Alex Amato <aj...@google.com> wrote:
>>>>
>>>>> @Michael, no particular reason. I think Ken's suggestion makes more
>>>>> sense.
>>>>>
>>>>> On Mon, Feb 25, 2019 at 10:36 AM Udi Meiri <eh...@google.com> wrote:
>>>>>
>>>>>> Talking about Python:
>>>>>> I only know of "./gradlew lint", which include style and some py3
>>>>>> compliance checking.
>>>>>> There is no auto-fix like spotlessApply AFAIK.
>>>>>>
>>>>>> As a side-note, I really dislike our python line continuation indent
>>>>>> rule, since pycharm can't be configured to adhere to it and I find myself
>>>>>> manually adjusting whitespace all the time.
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 25, 2019 at 10:22 AM Kenneth Knowles <ke...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> FWIW gradle is a depgraph-based build system. You can gain a few
>>>>>>> seconds by putting all but spotlessApply in one command.
>>>>>>>
>>>>>>> ./gradlew spotlessApply && ./gradlew checkstyleMain checkstyleTest
>>>>>>> javadoc findbugsMain compileTestJava compileJava
>>>>>>>
>>>>>>> It might be clever to define a meta-task. Gradle "base plugin" has
>>>>>>> the notable check (build and run tests), assemble (make artifacts), and
>>>>>>> build (assemble + check, badly named!)
>>>>>>>
>>>>>>> I think something like "everything except running tests and building
>>>>>>> artifacts" might be helpful.
>>>>>>>
>>>>>>> Kenn
>>>>>>>
>>>>>>> On Mon, Feb 25, 2019 at 10:13 AM Alex Amato <aj...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I made a thread about this a while back for java, but I don't think
>>>>>>>> the same commands like sptoless work for python.
>>>>>>>>
>>>>>>>> auto fixing lint issues
>>>>>>>> running and quick checks which would fail the PR (without running
>>>>>>>> the whole precommit?)
>>>>>>>> Something like findbugs to detect common issues (i.e. py3
>>>>>>>> compliance)
>>>>>>>>
>>>>>>>> FWIW, this is what I have been using for java. It will catch pretty
>>>>>>>> much everything except presubmit test failures.
>>>>>>>>
>>>>>>>> ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
>>>>>>>> checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
>>>>>>>> compileTestJava && ./gradlew compileJava
>>>>>>>>
>>>>>>>
>>>>
>>>> --
>>>> ================
>>>> Ruoyun Huang
>>>>
>>>>
>>
>> --
>> ================
>> Ruoyun Huang
>>
>>
Re: What quick command to catch common issues before pushing a python PR?
Posted by Pablo Estrada <pa...@google.com>.
I use a selection of tox tasks. Here are the tox tasks that I use the most:
- py27-gcp
- py35-gcp
- py27-cython
- py35-cython
- py35-lint
- py27-lint
Most recently, I'll run `tox -e py3-gcp,py3-lint`, which run fairly
quickly. You can choose which subset works for you.
My insight is: Lints are pretty fast, so it's fine to add a couple
different lints. Unittest runs are pretty slow, so I usually go for the one
with most coverage for my change (x-gcp, or x-cython).
Best
-P.
On Mon, Feb 25, 2019 at 4:33 PM Ruoyun Huang <ru...@google.com> wrote:
> nvm. Don't take my previous non-scientific comparison (only ran it once)
> too seriously. :-)
>
> I tried to repeat each for multiple times and now the difference
> diminishes. likely there was a transient error in caching.
>
> On Mon, Feb 25, 2019 at 3:38 PM Kenneth Knowles <ke...@apache.org> wrote:
>
>> Ah, that is likely caused by us having ill-defined tasks that cannot be
>> cached. Or is it that the configuration time is so significant?
>>
>> Kenn
>>
>> On Mon, Feb 25, 2019 at 11:05 AM Ruoyun Huang <ru...@google.com> wrote:
>>
>>> Out of curiosity as a light gradle user, I did a side by side
>>> comparison, and the readings confirm what Ken and Michael suggests.
>>>
>>> In the same repository, do gradle clean then followed by either of the
>>> two commands. Measure their runtime respectively. The latter one takes
>>> *1/3* running time.
>>>
>>> time ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
>>> checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
>>> compileTestJava && ./gradlew compileJava
>>> real 9m29.330s user 0m11.330s sys 0m1.239s
>>>
>>> time ./gradlew spotlessApply checkstyleMain checkstyleTest javadoc
>>> findbugsMain compileJava compileTestJava
>>> real 3m35.573s
>>> user 0m2.701s
>>> sys 0m0.327s
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Feb 25, 2019 at 10:47 AM Alex Amato <aj...@google.com> wrote:
>>>
>>>> @Michael, no particular reason. I think Ken's suggestion makes more
>>>> sense.
>>>>
>>>> On Mon, Feb 25, 2019 at 10:36 AM Udi Meiri <eh...@google.com> wrote:
>>>>
>>>>> Talking about Python:
>>>>> I only know of "./gradlew lint", which include style and some py3
>>>>> compliance checking.
>>>>> There is no auto-fix like spotlessApply AFAIK.
>>>>>
>>>>> As a side-note, I really dislike our python line continuation indent
>>>>> rule, since pycharm can't be configured to adhere to it and I find myself
>>>>> manually adjusting whitespace all the time.
>>>>>
>>>>>
>>>>> On Mon, Feb 25, 2019 at 10:22 AM Kenneth Knowles <ke...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> FWIW gradle is a depgraph-based build system. You can gain a few
>>>>>> seconds by putting all but spotlessApply in one command.
>>>>>>
>>>>>> ./gradlew spotlessApply && ./gradlew checkstyleMain checkstyleTest
>>>>>> javadoc findbugsMain compileTestJava compileJava
>>>>>>
>>>>>> It might be clever to define a meta-task. Gradle "base plugin" has
>>>>>> the notable check (build and run tests), assemble (make artifacts), and
>>>>>> build (assemble + check, badly named!)
>>>>>>
>>>>>> I think something like "everything except running tests and building
>>>>>> artifacts" might be helpful.
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>> On Mon, Feb 25, 2019 at 10:13 AM Alex Amato <aj...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I made a thread about this a while back for java, but I don't think
>>>>>>> the same commands like sptoless work for python.
>>>>>>>
>>>>>>> auto fixing lint issues
>>>>>>> running and quick checks which would fail the PR (without running
>>>>>>> the whole precommit?)
>>>>>>> Something like findbugs to detect common issues (i.e. py3 compliance)
>>>>>>>
>>>>>>> FWIW, this is what I have been using for java. It will catch pretty
>>>>>>> much everything except presubmit test failures.
>>>>>>>
>>>>>>> ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
>>>>>>> checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
>>>>>>> compileTestJava && ./gradlew compileJava
>>>>>>>
>>>>>>
>>>
>>> --
>>> ================
>>> Ruoyun Huang
>>>
>>>
>
> --
> ================
> Ruoyun Huang
>
>
Re: What quick command to catch common issues before pushing a python PR?
Posted by Ruoyun Huang <ru...@google.com>.
nvm. Don't take my previous non-scientific comparison (only ran it once)
too seriously. :-)
I tried to repeat each for multiple times and now the difference
diminishes. likely there was a transient error in caching.
On Mon, Feb 25, 2019 at 3:38 PM Kenneth Knowles <ke...@apache.org> wrote:
> Ah, that is likely caused by us having ill-defined tasks that cannot be
> cached. Or is it that the configuration time is so significant?
>
> Kenn
>
> On Mon, Feb 25, 2019 at 11:05 AM Ruoyun Huang <ru...@google.com> wrote:
>
>> Out of curiosity as a light gradle user, I did a side by side comparison,
>> and the readings confirm what Ken and Michael suggests.
>>
>> In the same repository, do gradle clean then followed by either of the
>> two commands. Measure their runtime respectively. The latter one takes
>> *1/3* running time.
>>
>> time ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
>> checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
>> compileTestJava && ./gradlew compileJava
>> real 9m29.330s user 0m11.330s sys 0m1.239s
>>
>> time ./gradlew spotlessApply checkstyleMain checkstyleTest javadoc
>> findbugsMain compileJava compileTestJava
>> real 3m35.573s
>> user 0m2.701s
>> sys 0m0.327s
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Feb 25, 2019 at 10:47 AM Alex Amato <aj...@google.com> wrote:
>>
>>> @Michael, no particular reason. I think Ken's suggestion makes more
>>> sense.
>>>
>>> On Mon, Feb 25, 2019 at 10:36 AM Udi Meiri <eh...@google.com> wrote:
>>>
>>>> Talking about Python:
>>>> I only know of "./gradlew lint", which include style and some py3
>>>> compliance checking.
>>>> There is no auto-fix like spotlessApply AFAIK.
>>>>
>>>> As a side-note, I really dislike our python line continuation indent
>>>> rule, since pycharm can't be configured to adhere to it and I find myself
>>>> manually adjusting whitespace all the time.
>>>>
>>>>
>>>> On Mon, Feb 25, 2019 at 10:22 AM Kenneth Knowles <ke...@apache.org>
>>>> wrote:
>>>>
>>>>> FWIW gradle is a depgraph-based build system. You can gain a few
>>>>> seconds by putting all but spotlessApply in one command.
>>>>>
>>>>> ./gradlew spotlessApply && ./gradlew checkstyleMain checkstyleTest
>>>>> javadoc findbugsMain compileTestJava compileJava
>>>>>
>>>>> It might be clever to define a meta-task. Gradle "base plugin" has the
>>>>> notable check (build and run tests), assemble (make artifacts), and build
>>>>> (assemble + check, badly named!)
>>>>>
>>>>> I think something like "everything except running tests and building
>>>>> artifacts" might be helpful.
>>>>>
>>>>> Kenn
>>>>>
>>>>> On Mon, Feb 25, 2019 at 10:13 AM Alex Amato <aj...@google.com>
>>>>> wrote:
>>>>>
>>>>>> I made a thread about this a while back for java, but I don't think
>>>>>> the same commands like sptoless work for python.
>>>>>>
>>>>>> auto fixing lint issues
>>>>>> running and quick checks which would fail the PR (without running the
>>>>>> whole precommit?)
>>>>>> Something like findbugs to detect common issues (i.e. py3 compliance)
>>>>>>
>>>>>> FWIW, this is what I have been using for java. It will catch pretty
>>>>>> much everything except presubmit test failures.
>>>>>>
>>>>>> ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
>>>>>> checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
>>>>>> compileTestJava && ./gradlew compileJava
>>>>>>
>>>>>
>>
>> --
>> ================
>> Ruoyun Huang
>>
>>
--
================
Ruoyun Huang
Re: What quick command to catch common issues before pushing a python PR?
Posted by Kenneth Knowles <ke...@apache.org>.
Ah, that is likely caused by us having ill-defined tasks that cannot be
cached. Or is it that the configuration time is so significant?
Kenn
On Mon, Feb 25, 2019 at 11:05 AM Ruoyun Huang <ru...@google.com> wrote:
> Out of curiosity as a light gradle user, I did a side by side comparison,
> and the readings confirm what Ken and Michael suggests.
>
> In the same repository, do gradle clean then followed by either of the two
> commands. Measure their runtime respectively. The latter one takes *1/3*
> running time.
>
> time ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
> checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
> compileTestJava && ./gradlew compileJava
> real 9m29.330s user 0m11.330s sys 0m1.239s
>
> time ./gradlew spotlessApply checkstyleMain checkstyleTest javadoc
> findbugsMain compileJava compileTestJava
> real 3m35.573s
> user 0m2.701s
> sys 0m0.327s
>
>
>
>
>
>
>
> On Mon, Feb 25, 2019 at 10:47 AM Alex Amato <aj...@google.com> wrote:
>
>> @Michael, no particular reason. I think Ken's suggestion makes more sense.
>>
>> On Mon, Feb 25, 2019 at 10:36 AM Udi Meiri <eh...@google.com> wrote:
>>
>>> Talking about Python:
>>> I only know of "./gradlew lint", which include style and some py3
>>> compliance checking.
>>> There is no auto-fix like spotlessApply AFAIK.
>>>
>>> As a side-note, I really dislike our python line continuation indent
>>> rule, since pycharm can't be configured to adhere to it and I find myself
>>> manually adjusting whitespace all the time.
>>>
>>>
>>> On Mon, Feb 25, 2019 at 10:22 AM Kenneth Knowles <ke...@apache.org>
>>> wrote:
>>>
>>>> FWIW gradle is a depgraph-based build system. You can gain a few
>>>> seconds by putting all but spotlessApply in one command.
>>>>
>>>> ./gradlew spotlessApply && ./gradlew checkstyleMain checkstyleTest
>>>> javadoc findbugsMain compileTestJava compileJava
>>>>
>>>> It might be clever to define a meta-task. Gradle "base plugin" has the
>>>> notable check (build and run tests), assemble (make artifacts), and build
>>>> (assemble + check, badly named!)
>>>>
>>>> I think something like "everything except running tests and building
>>>> artifacts" might be helpful.
>>>>
>>>> Kenn
>>>>
>>>> On Mon, Feb 25, 2019 at 10:13 AM Alex Amato <aj...@google.com> wrote:
>>>>
>>>>> I made a thread about this a while back for java, but I don't think
>>>>> the same commands like sptoless work for python.
>>>>>
>>>>> auto fixing lint issues
>>>>> running and quick checks which would fail the PR (without running the
>>>>> whole precommit?)
>>>>> Something like findbugs to detect common issues (i.e. py3 compliance)
>>>>>
>>>>> FWIW, this is what I have been using for java. It will catch pretty
>>>>> much everything except presubmit test failures.
>>>>>
>>>>> ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
>>>>> checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
>>>>> compileTestJava && ./gradlew compileJava
>>>>>
>>>>
>
> --
> ================
> Ruoyun Huang
>
>
Re: What quick command to catch common issues before pushing a python PR?
Posted by Ruoyun Huang <ru...@google.com>.
Out of curiosity as a light gradle user, I did a side by side comparison,
and the readings confirm what Ken and Michael suggests.
In the same repository, do gradle clean then followed by either of the two
commands. Measure their runtime respectively. The latter one takes *1/3*
running time.
time ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
compileTestJava && ./gradlew compileJava
real 9m29.330s user 0m11.330s sys 0m1.239s
time ./gradlew spotlessApply checkstyleMain checkstyleTest javadoc
findbugsMain compileJava compileTestJava
real 3m35.573s
user 0m2.701s
sys 0m0.327s
On Mon, Feb 25, 2019 at 10:47 AM Alex Amato <aj...@google.com> wrote:
> @Michael, no particular reason. I think Ken's suggestion makes more sense.
>
> On Mon, Feb 25, 2019 at 10:36 AM Udi Meiri <eh...@google.com> wrote:
>
>> Talking about Python:
>> I only know of "./gradlew lint", which include style and some py3
>> compliance checking.
>> There is no auto-fix like spotlessApply AFAIK.
>>
>> As a side-note, I really dislike our python line continuation indent
>> rule, since pycharm can't be configured to adhere to it and I find myself
>> manually adjusting whitespace all the time.
>>
>>
>> On Mon, Feb 25, 2019 at 10:22 AM Kenneth Knowles <ke...@apache.org> wrote:
>>
>>> FWIW gradle is a depgraph-based build system. You can gain a few seconds
>>> by putting all but spotlessApply in one command.
>>>
>>> ./gradlew spotlessApply && ./gradlew checkstyleMain checkstyleTest
>>> javadoc findbugsMain compileTestJava compileJava
>>>
>>> It might be clever to define a meta-task. Gradle "base plugin" has the
>>> notable check (build and run tests), assemble (make artifacts), and build
>>> (assemble + check, badly named!)
>>>
>>> I think something like "everything except running tests and building
>>> artifacts" might be helpful.
>>>
>>> Kenn
>>>
>>> On Mon, Feb 25, 2019 at 10:13 AM Alex Amato <aj...@google.com> wrote:
>>>
>>>> I made a thread about this a while back for java, but I don't think the
>>>> same commands like sptoless work for python.
>>>>
>>>> auto fixing lint issues
>>>> running and quick checks which would fail the PR (without running the
>>>> whole precommit?)
>>>> Something like findbugs to detect common issues (i.e. py3 compliance)
>>>>
>>>> FWIW, this is what I have been using for java. It will catch pretty
>>>> much everything except presubmit test failures.
>>>>
>>>> ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
>>>> checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
>>>> compileTestJava && ./gradlew compileJava
>>>>
>>>
--
================
Ruoyun Huang
Re: What quick command to catch common issues before pushing a python PR?
Posted by Alex Amato <aj...@google.com>.
@Michael, no particular reason. I think Ken's suggestion makes more sense.
On Mon, Feb 25, 2019 at 10:36 AM Udi Meiri <eh...@google.com> wrote:
> Talking about Python:
> I only know of "./gradlew lint", which include style and some py3
> compliance checking.
> There is no auto-fix like spotlessApply AFAIK.
>
> As a side-note, I really dislike our python line continuation indent rule,
> since pycharm can't be configured to adhere to it and I find myself
> manually adjusting whitespace all the time.
>
>
> On Mon, Feb 25, 2019 at 10:22 AM Kenneth Knowles <ke...@apache.org> wrote:
>
>> FWIW gradle is a depgraph-based build system. You can gain a few seconds
>> by putting all but spotlessApply in one command.
>>
>> ./gradlew spotlessApply && ./gradlew checkstyleMain checkstyleTest
>> javadoc findbugsMain compileTestJava compileJava
>>
>> It might be clever to define a meta-task. Gradle "base plugin" has the
>> notable check (build and run tests), assemble (make artifacts), and build
>> (assemble + check, badly named!)
>>
>> I think something like "everything except running tests and building
>> artifacts" might be helpful.
>>
>> Kenn
>>
>> On Mon, Feb 25, 2019 at 10:13 AM Alex Amato <aj...@google.com> wrote:
>>
>>> I made a thread about this a while back for java, but I don't think the
>>> same commands like sptoless work for python.
>>>
>>> auto fixing lint issues
>>> running and quick checks which would fail the PR (without running the
>>> whole precommit?)
>>> Something like findbugs to detect common issues (i.e. py3 compliance)
>>>
>>> FWIW, this is what I have been using for java. It will catch pretty much
>>> everything except presubmit test failures.
>>>
>>> ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
>>> checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
>>> compileTestJava && ./gradlew compileJava
>>>
>>
Re: What quick command to catch common issues before pushing a python PR?
Posted by Udi Meiri <eh...@google.com>.
Talking about Python:
I only know of "./gradlew lint", which include style and some py3
compliance checking.
There is no auto-fix like spotlessApply AFAIK.
As a side-note, I really dislike our python line continuation indent rule,
since pycharm can't be configured to adhere to it and I find myself
manually adjusting whitespace all the time.
On Mon, Feb 25, 2019 at 10:22 AM Kenneth Knowles <ke...@apache.org> wrote:
> FWIW gradle is a depgraph-based build system. You can gain a few seconds
> by putting all but spotlessApply in one command.
>
> ./gradlew spotlessApply && ./gradlew checkstyleMain checkstyleTest javadoc
> findbugsMain compileTestJava compileJava
>
> It might be clever to define a meta-task. Gradle "base plugin" has the
> notable check (build and run tests), assemble (make artifacts), and build
> (assemble + check, badly named!)
>
> I think something like "everything except running tests and building
> artifacts" might be helpful.
>
> Kenn
>
> On Mon, Feb 25, 2019 at 10:13 AM Alex Amato <aj...@google.com> wrote:
>
>> I made a thread about this a while back for java, but I don't think the
>> same commands like sptoless work for python.
>>
>> auto fixing lint issues
>> running and quick checks which would fail the PR (without running the
>> whole precommit?)
>> Something like findbugs to detect common issues (i.e. py3 compliance)
>>
>> FWIW, this is what I have been using for java. It will catch pretty much
>> everything except presubmit test failures.
>>
>> ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
>> checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
>> compileTestJava && ./gradlew compileJava
>>
>
Re: What quick command to catch common issues before pushing a python PR?
Posted by Kenneth Knowles <ke...@apache.org>.
FWIW gradle is a depgraph-based build system. You can gain a few seconds by
putting all but spotlessApply in one command.
./gradlew spotlessApply && ./gradlew checkstyleMain checkstyleTest javadoc
findbugsMain compileTestJava compileJava
It might be clever to define a meta-task. Gradle "base plugin" has the
notable check (build and run tests), assemble (make artifacts), and build
(assemble + check, badly named!)
I think something like "everything except running tests and building
artifacts" might be helpful.
Kenn
On Mon, Feb 25, 2019 at 10:13 AM Alex Amato <aj...@google.com> wrote:
> I made a thread about this a while back for java, but I don't think the
> same commands like sptoless work for python.
>
> auto fixing lint issues
> running and quick checks which would fail the PR (without running the
> whole precommit?)
> Something like findbugs to detect common issues (i.e. py3 compliance)
>
> FWIW, this is what I have been using for java. It will catch pretty much
> everything except presubmit test failures.
>
> ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
> checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
> compileTestJava && ./gradlew compileJava
>
Re: What quick command to catch common issues before pushing a python PR?
Posted by Michael Luckey <ad...@gmail.com>.
Hi,
just curious to know, why you prefer those && instead of a plain
./gradlew spotlessApply checkstyleMain checkstyleTest javadoc findbugsMain
compileJava compileTestJava
michel
On Mon, Feb 25, 2019 at 7:13 PM Alex Amato <aj...@google.com> wrote:
> I made a thread about this a while back for java, but I don't think the
> same commands like sptoless work for python.
>
> auto fixing lint issues
> running and quick checks which would fail the PR (without running the
> whole precommit?)
> Something like findbugs to detect common issues (i.e. py3 compliance)
>
> FWIW, this is what I have been using for java. It will catch pretty much
> everything except presubmit test failures.
>
> ./gradlew spotlessApply && ./gradlew checkstyleMain && ./gradlew
> checkstyleTest && ./gradlew javadoc && ./gradlew findbugsMain && ./gradlew
> compileTestJava && ./gradlew compileJava
>