You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Eugene Kirpichov <ki...@google.com.INVALID> on 2017/05/18 23:29:58 UTC
How can I disable running Python SDK tests when testing my Java change?
I've noticed that when I run "mvn verify", most of the time when I look at
the screen it's running Python tests.
Indeed, the Reactor Summary says:
...
[INFO] Apache Beam :: SDKs :: Python ...................... SUCCESS [11:56
min]
...
[INFO] Total time: 12:03 min (Wall Clock)
i.e. it's clearly on the critical path. The longest other project is
02:17min (Runners::Spark).
Are our .pom files customizable with an option to run only Java tests? (or,
respectively, only Python tests)
Thanks.
Re: How can I disable running Python SDK tests when testing my Java change?
Posted by Ted Yu <yu...@gmail.com>.
I logged BEAM-2335 with some of my findings.
Cheers
On Fri, May 19, 2017 at 12:38 PM, Borisa Zivkovic <
borisha.zivkovic@gmail.com> wrote:
> I think it should be added. I am compiling a list of useful maven commands
> to put there. But it takes time.
>
> For example, how do I execute only one test marked as @NeedsRunner?
> How do I execute one specific test in java io?
> How to execute one pecific test in any of the runners?
> How to use beamTestpipelineoptions with few json examples?
> Will mvn clean verify execute ALL tests against all runners?
>
> I think this kind of information would be very useful to speed up new
> developers.
> To figure this out one has to go through pom files.
>
> Cheers
>
> On Fri 19 May 2017 at 18:41, Ted Yu <yu...@gmail.com> wrote:
>
> > Should this tip be added to https://beam.apache.org/contribute/testing/
> ?
> >
> > Cheers
> >
> > On Fri, May 19, 2017 at 10:31 AM, Lukasz Cwik <lc...@google.com.invalid>
> > wrote:
> >
> > > Developers can use *-pl \!sdks/python* to skip the python module.
> > >
> > > Breaking it up would help developers working outside of Python and
> would
> > > decrease the precommit/postcommit execution times.
> > >
> > > On Thu, May 18, 2017 at 7:24 PM, Robert Bradshaw <
> > > robertwb@google.com.invalid> wrote:
> > >
> > > > We could consider splitting Python up into the four things it runs:
> > > > all tests with Cython, all tests without Cython, docs, and
> checkstyle.
> > > > However, I never use Maven when developing the python portions.
> > > >
> > > > On Thu, May 18, 2017 at 6:35 PM, Thomas Groh
> <tgroh@google.com.invalid
> > >
> > > > wrote:
> > > > > Generally I pass "-am -amd -pl sdks/java/core" to my maven
> > invocation.
> > > > -pl
> > > > > is the module to build, -am indicates to also make all modules my
> > > target
> > > > > depends upon, and -amd indicates to also make all of the
> > dependencies;
> > > so
> > > > > if you're only modifying java, that should hit everything. If
> you're
> > > > making
> > > > > another module, you can specify that as the -pl target, and if you
> > > > > 'install' instead of 'verify' you can resume arbitrarily.
> > > > >
> > > > > On Thu, May 18, 2017 at 4:29 PM, Eugene Kirpichov <
> > > > > kirpichov@google.com.invalid> wrote:
> > > > >
> > > > >> I've noticed that when I run "mvn verify", most of the time when I
> > > look
> > > > at
> > > > >> the screen it's running Python tests.
> > > > >>
> > > > >> Indeed, the Reactor Summary says:
> > > > >> ...
> > > > >> [INFO] Apache Beam :: SDKs :: Python ......................
> SUCCESS
> > > > [11:56
> > > > >> min]
> > > > >> ...
> > > > >> [INFO] Total time: 12:03 min (Wall Clock)
> > > > >>
> > > > >> i.e. it's clearly on the critical path. The longest other project
> is
> > > > >> 02:17min (Runners::Spark).
> > > > >>
> > > > >> Are our .pom files customizable with an option to run only Java
> > tests?
> > > > (or,
> > > > >> respectively, only Python tests)
> > > > >>
> > > > >> Thanks.
> > > > >>
> > > >
> > >
> >
>
Re: How can I disable running Python SDK tests when testing my Java change?
Posted by Borisa Zivkovic <bo...@gmail.com>.
I think it should be added. I am compiling a list of useful maven commands
to put there. But it takes time.
For example, how do I execute only one test marked as @NeedsRunner?
How do I execute one specific test in java io?
How to execute one pecific test in any of the runners?
How to use beamTestpipelineoptions with few json examples?
Will mvn clean verify execute ALL tests against all runners?
I think this kind of information would be very useful to speed up new
developers.
To figure this out one has to go through pom files.
Cheers
On Fri 19 May 2017 at 18:41, Ted Yu <yu...@gmail.com> wrote:
> Should this tip be added to https://beam.apache.org/contribute/testing/ ?
>
> Cheers
>
> On Fri, May 19, 2017 at 10:31 AM, Lukasz Cwik <lc...@google.com.invalid>
> wrote:
>
> > Developers can use *-pl \!sdks/python* to skip the python module.
> >
> > Breaking it up would help developers working outside of Python and would
> > decrease the precommit/postcommit execution times.
> >
> > On Thu, May 18, 2017 at 7:24 PM, Robert Bradshaw <
> > robertwb@google.com.invalid> wrote:
> >
> > > We could consider splitting Python up into the four things it runs:
> > > all tests with Cython, all tests without Cython, docs, and checkstyle.
> > > However, I never use Maven when developing the python portions.
> > >
> > > On Thu, May 18, 2017 at 6:35 PM, Thomas Groh <tgroh@google.com.invalid
> >
> > > wrote:
> > > > Generally I pass "-am -amd -pl sdks/java/core" to my maven
> invocation.
> > > -pl
> > > > is the module to build, -am indicates to also make all modules my
> > target
> > > > depends upon, and -amd indicates to also make all of the
> dependencies;
> > so
> > > > if you're only modifying java, that should hit everything. If you're
> > > making
> > > > another module, you can specify that as the -pl target, and if you
> > > > 'install' instead of 'verify' you can resume arbitrarily.
> > > >
> > > > On Thu, May 18, 2017 at 4:29 PM, Eugene Kirpichov <
> > > > kirpichov@google.com.invalid> wrote:
> > > >
> > > >> I've noticed that when I run "mvn verify", most of the time when I
> > look
> > > at
> > > >> the screen it's running Python tests.
> > > >>
> > > >> Indeed, the Reactor Summary says:
> > > >> ...
> > > >> [INFO] Apache Beam :: SDKs :: Python ...................... SUCCESS
> > > [11:56
> > > >> min]
> > > >> ...
> > > >> [INFO] Total time: 12:03 min (Wall Clock)
> > > >>
> > > >> i.e. it's clearly on the critical path. The longest other project is
> > > >> 02:17min (Runners::Spark).
> > > >>
> > > >> Are our .pom files customizable with an option to run only Java
> tests?
> > > (or,
> > > >> respectively, only Python tests)
> > > >>
> > > >> Thanks.
> > > >>
> > >
> >
>
Re: How can I disable running Python SDK tests when testing my Java change?
Posted by Ted Yu <yu...@gmail.com>.
Should this tip be added to https://beam.apache.org/contribute/testing/ ?
Cheers
On Fri, May 19, 2017 at 10:31 AM, Lukasz Cwik <lc...@google.com.invalid>
wrote:
> Developers can use *-pl \!sdks/python* to skip the python module.
>
> Breaking it up would help developers working outside of Python and would
> decrease the precommit/postcommit execution times.
>
> On Thu, May 18, 2017 at 7:24 PM, Robert Bradshaw <
> robertwb@google.com.invalid> wrote:
>
> > We could consider splitting Python up into the four things it runs:
> > all tests with Cython, all tests without Cython, docs, and checkstyle.
> > However, I never use Maven when developing the python portions.
> >
> > On Thu, May 18, 2017 at 6:35 PM, Thomas Groh <tg...@google.com.invalid>
> > wrote:
> > > Generally I pass "-am -amd -pl sdks/java/core" to my maven invocation.
> > -pl
> > > is the module to build, -am indicates to also make all modules my
> target
> > > depends upon, and -amd indicates to also make all of the dependencies;
> so
> > > if you're only modifying java, that should hit everything. If you're
> > making
> > > another module, you can specify that as the -pl target, and if you
> > > 'install' instead of 'verify' you can resume arbitrarily.
> > >
> > > On Thu, May 18, 2017 at 4:29 PM, Eugene Kirpichov <
> > > kirpichov@google.com.invalid> wrote:
> > >
> > >> I've noticed that when I run "mvn verify", most of the time when I
> look
> > at
> > >> the screen it's running Python tests.
> > >>
> > >> Indeed, the Reactor Summary says:
> > >> ...
> > >> [INFO] Apache Beam :: SDKs :: Python ...................... SUCCESS
> > [11:56
> > >> min]
> > >> ...
> > >> [INFO] Total time: 12:03 min (Wall Clock)
> > >>
> > >> i.e. it's clearly on the critical path. The longest other project is
> > >> 02:17min (Runners::Spark).
> > >>
> > >> Are our .pom files customizable with an option to run only Java tests?
> > (or,
> > >> respectively, only Python tests)
> > >>
> > >> Thanks.
> > >>
> >
>
Re: How can I disable running Python SDK tests when testing my Java change?
Posted by Lukasz Cwik <lc...@google.com.INVALID>.
Developers can use *-pl \!sdks/python* to skip the python module.
Breaking it up would help developers working outside of Python and would
decrease the precommit/postcommit execution times.
On Thu, May 18, 2017 at 7:24 PM, Robert Bradshaw <
robertwb@google.com.invalid> wrote:
> We could consider splitting Python up into the four things it runs:
> all tests with Cython, all tests without Cython, docs, and checkstyle.
> However, I never use Maven when developing the python portions.
>
> On Thu, May 18, 2017 at 6:35 PM, Thomas Groh <tg...@google.com.invalid>
> wrote:
> > Generally I pass "-am -amd -pl sdks/java/core" to my maven invocation.
> -pl
> > is the module to build, -am indicates to also make all modules my target
> > depends upon, and -amd indicates to also make all of the dependencies; so
> > if you're only modifying java, that should hit everything. If you're
> making
> > another module, you can specify that as the -pl target, and if you
> > 'install' instead of 'verify' you can resume arbitrarily.
> >
> > On Thu, May 18, 2017 at 4:29 PM, Eugene Kirpichov <
> > kirpichov@google.com.invalid> wrote:
> >
> >> I've noticed that when I run "mvn verify", most of the time when I look
> at
> >> the screen it's running Python tests.
> >>
> >> Indeed, the Reactor Summary says:
> >> ...
> >> [INFO] Apache Beam :: SDKs :: Python ...................... SUCCESS
> [11:56
> >> min]
> >> ...
> >> [INFO] Total time: 12:03 min (Wall Clock)
> >>
> >> i.e. it's clearly on the critical path. The longest other project is
> >> 02:17min (Runners::Spark).
> >>
> >> Are our .pom files customizable with an option to run only Java tests?
> (or,
> >> respectively, only Python tests)
> >>
> >> Thanks.
> >>
>
Re: How can I disable running Python SDK tests when testing my Java change?
Posted by Robert Bradshaw <ro...@google.com.INVALID>.
We could consider splitting Python up into the four things it runs:
all tests with Cython, all tests without Cython, docs, and checkstyle.
However, I never use Maven when developing the python portions.
On Thu, May 18, 2017 at 6:35 PM, Thomas Groh <tg...@google.com.invalid> wrote:
> Generally I pass "-am -amd -pl sdks/java/core" to my maven invocation. -pl
> is the module to build, -am indicates to also make all modules my target
> depends upon, and -amd indicates to also make all of the dependencies; so
> if you're only modifying java, that should hit everything. If you're making
> another module, you can specify that as the -pl target, and if you
> 'install' instead of 'verify' you can resume arbitrarily.
>
> On Thu, May 18, 2017 at 4:29 PM, Eugene Kirpichov <
> kirpichov@google.com.invalid> wrote:
>
>> I've noticed that when I run "mvn verify", most of the time when I look at
>> the screen it's running Python tests.
>>
>> Indeed, the Reactor Summary says:
>> ...
>> [INFO] Apache Beam :: SDKs :: Python ...................... SUCCESS [11:56
>> min]
>> ...
>> [INFO] Total time: 12:03 min (Wall Clock)
>>
>> i.e. it's clearly on the critical path. The longest other project is
>> 02:17min (Runners::Spark).
>>
>> Are our .pom files customizable with an option to run only Java tests? (or,
>> respectively, only Python tests)
>>
>> Thanks.
>>
Re: How can I disable running Python SDK tests when testing my Java change?
Posted by Thomas Groh <tg...@google.com.INVALID>.
Generally I pass "-am -amd -pl sdks/java/core" to my maven invocation. -pl
is the module to build, -am indicates to also make all modules my target
depends upon, and -amd indicates to also make all of the dependencies; so
if you're only modifying java, that should hit everything. If you're making
another module, you can specify that as the -pl target, and if you
'install' instead of 'verify' you can resume arbitrarily.
On Thu, May 18, 2017 at 4:29 PM, Eugene Kirpichov <
kirpichov@google.com.invalid> wrote:
> I've noticed that when I run "mvn verify", most of the time when I look at
> the screen it's running Python tests.
>
> Indeed, the Reactor Summary says:
> ...
> [INFO] Apache Beam :: SDKs :: Python ...................... SUCCESS [11:56
> min]
> ...
> [INFO] Total time: 12:03 min (Wall Clock)
>
> i.e. it's clearly on the critical path. The longest other project is
> 02:17min (Runners::Spark).
>
> Are our .pom files customizable with an option to run only Java tests? (or,
> respectively, only Python tests)
>
> Thanks.
>