You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Amit Sela <am...@gmail.com> on 2016/10/20 19:24:26 UTC
[DISCUSS] Executing (Jenkins) RunnableOnService tests more efficiently.
Hi all,
I'd like to discuss options to execute ROS tests (per runner) more
efficiently, and explore the option of running them on PreCommit, as
opposed to PostCommit as they run today.
The SDK provides a set of tests called "RunnableOnService" (aka ROS) that
can be applied to a runner and validate it (correctly) supports SDK
features.
It's 300+ tests in total (batch + streaming) and it clearly takes time, and
that is why it runs on PostCommit.
I think we should look for a configuration where this is executed
more efficiently and if possible run on PreCommit since runners are
encouraged to rely on those tests and it's better to know of breaking
changes before hand.
This came up somewhere in this
<https://github.com/apache/incubator-beam/pull/1055> conversation, and the
highlights are basically:
Kenneth Knowles suggested we might parallelize sub-builds in the following
manner:
1. Run unit tests.
2. (sub tasks) Run ROS tests for each runner in parallel, skipping unit
tests.
I was wondering if we could setup Jenkins to run ROS per runner only of
there was a code change for that runner - of course SDK changes will
probably have to run ROS for all runners, but that might still be an
optimization.
I think one of Beam's best sell-points is it's extensive testing framework,
and the fact that runners can be validated across capabilities, but it
would be best to know of runner-braking changes before merging to master.
Thoughts ?
Thanks,
Amit
Re: [DISCUSS] Executing (Jenkins) RunnableOnService tests more efficiently.
Posted by Robert Bradshaw <ro...@google.com.INVALID>.
+1 to making these cheaper.
Another idea I've had, though I'm not sure how hard it would be to
implement, is that many ROS tests are trivial pipelines and dominated
by the overhead of invoking the service itself. I wonder how hard it
would be to transparently "pack" many pipelines into single job
requests whose success would be a very strong (though not perfect)
indicator that each pipeline would have succeeded individually. Of
course not all tests are structured in this manner, but from my
experience most are, and this could be significant savings.
On Thu, Oct 20, 2016 at 12:24 PM, Amit Sela <am...@gmail.com> wrote:
> Hi all,
>
> I'd like to discuss options to execute ROS tests (per runner) more
> efficiently, and explore the option of running them on PreCommit, as
> opposed to PostCommit as they run today.
>
> The SDK provides a set of tests called "RunnableOnService" (aka ROS) that
> can be applied to a runner and validate it (correctly) supports SDK
> features.
> It's 300+ tests in total (batch + streaming) and it clearly takes time, and
> that is why it runs on PostCommit.
> I think we should look for a configuration where this is executed
> more efficiently and if possible run on PreCommit since runners are
> encouraged to rely on those tests and it's better to know of breaking
> changes before hand.
>
> This came up somewhere in this
> <https://github.com/apache/incubator-beam/pull/1055> conversation, and the
> highlights are basically:
>
> Kenneth Knowles suggested we might parallelize sub-builds in the following
> manner:
>
> 1. Run unit tests.
> 2. (sub tasks) Run ROS tests for each runner in parallel, skipping unit
> tests.
>
> I was wondering if we could setup Jenkins to run ROS per runner only of
> there was a code change for that runner - of course SDK changes will
> probably have to run ROS for all runners, but that might still be an
> optimization.
>
> I think one of Beam's best sell-points is it's extensive testing framework,
> and the fact that runners can be validated across capabilities, but it
> would be best to know of runner-braking changes before merging to master.
>
> Thoughts ?
>
> Thanks,
> Amit
Re: [DISCUSS] Executing (Jenkins) RunnableOnService tests more efficiently.
Posted by Lukasz Cwik <lc...@google.com.INVALID>.
Robert, it seems as though setting up the Flink/Spark/Gearpump cluster
locally for testing is pretty fast already and typically all the ROS tests
run within 5-10 mins.
The two outliers are Google Cloud Dataflow which takes over an hour, and
also the amount of time we have to wait before there is a free executor
available in Jenkins in the afternoon/evening PST time.
On Thu, Oct 20, 2016 at 9:26 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi,
>
> indeed, let me ping INFRA to try to get update.
>
> Regards
> JB
>
>
> On 10/20/2016 09:38 PM, Davor Bonaci wrote:
>
>> I'd be hugely in favor, however, this is not what the Apache Jenkins
>> supports right now, AFAIK. I've asked Infra about this awhile ago, but
>> nothing has moved yet. There was also a Jira issue about it, INFRA-11610.
>>
>> On Thu, Oct 20, 2016 at 12:24 PM, Amit Sela <am...@gmail.com> wrote:
>>
>> Hi all,
>>>
>>> I'd like to discuss options to execute ROS tests (per runner) more
>>> efficiently, and explore the option of running them on PreCommit, as
>>> opposed to PostCommit as they run today.
>>>
>>> The SDK provides a set of tests called "RunnableOnService" (aka ROS) that
>>> can be applied to a runner and validate it (correctly) supports SDK
>>> features.
>>> It's 300+ tests in total (batch + streaming) and it clearly takes time,
>>> and
>>> that is why it runs on PostCommit.
>>> I think we should look for a configuration where this is executed
>>> more efficiently and if possible run on PreCommit since runners are
>>> encouraged to rely on those tests and it's better to know of breaking
>>> changes before hand.
>>>
>>> This came up somewhere in this
>>> <https://github.com/apache/incubator-beam/pull/1055> conversation, and
>>> the
>>> highlights are basically:
>>>
>>> Kenneth Knowles suggested we might parallelize sub-builds in the
>>> following
>>> manner:
>>>
>>> 1. Run unit tests.
>>> 2. (sub tasks) Run ROS tests for each runner in parallel, skipping
>>> unit
>>> tests.
>>>
>>> I was wondering if we could setup Jenkins to run ROS per runner only of
>>> there was a code change for that runner - of course SDK changes will
>>> probably have to run ROS for all runners, but that might still be an
>>> optimization.
>>>
>>> I think one of Beam's best sell-points is it's extensive testing
>>> framework,
>>> and the fact that runners can be validated across capabilities, but it
>>> would be best to know of runner-braking changes before merging to master.
>>>
>>> Thoughts ?
>>>
>>> Thanks,
>>> Amit
>>>
>>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [DISCUSS] Executing (Jenkins) RunnableOnService tests more
efficiently.
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
indeed, let me ping INFRA to try to get update.
Regards
JB
On 10/20/2016 09:38 PM, Davor Bonaci wrote:
> I'd be hugely in favor, however, this is not what the Apache Jenkins
> supports right now, AFAIK. I've asked Infra about this awhile ago, but
> nothing has moved yet. There was also a Jira issue about it, INFRA-11610.
>
> On Thu, Oct 20, 2016 at 12:24 PM, Amit Sela <am...@gmail.com> wrote:
>
>> Hi all,
>>
>> I'd like to discuss options to execute ROS tests (per runner) more
>> efficiently, and explore the option of running them on PreCommit, as
>> opposed to PostCommit as they run today.
>>
>> The SDK provides a set of tests called "RunnableOnService" (aka ROS) that
>> can be applied to a runner and validate it (correctly) supports SDK
>> features.
>> It's 300+ tests in total (batch + streaming) and it clearly takes time, and
>> that is why it runs on PostCommit.
>> I think we should look for a configuration where this is executed
>> more efficiently and if possible run on PreCommit since runners are
>> encouraged to rely on those tests and it's better to know of breaking
>> changes before hand.
>>
>> This came up somewhere in this
>> <https://github.com/apache/incubator-beam/pull/1055> conversation, and the
>> highlights are basically:
>>
>> Kenneth Knowles suggested we might parallelize sub-builds in the following
>> manner:
>>
>> 1. Run unit tests.
>> 2. (sub tasks) Run ROS tests for each runner in parallel, skipping unit
>> tests.
>>
>> I was wondering if we could setup Jenkins to run ROS per runner only of
>> there was a code change for that runner - of course SDK changes will
>> probably have to run ROS for all runners, but that might still be an
>> optimization.
>>
>> I think one of Beam's best sell-points is it's extensive testing framework,
>> and the fact that runners can be validated across capabilities, but it
>> would be best to know of runner-braking changes before merging to master.
>>
>> Thoughts ?
>>
>> Thanks,
>> Amit
>>
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [DISCUSS] Executing (Jenkins) RunnableOnService tests more efficiently.
Posted by Davor Bonaci <da...@google.com.INVALID>.
I'd be hugely in favor, however, this is not what the Apache Jenkins
supports right now, AFAIK. I've asked Infra about this awhile ago, but
nothing has moved yet. There was also a Jira issue about it, INFRA-11610.
On Thu, Oct 20, 2016 at 12:24 PM, Amit Sela <am...@gmail.com> wrote:
> Hi all,
>
> I'd like to discuss options to execute ROS tests (per runner) more
> efficiently, and explore the option of running them on PreCommit, as
> opposed to PostCommit as they run today.
>
> The SDK provides a set of tests called "RunnableOnService" (aka ROS) that
> can be applied to a runner and validate it (correctly) supports SDK
> features.
> It's 300+ tests in total (batch + streaming) and it clearly takes time, and
> that is why it runs on PostCommit.
> I think we should look for a configuration where this is executed
> more efficiently and if possible run on PreCommit since runners are
> encouraged to rely on those tests and it's better to know of breaking
> changes before hand.
>
> This came up somewhere in this
> <https://github.com/apache/incubator-beam/pull/1055> conversation, and the
> highlights are basically:
>
> Kenneth Knowles suggested we might parallelize sub-builds in the following
> manner:
>
> 1. Run unit tests.
> 2. (sub tasks) Run ROS tests for each runner in parallel, skipping unit
> tests.
>
> I was wondering if we could setup Jenkins to run ROS per runner only of
> there was a code change for that runner - of course SDK changes will
> probably have to run ROS for all runners, but that might still be an
> optimization.
>
> I think one of Beam's best sell-points is it's extensive testing framework,
> and the fact that runners can be validated across capabilities, but it
> would be best to know of runner-braking changes before merging to master.
>
> Thoughts ?
>
> Thanks,
> Amit
>