You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Mikhail Gryzykhin <mi...@google.com> on 2018/08/13 16:35:24 UTC
Rename Nexmark jobs to Perf instead of PostCommit
Hi everyone,
As I can understand, a lot of tests in Nexmark set are performance tests. I
suggest to rename(or split) the set to performance tests.
Performance tests are much less reliable compared to post-commit tests and
should have different requirements. Additionally, they are much more flaky.
Splitting out performance tests to separate set will allow us to treat
failures with lower priority and add more tolerance for flakes compared to
what we have decided for post-commit tests
<https://beam.apache.org/contribute/postcommits-policies/>.
This will also be more organic to use different builder from
PostcommitJobBuilder
<https://github.com/apache/beam/tree/master/.test-infra/jenkins>, since we
will want different requirements for perf tests.
I do not believe we have a problem with this in current state, but I expect
this to become an issue in the future as amount of perf tests grows.
Regards,
--Mikhail
Have feedback <http://go/migryz-feedback>?
Re: Rename Nexmark jobs to Perf instead of PostCommit
Posted by Etienne Chauchot <ec...@apache.org>.
@Mikhail I must disagree. I have set Nexmark tests as PostCommit mainly because they were used to discover regressions
on all the beam model as soon as possible: when the new regression commit comes into master. The graphs (see https://bea
m.apache.org/documentation/sdks/java/nexmark/) help a lot in that way. As Reuven said, performance testing is more a
second aim even if it is part of regression.
Etienne
Le mardi 14 août 2018 à 15:40 -0700, Mikhail Gryzykhin a écrit :
> @Reuven Lax Thank you for clarifying. It makes sense to keep them as post-commits then.
> @Ahmet Altay I don't see them being flaky atm. So we are good here.
>
> I would prefer to separate perf tests mostly because of requirements, such as runtime, environment, trigger
> restrictions. Right now we have a separate setup with no GHPRB phrase trigger for Nexmark tests.
>
> My understanding was that due to the "performance" nature of Nexmark tests, that we can not treat them as post-commit
> tests and have to create special setup for them.
>
> --Mikhail
> Have feedback?
>
> On Mon, Aug 13, 2018 at 10:37 AM Ahmet Altay <al...@google.com> wrote:
> > I replied before seeing Reuven's comment. In that case it makes sense to keep them as post commit tests.
> > Are these test flaky because they fail some performance metrics? If that is the case it makes sense to separate them
> > from functional tests.
> > On Mon, Aug 13, 2018 at 10:30 AM, Reuven Lax <re...@google.com> wrote:
> > > Nexmark was primarily written as a set of functionality tests, not performance tests. In fact some of the Nexmark
> > > queries are written in knowingly inefficient ways, in order to better exercise more features of Beam. We also use
> > > Nexmark as performance tests (mostly out of convenience, because we already have those tests), however I believe
> > > that is a secondary use.
> > >
> > > Reuven
> > >
> > > On Mon, Aug 13, 2018 at 9:36 AM Mikhail Gryzykhin <mi...@google.com> wrote:
> > > > Hi everyone,
> > > >
> > > > As I can understand, a lot of tests in Nexmark set are performance tests. I suggest to rename(or split) the set
> > > > to performance tests.
> > > >
> > > > Performance tests are much less reliable compared to post-commit tests and should have different requirements.
> > > > Additionally, they are much more flaky.
> > > >
> > > > Splitting out performance tests to separate set will allow us to treat failures with lower priority and add more
> > > > tolerance for flakes compared to what we have decided for post-commit tests.
> > > >
> > > > This will also be more organic to use different builder from PostcommitJobBuilder, since we will want different
> > > > requirements for perf tests.
> > > >
> > > > I do not believe we have a problem with this in current state, but I expect this to become an issue in the
> > > > future as amount of perf tests grows.
> > > >
> > > > Regards,
> > > > --Mikhail
> > > >
> > > > Have feedback?
Re: Rename Nexmark jobs to Perf instead of PostCommit
Posted by Mikhail Gryzykhin <mi...@google.com>.
@Reuven Lax <re...@google.com> Thank you for clarifying. It makes sense to
keep them as post-commits then.
@Ahmet Altay <al...@google.com> I don't see them being flaky atm. So we are
good here.
I would prefer to separate perf tests mostly because of requirements, such
as runtime, environment, trigger restrictions. Right now we have a separate
setup with no GHPRB phrase trigger for Nexmark tests.
My understanding was that due to the "performance" nature of Nexmark tests,
that we can not treat them as post-commit tests and have to create special
setup for them.
--Mikhail
Have feedback <http://go/migryz-feedback>?
On Mon, Aug 13, 2018 at 10:37 AM Ahmet Altay <al...@google.com> wrote:
> I replied before seeing Reuven's comment. In that case it makes sense to
> keep them as post commit tests.
>
> Are these test flaky because they fail some performance metrics? If that
> is the case it makes sense to separate them from functional tests.
>
> On Mon, Aug 13, 2018 at 10:30 AM, Reuven Lax <re...@google.com> wrote:
>
>> Nexmark was primarily written as a set of functionality tests, not
>> performance tests. In fact some of the Nexmark queries are written in
>> knowingly inefficient ways, in order to better exercise more features of
>> Beam. We also use Nexmark as performance tests (mostly out of convenience,
>> because we already have those tests), however I believe that is a secondary
>> use.
>>
>> Reuven
>>
>> On Mon, Aug 13, 2018 at 9:36 AM Mikhail Gryzykhin <mi...@google.com>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> As I can understand, a lot of tests in Nexmark set are performance
>>> tests. I suggest to rename(or split) the set to performance tests.
>>>
>>> Performance tests are much less reliable compared to post-commit tests
>>> and should have different requirements. Additionally, they are much more
>>> flaky.
>>>
>>> Splitting out performance tests to separate set will allow us to treat
>>> failures with lower priority and add more tolerance for flakes compared to
>>> what we have decided for post-commit tests
>>> <https://beam.apache.org/contribute/postcommits-policies/>.
>>>
>>> This will also be more organic to use different builder from
>>> PostcommitJobBuilder
>>> <https://github.com/apache/beam/tree/master/.test-infra/jenkins>, since
>>> we will want different requirements for perf tests.
>>>
>>> I do not believe we have a problem with this in current state, but I
>>> expect this to become an issue in the future as amount of perf tests grows.
>>>
>>> Regards,
>>> --Mikhail
>>>
>>> Have feedback <http://go/migryz-feedback>?
>>>
>>
>
Re: Rename Nexmark jobs to Perf instead of PostCommit
Posted by Ahmet Altay <al...@google.com>.
I replied before seeing Reuven's comment. In that case it makes sense to
keep them as post commit tests.
Are these test flaky because they fail some performance metrics? If that is
the case it makes sense to separate them from functional tests.
On Mon, Aug 13, 2018 at 10:30 AM, Reuven Lax <re...@google.com> wrote:
> Nexmark was primarily written as a set of functionality tests, not
> performance tests. In fact some of the Nexmark queries are written in
> knowingly inefficient ways, in order to better exercise more features of
> Beam. We also use Nexmark as performance tests (mostly out of convenience,
> because we already have those tests), however I believe that is a secondary
> use.
>
> Reuven
>
> On Mon, Aug 13, 2018 at 9:36 AM Mikhail Gryzykhin <mi...@google.com>
> wrote:
>
>> Hi everyone,
>>
>> As I can understand, a lot of tests in Nexmark set are performance tests.
>> I suggest to rename(or split) the set to performance tests.
>>
>> Performance tests are much less reliable compared to post-commit tests
>> and should have different requirements. Additionally, they are much more
>> flaky.
>>
>> Splitting out performance tests to separate set will allow us to treat
>> failures with lower priority and add more tolerance for flakes compared to
>> what we have decided for post-commit tests
>> <https://beam.apache.org/contribute/postcommits-policies/>.
>>
>> This will also be more organic to use different builder from
>> PostcommitJobBuilder
>> <https://github.com/apache/beam/tree/master/.test-infra/jenkins>, since
>> we will want different requirements for perf tests.
>>
>> I do not believe we have a problem with this in current state, but I
>> expect this to become an issue in the future as amount of perf tests grows.
>>
>> Regards,
>> --Mikhail
>>
>> Have feedback <http://go/migryz-feedback>?
>>
>
Re: Rename Nexmark jobs to Perf instead of PostCommit
Posted by Reuven Lax <re...@google.com>.
Nexmark was primarily written as a set of functionality tests, not
performance tests. In fact some of the Nexmark queries are written in
knowingly inefficient ways, in order to better exercise more features of
Beam. We also use Nexmark as performance tests (mostly out of convenience,
because we already have those tests), however I believe that is a secondary
use.
Reuven
On Mon, Aug 13, 2018 at 9:36 AM Mikhail Gryzykhin <mi...@google.com> wrote:
> Hi everyone,
>
> As I can understand, a lot of tests in Nexmark set are performance tests.
> I suggest to rename(or split) the set to performance tests.
>
> Performance tests are much less reliable compared to post-commit tests and
> should have different requirements. Additionally, they are much more flaky.
>
> Splitting out performance tests to separate set will allow us to treat
> failures with lower priority and add more tolerance for flakes compared to
> what we have decided for post-commit tests
> <https://beam.apache.org/contribute/postcommits-policies/>.
>
> This will also be more organic to use different builder from
> PostcommitJobBuilder
> <https://github.com/apache/beam/tree/master/.test-infra/jenkins>, since
> we will want different requirements for perf tests.
>
> I do not believe we have a problem with this in current state, but I
> expect this to become an issue in the future as amount of perf tests grows.
>
> Regards,
> --Mikhail
>
> Have feedback <http://go/migryz-feedback>?
>
Re: Rename Nexmark jobs to Perf instead of PostCommit
Posted by Ahmet Altay <al...@google.com>.
+1 to rename.
I agree with Andrew, flakiness and reliability problems with performance
are important problems that needs to be fixed. Do we have a sense of what
is making them less reliable?
On Mon, Aug 13, 2018 at 10:19 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> +1
>
> I think it makes sense to rename.
>
> Regards
> JB
> Le 13 août 2018, à 19:14, Pablo Estrada <pa...@google.com> a écrit:
>>
>> I think it makes sense to rename.
>>
>> Also, although we should hold perf tests to a high reliability standard,
>> we should also prioritize fixing and triaging PostCommit tests earlier.
>>
>> Best
>> -P.
>>
>> On Mon, Aug 13, 2018 at 10:10 AM Andrew Pilloud < apilloud@google.com>
>> wrote:
>>
>>> I have no objections with renaming these to Perf instead of PostCommit.
>>>
>>> I do disagree with your assessment that "Performance tests are much less
>>> reliable ... they are much more flaky." I think we should be holding perf
>>> tests to the same reliability standards as PostCommit tests. I'm wondering
>>> why you think otherwise?
>>>
>>> Andrew
>>>
>>> On Mon, Aug 13, 2018, 9:36 AM Mikhail Gryzykhin < migryz@google.com>
>>> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> As I can understand, a lot of tests in Nexmark set are performance
>>>> tests. I suggest to rename(or split) the set to performance tests.
>>>>
>>>> Performance tests are much less reliable compared to post-commit tests
>>>> and should have different requirements. Additionally, they are much more
>>>> flaky.
>>>>
>>>> Splitting out performance tests to separate set will allow us to treat
>>>> failures with lower priority and add more tolerance for flakes compared to
>>>> what we have decided for post-commit tests
>>>> <https://beam.apache.org/contribute/postcommits-policies/>.
>>>>
>>>> This will also be more organic to use different builder from
>>>> PostcommitJobBuilder
>>>> <https://github.com/apache/beam/tree/master/.test-infra/jenkins>,
>>>> since we will want different requirements for perf tests.
>>>>
>>>> I do not believe we have a problem with this in current state, but I
>>>> expect this to become an issue in the future as amount of perf tests grows.
>>>>
>>>> Regards,
>>>> --Mikhail
>>>>
>>>> Have feedback <http://go/migryz-feedback>?
>>>>
>>> --
>> Got feedback? go/pabloem-feedback
>> <https://goto.google.com/pabloem-feedback>
>>
>
Re: Rename Nexmark jobs to Perf instead of PostCommit
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
+1
I think it makes sense to rename.
Regards
JB
Le 13 août 2018 à 19:14, à 19:14, Pablo Estrada <pa...@google.com> a écrit:
>I think it makes sense to rename.
>
>Also, although we should hold perf tests to a high reliability
>standard, we
>should also prioritize fixing and triaging PostCommit tests earlier.
>
>Best
>-P.
>
>On Mon, Aug 13, 2018 at 10:10 AM Andrew Pilloud <ap...@google.com>
>wrote:
>
>> I have no objections with renaming these to Perf instead of
>PostCommit.
>>
>> I do disagree with your assessment that "Performance tests are much
>less
>> reliable ... they are much more flaky." I think we should be holding
>perf
>> tests to the same reliability standards as PostCommit tests. I'm
>wondering
>> why you think otherwise?
>>
>> Andrew
>>
>> On Mon, Aug 13, 2018, 9:36 AM Mikhail Gryzykhin <mi...@google.com>
>wrote:
>>
>>> Hi everyone,
>>>
>>> As I can understand, a lot of tests in Nexmark set are performance
>tests.
>>> I suggest to rename(or split) the set to performance tests.
>>>
>>> Performance tests are much less reliable compared to post-commit
>tests
>>> and should have different requirements. Additionally, they are much
>more
>>> flaky.
>>>
>>> Splitting out performance tests to separate set will allow us to
>treat
>>> failures with lower priority and add more tolerance for flakes
>compared to
>>> what we have decided for post-commit tests
>>> <https://beam.apache.org/contribute/postcommits-policies/>.
>>>
>>> This will also be more organic to use different builder from
>>> PostcommitJobBuilder
>>> <https://github.com/apache/beam/tree/master/.test-infra/jenkins>,
>since
>>> we will want different requirements for perf tests.
>>>
>>> I do not believe we have a problem with this in current state, but I
>>> expect this to become an issue in the future as amount of perf tests
>grows.
>>>
>>> Regards,
>>> --Mikhail
>>>
>>> Have feedback <http://go/migryz-feedback>?
>>>
>> --
>Got feedback? go/pabloem-feedback
Re: Rename Nexmark jobs to Perf instead of PostCommit
Posted by Pablo Estrada <pa...@google.com>.
I think it makes sense to rename.
Also, although we should hold perf tests to a high reliability standard, we
should also prioritize fixing and triaging PostCommit tests earlier.
Best
-P.
On Mon, Aug 13, 2018 at 10:10 AM Andrew Pilloud <ap...@google.com> wrote:
> I have no objections with renaming these to Perf instead of PostCommit.
>
> I do disagree with your assessment that "Performance tests are much less
> reliable ... they are much more flaky." I think we should be holding perf
> tests to the same reliability standards as PostCommit tests. I'm wondering
> why you think otherwise?
>
> Andrew
>
> On Mon, Aug 13, 2018, 9:36 AM Mikhail Gryzykhin <mi...@google.com> wrote:
>
>> Hi everyone,
>>
>> As I can understand, a lot of tests in Nexmark set are performance tests.
>> I suggest to rename(or split) the set to performance tests.
>>
>> Performance tests are much less reliable compared to post-commit tests
>> and should have different requirements. Additionally, they are much more
>> flaky.
>>
>> Splitting out performance tests to separate set will allow us to treat
>> failures with lower priority and add more tolerance for flakes compared to
>> what we have decided for post-commit tests
>> <https://beam.apache.org/contribute/postcommits-policies/>.
>>
>> This will also be more organic to use different builder from
>> PostcommitJobBuilder
>> <https://github.com/apache/beam/tree/master/.test-infra/jenkins>, since
>> we will want different requirements for perf tests.
>>
>> I do not believe we have a problem with this in current state, but I
>> expect this to become an issue in the future as amount of perf tests grows.
>>
>> Regards,
>> --Mikhail
>>
>> Have feedback <http://go/migryz-feedback>?
>>
> --
Got feedback? go/pabloem-feedback
Re: Rename Nexmark jobs to Perf instead of PostCommit
Posted by Andrew Pilloud <ap...@google.com>.
I have no objections with renaming these to Perf instead of PostCommit.
I do disagree with your assessment that "Performance tests are much less
reliable ... they are much more flaky." I think we should be holding perf
tests to the same reliability standards as PostCommit tests. I'm wondering
why you think otherwise?
Andrew
On Mon, Aug 13, 2018, 9:36 AM Mikhail Gryzykhin <mi...@google.com> wrote:
> Hi everyone,
>
> As I can understand, a lot of tests in Nexmark set are performance tests.
> I suggest to rename(or split) the set to performance tests.
>
> Performance tests are much less reliable compared to post-commit tests and
> should have different requirements. Additionally, they are much more flaky.
>
> Splitting out performance tests to separate set will allow us to treat
> failures with lower priority and add more tolerance for flakes compared to
> what we have decided for post-commit tests
> <https://beam.apache.org/contribute/postcommits-policies/>.
>
> This will also be more organic to use different builder from
> PostcommitJobBuilder
> <https://github.com/apache/beam/tree/master/.test-infra/jenkins>, since
> we will want different requirements for perf tests.
>
> I do not believe we have a problem with this in current state, but I
> expect this to become an issue in the future as amount of perf tests grows.
>
> Regards,
> --Mikhail
>
> Have feedback <http://go/migryz-feedback>?
>