You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Mariia Mykhailova <ma...@microsoft.com.INVALID> on 2017/12/11 20:45:36 UTC
List of special tests for release
Hi,
REEF has features which are not covered by testing on local runtime and the few examples we run manually when testing release in clusters.
One example of such a feature is driver restart on .NET side - AFAIK it is only covered in O.A.R.Examples.DriverRestart and requires a cluster with special YARN settings to run (a comment in the code says "Can only be run on HDInsight clusters with yarn.resourcemanager.am.max-attempts set to greater than or equal to 2."). This feature was broken and went unnoticed for several months. I'm sure there are other features like this which are not documented enough for the release manager to be aware of them (and more importantly of how to verify that they still work).
What do you think about putting together a list of such features and the process of their testing to follow during the release? This way we'd have a checklist for the release manager to verify testers' reports against rather than rely on the authors of all individual features to remember to test them and report back.
-Mariia
RE: List of special tests for release
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com.INVALID>.
Those will be part of the yarn tests. Just like running local runtime tests, we should run those tests to ensure nothing is broken. If any, we should either modify the code or modify the tests accordingly. For any new features that we need to add new yarn tests, we can just add them into the list.
-Julia
-----Original Message-----
From: Saikat Kanjilal [mailto:sxk1969@gmail.com]
Sent: Monday, December 11, 2017 1:13 PM
To: dev@reef.apache.org
Subject: Re: List of special tests for release
I agree, however my question is whether we'll be able to keep this up to date in a disciplined manner or will this be yet another thing to maintain that will fall by the wayside and become eventually outdated.
On Mon, Dec 11, 2017 at 12:45 PM, Mariia Mykhailova < mamykhai@microsoft.com.invalid> wrote:
> Hi,
>
> REEF has features which are not covered by testing on local runtime
> and the few examples we run manually when testing release in clusters.
> One example of such a feature is driver restart on .NET side - AFAIK
> it is only covered in O.A.R.Examples.DriverRestart and requires a
> cluster with special YARN settings to run (a comment in the code says
> "Can only be run on HDInsight clusters with
> yarn.resourcemanager.am.max-attempts set to greater than or equal to
> 2."). This feature was broken and went unnoticed for several months.
> I'm sure there are other features like this which are not documented
> enough for the release manager to be aware of them (and more importantly of how to verify that they still work).
>
> What do you think about putting together a list of such features and
> the process of their testing to follow during the release? This way
> we'd have a checklist for the release manager to verify testers'
> reports against rather than rely on the authors of all individual
> features to remember to test them and report back.
>
> -Mariia
>
>
Re: List of special tests for release
Posted by Saikat Kanjilal <sx...@gmail.com>.
I agree, however my question is whether we'll be able to keep this up to
date in a disciplined manner or will this be yet another thing to maintain
that will fall by the wayside and become eventually outdated.
On Mon, Dec 11, 2017 at 12:45 PM, Mariia Mykhailova <
mamykhai@microsoft.com.invalid> wrote:
> Hi,
>
> REEF has features which are not covered by testing on local runtime and
> the few examples we run manually when testing release in clusters.
> One example of such a feature is driver restart on .NET side - AFAIK it is
> only covered in O.A.R.Examples.DriverRestart and requires a cluster with
> special YARN settings to run (a comment in the code says "Can only be run
> on HDInsight clusters with yarn.resourcemanager.am.max-attempts set to
> greater than or equal to 2."). This feature was broken and went unnoticed
> for several months. I'm sure there are other features like this which are
> not documented enough for the release manager to be aware of them (and more
> importantly of how to verify that they still work).
>
> What do you think about putting together a list of such features and the
> process of their testing to follow during the release? This way we'd have a
> checklist for the release manager to verify testers' reports against rather
> than rely on the authors of all individual features to remember to test
> them and report back.
>
> -Mariia
>
>
Re: List of special tests for release
Posted by Byung-Gon Chun <bg...@gmail.com>.
+1 on having a set of cluster tests to verify a release candidate.
On Tue, Dec 12, 2017 at 10:30 AM, Markus Weimer <ma...@weimo.de> wrote:
> +1
>
> It would be good to use the new test framework for those tests. Once I am
> done with it :)
>
> Markus
>
> On Mon, Dec 11, 2017 at 1:05 PM, Julia Wang (QIUHE) <
> Qiuhe.Wang@microsoft.com.invalid> wrote:
>
> > Good suggestion! If don't have time to modify all the YARN tests, we
> > should at least have a list of yarn must pass tests and run them
> frequently.
> >
> > -Julia
> >
> > -----Original Message-----
> > From: Mariia Mykhailova [mailto:mamykhai@microsoft.com.INVALID]
> > Sent: Monday, December 11, 2017 12:46 PM
> > To: dev@reef.apache.org
> > Subject: List of special tests for release
> >
> > Hi,
> >
> > REEF has features which are not covered by testing on local runtime and
> > the few examples we run manually when testing release in clusters.
> > One example of such a feature is driver restart on .NET side - AFAIK it
> is
> > only covered in O.A.R.Examples.DriverRestart and requires a cluster with
> > special YARN settings to run (a comment in the code says "Can only be run
> > on HDInsight clusters with yarn.resourcemanager.am.max-attempts set to
> > greater than or equal to 2."). This feature was broken and went unnoticed
> > for several months. I'm sure there are other features like this which are
> > not documented enough for the release manager to be aware of them (and
> more
> > importantly of how to verify that they still work).
> >
> > What do you think about putting together a list of such features and the
> > process of their testing to follow during the release? This way we'd
> have a
> > checklist for the release manager to verify testers' reports against
> rather
> > than rely on the authors of all individual features to remember to test
> > them and report back.
> >
> > -Mariia
> >
> >
>
--
Byung-Gon Chun
Re: List of special tests for release
Posted by Markus Weimer <ma...@weimo.de>.
+1
It would be good to use the new test framework for those tests. Once I am
done with it :)
Markus
On Mon, Dec 11, 2017 at 1:05 PM, Julia Wang (QIUHE) <
Qiuhe.Wang@microsoft.com.invalid> wrote:
> Good suggestion! If don't have time to modify all the YARN tests, we
> should at least have a list of yarn must pass tests and run them frequently.
>
> -Julia
>
> -----Original Message-----
> From: Mariia Mykhailova [mailto:mamykhai@microsoft.com.INVALID]
> Sent: Monday, December 11, 2017 12:46 PM
> To: dev@reef.apache.org
> Subject: List of special tests for release
>
> Hi,
>
> REEF has features which are not covered by testing on local runtime and
> the few examples we run manually when testing release in clusters.
> One example of such a feature is driver restart on .NET side - AFAIK it is
> only covered in O.A.R.Examples.DriverRestart and requires a cluster with
> special YARN settings to run (a comment in the code says "Can only be run
> on HDInsight clusters with yarn.resourcemanager.am.max-attempts set to
> greater than or equal to 2."). This feature was broken and went unnoticed
> for several months. I'm sure there are other features like this which are
> not documented enough for the release manager to be aware of them (and more
> importantly of how to verify that they still work).
>
> What do you think about putting together a list of such features and the
> process of their testing to follow during the release? This way we'd have a
> checklist for the release manager to verify testers' reports against rather
> than rely on the authors of all individual features to remember to test
> them and report back.
>
> -Mariia
>
>
RE: List of special tests for release
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com.INVALID>.
Good suggestion! If don't have time to modify all the YARN tests, we should at least have a list of yarn must pass tests and run them frequently.
-Julia
-----Original Message-----
From: Mariia Mykhailova [mailto:mamykhai@microsoft.com.INVALID]
Sent: Monday, December 11, 2017 12:46 PM
To: dev@reef.apache.org
Subject: List of special tests for release
Hi,
REEF has features which are not covered by testing on local runtime and the few examples we run manually when testing release in clusters.
One example of such a feature is driver restart on .NET side - AFAIK it is only covered in O.A.R.Examples.DriverRestart and requires a cluster with special YARN settings to run (a comment in the code says "Can only be run on HDInsight clusters with yarn.resourcemanager.am.max-attempts set to greater than or equal to 2."). This feature was broken and went unnoticed for several months. I'm sure there are other features like this which are not documented enough for the release manager to be aware of them (and more importantly of how to verify that they still work).
What do you think about putting together a list of such features and the process of their testing to follow during the release? This way we'd have a checklist for the release manager to verify testers' reports against rather than rely on the authors of all individual features to remember to test them and report back.
-Mariia