You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Markus Weimer <ma...@weimo.de> on 2015/08/10 17:25:38 UTC
[DISCUSS] Move to NUnit?
Hi,
MSTest is giving us a few issues. Most notably, it is very tied to
Visual Studio, which makes testing on machines without it impossible.
That hits us for instance when we try to run unit tests on YARN
clusters. Hence, maybe we should switch to another unit test framework.
NUnit comes to mind.
However, I have next to no experience with it (or MSTest, to be honest).
Does anyone on the list have experience? Would NUnit allow us to run
tests without having to install things on every cluster node?
Thanks,
Markus
RE: [DISCUSS] Move to NUnit?
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com>.
Shall we try the converted tests on YARN? I think that is one of the major motivations we would want to convert, right?
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Thursday, August 13, 2015 8:02 AM
To: dev@reef.incubator.apache.org
Subject: Re: [DISCUSS] Move to NUnit?
On 2015-08-10 17:44, Anupam wrote:
> I quickly converted Org.Apache.REEF.Tang.Tests from MSTest to NUnit
> locally and tried on a machine without Visual Studio and it worked.
Awesome! So, moving to NUnit wouldn't be hard and it would fix the reliance on Visual Studio for running our tests. Shall we move forward with that, then?
On a related note: While browsing the .NET repositories[0], I saw that their code formatter[1] has a feature to automatically convert tests to xUnit. The xUnit page[2] describes it as the successor to NUnit. Which makes me wonder: If .NET uses it, should we, too?
Markus
[0]: https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fdotnet&data=01%7c01%7cQiuhe.Wang%40microsoft.com%7c8ffb34f0b72a44a979b408d2a408111f%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=eqoWG1oxp5iSwWTwKGKp2LkjJccHuIRrYs4xbF%2fUXvo%3d
[1]: https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fdotnet%2fcodeformatter&data=01%7c01%7cQiuhe.Wang%40microsoft.com%7c8ffb34f0b72a44a979b408d2a408111f%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=ODTybk%2fybQf%2fRJJn6%2bDI24zNeyyxZdk3qXKbfLp6Lh8%3d
[2]:
Re: [DISCUSS] Move to NUnit?
Posted by Markus Weimer <ma...@weimo.de>.
Hi,
Thanks for the analysis!
I think we can elimate MSTest from this list as it can't be automated in
ways we care about. Namely, we won't be able to have Visual Studio
installed on all YARN clusters.
This leaves NUnit and XUnit. The latter seems like the better place to
be long term, right? It is easier to automate. Also, it is the "chosen
one" by the .NET team.
The downside is the one-time cost of converting to it, right? Is that
feasible for us to do? Maybe we can do it one test assembly at a time?
Markus
On 2015-08-28 13:12, Anupam wrote:
> Re-sending the table as plain text.
>
> VISUAL STUDIO SUPPORT
> MSTEST: Native
> NUNIT: With ReSharper natively. Standalone plugin.
> XUNIT: With ReSharper plugin. Or standalone plugin.
>
> STANDALONE RUNNER
> MSTEST: Needs Visual Studio
> NUNIT: Self-contained
> XUNIT: Self-contained
>
> RUNNING ON BUILD SERVER
> MSTEST: Needs Visual Studio
> NUNIT: Needs NUnit binaries
> XUNIT: Runner packaged in NuGet
>
> MIGRATION
> MSTEST: None; Currently in use
> NUNIT: Trivial find-replace. DeployItem support.
> XUNIT: Trivial but hard to automate.
>
>
> On 28 August 2015 at 11:23, Anupam <an...@gmail.com> wrote:
>
>> After looking around a bit more.
>>
>>
>>
>> *MSTest*
>>
>> *NUnit*
>>
>> *xUnit*
>>
>> *Visual Studio Support*
>>
>> Native
>>
>> With ReSharper natively. Standalone plugin.
>>
>> With ReSharper plugin. Or standalone plugin.
>>
>> *Standalone runner*
>>
>> Needs Visual Studio
>>
>> Self-contained
>>
>> Self-contained
>>
>> *Running on Build Server*
>>
>> Needs Visual Studio
>>
>> Needs NUnit binaries
>>
>> Runner packaged in NuGet
>>
>> *Migration*
>>
>> None: Currently in use
>>
>> Trivial find-replace. DeployItem support.
>>
>> Trivial but hard to automate.
>>
>> My opinion is to move to XUnit as it paves an easy path towards full test
>> automation.
>>
>>
>> On 15 August 2015 at 08:47, Markus Weimer <ma...@weimo.de> wrote:
>>
>>> On 2015-08-14 23:11, Brian Cho wrote:
>>>> Does removing the dependency to Visual Studio mean that we could
>>>> potentially run .NET tests as part of Apache Jenkins CI? That would
>>>> be pretty awesome. Or are there other prerequisites that aren’t met
>>>> here?
>>>
>>> Yes, that would be one potential consequence. The .NET Foundation has
>>> some tools for exactly that:
>>>
>>> https://github.com/dotnet/dotnet-ci
>>>
>>> However, they seem to use xUnit, not NUnit. Shouldn't make much of a
>>> difference, though.
>>>
>>> Markus
>>>
>>
>>
>>
>> --
>> Anupam
>> Bellevue, WA
>> Ph: +1 (425)-777-5570
>>
>
>
>
Re: [DISCUSS] Move to NUnit?
Posted by Anupam <an...@gmail.com>.
Re-sending the table as plain text.
VISUAL STUDIO SUPPORT
MSTEST: Native
NUNIT: With ReSharper natively. Standalone plugin.
XUNIT: With ReSharper plugin. Or standalone plugin.
STANDALONE RUNNER
MSTEST: Needs Visual Studio
NUNIT: Self-contained
XUNIT: Self-contained
RUNNING ON BUILD SERVER
MSTEST: Needs Visual Studio
NUNIT: Needs NUnit binaries
XUNIT: Runner packaged in NuGet
MIGRATION
MSTEST: None; Currently in use
NUNIT: Trivial find-replace. DeployItem support.
XUNIT: Trivial but hard to automate.
On 28 August 2015 at 11:23, Anupam <an...@gmail.com> wrote:
> After looking around a bit more.
>
>
>
> *MSTest*
>
> *NUnit*
>
> *xUnit*
>
> *Visual Studio Support*
>
> Native
>
> With ReSharper natively. Standalone plugin.
>
> With ReSharper plugin. Or standalone plugin.
>
> *Standalone runner*
>
> Needs Visual Studio
>
> Self-contained
>
> Self-contained
>
> *Running on Build Server*
>
> Needs Visual Studio
>
> Needs NUnit binaries
>
> Runner packaged in NuGet
>
> *Migration*
>
> None: Currently in use
>
> Trivial find-replace. DeployItem support.
>
> Trivial but hard to automate.
>
> My opinion is to move to XUnit as it paves an easy path towards full test
> automation.
>
>
> On 15 August 2015 at 08:47, Markus Weimer <ma...@weimo.de> wrote:
>
>> On 2015-08-14 23:11, Brian Cho wrote:
>> > Does removing the dependency to Visual Studio mean that we could
>> > potentially run .NET tests as part of Apache Jenkins CI? That would
>> > be pretty awesome. Or are there other prerequisites that aren’t met
>> > here?
>>
>> Yes, that would be one potential consequence. The .NET Foundation has
>> some tools for exactly that:
>>
>> https://github.com/dotnet/dotnet-ci
>>
>> However, they seem to use xUnit, not NUnit. Shouldn't make much of a
>> difference, though.
>>
>> Markus
>>
>
>
>
> --
> Anupam
> Bellevue, WA
> Ph: +1 (425)-777-5570
>
--
Anupam
Bellevue, WA
Ph: +1 (425)-777-5570
Re: [DISCUSS] Move to NUnit?
Posted by Anupam <an...@gmail.com>.
After looking around a bit more.
*MSTest*
*NUnit*
*xUnit*
*Visual Studio Support*
Native
With ReSharper natively. Standalone plugin.
With ReSharper plugin. Or standalone plugin.
*Standalone runner*
Needs Visual Studio
Self-contained
Self-contained
*Running on Build Server*
Needs Visual Studio
Needs NUnit binaries
Runner packaged in NuGet
*Migration*
None: Currently in use
Trivial find-replace. DeployItem support.
Trivial but hard to automate.
My opinion is to move to XUnit as it paves an easy path towards full test
automation.
On 15 August 2015 at 08:47, Markus Weimer <ma...@weimo.de> wrote:
> On 2015-08-14 23:11, Brian Cho wrote:
> > Does removing the dependency to Visual Studio mean that we could
> > potentially run .NET tests as part of Apache Jenkins CI? That would
> > be pretty awesome. Or are there other prerequisites that aren’t met
> > here?
>
> Yes, that would be one potential consequence. The .NET Foundation has
> some tools for exactly that:
>
> https://github.com/dotnet/dotnet-ci
>
> However, they seem to use xUnit, not NUnit. Shouldn't make much of a
> difference, though.
>
> Markus
>
--
Anupam
Bellevue, WA
Ph: +1 (425)-777-5570
Re: [DISCUSS] Move to NUnit?
Posted by Markus Weimer <ma...@weimo.de>.
On 2015-08-14 23:11, Brian Cho wrote:
> Does removing the dependency to Visual Studio mean that we could
> potentially run .NET tests as part of Apache Jenkins CI? That would
> be pretty awesome. Or are there other prerequisites that aren’t met
> here?
Yes, that would be one potential consequence. The .NET Foundation has
some tools for exactly that:
https://github.com/dotnet/dotnet-ci
However, they seem to use xUnit, not NUnit. Shouldn't make much of a
difference, though.
Markus
RE: [DISCUSS] Move to NUnit?
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com>.
Depending on what is required by NUnit. Today, to run .Net testing, we must have VS library installed on the machine.
Julia
-----Original Message-----
From: Brian Cho [mailto:chobrian@gmail.com]
Sent: Friday, August 14, 2015 11:11 PM
To: dev@reef.incubator.apache.org
Subject: Re: [DISCUSS] Move to NUnit?
Does removing the dependency to Visual Studio mean that we could potentially run .NET tests as part of Apache Jenkins CI? That would be pretty awesome. Or are there other prerequisites that aren’t met here?
-Brian
On Fri, Aug 14, 2015 at 12:02 AM, Markus Weimer <ma...@weimo.de> wrote:
> On 2015-08-10 17:44, Anupam wrote:
> > I quickly converted Org.Apache.REEF.Tang.Tests from MSTest to NUnit
> locally
> > and tried on a machine without Visual Studio and it worked.
>
> Awesome! So, moving to NUnit wouldn't be hard and it would fix the
> reliance on Visual Studio for running our tests. Shall we move forward
> with that, then?
>
> On a related note: While browsing the .NET repositories[0], I saw that
> their code formatter[1] has a feature to automatically convert tests
> to xUnit. The xUnit page[2] describes it as the successor to NUnit.
> Which makes me wonder: If .NET uses it, should we, too?
>
> Markus
>
> [0]:
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithu
> b.com%2fdotnet&data=01%7c01%7cQiuhe.Wang%40microsoft.com%7cdf4d97cdb2d
> 749e63cac08d2a5385da7%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=JZM
> AjJtIMFSoNkY0x8T99XIVSd%2bf3DTjetRF6qDAcHU%3d
> [1]:
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithu
> b.com%2fdotnet%2fcodeformatter&data=01%7c01%7cQiuhe.Wang%40microsoft.c
> om%7cdf4d97cdb2d749e63cac08d2a5385da7%7c72f988bf86f141af91ab2d7cd011db
> 47%7c1&sdata=ZCtIuvpwmeUBNoY15cSGHE3oDwS3lTey8hFKeR%2fKYmA%3d
> [2]:
>
Re: [DISCUSS] Move to NUnit?
Posted by Brian Cho <ch...@gmail.com>.
Does removing the dependency to Visual Studio mean that we could
potentially run .NET tests as part of Apache Jenkins CI? That would be
pretty awesome. Or are there other prerequisites that aren’t met here?
-Brian
On Fri, Aug 14, 2015 at 12:02 AM, Markus Weimer <ma...@weimo.de> wrote:
> On 2015-08-10 17:44, Anupam wrote:
> > I quickly converted Org.Apache.REEF.Tang.Tests from MSTest to NUnit
> locally
> > and tried on a machine without Visual Studio and it worked.
>
> Awesome! So, moving to NUnit wouldn't be hard and it would fix the
> reliance on Visual Studio for running our tests. Shall we move forward
> with that, then?
>
> On a related note: While browsing the .NET repositories[0], I saw that
> their code formatter[1] has a feature to automatically convert tests to
> xUnit. The xUnit page[2] describes it as the successor to NUnit. Which
> makes me wonder: If .NET uses it, should we, too?
>
> Markus
>
> [0]: https://github.com/dotnet
> [1]: https://github.com/dotnet/codeformatter
> [2]:
>
Re: [DISCUSS] Move to NUnit?
Posted by Markus Weimer <ma...@weimo.de>.
On 2015-08-10 17:44, Anupam wrote:
> I quickly converted Org.Apache.REEF.Tang.Tests from MSTest to NUnit locally
> and tried on a machine without Visual Studio and it worked.
Awesome! So, moving to NUnit wouldn't be hard and it would fix the
reliance on Visual Studio for running our tests. Shall we move forward
with that, then?
On a related note: While browsing the .NET repositories[0], I saw that
their code formatter[1] has a feature to automatically convert tests to
xUnit. The xUnit page[2] describes it as the successor to NUnit. Which
makes me wonder: If .NET uses it, should we, too?
Markus
[0]: https://github.com/dotnet
[1]: https://github.com/dotnet/codeformatter
[2]:
Re: [DISCUSS] Move to NUnit?
Posted by Anupam <an...@gmail.com>.
*>Would NUnit allow us to run tests without having to install things on
every cluster node?*
I quickly converted Org.Apache.REEF.Tang.Tests from MSTest to NUnit locally
and tried on a machine without Visual Studio and it worked. I was able to
convert all but one unittest in this project with simple text replacement.
I chose this project because it contains almost 3/4th of the unittests in
the .NET solution. It should be extremely trivial to switch *most* of the
tests to run with NUnit. The only tests that might need some effort are the
tests that use DeploymentItem attribute, but that has workarounds.
On 10 August 2015 at 16:45, Markus Weimer <ma...@weimo.de> wrote:
> On 2015-08-10 16:29, Byung-Gon Chun wrote:
> > I don't understand your concern: running tests without having to
> > install things on every cluster node.
>
> Allow me to expand:
>
> When we run the Java tests on YARN, all we need to do is copy a JAR to
> the cluster and run the command line wrapped in `runtests.ps1` and its
> Bash equivalent. Crucially, JUnit is part of the JAR we copy. Hence, the
> unit tests can be driven from whichever node we copy the JAR to.
>
> For driving the .NET tests, we need Visual Studio, as the actual test
> runner is bundled with it. Stand-alone MSTest doesn't cut it, as that
> doesn't support 64Bit projects. And all of REEF is 64Bit only in order
> to link with the 64Bit JVM.
>
> Markus
>
--
Anupam
Bellevue, WA
Ph: +1 (425)-777-5570
Re: [DISCUSS] Move to NUnit?
Posted by Markus Weimer <ma...@weimo.de>.
On 2015-08-10 16:29, Byung-Gon Chun wrote:
> I don't understand your concern: running tests without having to
> install things on every cluster node.
Allow me to expand:
When we run the Java tests on YARN, all we need to do is copy a JAR to
the cluster and run the command line wrapped in `runtests.ps1` and its
Bash equivalent. Crucially, JUnit is part of the JAR we copy. Hence, the
unit tests can be driven from whichever node we copy the JAR to.
For driving the .NET tests, we need Visual Studio, as the actual test
runner is bundled with it. Stand-alone MSTest doesn't cut it, as that
doesn't support 64Bit projects. And all of REEF is 64Bit only in order
to link with the 64Bit JVM.
Markus
Re: [DISCUSS] Move to NUnit?
Posted by Byung-Gon Chun <bg...@gmail.com>.
Hi,
I have used NUNit. NUnit works like JUnit. If someone's familiar with
JUnit, there's little learning cost.
I don't understand your concern: running tests without having to install
things on every cluster node.
Thanks,
Gon
On Tue, Aug 11, 2015 at 12:25 AM, Markus Weimer <ma...@weimo.de> wrote:
> Hi,
>
> MSTest is giving us a few issues. Most notably, it is very tied to
> Visual Studio, which makes testing on machines without it impossible.
> That hits us for instance when we try to run unit tests on YARN
> clusters. Hence, maybe we should switch to another unit test framework.
> NUnit comes to mind.
>
> However, I have next to no experience with it (or MSTest, to be honest).
> Does anyone on the list have experience? Would NUnit allow us to run
> tests without having to install things on every cluster node?
>
> Thanks,
>
> Markus
>
--
Byung-Gon Chun