Fast Unit Tests

[marcos rebelo <ol...@gmail.com>]

Hey all,

We are using Scala and SQL heavily, but I have a problem with VERY SLOW
Unit Tests.

Is there a way to do fast Unit Tests on Spark?

How are you guys going around it?

Best Regards
Marcos Rebelo

Re: Fast Unit Tests

[Geoff Von Allmen <ge...@ibleducation.com>]

I am pretty new to spark/scala myself, but I just recently implemented unit
tests to test my transformations/aggregations and such myself.

I’m using the mrpowers spark-fast-tests
<https://github.com/MrPowers/spark-fast-tests> and spark-daria
<https://github.com/MrPowers/spark-daria> libraries.

I am also using a JDBC sink in the foreach writer. I’ve mocked the sink to
place the generated MYSQL statements in a global object and then I compare
the output there to an expected set of mysql statements.

I’m running this with FlatSpec ScalaTests, where my spark inputs are
manually generated fixtures for each test case. Everything seems to be
running well and it's nice and quick.
​

On Tue, May 1, 2018 at 8:25 AM, marcos rebelo <ol...@gmail.com> wrote:

> Hey all,
>
> We are using Scala and SQL heavily, but I have a problem with VERY SLOW
> Unit Tests.
>
> Is there a way to do fast Unit Tests on Spark?
>
> How are you guys going around it?
>
> Best Regards
> Marcos Rebelo
>

RE: Fast Unit Tests

[Yeikel Santana <em...@yeikel.com>]

Can you share a sample test case? How are you doing the unit tests? Are you creating the session in a beforeAll block or similar? 

 

As far as I know, if you use spark you will end up with light integration tests rather than “real” unit tests (please correct me if I am wrong). 

 

From: marcos rebelo <ol...@gmail.com> 
Sent: Tuesday, May 1, 2018 11:25 AM
To: user <us...@spark.apache.org>
Subject: Fast Unit Tests

 

Hey all,

 

We are using Scala and SQL heavily, but I have a problem with VERY SLOW Unit Tests. 

 

Is there a way to do fast Unit Tests on Spark? 

 

How are you guys going around it?

 

Best Regards

Marcos Rebelo