You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Michal Michalski <mi...@boxever.com> on 2015/04/28 17:32:32 UTC
Best practices on testing Spark jobs
Hi,
I have two questions regarding testing Spark jobs:
1. Is it possible to use Mockito for that purpose? I tried to use it, but
it looks like there are no interactions with mocks. I didn't dive into the
details of how Mockito works, but I guess it might be because of the
serialization and how Spark distributes tasks. I'm not sure about it though
and I'm looking for confirmation.
2. If not mockito, what's the alternative? What's the recommended way to
test Spark jobs? Should I manually create mocks by e.g. extending all the
classes I'd normally mock and changing the implementation of some methods?
I don't like this idea but I can't really see any other options now.
Kind regards,
Michał Michalski,
michal.michalski@boxever.com
Re: Best practices on testing Spark jobs
Posted by Sourav Chandra <so...@livestream.com>.
Hi,
Can you give some tutorials/examples how to write test case based on the
mentioned framework?
Thanks,
Sourav
On Tue, Apr 28, 2015 at 9:22 PM, Silvio Fiorito <
silvio.fiorito@granturing.com> wrote:
> Sorry that’s correct, I was thinking you were maybe trying to mock
> certain aspects of Spark core to write your tests. This is a library to
> help write unit tests by managing the SparkContext and StreamingContext. So
> you can test your transformations as necessary. More importantly on the
> streaming side it really helps simplify running tests on batch outputs.
>
> If you’re having serialization issues you may need to look at using
> transient lazy initializers, to see if that helps?
>
> From: Michal Michalski
> Date: Tuesday, April 28, 2015 at 11:42 AM
> To: Silvio Fiorito
> Cc: user
> Subject: Re: Best practices on testing Spark jobs
>
> Thanks Silvio. I might be missing something, but it looks like this
> project is a kind of a "framework" for setting up Spark for a testing, but
> after taking a quick look at the code it doesn't seem like it's solving the
> problem with mocking which is my main concern now - am I wrong?
>
> Kind regards,
> Michał Michalski,
> michal.michalski@boxever.com
>
> On 28 April 2015 at 16:35, Silvio Fiorito <si...@granturing.com>
> wrote:
>
>> Hi Michal,
>>
>> Please try spark-testing-base by Holden. I’ve used it and it works well
>> for unit testing batch and streaming jobs
>>
>> https://github.com/holdenk/spark-testing-base
>>
>> Thanks,
>> Silvio
>>
>> From: Michal Michalski
>> Date: Tuesday, April 28, 2015 at 11:32 AM
>> To: user
>> Subject: Best practices on testing Spark jobs
>>
>> Hi,
>>
>> I have two questions regarding testing Spark jobs:
>>
>> 1. Is it possible to use Mockito for that purpose? I tried to use it,
>> but it looks like there are no interactions with mocks. I didn't dive into
>> the details of how Mockito works, but I guess it might be because of the
>> serialization and how Spark distributes tasks. I'm not sure about it though
>> and I'm looking for confirmation.
>>
>> 2. If not mockito, what's the alternative? What's the recommended way
>> to test Spark jobs? Should I manually create mocks by e.g. extending all
>> the classes I'd normally mock and changing the implementation of some
>> methods? I don't like this idea but I can't really see any other options
>> now.
>>
>>
>> Kind regards,
>> Michał Michalski,
>> michal.michalski@boxever.com
>>
>
>
--
Sourav Chandra
Senior Software Engineer
· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
sourav.chandra@livestream.com
o: +91 80 4121 8723
m: +91 988 699 3746
skype: sourav.chandra
Livestream
"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
Block, Koramangala Industrial Area,
Bangalore 560034
www.livestream.com
Re: Best practices on testing Spark jobs
Posted by Silvio Fiorito <si...@granturing.com>.
Sorry that’s correct, I was thinking you were maybe trying to mock certain aspects of Spark core to write your tests. This is a library to help write unit tests by managing the SparkContext and StreamingContext. So you can test your transformations as necessary. More importantly on the streaming side it really helps simplify running tests on batch outputs.
If you’re having serialization issues you may need to look at using transient lazy initializers, to see if that helps?
From: Michal Michalski
Date: Tuesday, April 28, 2015 at 11:42 AM
To: Silvio Fiorito
Cc: user
Subject: Re: Best practices on testing Spark jobs
Thanks Silvio. I might be missing something, but it looks like this project is a kind of a "framework" for setting up Spark for a testing, but after taking a quick look at the code it doesn't seem like it's solving the problem with mocking which is my main concern now - am I wrong?
Kind regards,
Michał Michalski,
michal.michalski@boxever.com<ma...@boxever.com>
On 28 April 2015 at 16:35, Silvio Fiorito <si...@granturing.com>> wrote:
Hi Michal,
Please try spark-testing-base by Holden. I’ve used it and it works well for unit testing batch and streaming jobs
https://github.com/holdenk/spark-testing-base
Thanks,
Silvio
From: Michal Michalski
Date: Tuesday, April 28, 2015 at 11:32 AM
To: user
Subject: Best practices on testing Spark jobs
Hi,
I have two questions regarding testing Spark jobs:
1. Is it possible to use Mockito for that purpose? I tried to use it, but it looks like there are no interactions with mocks. I didn't dive into the details of how Mockito works, but I guess it might be because of the serialization and how Spark distributes tasks. I'm not sure about it though and I'm looking for confirmation.
2. If not mockito, what's the alternative? What's the recommended way to test Spark jobs? Should I manually create mocks by e.g. extending all the classes I'd normally mock and changing the implementation of some methods? I don't like this idea but I can't really see any other options now.
Kind regards,
Michał Michalski,
michal.michalski@boxever.com<ma...@boxever.com>
Re: Best practices on testing Spark jobs
Posted by Michal Michalski <mi...@boxever.com>.
Thanks Silvio. I might be missing something, but it looks like this project
is a kind of a "framework" for setting up Spark for a testing, but after
taking a quick look at the code it doesn't seem like it's solving the
problem with mocking which is my main concern now - am I wrong?
Kind regards,
Michał Michalski,
michal.michalski@boxever.com
On 28 April 2015 at 16:35, Silvio Fiorito <si...@granturing.com>
wrote:
> Hi Michal,
>
> Please try spark-testing-base by Holden. I’ve used it and it works well
> for unit testing batch and streaming jobs
>
> https://github.com/holdenk/spark-testing-base
>
> Thanks,
> Silvio
>
> From: Michal Michalski
> Date: Tuesday, April 28, 2015 at 11:32 AM
> To: user
> Subject: Best practices on testing Spark jobs
>
> Hi,
>
> I have two questions regarding testing Spark jobs:
>
> 1. Is it possible to use Mockito for that purpose? I tried to use it,
> but it looks like there are no interactions with mocks. I didn't dive into
> the details of how Mockito works, but I guess it might be because of the
> serialization and how Spark distributes tasks. I'm not sure about it though
> and I'm looking for confirmation.
>
> 2. If not mockito, what's the alternative? What's the recommended way to
> test Spark jobs? Should I manually create mocks by e.g. extending all the
> classes I'd normally mock and changing the implementation of some methods?
> I don't like this idea but I can't really see any other options now.
>
>
> Kind regards,
> Michał Michalski,
> michal.michalski@boxever.com
>
Re: Best practices on testing Spark jobs
Posted by Silvio Fiorito <si...@granturing.com>.
Hi Michal,
Please try spark-testing-base by Holden. I’ve used it and it works well for unit testing batch and streaming jobs
https://github.com/holdenk/spark-testing-base
Thanks,
Silvio
From: Michal Michalski
Date: Tuesday, April 28, 2015 at 11:32 AM
To: user
Subject: Best practices on testing Spark jobs
Hi,
I have two questions regarding testing Spark jobs:
1. Is it possible to use Mockito for that purpose? I tried to use it, but it looks like there are no interactions with mocks. I didn't dive into the details of how Mockito works, but I guess it might be because of the serialization and how Spark distributes tasks. I'm not sure about it though and I'm looking for confirmation.
2. If not mockito, what's the alternative? What's the recommended way to test Spark jobs? Should I manually create mocks by e.g. extending all the classes I'd normally mock and changing the implementation of some methods? I don't like this idea but I can't really see any other options now.
Kind regards,
Michał Michalski,
michal.michalski@boxever.com<ma...@boxever.com>