You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Naveen Madhire <vm...@umail.iu.edu> on 2015/07/10 19:41:55 UTC
Unit tests of spark application
Hi,
I want to write junit test cases in scala for testing spark application. Is
there any guide or link which I can refer.
Thank you very much.
-Naveen
Re: Unit tests of spark application
Posted by Naveen Madhire <vm...@umail.iu.edu>.
Thanks. Spark-testing-base works pretty well.
On Fri, Jul 10, 2015 at 3:23 PM, Burak Yavuz <br...@gmail.com> wrote:
> I can +1 Holden's spark-testing-base package.
>
> Burak
>
> On Fri, Jul 10, 2015 at 12:23 PM, Holden Karau <ho...@pigscanfly.ca>
> wrote:
>
>> Somewhat biased of course, but you can also use spark-testing-base from
>> spark-packages.org as a basis for your unittests.
>>
>> On Fri, Jul 10, 2015 at 12:03 PM, Daniel Siegmann <
>> daniel.siegmann@teamaol.com> wrote:
>>
>>> On Fri, Jul 10, 2015 at 1:41 PM, Naveen Madhire <vm...@umail.iu.edu>
>>> wrote:
>>>
>>>> I want to write junit test cases in scala for testing spark
>>>> application. Is there any guide or link which I can refer.
>>>>
>>>
>>> https://spark.apache.org/docs/latest/programming-guide.html#unit-testing
>>>
>>> Typically I create test data using SparkContext.parallelize and then
>>> call RDD.collect to get the results to assert.
>>>
>>
>>
>>
>> --
>> Cell : 425-233-8271
>> Twitter: https://twitter.com/holdenkarau
>> Linked In: https://www.linkedin.com/in/holdenkarau
>>
>
>
Re: Unit tests of spark application
Posted by Burak Yavuz <br...@gmail.com>.
I can +1 Holden's spark-testing-base package.
Burak
On Fri, Jul 10, 2015 at 12:23 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
> Somewhat biased of course, but you can also use spark-testing-base from
> spark-packages.org as a basis for your unittests.
>
> On Fri, Jul 10, 2015 at 12:03 PM, Daniel Siegmann <
> daniel.siegmann@teamaol.com> wrote:
>
>> On Fri, Jul 10, 2015 at 1:41 PM, Naveen Madhire <vm...@umail.iu.edu>
>> wrote:
>>
>>> I want to write junit test cases in scala for testing spark application.
>>> Is there any guide or link which I can refer.
>>>
>>
>> https://spark.apache.org/docs/latest/programming-guide.html#unit-testing
>>
>> Typically I create test data using SparkContext.parallelize and then
>> call RDD.collect to get the results to assert.
>>
>
>
>
> --
> Cell : 425-233-8271
> Twitter: https://twitter.com/holdenkarau
> Linked In: https://www.linkedin.com/in/holdenkarau
>
Re: Unit tests of spark application
Posted by Holden Karau <ho...@pigscanfly.ca>.
Somewhat biased of course, but you can also use spark-testing-base from
spark-packages.org as a basis for your unittests.
On Fri, Jul 10, 2015 at 12:03 PM, Daniel Siegmann <
daniel.siegmann@teamaol.com> wrote:
> On Fri, Jul 10, 2015 at 1:41 PM, Naveen Madhire <vm...@umail.iu.edu>
> wrote:
>
>> I want to write junit test cases in scala for testing spark application.
>> Is there any guide or link which I can refer.
>>
>
> https://spark.apache.org/docs/latest/programming-guide.html#unit-testing
>
> Typically I create test data using SparkContext.parallelize and then call
> RDD.collect to get the results to assert.
>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
Linked In: https://www.linkedin.com/in/holdenkarau
Re: Unit tests of spark application
Posted by Daniel Siegmann <da...@teamaol.com>.
On Fri, Jul 10, 2015 at 1:41 PM, Naveen Madhire <vm...@umail.iu.edu>
wrote:
> I want to write junit test cases in scala for testing spark application.
> Is there any guide or link which I can refer.
>
https://spark.apache.org/docs/latest/programming-guide.html#unit-testing
Typically I create test data using SparkContext.parallelize and then call
RDD.collect to get the results to assert.
Re: Unit tests of spark application
Posted by Richard Marscher <rm...@localytics.com>.
Unless you had something specific in mind, it should be as simple as
creating a SparkContext object using a master of local[2] in your tests
On Fri, Jul 10, 2015 at 1:41 PM, Naveen Madhire <vm...@umail.iu.edu>
wrote:
> Hi,
>
> I want to write junit test cases in scala for testing spark application.
> Is there any guide or link which I can refer.
>
> Thank you very much.
>
> -Naveen
>
>
>
>
--
--
*Richard Marscher*
Software Engineer
Localytics
Localytics.com <http://localytics.com/> | Our Blog
<http://localytics.com/blog> | Twitter <http://twitter.com/localytics> |
Facebook <http://facebook.com/localytics> | LinkedIn
<http://www.linkedin.com/company/1148792?trk=tyah>