You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by James King <ja...@gmail.com> on 2015/04/21 13:26:39 UTC

Spark Unit Testing

I'm trying to write some unit tests for my spark code.

I need to pass a JavaPairDStream<String, String> to my spark class.

Is there a way to create a JavaPairDStream using Java API?

Also is there a good resource that covers an approach (or approaches) for
unit testing using Java.

Regards
jk

Re: Spark Unit Testing

Posted by James King <ja...@gmail.com>.

Hi Emre, thanks for the help will have a look. Cheers!

On Tue, Apr 21, 2015 at 1:46 PM, Emre Sevinc <em...@gmail.com> wrote:

> Hello James,
>
> Did you check the following resources:
>
>  -
> https://github.com/apache/spark/tree/master/streaming/src/test/java/org/apache/spark/streaming
>
>  -
> http://www.slideshare.net/databricks/strata-sj-everyday-im-shuffling-tips-for-writing-better-spark-programs
>
> --
> Emre Sevinç
> http://www.bigindustries.be/
>
>
> On Tue, Apr 21, 2015 at 1:26 PM, James King <ja...@gmail.com> wrote:
>
>> I'm trying to write some unit tests for my spark code.
>>
>> I need to pass a JavaPairDStream<String, String> to my spark class.
>>
>> Is there a way to create a JavaPairDStream using Java API?
>>
>> Also is there a good resource that covers an approach (or approaches) for
>> unit testing using Java.
>>
>> Regards
>> jk
>>
>
>
>
> --
> Emre Sevinc
>

Re: Spark Unit Testing

Posted by Emre Sevinc <em...@gmail.com>.

Hello James,

Did you check the following resources:

 -
https://github.com/apache/spark/tree/master/streaming/src/test/java/org/apache/spark/streaming

 -
http://www.slideshare.net/databricks/strata-sj-everyday-im-shuffling-tips-for-writing-better-spark-programs

--
Emre Sevinç
http://www.bigindustries.be/


On Tue, Apr 21, 2015 at 1:26 PM, James King <ja...@gmail.com> wrote:

> I'm trying to write some unit tests for my spark code.
>
> I need to pass a JavaPairDStream<String, String> to my spark class.
>
> Is there a way to create a JavaPairDStream using Java API?
>
> Also is there a good resource that covers an approach (or approaches) for
> unit testing using Java.
>
> Regards
> jk
>



-- 
Emre Sevinc