You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mark Vervuurt <m....@gmail.com> on 2015/10/21 12:16:30 UTC
Spark-Testing-Base Q/A
Hi Everyone,
I am busy trying out ‘Spark-Testing-Base <https://github.com/holdenk/spark-testing-base>’. I have the following questions?
Can you test Spark Streaming Jobs using Java?
Can I use Spark-Testing-Base 1.3.0_0.1.1 together with Spark 1.3.1?
Thanks.
Greetings,
Mark
Re: Spark-Testing-Base Q/A
Posted by Holden Karau <ho...@pigscanfly.ca>.
And now (before 1am California time :p) there is a new version of
spark-testing base which adds a java base class for streaming tests. I
noticed you were using 1.3 so I put in the effort to make this release for
Spark 1.3 to 1.5 inclusive).
On Wed, Oct 21, 2015 at 4:16 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
>
>
> On Wednesday, October 21, 2015, Mark Vervuurt <m....@gmail.com>
> wrote:
>
>> Hi Holden,
>>
>> Thanks for the information, I think that a Java Base Class in order to
>> test SparkStreaming using Java would be useful for the community.
>> Unfortunately not all of our customers are willing to use Scala or Python.
>>
> Sounds reasonable, I'll add it this week.
>
>>
>> If i am not wrong it’s 4:00 AM for you in California ;)
>>
>> Yup, I'm not great a regular schedules but I make up for it by doing
> stuff when I've had too much coffee to sleep :p
>
>> Regards,
>> Mark
>>
>> On 21 Oct 2015, at 12:42, Holden Karau <ho...@pigscanfly.ca> wrote:
>>
>>
>>
>> On Wednesday, October 21, 2015, Mark Vervuurt <m....@gmail.com>
>> wrote:
>>
>>> Hi Everyone,
>>>
>>> I am busy trying out ‘Spark-Testing-Base
>>> <https://github.com/holdenk/spark-testing-base>’. I have the following
>>> questions?
>>>
>>>
>>> - Can you test Spark Streaming Jobs using Java?
>>>
>>> The current base class for testing streaming jobs is implemented using a
>> Scala test library (and one in Python too), I can add one using a junit
>> base for streaming if it would be useful for you.
>>
>>>
>>> - Can I use Spark-Testing-Base 1.3.0_0.1.1 together with Spark 1.3.1?
>>>
>>> You should be able to, the API changes were small enough I didn't
>> publish a seperate package, but if you run into any issues let me know.
>>
>>>
>>>
>>>
>>> Thanks.
>>>
>>> Greetings,
>>> Mark
>>>
>>
>>
>> --
>> Cell : 425-233-8271
>> Twitter: https://twitter.com/holdenkarau
>>
>>
>>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
Re: Spark-Testing-Base Q/A
Posted by Holden Karau <ho...@pigscanfly.ca>.
On Wednesday, October 21, 2015, Mark Vervuurt <m....@gmail.com>
wrote:
> Hi Holden,
>
> Thanks for the information, I think that a Java Base Class in order to
> test SparkStreaming using Java would be useful for the community.
> Unfortunately not all of our customers are willing to use Scala or Python.
>
Sounds reasonable, I'll add it this week.
>
> If i am not wrong it’s 4:00 AM for you in California ;)
>
> Yup, I'm not great a regular schedules but I make up for it by doing stuff
when I've had too much coffee to sleep :p
> Regards,
> Mark
>
> On 21 Oct 2015, at 12:42, Holden Karau <ho...@pigscanfly.ca> wrote:
>
>
>
> On Wednesday, October 21, 2015, Mark Vervuurt <m....@gmail.com>
> wrote:
>
>> Hi Everyone,
>>
>> I am busy trying out ‘Spark-Testing-Base
>> <https://github.com/holdenk/spark-testing-base>’. I have the following
>> questions?
>>
>>
>> - Can you test Spark Streaming Jobs using Java?
>>
>> The current base class for testing streaming jobs is implemented using a
> Scala test library (and one in Python too), I can add one using a junit
> base for streaming if it would be useful for you.
>
>>
>> - Can I use Spark-Testing-Base 1.3.0_0.1.1 together with Spark 1.3.1?
>>
>> You should be able to, the API changes were small enough I didn't
> publish a seperate package, but if you run into any issues let me know.
>
>>
>>
>>
>> Thanks.
>>
>> Greetings,
>> Mark
>>
>
>
> --
> Cell : 425-233-8271
> Twitter: https://twitter.com/holdenkarau
>
>
>
Re: Spark-Testing-Base Q/A
Posted by Mark Vervuurt <m....@gmail.com>.
Hi Holden,
Thanks for the information, I think that a Java Base Class in order to test SparkStreaming using Java would be useful for the community. Unfortunately not all of our customers are willing to use Scala or Python.
If i am not wrong it’s 4:00 AM for you in California ;)
Regards,
Mark
> On 21 Oct 2015, at 12:42, Holden Karau <ho...@pigscanfly.ca> wrote:
>
>
>
> On Wednesday, October 21, 2015, Mark Vervuurt <m.a.vervuurt@gmail.com <ma...@gmail.com>> wrote:
> Hi Everyone,
>
> I am busy trying out ‘Spark-Testing-Base <https://github.com/holdenk/spark-testing-base>’. I have the following questions?
>
> Can you test Spark Streaming Jobs using Java?
> The current base class for testing streaming jobs is implemented using a Scala test library (and one in Python too), I can add one using a junit base for streaming if it would be useful for you.
> Can I use Spark-Testing-Base 1.3.0_0.1.1 together with Spark 1.3.1?
> You should be able to, the API changes were small enough I didn't publish a seperate package, but if you run into any issues let me know.
>
> Thanks.
>
> Greetings,
> Mark
>
>
> --
> Cell : 425-233-8271
> Twitter: https://twitter.com/holdenkarau <https://twitter.com/holdenkarau>
Re: Spark-Testing-Base Q/A
Posted by Holden Karau <ho...@pigscanfly.ca>.
On Wednesday, October 21, 2015, Mark Vervuurt <m....@gmail.com>
wrote:
> Hi Everyone,
>
> I am busy trying out ‘Spark-Testing-Base
> <https://github.com/holdenk/spark-testing-base>’. I have the following
> questions?
>
>
> - Can you test Spark Streaming Jobs using Java?
>
> The current base class for testing streaming jobs is implemented using a
Scala test library (and one in Python too), I can add one using a junit
base for streaming if it would be useful for you.
>
> - Can I use Spark-Testing-Base 1.3.0_0.1.1 together with Spark 1.3.1?
>
> You should be able to, the API changes were small enough I didn't publish
a seperate package, but if you run into any issues let me know.
>
>
>
> Thanks.
>
> Greetings,
> Mark
>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau