RE: [EXTERNAL] - Re: testing frameworks

[Steve Pruitt <bp...@opentext.com>]

Something more on the lines of integration I believe.  Run one or more Spark jobs and verify the output results.  If this makes sense.

I am very new to the world of Spark.  We want to include pipeline testing from the get go.  I will check out spark-testing-base.


Thanks.

From: Holden Karau [mailto:holden@pigscanfly.ca]
Sent: Monday, May 21, 2018 11:32 AM
To: Steve Pruitt <bp...@opentext.com>
Cc: user@spark.apache.org
Subject: [EXTERNAL] - Re: testing frameworks

So I’m biased as the author of spark-testing-base but I think it’s pretty ok. Are you looking for unit or integration or something else?

On Mon, May 21, 2018 at 5:24 AM Steve Pruitt <bp...@opentext.com>> wrote:
Hi,

Can anyone recommend testing frameworks suitable for Spark jobs.  Something that can be integrated into a CI tool would be great.

Thanks.

--
Twitter: https://twitter.com/holdenkarau<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_holdenkarau&d=DwMFaQ&c=ZgVRmm3mf2P1-XDAyDsu4A&r=ksx9qnQFG3QvxkP54EBPEzv1HHDjlk-MFO-7EONGCtY&m=YTdxEm6qmXE1TQvlRzPccMkNLcynfxhC32Uj91HcaXA&s=a_ORg1aB6eKT2ZYxtSJw3oOQnHmi07gjf9whuROeNYw&e=>

Re: [EXTERNAL] - Re: testing frameworks

[Joel D <ga...@gmail.com>]

We’ve developed our own version of testing framework consisting of
different areas of checking, sometimes providing expected data and
comparing with the resultant data from the data object.

Cheers.

On Tue, May 22, 2018 at 1:48 PM Steve Pruitt <bp...@opentext.com> wrote:

> Something more on the lines of integration I believe.  Run one or more
> Spark jobs and verify the output results.  If this makes sense.
>
>
>
> I am very new to the world of Spark.  We want to include pipeline testing
> from the get go.  I will check out spark-testing-base.
>
>
>
>
>
> Thanks.
>
>
>
> *From:* Holden Karau [mailto:holden@pigscanfly.ca]
> *Sent:* Monday, May 21, 2018 11:32 AM
> *To:* Steve Pruitt <bp...@opentext.com>
> *Cc:* user@spark.apache.org
> *Subject:* [EXTERNAL] - Re: testing frameworks
>
>
>
> So I’m biased as the author of spark-testing-base but I think it’s pretty
> ok. Are you looking for unit or integration or something else?
>
>
>
> On Mon, May 21, 2018 at 5:24 AM Steve Pruitt <bp...@opentext.com> wrote:
>
> Hi,
>
>
>
> Can anyone recommend testing frameworks suitable for Spark jobs.
> Something that can be integrated into a CI tool would be great.
>
>
>
> Thanks.
>
>
>
> --
>
> Twitter: https://twitter.com/holdenkarau
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_holdenkarau&d=DwMFaQ&c=ZgVRmm3mf2P1-XDAyDsu4A&r=ksx9qnQFG3QvxkP54EBPEzv1HHDjlk-MFO-7EONGCtY&m=YTdxEm6qmXE1TQvlRzPccMkNLcynfxhC32Uj91HcaXA&s=a_ORg1aB6eKT2ZYxtSJw3oOQnHmi07gjf9whuROeNYw&e=>
>