You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Holden Karau <ho...@pigscanfly.ca> on 2015/10/07 00:12:00 UTC
Adding Spark Testing functionality
Hi Spark Devs,
So this has been brought up a few times before, and generally on the user
list people get directed to use spark-testing-base. I'd like to start
moving some of spark-testing-base's functionality into Spark so that people
don't need a library to do what is (hopefully :p) a very common requirement
across all Spark projects.
To that end I was wondering what peoples thoughts are on where this should
live inside of Spark. I was thinking it could either be a separate testing
project (like sql or similar), or just put the bits to enable testing
inside of each relevant project.
I was also thinking it probably makes sense to only move the unit testing
parts at the start and leave things like integration testing in a testing
project since that could vary depending on the users environment.
What are peoples thoughts?
Cheers,
Holden :)
Re: Adding Spark Testing functionality
Posted by Holden Karau <ho...@pigscanfly.ca>.
So here is a quick description of the current testing bits (I can expand on
it if people are interested) http://bit.ly/pandaPandaPanda .
On Tue, Oct 6, 2015 at 3:49 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
> I'll put together a google doc and send that out (in the meantime a quick
> guide of sort of how the current package can be used is in the blog post I
> did at
> http://blog.cloudera.com/blog/2015/09/making-apache-spark-testing-easy-with-spark-testing-base/
> ) If people think its better to keep as a package I am of course happy to
> keep doing that. It feels a little strange to have something as core as
> being able to test your code live outside.
>
> On Tue, Oct 6, 2015 at 3:44 PM, Patrick Wendell <pw...@gmail.com>
> wrote:
>
>> Hey Holden,
>>
>> It would be helpful if you could outline the set of features you'd
>> imagine being part of Spark in a short doc. I didn't see a README on the
>> existing repo, so it's hard to know exactly what is being proposed.
>>
>> As a general point of process, we've typically avoided merging modules
>> into Spark that can exist outside of the project. A testing utility package
>> that is based on Spark's public API's seems like a really useful thing for
>> the community, but it does seem like a good fit for a package library. At
>> least, this is my first question after taking a look at the project.
>>
>> In any case, getting some high level view of the functionality you
>> imagine would be helpful to give more detailed feedback.
>>
>> - Patrick
>>
>> On Tue, Oct 6, 2015 at 3:12 PM, Holden Karau <ho...@pigscanfly.ca>
>> wrote:
>>
>>> Hi Spark Devs,
>>>
>>> So this has been brought up a few times before, and generally on the
>>> user list people get directed to use spark-testing-base. I'd like to start
>>> moving some of spark-testing-base's functionality into Spark so that people
>>> don't need a library to do what is (hopefully :p) a very common requirement
>>> across all Spark projects.
>>>
>>> To that end I was wondering what peoples thoughts are on where this
>>> should live inside of Spark. I was thinking it could either be a separate
>>> testing project (like sql or similar), or just put the bits to enable
>>> testing inside of each relevant project.
>>>
>>> I was also thinking it probably makes sense to only move the unit
>>> testing parts at the start and leave things like integration testing in a
>>> testing project since that could vary depending on the users environment.
>>>
>>> What are peoples thoughts?
>>>
>>> Cheers,
>>>
>>> Holden :)
>>>
>>
>>
>
>
> --
> Cell : 425-233-8271
> Twitter: https://twitter.com/holdenkarau
> Linked In: https://www.linkedin.com/in/holdenkarau
>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
Linked In: https://www.linkedin.com/in/holdenkarau
Re: Adding Spark Testing functionality
Posted by Holden Karau <ho...@pigscanfly.ca>.
I'll put together a google doc and send that out (in the meantime a quick
guide of sort of how the current package can be used is in the blog post I
did at
http://blog.cloudera.com/blog/2015/09/making-apache-spark-testing-easy-with-spark-testing-base/
) If people think its better to keep as a package I am of course happy to
keep doing that. It feels a little strange to have something as core as
being able to test your code live outside.
On Tue, Oct 6, 2015 at 3:44 PM, Patrick Wendell <pw...@gmail.com> wrote:
> Hey Holden,
>
> It would be helpful if you could outline the set of features you'd imagine
> being part of Spark in a short doc. I didn't see a README on the existing
> repo, so it's hard to know exactly what is being proposed.
>
> As a general point of process, we've typically avoided merging modules
> into Spark that can exist outside of the project. A testing utility package
> that is based on Spark's public API's seems like a really useful thing for
> the community, but it does seem like a good fit for a package library. At
> least, this is my first question after taking a look at the project.
>
> In any case, getting some high level view of the functionality you imagine
> would be helpful to give more detailed feedback.
>
> - Patrick
>
> On Tue, Oct 6, 2015 at 3:12 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
>
>> Hi Spark Devs,
>>
>> So this has been brought up a few times before, and generally on the user
>> list people get directed to use spark-testing-base. I'd like to start
>> moving some of spark-testing-base's functionality into Spark so that people
>> don't need a library to do what is (hopefully :p) a very common requirement
>> across all Spark projects.
>>
>> To that end I was wondering what peoples thoughts are on where this
>> should live inside of Spark. I was thinking it could either be a separate
>> testing project (like sql or similar), or just put the bits to enable
>> testing inside of each relevant project.
>>
>> I was also thinking it probably makes sense to only move the unit testing
>> parts at the start and leave things like integration testing in a testing
>> project since that could vary depending on the users environment.
>>
>> What are peoples thoughts?
>>
>> Cheers,
>>
>> Holden :)
>>
>
>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
Linked In: https://www.linkedin.com/in/holdenkarau
Re: Adding Spark Testing functionality
Posted by Patrick Wendell <pw...@gmail.com>.
Hey Holden,
It would be helpful if you could outline the set of features you'd imagine
being part of Spark in a short doc. I didn't see a README on the existing
repo, so it's hard to know exactly what is being proposed.
As a general point of process, we've typically avoided merging modules into
Spark that can exist outside of the project. A testing utility package that
is based on Spark's public API's seems like a really useful thing for the
community, but it does seem like a good fit for a package library. At
least, this is my first question after taking a look at the project.
In any case, getting some high level view of the functionality you imagine
would be helpful to give more detailed feedback.
- Patrick
On Tue, Oct 6, 2015 at 3:12 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
> Hi Spark Devs,
>
> So this has been brought up a few times before, and generally on the user
> list people get directed to use spark-testing-base. I'd like to start
> moving some of spark-testing-base's functionality into Spark so that people
> don't need a library to do what is (hopefully :p) a very common requirement
> across all Spark projects.
>
> To that end I was wondering what peoples thoughts are on where this should
> live inside of Spark. I was thinking it could either be a separate testing
> project (like sql or similar), or just put the bits to enable testing
> inside of each relevant project.
>
> I was also thinking it probably makes sense to only move the unit testing
> parts at the start and leave things like integration testing in a testing
> project since that could vary depending on the users environment.
>
> What are peoples thoughts?
>
> Cheers,
>
> Holden :)
>