You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Shiva Ramagopal <tr...@gmail.com> on 2017/04/05 11:32:15 UTC

Re: unit testing in spark

Hi,

I've been following this thread for a while.

I'm trying to bring in a test strategy in my team to test a number of data
pipelines before production. I have watched Lars' presentation and find it
great. However I'm debating whether unit tests are worth the effort if
there are good job-level and pipeline-level tests. Does anybody have any
experiences benefitting from unit-tests in such a case?

Cheers,
Shiv

On Mon, Dec 12, 2016 at 6:00 AM, Juan Rodríguez Hortalá <
juan.rodriguez.hortala@gmail.com> wrote:

> Hi all,
>
> I would also would like to participate on that.
>
> Greetings,
>
> Juan
>
> On Fri, Dec 9, 2016 at 6:03 AM, Michael Stratton <michael.stratton@
> komodohealth.com> wrote:
>
>> That sounds great, please include me so I can get involved.
>>
>> On Fri, Dec 9, 2016 at 7:39 AM, Marco Mistroni <mm...@gmail.com>
>> wrote:
>>
>>> Me too as I spent most of my time writing unit/integ tests....  pls
>>> advise on where I  can start
>>> Kr
>>>
>>> On 9 Dec 2016 12:15 am, "Miguel Morales" <th...@gmail.com>
>>> wrote:
>>>
>>>> I would be interested in contributing.  Ive created my own library for
>>>> this as well.  In my blog post I talk about testing with Spark in RSpec
>>>> style:
>>>> https://medium.com/@therevoltingx/test-driven-development-w-
>>>> apache-spark-746082b44941
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Dec 8, 2016, at 4:09 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
>>>>
>>>> There are also libraries designed to simplify testing Spark in the
>>>> various platforms, spark-testing-base
>>>> <http://github.com/holdenk/spark-testing-base> for Scala/Java/Python
>>>> (& video https://www.youtube.com/watch?v=f69gSGSLGrY), sscheck
>>>> <https://github.com/juanrh/sscheck> (scala focused property based),
>>>> pyspark.test (python focused with py.test instead of unittest2) (&
>>>> blog post from nextdoor https://engblog.nextd
>>>> oor.com/unit-testing-apache-spark-with-py-test-3b8970dc013b#.jw3bdcej9
>>>>  )
>>>>
>>>> Good luck on your Spark Adventures :)
>>>>
>>>> P.S.
>>>>
>>>> If anyone is interested in helping improve spark testing libraries I'm
>>>> always looking for more people to be involved with spark-testing-base
>>>> because I'm lazy :p
>>>>
>>>> On Thu, Dec 8, 2016 at 2:05 PM, Lars Albertsson <la...@mapflat.com>
>>>> wrote:
>>>>
>>>>> I wrote some advice in a previous post on the list:
>>>>> http://markmail.org/message/bbs5acrnksjxsrrs
>>>>>
>>>>> It does not mention python, but the strategy advice is the same. Just
>>>>> replace JUnit/Scalatest with pytest, unittest, or your favourite
>>>>> python test framework.
>>>>>
>>>>>
>>>>> I recently held a presentation on the subject. There is a video
>>>>> recording at https://vimeo.com/192429554 and slides at
>>>>> http://www.slideshare.net/lallea/test-strategies-for-data-pr
>>>>> ocessing-pipelines-67244458
>>>>>
>>>>> You can find more material on test strategies at
>>>>> http://www.mapflat.com/lands/resources/reading-list/index.html
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Lars Albertsson
>>>>> Data engineering consultant
>>>>> www.mapflat.com
>>>>> https://twitter.com/lalleal
>>>>> +46 70 7687109
>>>>> Calendar: https://goo.gl/6FBtlS, https://freebusy.io/lalle@mapflat.com
>>>>>
>>>>>
>>>>> On Thu, Dec 8, 2016 at 4:14 PM, pseudo oduesp <ps...@gmail.com>
>>>>> wrote:
>>>>> > somone can tell me how i can make unit test on pyspark ?
>>>>> > (book, tutorial ...)
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Cell : 425-233-8271 <(425)%20233-8271>
>>>> Twitter: https://twitter.com/holdenkarau
>>>>
>>>>
>>
>

Re: unit testing in spark

Posted by Steve Loughran <st...@hortonworks.com>.

(sorry sent an empty reply by accident)

Unit testing is one of the easiest ways to isolate problems in an an internal class, things you can get wrong. But: time spent writing unit tests is time *not* spent writing integration tests. Which biases me towards the integration.

What I do find is good is writing integration tests to debug things: if something is playing up, if you can write a unit test to replicate then not only can you isolate the problem, you can verify it is fixed and stays fixed. And as they are fast & often runnable in parallel, easy to do repetitively.

But: Tests have a maintenance cost, especially if the tests go into the internals, making them very brittle to change. Mocking is the real troublespot here. It's good to be able to simulate failures, but given the choice between "integration test against real code" and "something using mocks which produce "impossible' stack traces and, after a code rework, fail so badly you can't tell if it's a regression or just the tests are obsolete", I'd go for production, even if runs up some bills.

I really liked Lar's slides; gave me some ideas. One thing I've been exploring is using system metrics in testing, adding more metrics to help note what is happening

https://steveloughran.blogspot.co.uk/2016/04/distributed-testing-making-use-of.html

Strengths: encourages me to write metrics, can be used in in-VM tests, and collected from a distributed SUT integration tests, both for asserts and logging. Weakness1. : exposing internal state which, again, can be brittle. 2. in integration tests the results can vary a lot, so you can't really make assertions on it. Better there to collect things and use in test reports.

Which brings me to a real issue with integration tests, which isn't a fault of the apps or the tests, but in today's test runners: log capture and reporting dates from the era where we were running unit tests, so thinking about the reporting problems there: standard out and error for a single process, no standard log format so naive stream capture over structured log entries; test runners which don't repot much on a failure but the stack trace, or, with scalatest, half the stack trace (*), missing out on those of the remote systems. Systems which, if you are playing with cloud infra, may not be there when you get to analyse the test results. You are left trying to compare 9 logs across 3 destroyed VMs to work out why the test runner through an assertion failure.

This is tractable, and indeed, the Kakfa people have been advocating "use kafka as the collector of test results" to address it: the logs, metrics, events raised by the SUT., etc, and then somehow correlate them into test reports, or at least provide the ordering of events and state across parts of the system so that you can work back from a test failure. Yes, that means moving way beyond the usual ant-JUnit XML report everything creates, but like I said: that was written for a different era. It's time to move on, generating the XML report as one of the outputs if you want, but not the one you use for diagnosing why a test fails.

I'd love to see what people have been up to in that area. If anyone has insights there, it'd be topic for a hangout.

-Steve

(*) Scaltest opinions: https://steveloughran.blogspot.co.uk/2016/09/scalatest-thoughts-and-ideas.html