You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Shmuel Blitz <sh...@similarweb.com> on 2018/09/16 19:09:51 UTC

Run spark tests on Windows/docker

Hi,

I'd like to build and run spark tests on my PC.

Build works fine on my Windows machine, but the tests can't run for various
reasons.

1. Is it possible to run the tests on a Windows without special magic?
2. If you need some magic, how complicated is it?
3. I thought about running the tests on a docker linux container with the
Spark build mounted from the host PC. Has anyone done this? Do you have a
recommended docker image to work with?
4. any special consideration I should think of?

Thanks,
Shmuel

-- 
Shmuel Blitz
Big Data Developer
Email: shmuel.blitz@similarweb.com
www.similarweb.com
<https://www.similarweb.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://www.facebook.com/SimilarWeb/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://www.linkedin.com/company/429838/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://twitter.com/similarweb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>

Re: Run spark tests on Windows/docker

Posted by Shmuel Blitz <sh...@similarweb.com>.
Since I got no feedback, I'll try asking differently:

Can anyone point me to any resources regarding how to run the project's
tests?
Where can I find a good Docker image that would serve as a YARN cluster for
submitting jobs?

Thanks,
Shmuel

On Sun, Sep 16, 2018 at 10:09 PM Shmuel Blitz <sh...@similarweb.com>
wrote:

> Hi,
>
> I'd like to build and run spark tests on my PC.
>
> Build works fine on my Windows machine, but the tests can't run for
> various reasons.
>
> 1. Is it possible to run the tests on a Windows without special magic?
> 2. If you need some magic, how complicated is it?
> 3. I thought about running the tests on a docker linux container with the
> Spark build mounted from the host PC. Has anyone done this? Do you have a
> recommended docker image to work with?
> 4. any special consideration I should think of?
>
> Thanks,
> Shmuel
>
> --
> Shmuel Blitz
> Big Data Developer
> Email: shmuel.blitz@similarweb.com
> www.similarweb.com
> <https://www.similarweb.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> <https://www.facebook.com/SimilarWeb/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <https://www.linkedin.com/company/429838/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> <https://twitter.com/similarweb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>


-- 
Shmuel Blitz
Big Data Developer
Email: shmuel.blitz@similarweb.com
www.similarweb.com
<https://www.similarweb.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://www.facebook.com/SimilarWeb/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://www.linkedin.com/company/429838/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://twitter.com/similarweb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>