You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Eugene Morozov <fa...@list.ru> on 2015/04/10 11:17:02 UTC

Unit tests with spark local[4] on Jenkins

Hi, 

I’m trying to set up my testing infrastructure, which currently depends on Spark being run with master “local[4]”.
I’m using Jenkins and on my local environment everything is just fine. But on Jenkins machine I ran into issue with "Failed to bind to: /X.X.X.X:0"

As far as I understand the issue, X.X.X.X is external IP of my Jenkins, for some reason it binds to this interface. Unfortunately all ports on this interface are behind the firewall, so it’s not possible to get any random available one. It is possible to run tests on Jenkins machine (out of sources by running mvn clean package). But in order to do that it’s a must to set SPARK_LOCAL_IP. Then everything works as it should.

The problem with Jenkins though is that for some reason I cannot pass this env property to the build itself. I’m using envinject plugin and in debug I see there is a set for this variable. So, it should be fine. I also tried to use maven surefire plugin and specified there the variable as a system property, but still nothing.

Not sure, though, it’s an issue with Spark, but may be someone already set up mvn project on Spark with some testing around it?

Eugene Morozov
fathersson@list.ru





Re: Unit tests with spark local[4] on Jenkins

Posted by Eugene Morozov <fa...@list.ru>.
I’ve used Jenkins envinject plugin for Jenkins and it seems it actually inject this variable into the build, but for some reason Spark does not take a look at it. It still resolves to the external IP of the machine. I’ve been told that this is how AWS cloud works. So, it resolves hostname into its external IP.
So, we’ve just updated /etc/hosts to map the hostname to the localhost and the problem has gone away.

Hope it’ll help someone.


On 10 Apr 2015, at 02:17, Eugene Morozov <fa...@list.ru> wrote:

> Hi, 
> 
> I’m trying to set up my testing infrastructure, which currently depends on Spark being run with master “local[4]”.
> I’m using Jenkins and on my local environment everything is just fine. But on Jenkins machine I ran into issue with "Failed to bind to: /X.X.X.X:0"
> 
> As far as I understand the issue, X.X.X.X is external IP of my Jenkins, for some reason it binds to this interface. Unfortunately all ports on this interface are behind the firewall, so it’s not possible to get any random available one. It is possible to run tests on Jenkins machine (out of sources by running mvn clean package). But in order to do that it’s a must to set SPARK_LOCAL_IP. Then everything works as it should.
> 
> The problem with Jenkins though is that for some reason I cannot pass this env property to the build itself. I’m using envinject plugin and in debug I see there is a set for this variable. So, it should be fine. I also tried to use maven surefire plugin and specified there the variable as a system property, but still nothing.
> 
> Not sure, though, it’s an issue with Spark, but may be someone already set up mvn project on Spark with some testing around it?
> 
> Eugene Morozov
> fathersson@list.ru
> 
> 
> 
> 

Eugene Morozov
fathersson@list.ru