You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Konstantin Kudryavtsev <ku...@gmail.com> on 2014/07/31 12:04:08 UTC
Ports required for running spark
Hi there,
I'm trying to run Spark on YARN cluster and face with issued that some
ports are closed, particularly port 33007 (I suppose it's used by Akka)
Could you please provide me with a list of all ports required for Spark?
Also, is it possible to set up these ports?
Thank you in advance,
Konstantin Kudryavtsev
Re: Ports required for running spark
Posted by Larry Xiao <xi...@sjtu.edu.cn>.
Sorry, I don't have experience with YARN.
I checked the YARN page
http://spark.apache.org/docs/latest/running-on-yarn.html
And for configuration, it refers to
http://spark.apache.org/docs/latest/configuration.html " Most of the
configs are the same for Spark on YARN as for other deployment modes.
See theconfiguration page
<http://spark.apache.org/docs/latest/configuration.html>for more
information on those. These are configs that are specific to Spark on YARN."
And in Networking, there's spark.driver.port.
Don't know if it's helpful.
Larry
On 7/31/14, 6:17 PM, Konstantin Kudryavtsev wrote:
> Hi Larry,
>
> I'm afraid this is standalone mode, I'm interesting in YARN
>
> Also, I don't see port-in-trouble 33007which i believe related to Akka
>
> Thank you,
> Konstantin Kudryavtsev
>
>
> On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao <xiaodi@sjtu.edu.cn
> <ma...@sjtu.edu.cn>> wrote:
>
> Hi Konstantin,
>
> I think you can find it at
> https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
> and you can specify port for master or worker at conf/spark-env.sh
>
> Larry
>
>
> On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:
>> Hi there,
>>
>> I'm trying to run Spark on YARN cluster and face with issued that
>> some ports are closed, particularly port 33007 (I suppose it's
>> used by Akka)
>>
>> Could you please provide me with a list of all ports required for
>> Spark?
>> Also, is it possible to set up these ports?
>>
>> Thank you in advance,
>> Konstantin Kudryavtsev
>
>
Re: Ports required for running spark
Posted by Andrew Ash <an...@andrewash.com>.
Also Konstantin do you have a firewall between your Spark services? If
that's what's causing these issues, then you may be interested in the
ability to configure every port a Spark service listens on --
https://issues.apache.org/jira/browse/SPARK-2157
On Thu, Jul 31, 2014 at 8:47 AM, Haiyang Fu <ha...@gmail.com> wrote:
> Hi Konstantin,
>
> Could you please post your first container's stderr log here which is
> always the AM log?As far as I know, ports except 8020,8080,8081,50070,50071
> are all random socket ports determined by each job. So 33007 maybe just
> a temporary port for data transferation. The deeper reason for 33007
> ConnectionRefused maybe something else.would you please check If it is
> always 33007 when you submit your job again ?
>
>
> haiy
>
>
> On Thu, Jul 31, 2014 at 8:17 PM, Konstantin Kudryavtsev <
> kudryavtsev.konstantin@gmail.com> wrote:
>
>> Hi Haiyang,
>>
>> you are right, YARN takes over the resource management, bot I constantly
>> got Exception ConnectionRefused on mentioned port. So, I suppose some spark
>> internal communications are done via this port... but I don't know what
>> exactly and how can I change it...
>>
>> Thank you,
>> Konstantin Kudryavtsev
>>
>>
>> On Thu, Jul 31, 2014 at 2:53 PM, Haiyang Fu <ha...@gmail.com>
>> wrote:
>>
>>> Hi Konstantin,
>>> Would you please post some more details? Error info or exception from
>>> the log on what situation？when you run spark job on yarn cluster mode ,yarn
>>> will take over all the resource management.
>>>
>>>
>>> On Thu, Jul 31, 2014 at 6:17 PM, Konstantin Kudryavtsev <
>>> kudryavtsev.konstantin@gmail.com> wrote:
>>>
>>>> Hi Larry,
>>>>
>>>> I'm afraid this is standalone mode, I'm interesting in YARN
>>>>
>>>> Also, I don't see port-in-trouble 33007 which i believe related to
>>>> Akka
>>>>
>>>> Thank you,
>>>> Konstantin Kudryavtsev
>>>>
>>>>
>>>> On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao <xi...@sjtu.edu.cn> wrote:
>>>>
>>>>> Hi Konstantin,
>>>>>
>>>>> I think you can find it at
>>>>> https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
>>>>> and you can specify port for master or worker at conf/spark-env.sh
>>>>>
>>>>> Larry
>>>>>
>>>>>
>>>>> On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:
>>>>>
>>>>> Hi there,
>>>>>
>>>>> I'm trying to run Spark on YARN cluster and face with issued that
>>>>> some ports are closed, particularly port 33007 (I suppose it's used
>>>>> by Akka)
>>>>>
>>>>> Could you please provide me with a list of all ports required for
>>>>> Spark?
>>>>> Also, is it possible to set up these ports?
>>>>>
>>>>> Thank you in advance,
>>>>> Konstantin Kudryavtsev
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Ports required for running spark
Posted by Haiyang Fu <ha...@gmail.com>.
Hi Konstantin,
Could you please post your first container's stderr log here which is
always the AM log?As far as I know, ports except 8020,8080,8081,50070,50071
are all random socket ports determined by each job. So 33007 maybe just a
temporary port for data transferation. The deeper reason for 33007
ConnectionRefused maybe something else.would you please check If it is
always 33007 when you submit your job again ?
haiy
On Thu, Jul 31, 2014 at 8:17 PM, Konstantin Kudryavtsev <
kudryavtsev.konstantin@gmail.com> wrote:
> Hi Haiyang,
>
> you are right, YARN takes over the resource management, bot I constantly
> got Exception ConnectionRefused on mentioned port. So, I suppose some spark
> internal communications are done via this port... but I don't know what
> exactly and how can I change it...
>
> Thank you,
> Konstantin Kudryavtsev
>
>
> On Thu, Jul 31, 2014 at 2:53 PM, Haiyang Fu <ha...@gmail.com>
> wrote:
>
>> Hi Konstantin,
>> Would you please post some more details? Error info or exception from the
>> log on what situation？when you run spark job on yarn cluster mode ,yarn
>> will take over all the resource management.
>>
>>
>> On Thu, Jul 31, 2014 at 6:17 PM, Konstantin Kudryavtsev <
>> kudryavtsev.konstantin@gmail.com> wrote:
>>
>>> Hi Larry,
>>>
>>> I'm afraid this is standalone mode, I'm interesting in YARN
>>>
>>> Also, I don't see port-in-trouble 33007 which i believe related to
>>> Akka
>>>
>>> Thank you,
>>> Konstantin Kudryavtsev
>>>
>>>
>>> On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao <xi...@sjtu.edu.cn> wrote:
>>>
>>>> Hi Konstantin,
>>>>
>>>> I think you can find it at
>>>> https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
>>>> and you can specify port for master or worker at conf/spark-env.sh
>>>>
>>>> Larry
>>>>
>>>>
>>>> On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:
>>>>
>>>> Hi there,
>>>>
>>>> I'm trying to run Spark on YARN cluster and face with issued that
>>>> some ports are closed, particularly port 33007 (I suppose it's used by
>>>> Akka)
>>>>
>>>> Could you please provide me with a list of all ports required for
>>>> Spark?
>>>> Also, is it possible to set up these ports?
>>>>
>>>> Thank you in advance,
>>>> Konstantin Kudryavtsev
>>>>
>>>>
>>>>
>>>
>>
>
Re: Ports required for running spark
Posted by Konstantin Kudryavtsev <ku...@gmail.com>.
Hi Haiyang,
you are right, YARN takes over the resource management, bot I constantly
got Exception ConnectionRefused on mentioned port. So, I suppose some spark
internal communications are done via this port... but I don't know what
exactly and how can I change it...
Thank you,
Konstantin Kudryavtsev
On Thu, Jul 31, 2014 at 2:53 PM, Haiyang Fu <ha...@gmail.com> wrote:
> Hi Konstantin,
> Would you please post some more details? Error info or exception from the
> log on what situation？when you run spark job on yarn cluster mode ,yarn
> will take over all the resource management.
>
>
> On Thu, Jul 31, 2014 at 6:17 PM, Konstantin Kudryavtsev <
> kudryavtsev.konstantin@gmail.com> wrote:
>
>> Hi Larry,
>>
>> I'm afraid this is standalone mode, I'm interesting in YARN
>>
>> Also, I don't see port-in-trouble 33007 which i believe related to Akka
>>
>> Thank you,
>> Konstantin Kudryavtsev
>>
>>
>> On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao <xi...@sjtu.edu.cn> wrote:
>>
>>> Hi Konstantin,
>>>
>>> I think you can find it at
>>> https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
>>> and you can specify port for master or worker at conf/spark-env.sh
>>>
>>> Larry
>>>
>>>
>>> On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:
>>>
>>> Hi there,
>>>
>>> I'm trying to run Spark on YARN cluster and face with issued that some
>>> ports are closed, particularly port 33007 (I suppose it's used by Akka)
>>>
>>> Could you please provide me with a list of all ports required for
>>> Spark?
>>> Also, is it possible to set up these ports?
>>>
>>> Thank you in advance,
>>> Konstantin Kudryavtsev
>>>
>>>
>>>
>>
>
Re: Ports required for running spark
Posted by Haiyang Fu <ha...@gmail.com>.
Hi Konstantin,
Would you please post some more details? Error info or exception from the
log on what situation？when you run spark job on yarn cluster mode ,yarn
will take over all the resource management.
On Thu, Jul 31, 2014 at 6:17 PM, Konstantin Kudryavtsev <
kudryavtsev.konstantin@gmail.com> wrote:
> Hi Larry,
>
> I'm afraid this is standalone mode, I'm interesting in YARN
>
> Also, I don't see port-in-trouble 33007 which i believe related to Akka
>
> Thank you,
> Konstantin Kudryavtsev
>
>
> On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao <xi...@sjtu.edu.cn> wrote:
>
>> Hi Konstantin,
>>
>> I think you can find it at
>> https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
>> and you can specify port for master or worker at conf/spark-env.sh
>>
>> Larry
>>
>>
>> On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:
>>
>> Hi there,
>>
>> I'm trying to run Spark on YARN cluster and face with issued that some
>> ports are closed, particularly port 33007 (I suppose it's used by Akka)
>>
>> Could you please provide me with a list of all ports required for
>> Spark?
>> Also, is it possible to set up these ports?
>>
>> Thank you in advance,
>> Konstantin Kudryavtsev
>>
>>
>>
>
Re: Ports required for running spark
Posted by Konstantin Kudryavtsev <ku...@gmail.com>.
Hi Larry,
I'm afraid this is standalone mode, I'm interesting in YARN
Also, I don't see port-in-trouble 33007 which i believe related to Akka
Thank you,
Konstantin Kudryavtsev
On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao <xi...@sjtu.edu.cn> wrote:
> Hi Konstantin,
>
> I think you can find it at
> https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
> and you can specify port for master or worker at conf/spark-env.sh
>
> Larry
>
>
> On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:
>
> Hi there,
>
> I'm trying to run Spark on YARN cluster and face with issued that some
> ports are closed, particularly port 33007 (I suppose it's used by Akka)
>
> Could you please provide me with a list of all ports required for Spark?
> Also, is it possible to set up these ports?
>
> Thank you in advance,
> Konstantin Kudryavtsev
>
>
>
Re: Ports required for running spark
Posted by Larry Xiao <xi...@sjtu.edu.cn>.
Hi Konstantin,
I think you can find it at
https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
and you can specify port for master or worker at conf/spark-env.sh
Larry
On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:
> Hi there,
>
> I'm trying to run Spark on YARN cluster and face with issued that some
> ports are closed, particularly port 33007 (I suppose it's used by Akka)
>
> Could you please provide me with a list of all ports required for Spark?
> Also, is it possible to set up these ports?
>
> Thank you in advance,
> Konstantin Kudryavtsev