You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Garry Chen <gc...@cornell.edu> on 2015/08/21 16:55:59 UTC
Spark ec2 lunch problem
Hi All,
I am trying to lunch a spark ec2 cluster by running spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but getting following message endless. Please help.
Warning: SSH connection error. (This could be temporary.)
Host:
SSH return code: 255
SSH output: ssh: Could not resolve hostname : Name or service not known
RE: Spark ec2 lunch problem
Posted by Garry Chen <gc...@cornell.edu>.
No, the message never end. I have to ctrl-c out of it.
Garry
From: shahid ashraf [mailto:shahid@trialx.com]
Sent: Friday, August 21, 2015 11:13 AM
To: Garry Chen <gc...@cornell.edu>
Cc: user@spark.apache.org
Subject: Re: Spark ec2 lunch problem
Does the cluster work at the end ?
On Fri, Aug 21, 2015 at 8:25 PM, Garry Chen <gc...@cornell.edu>> wrote:
Hi All,
I am trying to lunch a spark ec2 cluster by running spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but getting following message endless. Please help.
Warning: SSH connection error. (This could be temporary.)
Host:
SSH return code: 255
SSH output: ssh: Could not resolve hostname : Name or service not known
--
with Regards
Shahid Ashraf
Re: Spark ec2 lunch problem
Posted by shahid ashraf <sh...@trialx.com>.
Does the cluster work at the end ?
On Fri, Aug 21, 2015 at 8:25 PM, Garry Chen <gc...@cornell.edu> wrote:
> Hi All,
>
> I am trying to lunch a spark ec2 cluster by running
> spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc
> --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but
> getting following message endless. Please help.
>
>
>
>
>
> Warning: SSH connection error. (This could be temporary.)
>
> Host:
>
> SSH return code: 255
>
> SSH output: ssh: Could not resolve hostname : Name or service not known
>
--
with Regards
Shahid Ashraf
Re: Spark ec2 lunch problem
Posted by Andrew Or <an...@databricks.com>.
Hey Garry,
Have you verified that your particular VPC and subnet are open to the
world? In particular, have you verified the route table attached to your
VPC / subnet contains an internet gateway open to the public?
I've run into this issue myself recently and that was the problem for me.
-Andrew
2015-08-24 5:58 GMT-07:00 Robin East <ro...@xense.co.uk>:
> spark-ec2 is the way to go however you may need to debug connectivity
> issues. For example do you know that the servers were correctly setup in
> AWS and can you access each node using ssh? If no then you need to work out
> why (it’s not a spark issue). If yes then you will need to work out why ssh
> via the spark-ec2 script is not working.
>
> I’ve used spark-ec2 successfully many times but have never used the
> —vpc-id and —subnet-id options and that may be the source of your problems,
> especially since it appears to be a hostname resolution issue. If you could
> confirm the above questions then maybe someone on the list can help
> diagnose the specific problem.
>
>
>
> -------------------------------------------------------------------------------
> Robin East
> *Spark GraphX in Action* Michael Malak and Robin East
> Manning Publications Co.
> http://www.manning.com/malak/
>
> On 24 Aug 2015, at 13:45, Garry Chen <gc...@cornell.edu> wrote:
>
> So what is the best way to deploy spark cluster in EC2 environment any
> suggestions?
>
> Garry
>
> *From:* Akhil Das [mailto:akhil@sigmoidanalytics.com
> <ak...@sigmoidanalytics.com>]
> *Sent:* Friday, August 21, 2015 4:27 PM
> *To:* Garry Chen <gc...@cornell.edu>
> *Cc:* user@spark.apache.org
> *Subject:* Re: Spark ec2 lunch problem
>
>
> It may happen that the version of spark-ec2 script you are using is buggy
> or sometime AWS have problem provisioning machines.
> On Aug 21, 2015 7:56 AM, "Garry Chen" <gc...@cornell.edu> wrote:
>
> Hi All,
> I am trying to lunch a spark ec2 cluster by running
> spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc
> --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but
> getting following message endless. Please help.
>
>
> Warning: SSH connection error. (This could be temporary.)
> Host:
> SSH return code: 255
> SSH output: ssh: Could not resolve hostname : Name or service not known
>
>
>
Re: Spark ec2 lunch problem
Posted by Robin East <ro...@xense.co.uk>.
spark-ec2 is the way to go however you may need to debug connectivity issues. For example do you know that the servers were correctly setup in AWS and can you access each node using ssh? If no then you need to work out why (it’s not a spark issue). If yes then you will need to work out why ssh via the spark-ec2 script is not working.
I’ve used spark-ec2 successfully many times but have never used the —vpc-id and —subnet-id options and that may be the source of your problems, especially since it appears to be a hostname resolution issue. If you could confirm the above questions then maybe someone on the list can help diagnose the specific problem.
-------------------------------------------------------------------------------
Robin East
Spark GraphX in Action Michael Malak and Robin East
Manning Publications Co.
http://www.manning.com/malak/ <http://www.manning.com/malak/>
> On 24 Aug 2015, at 13:45, Garry Chen <gc...@cornell.edu> wrote:
>
> So what is the best way to deploy spark cluster in EC2 environment any suggestions?
>
> Garry
>
> From: Akhil Das [mailto:akhil@sigmoidanalytics.com <ma...@sigmoidanalytics.com>]
> Sent: Friday, August 21, 2015 4:27 PM
> To: Garry Chen <gc92@cornell.edu <ma...@cornell.edu>>
> Cc: user@spark.apache.org <ma...@spark.apache.org>
> Subject: Re: Spark ec2 lunch problem
>
> It may happen that the version of spark-ec2 script you are using is buggy or sometime AWS have problem provisioning machines.
>
> On Aug 21, 2015 7:56 AM, "Garry Chen" <gc92@cornell.edu <ma...@cornell.edu>> wrote:
> Hi All,
> I am trying to lunch a spark ec2 cluster by running spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but getting following message endless. Please help.
>
>
> Warning: SSH connection error. (This could be temporary.)
> Host:
> SSH return code: 255
> SSH output: ssh: Could not resolve hostname : Name or service not known
RE: Spark ec2 lunch problem
Posted by Garry Chen <gc...@cornell.edu>.
So what is the best way to deploy spark cluster in EC2 environment any suggestions?
Garry
From: Akhil Das [mailto:akhil@sigmoidanalytics.com]
Sent: Friday, August 21, 2015 4:27 PM
To: Garry Chen <gc...@cornell.edu>
Cc: user@spark.apache.org
Subject: Re: Spark ec2 lunch problem
It may happen that the version of spark-ec2 script you are using is buggy or sometime AWS have problem provisioning machines.
On Aug 21, 2015 7:56 AM, "Garry Chen" <gc...@cornell.edu>> wrote:
Hi All,
I am trying to lunch a spark ec2 cluster by running spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but getting following message endless. Please help.
Warning: SSH connection error. (This could be temporary.)
Host:
SSH return code: 255
SSH output: ssh: Could not resolve hostname : Name or service not known
Re: Spark ec2 lunch problem
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
It may happen that the version of spark-ec2 script you are using is buggy
or sometime AWS have problem provisioning machines.
On Aug 21, 2015 7:56 AM, "Garry Chen" <gc...@cornell.edu> wrote:
> Hi All,
>
> I am trying to lunch a spark ec2 cluster by running
> spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc
> --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but
> getting following message endless. Please help.
>
>
>
>
>
> Warning: SSH connection error. (This could be temporary.)
>
> Host:
>
> SSH return code: 255
>
> SSH output: ssh: Could not resolve hostname : Name or service not known
>