You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sam Taylor Steyer <ss...@stanford.edu> on 2014/06/04 16:45:14 UTC
Trouble launching EC2 Cluster with Spark
Hi,
I am trying to launch an EC2 cluster from spark using the following command:
./spark-ec2 -k HackerPair -i [path]/HackerPair.pem -s 2 launch HackerCluster
I set my access key id and secret access key. I have been getting an error in the "setting up security groups..." phase:
Invalid value 'null' for protocol. VPC security groups must specify protocols explicitly.
My project partner gets one step further and then gets the error
The key pair 'JamesAndSamTest' does not exist.
Any thoughts as to how we could fix these problems? Thanks a lot!
Sam
Re: Trouble launching EC2 Cluster with Spark
Posted by Patrick Wendell <pw...@gmail.com>.
Hey Sam,
You mentioned two problems here, did your VPC error message get fixed
or only the key permissions problem?
I noticed we had some report a similar issue with the VPC stuff a long
time back (but there is no real resolution here):
https://spark-project.atlassian.net/browse/SPARK-1166
If that's still an issue, one thing to try is just changing the name
of the cluster. We create groups that are identified with the cluster
name, and there might be something that just got screwed up with the
original group creation and AWS isn't happy.
- Patrick
On Wed, Jun 4, 2014 at 12:55 PM, Sam Taylor Steyer <ss...@stanford.edu> wrote:
> Awesome, that worked. Thank you!
>
> ----- Original Message -----
> From: "Krishna Sankar" <ks...@gmail.com>
> To: user@spark.apache.org
> Sent: Wednesday, June 4, 2014 12:52:00 PM
> Subject: Re: Trouble launching EC2 Cluster with Spark
>
> chmod 600 <path>/FinalKey.pem
>
> Cheers
>
> <k/>
>
>
> On Wed, Jun 4, 2014 at 12:49 PM, Sam Taylor Steyer <ss...@stanford.edu>
> wrote:
>
>> Also, once my friend logged in to his cluster he received the error
>> "Permissions 0644 for 'FinalKey.pem' are too open." This sounds like the
>> other problem described. How do we make the permissions more private?
>>
>> Thanks very much,
>> Sam
>>
>> ----- Original Message -----
>> From: "Sam Taylor Steyer" <ss...@stanford.edu>
>> To: user@spark.apache.org
>> Sent: Wednesday, June 4, 2014 12:42:04 PM
>> Subject: Re: Trouble launching EC2 Cluster with Spark
>>
>> Thanks you! The regions advice solved the problem for my friend who was
>> getting the key pair does not exist problem. I am still getting the error:
>>
>> ERROR:boto:400 Bad Request
>> ERROR:boto:<?xml version="1.0" encoding="UTF-8"?>
>> <Response><Errors><Error><Code>InvalidParameterValue</Code><Message>Invalid
>> value 'null' for protocol. VPC security group rules must specify protocols
>> explicitly.</Message></Error></Errors><RequestID>7ff92687-b95a-4a39-94cb-e2d00a6928fd</RequestID></Response>
>>
>> This sounds like it could have to do with the access settings of the
>> security group, but I don't know how to change. Any advice would be much
>> appreciated!
>>
>> Sam
>>
>> ----- Original Message -----
>> From: "Krishna Sankar" <ks...@gmail.com>
>> To: user@spark.apache.org
>> Sent: Wednesday, June 4, 2014 8:52:59 AM
>> Subject: Re: Trouble launching EC2 Cluster with Spark
>>
>> One reason could be that the keys are in a different region. Need to create
>> the keys in us-east-1-North Virginia.
>> Cheers
>> <k/>
>>
>>
>> On Wed, Jun 4, 2014 at 7:45 AM, Sam Taylor Steyer <ss...@stanford.edu>
>> wrote:
>>
>> > Hi,
>> >
>> > I am trying to launch an EC2 cluster from spark using the following
>> > command:
>> >
>> > ./spark-ec2 -k HackerPair -i [path]/HackerPair.pem -s 2 launch
>> > HackerCluster
>> >
>> > I set my access key id and secret access key. I have been getting an
>> error
>> > in the "setting up security groups..." phase:
>> >
>> > Invalid value 'null' for protocol. VPC security groups must specify
>> > protocols explicitly.
>> >
>> > My project partner gets one step further and then gets the error
>> >
>> > The key pair 'JamesAndSamTest' does not exist.
>> >
>> > Any thoughts as to how we could fix these problems? Thanks a lot!
>> > Sam
>> >
>>
Re: Trouble launching EC2 Cluster with Spark
Posted by Sam Taylor Steyer <ss...@stanford.edu>.
Awesome, that worked. Thank you!
----- Original Message -----
From: "Krishna Sankar" <ks...@gmail.com>
To: user@spark.apache.org
Sent: Wednesday, June 4, 2014 12:52:00 PM
Subject: Re: Trouble launching EC2 Cluster with Spark
chmod 600 <path>/FinalKey.pem
Cheers
<k/>
On Wed, Jun 4, 2014 at 12:49 PM, Sam Taylor Steyer <ss...@stanford.edu>
wrote:
> Also, once my friend logged in to his cluster he received the error
> "Permissions 0644 for 'FinalKey.pem' are too open." This sounds like the
> other problem described. How do we make the permissions more private?
>
> Thanks very much,
> Sam
>
> ----- Original Message -----
> From: "Sam Taylor Steyer" <ss...@stanford.edu>
> To: user@spark.apache.org
> Sent: Wednesday, June 4, 2014 12:42:04 PM
> Subject: Re: Trouble launching EC2 Cluster with Spark
>
> Thanks you! The regions advice solved the problem for my friend who was
> getting the key pair does not exist problem. I am still getting the error:
>
> ERROR:boto:400 Bad Request
> ERROR:boto:<?xml version="1.0" encoding="UTF-8"?>
> <Response><Errors><Error><Code>InvalidParameterValue</Code><Message>Invalid
> value 'null' for protocol. VPC security group rules must specify protocols
> explicitly.</Message></Error></Errors><RequestID>7ff92687-b95a-4a39-94cb-e2d00a6928fd</RequestID></Response>
>
> This sounds like it could have to do with the access settings of the
> security group, but I don't know how to change. Any advice would be much
> appreciated!
>
> Sam
>
> ----- Original Message -----
> From: "Krishna Sankar" <ks...@gmail.com>
> To: user@spark.apache.org
> Sent: Wednesday, June 4, 2014 8:52:59 AM
> Subject: Re: Trouble launching EC2 Cluster with Spark
>
> One reason could be that the keys are in a different region. Need to create
> the keys in us-east-1-North Virginia.
> Cheers
> <k/>
>
>
> On Wed, Jun 4, 2014 at 7:45 AM, Sam Taylor Steyer <ss...@stanford.edu>
> wrote:
>
> > Hi,
> >
> > I am trying to launch an EC2 cluster from spark using the following
> > command:
> >
> > ./spark-ec2 -k HackerPair -i [path]/HackerPair.pem -s 2 launch
> > HackerCluster
> >
> > I set my access key id and secret access key. I have been getting an
> error
> > in the "setting up security groups..." phase:
> >
> > Invalid value 'null' for protocol. VPC security groups must specify
> > protocols explicitly.
> >
> > My project partner gets one step further and then gets the error
> >
> > The key pair 'JamesAndSamTest' does not exist.
> >
> > Any thoughts as to how we could fix these problems? Thanks a lot!
> > Sam
> >
>
Re: Trouble launching EC2 Cluster with Spark
Posted by Krishna Sankar <ks...@gmail.com>.
chmod 600 <path>/FinalKey.pem
Cheers
<k/>
On Wed, Jun 4, 2014 at 12:49 PM, Sam Taylor Steyer <ss...@stanford.edu>
wrote:
> Also, once my friend logged in to his cluster he received the error
> "Permissions 0644 for 'FinalKey.pem' are too open." This sounds like the
> other problem described. How do we make the permissions more private?
>
> Thanks very much,
> Sam
>
> ----- Original Message -----
> From: "Sam Taylor Steyer" <ss...@stanford.edu>
> To: user@spark.apache.org
> Sent: Wednesday, June 4, 2014 12:42:04 PM
> Subject: Re: Trouble launching EC2 Cluster with Spark
>
> Thanks you! The regions advice solved the problem for my friend who was
> getting the key pair does not exist problem. I am still getting the error:
>
> ERROR:boto:400 Bad Request
> ERROR:boto:<?xml version="1.0" encoding="UTF-8"?>
> <Response><Errors><Error><Code>InvalidParameterValue</Code><Message>Invalid
> value 'null' for protocol. VPC security group rules must specify protocols
> explicitly.</Message></Error></Errors><RequestID>7ff92687-b95a-4a39-94cb-e2d00a6928fd</RequestID></Response>
>
> This sounds like it could have to do with the access settings of the
> security group, but I don't know how to change. Any advice would be much
> appreciated!
>
> Sam
>
> ----- Original Message -----
> From: "Krishna Sankar" <ks...@gmail.com>
> To: user@spark.apache.org
> Sent: Wednesday, June 4, 2014 8:52:59 AM
> Subject: Re: Trouble launching EC2 Cluster with Spark
>
> One reason could be that the keys are in a different region. Need to create
> the keys in us-east-1-North Virginia.
> Cheers
> <k/>
>
>
> On Wed, Jun 4, 2014 at 7:45 AM, Sam Taylor Steyer <ss...@stanford.edu>
> wrote:
>
> > Hi,
> >
> > I am trying to launch an EC2 cluster from spark using the following
> > command:
> >
> > ./spark-ec2 -k HackerPair -i [path]/HackerPair.pem -s 2 launch
> > HackerCluster
> >
> > I set my access key id and secret access key. I have been getting an
> error
> > in the "setting up security groups..." phase:
> >
> > Invalid value 'null' for protocol. VPC security groups must specify
> > protocols explicitly.
> >
> > My project partner gets one step further and then gets the error
> >
> > The key pair 'JamesAndSamTest' does not exist.
> >
> > Any thoughts as to how we could fix these problems? Thanks a lot!
> > Sam
> >
>
Re: Trouble launching EC2 Cluster with Spark
Posted by Sam Taylor Steyer <ss...@stanford.edu>.
Also, once my friend logged in to his cluster he received the error "Permissions 0644 for 'FinalKey.pem' are too open." This sounds like the other problem described. How do we make the permissions more private?
Thanks very much,
Sam
----- Original Message -----
From: "Sam Taylor Steyer" <ss...@stanford.edu>
To: user@spark.apache.org
Sent: Wednesday, June 4, 2014 12:42:04 PM
Subject: Re: Trouble launching EC2 Cluster with Spark
Thanks you! The regions advice solved the problem for my friend who was getting the key pair does not exist problem. I am still getting the error:
ERROR:boto:400 Bad Request
ERROR:boto:<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterValue</Code><Message>Invalid value 'null' for protocol. VPC security group rules must specify protocols explicitly.</Message></Error></Errors><RequestID>7ff92687-b95a-4a39-94cb-e2d00a6928fd</RequestID></Response>
This sounds like it could have to do with the access settings of the security group, but I don't know how to change. Any advice would be much appreciated!
Sam
----- Original Message -----
From: "Krishna Sankar" <ks...@gmail.com>
To: user@spark.apache.org
Sent: Wednesday, June 4, 2014 8:52:59 AM
Subject: Re: Trouble launching EC2 Cluster with Spark
One reason could be that the keys are in a different region. Need to create
the keys in us-east-1-North Virginia.
Cheers
<k/>
On Wed, Jun 4, 2014 at 7:45 AM, Sam Taylor Steyer <ss...@stanford.edu>
wrote:
> Hi,
>
> I am trying to launch an EC2 cluster from spark using the following
> command:
>
> ./spark-ec2 -k HackerPair -i [path]/HackerPair.pem -s 2 launch
> HackerCluster
>
> I set my access key id and secret access key. I have been getting an error
> in the "setting up security groups..." phase:
>
> Invalid value 'null' for protocol. VPC security groups must specify
> protocols explicitly.
>
> My project partner gets one step further and then gets the error
>
> The key pair 'JamesAndSamTest' does not exist.
>
> Any thoughts as to how we could fix these problems? Thanks a lot!
> Sam
>
Re: Trouble launching EC2 Cluster with Spark
Posted by Sam Taylor Steyer <ss...@stanford.edu>.
Thanks you! The regions advice solved the problem for my friend who was getting the key pair does not exist problem. I am still getting the error:
ERROR:boto:400 Bad Request
ERROR:boto:<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterValue</Code><Message>Invalid value 'null' for protocol. VPC security group rules must specify protocols explicitly.</Message></Error></Errors><RequestID>7ff92687-b95a-4a39-94cb-e2d00a6928fd</RequestID></Response>
This sounds like it could have to do with the access settings of the security group, but I don't know how to change. Any advice would be much appreciated!
Sam
----- Original Message -----
From: "Krishna Sankar" <ks...@gmail.com>
To: user@spark.apache.org
Sent: Wednesday, June 4, 2014 8:52:59 AM
Subject: Re: Trouble launching EC2 Cluster with Spark
One reason could be that the keys are in a different region. Need to create
the keys in us-east-1-North Virginia.
Cheers
<k/>
On Wed, Jun 4, 2014 at 7:45 AM, Sam Taylor Steyer <ss...@stanford.edu>
wrote:
> Hi,
>
> I am trying to launch an EC2 cluster from spark using the following
> command:
>
> ./spark-ec2 -k HackerPair -i [path]/HackerPair.pem -s 2 launch
> HackerCluster
>
> I set my access key id and secret access key. I have been getting an error
> in the "setting up security groups..." phase:
>
> Invalid value 'null' for protocol. VPC security groups must specify
> protocols explicitly.
>
> My project partner gets one step further and then gets the error
>
> The key pair 'JamesAndSamTest' does not exist.
>
> Any thoughts as to how we could fix these problems? Thanks a lot!
> Sam
>
Re: Trouble launching EC2 Cluster with Spark
Posted by Krishna Sankar <ks...@gmail.com>.
One reason could be that the keys are in a different region. Need to create
the keys in us-east-1-North Virginia.
Cheers
<k/>
On Wed, Jun 4, 2014 at 7:45 AM, Sam Taylor Steyer <ss...@stanford.edu>
wrote:
> Hi,
>
> I am trying to launch an EC2 cluster from spark using the following
> command:
>
> ./spark-ec2 -k HackerPair -i [path]/HackerPair.pem -s 2 launch
> HackerCluster
>
> I set my access key id and secret access key. I have been getting an error
> in the "setting up security groups..." phase:
>
> Invalid value 'null' for protocol. VPC security groups must specify
> protocols explicitly.
>
> My project partner gets one step further and then gets the error
>
> The key pair 'JamesAndSamTest' does not exist.
>
> Any thoughts as to how we could fix these problems? Thanks a lot!
> Sam
>
Re: Trouble launching EC2 Cluster with Spark
Posted by Gianluca Privitera <gi...@studio.unibo.it>.
Hi,
if you say you correctly setted your access key id and secret access key
then probably it's a problem related to the key.pem file.
Try generate a new one, and be sure to be the only one with the right to
read it or it wont work.
Gianluca
On 04/06/2014 09:45, Sam Taylor Steyer wrote:
> Hi,
>
> I am trying to launch an EC2 cluster from spark using the following command:
>
> ./spark-ec2 -k HackerPair -i [path]/HackerPair.pem -s 2 launch HackerCluster
>
> I set my access key id and secret access key. I have been getting an error in the "setting up security groups..." phase:
>
> Invalid value 'null' for protocol. VPC security groups must specify protocols explicitly.
>
> My project partner gets one step further and then gets the error
>
> The key pair 'JamesAndSamTest' does not exist.
>
> Any thoughts as to how we could fix these problems? Thanks a lot!
> Sam