You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Nicholas Chammas (JIRA)" <ji...@apache.org> on 2015/05/28 17:19:28 UTC
[jira] [Commented] (SPARK-7900) Reduce number of tagging calls in
spark-ec2
[ https://issues.apache.org/jira/browse/SPARK-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14563053#comment-14563053 ]
Nicholas Chammas commented on SPARK-7900:
-----------------------------------------
An alternative approach would be to just [increase the wait time|https://github.com/apache/spark/blob/e838a25bdb5603ef05e779225704c972ce436145/ec2/spark_ec2.py#L681-L683] to 30 or 60 seconds before trying to tag instances.
[~shivaram] / [~joshrosen]: Any preference on an approach?
> Reduce number of tagging calls in spark-ec2
> -------------------------------------------
>
> Key: SPARK-7900
> URL: https://issues.apache.org/jira/browse/SPARK-7900
> Project: Spark
> Issue Type: Bug
> Components: EC2
> Affects Versions: 1.4.0
> Reporter: Nicholas Chammas
> Priority: Minor
>
> spark-ec2 currently tags each instance with its own name:
> https://github.com/apache/spark/blob/4615081d7a10b023491e25478d19b8161e030974/ec2/spark_ec2.py#L684-L692
> Quite often, one of these tagging calls will fail:
> {code}
> Launching instances...
> Launched 10 slaves in us-west-2a, regid = r-89656e83
> Launched master in us-west-2a, regid = r-07646f0d
> Waiting for AWS to propagate instance metadata...
> Traceback (most recent call last):
> File "../spark/ec2/spark_ec2.py", line 1395, in <module>
> main()
> File "../spark/ec2/spark_ec2.py", line 1387, in main
> real_main()
> File "../spark/ec2/spark_ec2.py", line 1222, in real_main
> (master_nodes, slave_nodes) = launch_cluster(conn, opts, cluster_name)
> File "../spark/ec2/spark_ec2.py", line 667, in launch_cluster
> value='{cn}-slave-{iid}'.format(cn=cluster_name, iid=slave.id))
> File "</path/spark/ec2/lib/boto-2.34.0/boto/ec2/ec2object.py",> line 80, in add_tag
> self.add_tags({key: value}, dry_run)
> File "</path/spark/ec2/lib/boto-2.34.0/boto/ec2/ec2object.py",> line 97, in add_tags
> dry_run=dry_run
> File "</path/spark/ec2/lib/boto-2.34.0/boto/ec2/connection.py",> line 4202, in create_tags
> return self.get_status('CreateTags', params, verb='POST')
> File "</path/spark/ec2/lib/boto-2.34.0/boto/connection.py",> line 1223, in get_status
> raise self.ResponseError(response.status, response.reason, body)
> boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
> <?xml version="1.0" encoding="UTF-8"?>
> <Response><Errors><Error><Code>InvalidInstanceID.NotFound</Code><Message>The instance ID 'i-d3b72524' does not exist</Message></Error></Errors><RequestID>f0936ab5-4d10-46d1-a35d-cefaf8a68adc</RequestID></Response>
> {code}
> This is presumably a problem with AWS metadata taking time to become available on all the servers that spark-ec2 hits as it makes the several tagging calls.
> Instead of retrying the tagging calls, we should just reduce them to 2 calls--one for the master, one for the slaves.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org