You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Nicholas Chammas (JIRA)" <ji...@apache.org> on 2014/09/04 06:56:51 UTC
[jira] [Updated] (SPARK-3398) Have spark-ec2 intelligently wait for
specific cluster states
[ https://issues.apache.org/jira/browse/SPARK-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicholas Chammas updated SPARK-3398:
------------------------------------
Description:
{{spark-ec2}} currently has retry logic for when it tries to install stuff on a cluster and for when it tries to destroy security groups.
It would be better to have some logic that allows {{spark-ec2}} to explicitly wait for when all the nodes in a cluster it is working on have reached a specific state.
Examples:
* Wait for all nodes to be up
* Wait for all nodes to be up and accepting SSH connections (then start installing stuff)
* Wait for all nodes to be down
* Wait for all nodes to be terminated (then delete the security groups)
Having a function in the {{spark_ec2.py}} script that blocks until the desired cluster state is reached would reduce the need for various retry logic. It would probably also eliminate the need for the {{--wait}} parameter.
was:
{{spark-ec2}} currently has retry logic for when it tries to install stuff on a cluster and for when it tries to destroy security groups.
It would be better to have some logic that allows {{spark-ec2}} to explicitly wait for when all the nodes in a cluster it is working on have reached a specific state.
Examples:
* Wait for all nodes to be up
* Wait for all nodes to be up and accepting SSH connections (then start installing stuff)
* Wait for all nodes to be down
* Wait for all nodes to be terminated (then delete the security groups)
Having a function that blocks until the desired cluster state is reached would reduce the need for various retry logic and probably also eliminate the need for the {{--wait}} parameter.
> Have spark-ec2 intelligently wait for specific cluster states
> -------------------------------------------------------------
>
> Key: SPARK-3398
> URL: https://issues.apache.org/jira/browse/SPARK-3398
> Project: Spark
> Issue Type: Improvement
> Components: EC2
> Reporter: Nicholas Chammas
> Priority: Minor
>
> {{spark-ec2}} currently has retry logic for when it tries to install stuff on a cluster and for when it tries to destroy security groups.
> It would be better to have some logic that allows {{spark-ec2}} to explicitly wait for when all the nodes in a cluster it is working on have reached a specific state.
> Examples:
> * Wait for all nodes to be up
> * Wait for all nodes to be up and accepting SSH connections (then start installing stuff)
> * Wait for all nodes to be down
> * Wait for all nodes to be terminated (then delete the security groups)
> Having a function in the {{spark_ec2.py}} script that blocks until the desired cluster state is reached would reduce the need for various retry logic. It would probably also eliminate the need for the {{--wait}} parameter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org