[jira] [Created] (SPARK-3332) Tags shouldn't be the main strategy for machine membership on clusters

["Allan Douglas R. de Oliveira (JIRA)" <ji...@apache.org>]

Allan Douglas R. de Oliveira created SPARK-3332:
---------------------------------------------------

             Summary: Tags shouldn't be the main strategy for machine membership on clusters
                 Key: SPARK-3332
                 URL: https://issues.apache.org/jira/browse/SPARK-3332
             Project: Spark
          Issue Type: Bug
          Components: EC2
            Reporter: Allan Douglas R. de Oliveira


The implementation for SPARK-2333 changed the machine membership mechanism from security groups to tags.

This is a fundamentally flawed strategy as there aren't guarantees at all the machines will have a tag (even with a retry mechanism).

For instance, if the script is killed after launching the instances but before setting the tags the machines will be "invisible" to a destroy command, leaving a unmanageable cluster behind.

The initial proposal is to go back to the previous behavior for all cases but when the new flag (--security-group-prefix) is used.

Also it's worthwhile to mention that SPARK-3180 introduced the --additional-security-group flag which is a reasonable solution to SPARK-2333 (but isn't a full replacement to all use cases of --security-group-prefix).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org