[jira] [Created] (SPARK-9688) Improve spark-ec2 script to handle users that are not root

["Karina Uribe (JIRA)" <ji...@apache.org>]

Karina Uribe created SPARK-9688:
-----------------------------------

             Summary: Improve spark-ec2 script to handle users that are not root
                 Key: SPARK-9688
                 URL: https://issues.apache.org/jira/browse/SPARK-9688
             Project: Spark
          Issue Type: Improvement
          Components: EC2
    Affects Versions: 1.4.1, 1.4.0
         Environment: All
            Reporter: Karina Uribe


Hi, 

I was trying to use the spark-ec2 script from Spark to create a new Spark cluster wit an user different than root (--user=ec2-user). Unfortunately the part of the script that attempts to copy the templates into the target machines fail because it tries to rsync /etc/* and /root/* 

This is the full traceback

rsync: recv_generator: mkdir "/root/spark-ec2" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***

sent 95 bytes  received 17 bytes  224.00 bytes/sec
total size is 1444  speedup is 12.89
rsync error: some files/attrs were not transferred (see previous errors) (code 2                  3) at main.c(1039) [sender=3.0.6]
Traceback (most recent call last):
  File "/home/ec2-user/spark-1.4.0/ec2/spark_ec2.py", line 1455, in <module>
    main()
  File "/home/ec2-user/spark-1.4.0/ec2/spark_ec2.py", line 1447, in main
    real_main()
  File "/home/ec2-user/spark-1.4.0/ec2/spark_ec2.py", line 1283, in real_main
    setup_cluster(conn, master_nodes, slave_nodes, opts, True)
  File "/home/ec2-user/spark-1.4.0/ec2/spark_ec2.py", line 785, in setup_cluster
    modules=modules
  File "/home/ec2-user/spark-1.4.0/ec2/spark_ec2.py", line 1049, in deploy_files
    subprocess.check_call(command)
  File "/usr/lib64/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['rsync', '-rv', '-e', 'ssh -o StrictHos                  tKeyChecking=no -o UserKnownHostsFile=/dev/null -i /home/ec2-user/.ssh/sparkclus                  terkey_us_east.pem', '/tmp/tmpT4Iw54/', u'ec2-user@ec2-52-2-96-193.compute-1.ama                  zonaws.com:/']' returned non-zero exit status 23


Is there a workaround for this? I want to improve security of our operations by avoiding user root on the instances. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org