[jira] [Created] (AURORA-367) simple commands using sudo do not go through FINALIZING state

["Jay Buffington (JIRA)" <ji...@apache.org>]

Jay Buffington created AURORA-367:
-------------------------------------

             Summary: simple commands using sudo do not go through FINALIZING state
                 Key: AURORA-367
                 URL: https://issues.apache.org/jira/browse/AURORA-367
             Project: Aurora
          Issue Type: Bug
          Components: Executor
            Reporter: Jay Buffington
            Priority: Minor


The below .aurora file causes the FINALIZING state to be skipped.  It goes
through these states:

{noformat}
    ACTIVE   04/28 18:33:29
    ACTIVE   04/28 18:35:51
    CLEANING     04/28 18:35:51
    KILLED   04/28 18:36:51
{noformat}


Here's the definition of the job:
{noformat}
    $ cat fail_finalize.aurora
    jobs = [Job(
      task=SimpleTask(name="fail_finalize", command="""
         sudo sleep 600
    """),
      role='jaybuff',
      environment="prod",
      cluster="vp21d01cp")]
{noformat}
{noformat}
    $ aurora inspect  vp21d01cp/jaybuff/prod/fail_finalize fail_finalize.aurora
    Job level information
      name:       fail_finalize
      role:       jaybuff
      contact:    <class 'pystachio.composite.Empty'>
      cluster:    vp21d01cp
      instances:  1
      service:    False
      production: False

    Task level information
      name: fail_finalize

    Process fail_finalize:
      cmdline:

             sudo sleep 600

    $
{noformat}

It looks like this is caused due to bash exec'ing "simple commands" rather than "fork+exec" like it does with a "complex command" like {{sudo sleep 600; 1}}  To demonstrate what I'm talking about:

{{/bin/bash -c "(sudo sleep 400)"}} forces the command to run in a subshell (effectively doing a fork+exec), so we see this:

{noformat}
    $ ps afo pid,user,cmd
      PID USER     CMD
     5320 jaybuff  -bash
    18651 jaybuff   \_ /bin/bash -c (sudo sleep 400)
    18652 root          \_ sudo sleep 400
    18653 root              \_ sleep 400
{noformat}


Whereas {{/bin/bash -c "sudo sleep 400"}} doesn't use a subshell:

{noformat}
    $ ps afo pid,user,cmd
      PID USER     CMD
     5320 jaybuff  -bash
    19805 root      \_ sudo sleep 400
    19806 root          \_ sleep 400
{noformat}


The problem with this is that when the executor goes to kill the task it sends a SIGTERM to the pid it forked.  If bash didn't use a subshell and it exec'ed sudo then that process is running as root and the SIGTERM comes from an unprivileged user, so the SIGTERM is ignored.  I suspect after some timeout the executor is killed by mesos-slave and the FINALIZING state is never reached.

There doesn't look to be a command line flag to tell bash to always use a subshell.  The best option I could find was the bash manual says "Placing a list of commands between parentheses causes a subshell environment to be created"   See http://www.gnu.org/software/bash/manual/bashref.html#Command-Grouping



--
This message was sent by Atlassian JIRA
(v6.2#6252)