[jira] [Commented] (SPARK-4465) runAsSparkUser doesn't affect TaskRunner in Mesos environment at all.

["Jongyoul Lee (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14253059#comment-14253059 ] 

Jongyoul Lee commented on SPARK-4465:
-------------------------------------

I've dug into mesos code and jni and java.security which is used by UserGroupInformation in runAsSparkUser. Finally I've found an exact wrong point of MesosExecutorBackend, these are about java.security scope and how mesos c++ executor works. Unlike to standalone and yarn, in a task under mesos cluster, jni runs each task in a native method via callback. There are several problems about running java code under jni environment, especially it's not covered by java.security. JDK spec warn the following
{quote}
Some important points about being privileged: Firstly, this concept only exists within a single thread. As soon as the privileged code completes, the privilege is guaranteed to be erased or revoked.
-- http://docs.oracle.com/javase/7/docs/technotes/guides/security/spec/security-spec.doc4.html#20573
{quote}

{code:title=MesosExecutorBackend.scala}
    SparkHadoopUtil.get.runAsSparkUser { () =>
        MesosNativeLibrary.load()
        // Create a new Executor and start it running
        val runner = new MesosExecutorBackend()
        new MesosExecutorDriver(runner).run()
    }
{code}

Show that code. Only run() method is only covered by runAsSparkUser, however, the actual code running tasks is not covered by runAsSparkUser. As a result, MesosExecutorBackend.launchTask doesn't affect by runAsSparkUser.

I'll fix that bug by adding runAsSparkUser into MesosExecutorBackend.launchTask rather and adding them into Executor.launchTask, because Executor.launchTask is already covered in case of standalone and yarn. Thus I judged that code is exact point of fix that bug.

> runAsSparkUser doesn't affect TaskRunner in Mesos environment at all.
> ---------------------------------------------------------------------
>
>                 Key: SPARK-4465
>                 URL: https://issues.apache.org/jira/browse/SPARK-4465
>             Project: Spark
>          Issue Type: Bug
>          Components: Input/Output, Mesos
>            Reporter: Jongyoul Lee
>             Fix For: 1.2.0, 1.3.0
>
>
> runAsSparkUser enable classes using hadoop library to change an active user to spark User, however in Mesos environment, because the function calls before running within JNI, runAsSparkUser doesn't affect anything, and meaningless code. fix the Appropriate scope of function and remove meaningless code. That's a bug because of running program incorrectly. That's related to SPARK-3223 but setting framework user is not perfect solution in my tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org