You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Peter Rudenko <pe...@gmail.com> on 2015/11/02 11:16:07 UTC
How to set user in ContainerLaunchContext?
Hi, i have an ApplicationMaster which accepts requests and launches
spark jobs. In request i have a field "username" - the user i want to
laucnh a job from. How can i set a user which will be run conmmand on a
container? Currently they all running as yarn user even though AM is
running as a root user.
Thanks,
Peter Rudenko
Re: How to set user in ContainerLaunchContext?
Posted by Varun Vasudev <vv...@apache.org>.
Hi Peter,
What’s the container executor you’re using? If you’re using the DefaultContainerExecutor, all jobs are run as the yarn user. You’ll have to use the LinuxContainerExecutor to run containers as the user who submitted them.
-Varun
On 11/2/15, 3:46 PM, "Peter Rudenko" <pe...@gmail.com> wrote:
>Hi, i have an ApplicationMaster which accepts requests and launches
>spark jobs. In request i have a field "username" - the user i want to
>laucnh a job from. How can i set a user which will be run conmmand on a
>container? Currently they all running as yarn user even though AM is
>running as a root user.
>
>Thanks,
>Peter Rudenko
Re: How to set user in ContainerLaunchContext?
Posted by Varun Vasudev <vv...@apache.org>.
Hi Peter,
What’s the container executor you’re using? If you’re using the DefaultContainerExecutor, all jobs are run as the yarn user. You’ll have to use the LinuxContainerExecutor to run containers as the user who submitted them.
-Varun
On 11/2/15, 3:46 PM, "Peter Rudenko" <pe...@gmail.com> wrote:
>Hi, i have an ApplicationMaster which accepts requests and launches
>spark jobs. In request i have a field "username" - the user i want to
>laucnh a job from. How can i set a user which will be run conmmand on a
>container? Currently they all running as yarn user even though AM is
>running as a root user.
>
>Thanks,
>Peter Rudenko
Re: [Yarn] How to set user in ContainerLaunchContext?
Posted by Marcelo Vanzin <va...@cloudera.com>.
You can try the "--proxy-user" command line argument for spark-submit.
That requires that your RM configuration allows the user running your
AM to "proxy" other users. And I'm not completely sure it works
without Kerberos.
See: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html
On Mon, Nov 2, 2015 at 8:02 AM, Peter Rudenko <pe...@gmail.com> wrote:
> Hi, i have an ApplicationMaster which accepts requests and launches
> container on which it launches spark-submit --master yarn. In request i have
> a field "username" - the user i want to laucnh a job from. How can i set a
> user which will be run conmmand on a container? Currently they all running
> as yarn user even though AM is running as a root user.
>
> Here's my code:
>
>
> private def setupTokens(username: String): ByteBuffer = {
> val credentials = UserGroupInformation.createProxyUser(username,
> UserGroupInformation.getCurrentUser).getCredentials
> val dob = new DataOutputBuffer();
> credentials.writeTokenStorageToStream(dob);
> ByteBuffer.wrap(dob.getData(), 0, dob.getLength()).duplicate();
> }
>
> val cCLC = Records.newRecord(classOf[ContainerLaunchContext])
>
> cCLC.setCommands(List("spark-submit --master yarn ..."))
>
> cCLC.setTokens(setupTokens(user))
>
> Thanks, Peter Rudenko
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
[Yarn] How to set user in ContainerLaunchContext?
Posted by Peter Rudenko <pe...@gmail.com>.
Hi, i have an ApplicationMaster which accepts requests and launches
container on which it launches spark-submit --master yarn. In request i
have a field "username" - the user i want to laucnh a job from. How can
i set a user which will be run conmmand on a container? Currently they
all running as yarn user even though AM is running as a root user.
Here's my code:
private def setupTokens(username:String): ByteBuffer = {
val credentials = UserGroupInformation.createProxyUser(username, UserGroupInformation.getCurrentUser).getCredentials
val dob =new DataOutputBuffer();
credentials.writeTokenStorageToStream(dob);
ByteBuffer.wrap(dob.getData(),0, dob.getLength()).duplicate();
}
val cCLC = Records.newRecord(classOf[ContainerLaunchContext])
cCLC.setCommands(List("spark-submit --master yarn ..."))
cCLC.setTokens(setupTokens(user))
Thanks, Peter Rudenko
Re: How to set user in ContainerLaunchContext?
Posted by Varun Vasudev <vv...@apache.org>.
Hi Peter,
What’s the container executor you’re using? If you’re using the DefaultContainerExecutor, all jobs are run as the yarn user. You’ll have to use the LinuxContainerExecutor to run containers as the user who submitted them.
-Varun
On 11/2/15, 3:46 PM, "Peter Rudenko" <pe...@gmail.com> wrote:
>Hi, i have an ApplicationMaster which accepts requests and launches
>spark jobs. In request i have a field "username" - the user i want to
>laucnh a job from. How can i set a user which will be run conmmand on a
>container? Currently they all running as yarn user even though AM is
>running as a root user.
>
>Thanks,
>Peter Rudenko
Re: How to set user in ContainerLaunchContext?
Posted by Varun Vasudev <vv...@apache.org>.
Hi Peter,
What’s the container executor you’re using? If you’re using the DefaultContainerExecutor, all jobs are run as the yarn user. You’ll have to use the LinuxContainerExecutor to run containers as the user who submitted them.
-Varun
On 11/2/15, 3:46 PM, "Peter Rudenko" <pe...@gmail.com> wrote:
>Hi, i have an ApplicationMaster which accepts requests and launches
>spark jobs. In request i have a field "username" - the user i want to
>laucnh a job from. How can i set a user which will be run conmmand on a
>container? Currently they all running as yarn user even though AM is
>running as a root user.
>
>Thanks,
>Peter Rudenko