You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Michel Sumbul <mi...@gmail.com> on 2020/03/30 23:26:44 UTC
Question about launch_container.sh script
Hi Guys,
I would like to know if there is a way to modify the launch_container.sh
script on a yarn platform already installed?
if my understanding is correct, that script is called each time a container
is started on the node manager. My goal is to be sure that a specific
command is executed at the start (or at the end) of every yarn container.
I was not able to find the file except in the log folder, so a script
that's not going to be use anymore.
Also, is there a script executed at the end of each container? If yes is
there a way to modify it?
Thanks for the help,
Michel
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Garanti
sans virus. www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Unsubscribe
Posted by S B <sa...@gmail.com>.
Unsubscribe
--
When the going gets TOUGH, the TOUGH get going!
Re: Question about launch_container.sh script
Posted by Eric Badger <eb...@verizonmedia.com.INVALID>.
Yea it would probably require quite a bit of modification to your current
setup to get the DockerLinuxContainerRuntime working. And the entrypoint
would only work as a prestart script. It wouldn't give you an avenue to run
something after the task finished.
It's an interesting concept though. If you file a YARN JIRA it might gain
some traction.
Eric
On Tue, Mar 31, 2020 at 7:57 AM Michel Sumbul <mi...@gmail.com>
wrote:
> Hi Eric,
> Thanks for your reply, its really clear!
>
> Apparently it will be difficult to solve that, as the job that we submit
> are spark job. So I don't really see how to combine
> DockerLinuxContainerRuntime with a specific entry point running at the end
> a spark job on top of that container runtime. Moreover to have all of that
> production ready/stable.
>
> Again thanks for your reply,
> Michel
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Garanti
> sans virus. www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_4064178667749827955_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> Le mar. 31 mars 2020 à 00:40, Eric Badger <eb...@verizonmedia.com> a
> écrit :
>
>> The launch_container.sh script is created on the fly by the Nodemanager
>> for each task that is run. So you would need to modify the Nodemanager code
>> to modify the launch_container.sh script.
>>
>> The control flow is:
>> Nodemanager -> container-executor -> launch_container.sh
>>
>> The nodemanager launches the container-executor, which is a setuid binary
>> so that it can change users to the user that the process should be run as.
>> Then the container-executor execs the launch_container.sh script. The
>> container-executor waits around for the task to finish and reads its
>> stdout/stderr. Once the task is done, the container-executor cleans up as
>> well.
>>
>> The only way I know of to inject a pre-launch script would be to use the
>> DockerLinuxContainerRuntime and create an image with an Entrypoint. You
>> could set that Entrypoint as the script that you want to run and then the
>> Entrypoint script could end by exec'ing into launch_container.sh.
>>
>> Hope this helps,
>>
>> Eric
>>
>
Re: Question about launch_container.sh script
Posted by Michel Sumbul <mi...@gmail.com>.
Hi Eric,
Thanks for your reply, its really clear!
Apparently it will be difficult to solve that, as the job that we submit
are spark job. So I don't really see how to combine
DockerLinuxContainerRuntime with a specific entry point running at the end
a spark job on top of that container runtime. Moreover to have all of that
production ready/stable.
Again thanks for your reply,
Michel
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Garanti
sans virus. www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Le mar. 31 mars 2020 à 00:40, Eric Badger <eb...@verizonmedia.com> a
écrit :
> The launch_container.sh script is created on the fly by the Nodemanager
> for each task that is run. So you would need to modify the Nodemanager code
> to modify the launch_container.sh script.
>
> The control flow is:
> Nodemanager -> container-executor -> launch_container.sh
>
> The nodemanager launches the container-executor, which is a setuid binary
> so that it can change users to the user that the process should be run as.
> Then the container-executor execs the launch_container.sh script. The
> container-executor waits around for the task to finish and reads its
> stdout/stderr. Once the task is done, the container-executor cleans up as
> well.
>
> The only way I know of to inject a pre-launch script would be to use the
> DockerLinuxContainerRuntime and create an image with an Entrypoint. You
> could set that Entrypoint as the script that you want to run and then the
> Entrypoint script could end by exec'ing into launch_container.sh.
>
> Hope this helps,
>
> Eric
>
Re: Question about launch_container.sh script
Posted by Eric Badger <eb...@verizonmedia.com.INVALID>.
The launch_container.sh script is created on the fly by the Nodemanager for
each task that is run. So you would need to modify the Nodemanager code to
modify the launch_container.sh script.
The control flow is:
Nodemanager -> container-executor -> launch_container.sh
The nodemanager launches the container-executor, which is a setuid binary
so that it can change users to the user that the process should be run as.
Then the container-executor execs the launch_container.sh script. The
container-executor waits around for the task to finish and reads its
stdout/stderr. Once the task is done, the container-executor cleans up as
well.
The only way I know of to inject a pre-launch script would be to use the
DockerLinuxContainerRuntime and create an image with an Entrypoint. You
could set that Entrypoint as the script that you want to run and then the
Entrypoint script could end by exec'ing into launch_container.sh.
Hope this helps,
Eric