You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Shane Kumpf (JIRA)" <ji...@apache.org> on 2018/08/10 18:08:00 UTC
[jira] [Comment Edited] (YARN-8623) Update Docker examples to use
image which exists
[ https://issues.apache.org/jira/browse/YARN-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576655#comment-16576655 ]
Shane Kumpf edited comment on YARN-8623 at 8/10/18 6:07 PM:
------------------------------------------------------------
[~ccondit-target] - thanks for looking into this. I see what you mean about the challenge with using that image. I think you are correct that the existing apache/hadoop-runner image serves a different type of use case than we need here.
IMO, our target should be an image capable of running MapReduce pi, as that's the example we provide in the docs. If it also works for the Spark shell example we provide in our docs, with the appropriate spark install/config, that would be great, but I don't think it's a requirement to start.
Thinking about what we need to meet that goal, I think a majority of the users we would be targeting with this guide will have all of Hadoop installed on the nodes where these containers are running. Instead of trying to package the latest version of Apache Hadoop as an image, I think our example would be easier to maintain if we guide the user towards bind mounting the Hadoop binaries and configuration from the NodeManager hosts. If we take that approach, I believe the image should only need to include a JDK and set up JAVA_HOME. We might even be able to use an existing openjdk image.
Assuming we can't leverage an existing image, one question I'm unsure about is the process of creating an "official" image under the apache docker hub namespace. [~elek] - can you share any insights around this process?
was (Author: shanekumpf@gmail.com):
[~ccondit-target] - thanks for looking into this. I see what you mean about the challenge with using that image. I think you are correct that the existing apache/hadoop-runner image serves a different type of use case than we need here.
IMO, our target should be an image capable of running MapReduce pi, as that's the example we provide in the docs. If it also works for Spark shell example we provide in our docs, with the appropriate spark install/config, that would be great, but I don't think it's a requirement to start. !/jira/images/icons/emoticons/smile.png!
Thinking about what we need to meet that goal, I think a majority of the users we would be targeting with this guide will have all of Hadoop installed on the nodes where these containers are running. Instead of trying to package the latest version of Apache Hadoop as an image, I think our example would be easier to maintain if we guide the user towards bind mounting the Hadoop binaries and configuration from the NodeManager hosts. If we take that approach, I believe the image should only need to include a JDK and set up JAVA_HOME. We might even be able to use an existing openjdk image.
Assuming we can't leverage an existing image, one question I'm unsure about is the process of creating an "official" image under the apache docker hub namespace. [~elek] - can you share any insights around this process?
> Update Docker examples to use image which exists
> ------------------------------------------------
>
> Key: YARN-8623
> URL: https://issues.apache.org/jira/browse/YARN-8623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Craig Condit
> Priority: Minor
> Labels: Docker
>
> The example Docker image given in the documentation (images/hadoop-docker:latest) does not exist. We could change images/hadoop-docker:latest to apache/hadoop-runner:latest, which does exist. We'd need to do a quick sanity test to see if the image works with YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org