You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@myriad.apache.org by John Omernik <jo...@omernik.com> on 2015/09/09 15:41:45 UTC
Question about the Wiki Instructions on yarn-site.xml
This was another "slipped in" question in my other thread, I am breaking
out for specific instructions. Basically, I was struggling with with some
things in the wiki on this page:
https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
In step 5:
Step 5: Configure YARN to use Myriad
Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as instructed
in Sample: myriad-config-default.yml
<https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml>
.
Issue 1: It should link to the yarn-site.xml page, not hte
myriad-config.default.yml page
Issue 2:
It has us put that information in the yarn-site.xml This makes sense. The
resource manager needs to be aware of the myriad stuff.
Then I go to create a tarball, (which I SHOULD be able to use for both
resource manager and nodemanager... right?) However, the instructions state
to remove the *.xml files.
Step 6: Create the Tarball
The tarball has all of the files needed for the Node Managers and Resource
Managers. The following shows how to create the tarball and place it in
HDFS:
cd ~
sudo cp -rp /opt/hadoop-2.7.0 .
sudo rm hadoop-2.7.0/etc/hadoop/*.xml
sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
What I ended up doing... since I am running the resourcemanager (myriad) in
marathon, is I created two tarballs. One is my hadoop-2.7.0-RM.tar.gz which
has the all the xml files still in the tar ball for shipping to marathon.
Then other is hadoop-2.7.0-NM.tar.gz which per the instructions removes the
*.xml files from the /etc/hadoop/ directory.
I guess... my logic is that myriad creates the conf directory for the
nodemanagers... but then I thought, and I overthinking something? Am I
missing something? Could that be factoring into what I am doing here?
Obviously my first steps are to add the extra yarn-site.xml entries, but in
this current setup, they are only going into the resource manager yarn-site
as the the node-managers don't have a yarn-site in their directories. Am I
looking at this correctly? Perhaps we could rethink the removal process of
the XML files in the tarball to allow this to work correctly with a single
tarball?
If I am missing something here, please advise!
John
Re: Question about the Wiki Instructions on yarn-site.xml
Posted by Ruth Harris <rh...@maprtech.com>.
Thanks Darin1
--Ruth
On Wed, Sep 9, 2015 at 12:10 PM, John Omernik <jo...@omernik.com> wrote:
> As to editing directly, I am at a new employer, and we are trying to hash
> out if I can sign the Apache Committer Agreement stuff. Thus, my thoughts,
> if the group wants them, will have to be in informal forum posts which I
> can't make any claim to from an IP perspective. I will work on the
> committers document approved, and do more directly with the Wiki, sorry for
> the roundaboutness.
>
> On Wed, Sep 9, 2015 at 1:07 PM, Ruth Harris <rh...@maprtech.com> wrote:
>
> > Hi all,
> >
> > If you can clarify for me also about what the original instructions for
> the
> > Admin was trying to do and provide clearer information, I can update with
> > wiki information and then update the .markdown file in github.
> > Alternatively, the SME can update the wiki directly.
> >
> > I only walked through the config and build information associated with
> the
> > Developer information.
> >
> > Thanks, Ruth
> >
> > On Wed, Sep 9, 2015 at 8:52 AM, Darin Johnson <db...@gmail.com>
> > wrote:
> >
> > > John,
> > > Understood I don't think making the tempdir be setup that way is ideal.
> > > We've had issues with other frameworks in the past.
> > > Darin
> > > On Sep 9, 2015 11:48 AM, "John Omernik" <jo...@omernik.com> wrote:
> > >
> > > > Well at this point my biggest issue the root user stuff in the other
> > > thread
> > > > and figuring out how to get it to work without making my slave's
> mesos
> > > temp
> > > > only writable by root (is there a work around? And is this a best
> > > practice
> > > > anyhow? what are the down stream effects of this etc)
> > > >
> > > > On Wed, Sep 9, 2015 at 10:45 AM, Darin Johnson <
> > dbjohnson1978@gmail.com>
> > > > wrote:
> > > >
> > > > > Hey John I'm going to try to recreate issue using vanilla hadoop
> > later
> > > > > today. Any other settings I should know about?
> > > > > Darin
> > > > > On Sep 9, 2015 9:42 AM, "John Omernik" <jo...@omernik.com> wrote:
> > > > >
> > > > > > This was another "slipped in" question in my other thread, I am
> > > > breaking
> > > > > > out for specific instructions. Basically, I was struggling with
> > with
> > > > > some
> > > > > > things in the wiki on this page:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> > > > > >
> > > > > > In step 5:
> > > > > > Step 5: Configure YARN to use Myriad
> > > > > >
> > > > > > Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as
> > > > > instructed
> > > > > > in Sample: myriad-config-default.yml
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> > > > > > >
> > > > > > .
> > > > > >
> > > > > >
> > > > > > Issue 1: It should link to the yarn-site.xml page, not hte
> > > > > > myriad-config.default.yml page
> > > > > >
> > > > > > Issue 2:
> > > > > > It has us put that information in the yarn-site.xml This makes
> > sense.
> > > > > The
> > > > > > resource manager needs to be aware of the myriad stuff.
> > > > > >
> > > > > > Then I go to create a tarball, (which I SHOULD be able to use for
> > > both
> > > > > > resource manager and nodemanager... right?) However, the
> > instructions
> > > > > state
> > > > > > to remove the *.xml files.
> > > > > >
> > > > > > Step 6: Create the Tarball
> > > > > >
> > > > > > The tarball has all of the files needed for the Node Managers and
> > > > > Resource
> > > > > > Managers. The following shows how to create the tarball and place
> > it
> > > in
> > > > > > HDFS:
> > > > > > cd ~
> > > > > > sudo cp -rp /opt/hadoop-2.7.0 .
> > > > > > sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> > > > > > sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> > > > > > hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
> > > > > >
> > > > > >
> > > > > > What I ended up doing... since I am running the resourcemanager
> > > > (myriad)
> > > > > in
> > > > > > marathon, is I created two tarballs. One is my
> > hadoop-2.7.0-RM.tar.gz
> > > > > which
> > > > > > has the all the xml files still in the tar ball for shipping to
> > > > marathon.
> > > > > > Then other is hadoop-2.7.0-NM.tar.gz which per the instructions
> > > removes
> > > > > the
> > > > > > *.xml files from the /etc/hadoop/ directory.
> > > > > >
> > > > > >
> > > > > > I guess... my logic is that myriad creates the conf directory for
> > the
> > > > > > nodemanagers... but then I thought, and I overthinking something?
> > Am
> > > I
> > > > > > missing something? Could that be factoring into what I am doing
> > here?
> > > > > >
> > > > > >
> > > > > > Obviously my first steps are to add the extra yarn-site.xml
> > entries,
> > > > but
> > > > > in
> > > > > > this current setup, they are only going into the resource manager
> > > > > yarn-site
> > > > > > as the the node-managers don't have a yarn-site in their
> > directories.
> > > > > Am I
> > > > > > looking at this correctly? Perhaps we could rethink the removal
> > > > process
> > > > > of
> > > > > > the XML files in the tarball to allow this to work correctly
> with a
> > > > > single
> > > > > > tarball?
> > > > > >
> > > > > > If I am missing something here, please advise!
> > > > > >
> > > > > >
> > > > > > John
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Ruth Harris
> > Sr. Technical Writer, MapR
> >
>
--
Ruth Harris
Sr. Technical Writer, MapR
Re: Question about the Wiki Instructions on yarn-site.xml
Posted by John Omernik <jo...@omernik.com>.
As to editing directly, I am at a new employer, and we are trying to hash
out if I can sign the Apache Committer Agreement stuff. Thus, my thoughts,
if the group wants them, will have to be in informal forum posts which I
can't make any claim to from an IP perspective. I will work on the
committers document approved, and do more directly with the Wiki, sorry for
the roundaboutness.
On Wed, Sep 9, 2015 at 1:07 PM, Ruth Harris <rh...@maprtech.com> wrote:
> Hi all,
>
> If you can clarify for me also about what the original instructions for the
> Admin was trying to do and provide clearer information, I can update with
> wiki information and then update the .markdown file in github.
> Alternatively, the SME can update the wiki directly.
>
> I only walked through the config and build information associated with the
> Developer information.
>
> Thanks, Ruth
>
> On Wed, Sep 9, 2015 at 8:52 AM, Darin Johnson <db...@gmail.com>
> wrote:
>
> > John,
> > Understood I don't think making the tempdir be setup that way is ideal.
> > We've had issues with other frameworks in the past.
> > Darin
> > On Sep 9, 2015 11:48 AM, "John Omernik" <jo...@omernik.com> wrote:
> >
> > > Well at this point my biggest issue the root user stuff in the other
> > thread
> > > and figuring out how to get it to work without making my slave's mesos
> > temp
> > > only writable by root (is there a work around? And is this a best
> > practice
> > > anyhow? what are the down stream effects of this etc)
> > >
> > > On Wed, Sep 9, 2015 at 10:45 AM, Darin Johnson <
> dbjohnson1978@gmail.com>
> > > wrote:
> > >
> > > > Hey John I'm going to try to recreate issue using vanilla hadoop
> later
> > > > today. Any other settings I should know about?
> > > > Darin
> > > > On Sep 9, 2015 9:42 AM, "John Omernik" <jo...@omernik.com> wrote:
> > > >
> > > > > This was another "slipped in" question in my other thread, I am
> > > breaking
> > > > > out for specific instructions. Basically, I was struggling with
> with
> > > > some
> > > > > things in the wiki on this page:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> > > > >
> > > > > In step 5:
> > > > > Step 5: Configure YARN to use Myriad
> > > > >
> > > > > Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as
> > > > instructed
> > > > > in Sample: myriad-config-default.yml
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> > > > > >
> > > > > .
> > > > >
> > > > >
> > > > > Issue 1: It should link to the yarn-site.xml page, not hte
> > > > > myriad-config.default.yml page
> > > > >
> > > > > Issue 2:
> > > > > It has us put that information in the yarn-site.xml This makes
> sense.
> > > > The
> > > > > resource manager needs to be aware of the myriad stuff.
> > > > >
> > > > > Then I go to create a tarball, (which I SHOULD be able to use for
> > both
> > > > > resource manager and nodemanager... right?) However, the
> instructions
> > > > state
> > > > > to remove the *.xml files.
> > > > >
> > > > > Step 6: Create the Tarball
> > > > >
> > > > > The tarball has all of the files needed for the Node Managers and
> > > > Resource
> > > > > Managers. The following shows how to create the tarball and place
> it
> > in
> > > > > HDFS:
> > > > > cd ~
> > > > > sudo cp -rp /opt/hadoop-2.7.0 .
> > > > > sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> > > > > sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> > > > > hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
> > > > >
> > > > >
> > > > > What I ended up doing... since I am running the resourcemanager
> > > (myriad)
> > > > in
> > > > > marathon, is I created two tarballs. One is my
> hadoop-2.7.0-RM.tar.gz
> > > > which
> > > > > has the all the xml files still in the tar ball for shipping to
> > > marathon.
> > > > > Then other is hadoop-2.7.0-NM.tar.gz which per the instructions
> > removes
> > > > the
> > > > > *.xml files from the /etc/hadoop/ directory.
> > > > >
> > > > >
> > > > > I guess... my logic is that myriad creates the conf directory for
> the
> > > > > nodemanagers... but then I thought, and I overthinking something?
> Am
> > I
> > > > > missing something? Could that be factoring into what I am doing
> here?
> > > > >
> > > > >
> > > > > Obviously my first steps are to add the extra yarn-site.xml
> entries,
> > > but
> > > > in
> > > > > this current setup, they are only going into the resource manager
> > > > yarn-site
> > > > > as the the node-managers don't have a yarn-site in their
> directories.
> > > > Am I
> > > > > looking at this correctly? Perhaps we could rethink the removal
> > > process
> > > > of
> > > > > the XML files in the tarball to allow this to work correctly with a
> > > > single
> > > > > tarball?
> > > > >
> > > > > If I am missing something here, please advise!
> > > > >
> > > > >
> > > > > John
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Ruth Harris
> Sr. Technical Writer, MapR
>
Re: Question about the Wiki Instructions on yarn-site.xml
Posted by Darin Johnson <db...@gmail.com>.
Ruth, I wrote most of the remote distribution code and am going over John's
issue now. Will inform you later today/tomorrow and/or update the wiki.
Darin
On Wed, Sep 9, 2015 at 2:07 PM, Ruth Harris <rh...@maprtech.com> wrote:
> Hi all,
>
> If you can clarify for me also about what the original instructions for the
> Admin was trying to do and provide clearer information, I can update with
> wiki information and then update the .markdown file in github.
> Alternatively, the SME can update the wiki directly.
>
> I only walked through the config and build information associated with the
> Developer information.
>
> Thanks, Ruth
>
> On Wed, Sep 9, 2015 at 8:52 AM, Darin Johnson <db...@gmail.com>
> wrote:
>
> > John,
> > Understood I don't think making the tempdir be setup that way is ideal.
> > We've had issues with other frameworks in the past.
> > Darin
> > On Sep 9, 2015 11:48 AM, "John Omernik" <jo...@omernik.com> wrote:
> >
> > > Well at this point my biggest issue the root user stuff in the other
> > thread
> > > and figuring out how to get it to work without making my slave's mesos
> > temp
> > > only writable by root (is there a work around? And is this a best
> > practice
> > > anyhow? what are the down stream effects of this etc)
> > >
> > > On Wed, Sep 9, 2015 at 10:45 AM, Darin Johnson <
> dbjohnson1978@gmail.com>
> > > wrote:
> > >
> > > > Hey John I'm going to try to recreate issue using vanilla hadoop
> later
> > > > today. Any other settings I should know about?
> > > > Darin
> > > > On Sep 9, 2015 9:42 AM, "John Omernik" <jo...@omernik.com> wrote:
> > > >
> > > > > This was another "slipped in" question in my other thread, I am
> > > breaking
> > > > > out for specific instructions. Basically, I was struggling with
> with
> > > > some
> > > > > things in the wiki on this page:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> > > > >
> > > > > In step 5:
> > > > > Step 5: Configure YARN to use Myriad
> > > > >
> > > > > Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as
> > > > instructed
> > > > > in Sample: myriad-config-default.yml
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> > > > > >
> > > > > .
> > > > >
> > > > >
> > > > > Issue 1: It should link to the yarn-site.xml page, not hte
> > > > > myriad-config.default.yml page
> > > > >
> > > > > Issue 2:
> > > > > It has us put that information in the yarn-site.xml This makes
> sense.
> > > > The
> > > > > resource manager needs to be aware of the myriad stuff.
> > > > >
> > > > > Then I go to create a tarball, (which I SHOULD be able to use for
> > both
> > > > > resource manager and nodemanager... right?) However, the
> instructions
> > > > state
> > > > > to remove the *.xml files.
> > > > >
> > > > > Step 6: Create the Tarball
> > > > >
> > > > > The tarball has all of the files needed for the Node Managers and
> > > > Resource
> > > > > Managers. The following shows how to create the tarball and place
> it
> > in
> > > > > HDFS:
> > > > > cd ~
> > > > > sudo cp -rp /opt/hadoop-2.7.0 .
> > > > > sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> > > > > sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> > > > > hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
> > > > >
> > > > >
> > > > > What I ended up doing... since I am running the resourcemanager
> > > (myriad)
> > > > in
> > > > > marathon, is I created two tarballs. One is my
> hadoop-2.7.0-RM.tar.gz
> > > > which
> > > > > has the all the xml files still in the tar ball for shipping to
> > > marathon.
> > > > > Then other is hadoop-2.7.0-NM.tar.gz which per the instructions
> > removes
> > > > the
> > > > > *.xml files from the /etc/hadoop/ directory.
> > > > >
> > > > >
> > > > > I guess... my logic is that myriad creates the conf directory for
> the
> > > > > nodemanagers... but then I thought, and I overthinking something?
> Am
> > I
> > > > > missing something? Could that be factoring into what I am doing
> here?
> > > > >
> > > > >
> > > > > Obviously my first steps are to add the extra yarn-site.xml
> entries,
> > > but
> > > > in
> > > > > this current setup, they are only going into the resource manager
> > > > yarn-site
> > > > > as the the node-managers don't have a yarn-site in their
> directories.
> > > > Am I
> > > > > looking at this correctly? Perhaps we could rethink the removal
> > > process
> > > > of
> > > > > the XML files in the tarball to allow this to work correctly with a
> > > > single
> > > > > tarball?
> > > > >
> > > > > If I am missing something here, please advise!
> > > > >
> > > > >
> > > > > John
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Ruth Harris
> Sr. Technical Writer, MapR
>
Re: Question about the Wiki Instructions on yarn-site.xml
Posted by Ruth Harris <rh...@maprtech.com>.
Hi all,
If you can clarify for me also about what the original instructions for the
Admin was trying to do and provide clearer information, I can update with
wiki information and then update the .markdown file in github.
Alternatively, the SME can update the wiki directly.
I only walked through the config and build information associated with the
Developer information.
Thanks, Ruth
On Wed, Sep 9, 2015 at 8:52 AM, Darin Johnson <db...@gmail.com>
wrote:
> John,
> Understood I don't think making the tempdir be setup that way is ideal.
> We've had issues with other frameworks in the past.
> Darin
> On Sep 9, 2015 11:48 AM, "John Omernik" <jo...@omernik.com> wrote:
>
> > Well at this point my biggest issue the root user stuff in the other
> thread
> > and figuring out how to get it to work without making my slave's mesos
> temp
> > only writable by root (is there a work around? And is this a best
> practice
> > anyhow? what are the down stream effects of this etc)
> >
> > On Wed, Sep 9, 2015 at 10:45 AM, Darin Johnson <db...@gmail.com>
> > wrote:
> >
> > > Hey John I'm going to try to recreate issue using vanilla hadoop later
> > > today. Any other settings I should know about?
> > > Darin
> > > On Sep 9, 2015 9:42 AM, "John Omernik" <jo...@omernik.com> wrote:
> > >
> > > > This was another "slipped in" question in my other thread, I am
> > breaking
> > > > out for specific instructions. Basically, I was struggling with with
> > > some
> > > > things in the wiki on this page:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> > > >
> > > > In step 5:
> > > > Step 5: Configure YARN to use Myriad
> > > >
> > > > Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as
> > > instructed
> > > > in Sample: myriad-config-default.yml
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> > > > >
> > > > .
> > > >
> > > >
> > > > Issue 1: It should link to the yarn-site.xml page, not hte
> > > > myriad-config.default.yml page
> > > >
> > > > Issue 2:
> > > > It has us put that information in the yarn-site.xml This makes sense.
> > > The
> > > > resource manager needs to be aware of the myriad stuff.
> > > >
> > > > Then I go to create a tarball, (which I SHOULD be able to use for
> both
> > > > resource manager and nodemanager... right?) However, the instructions
> > > state
> > > > to remove the *.xml files.
> > > >
> > > > Step 6: Create the Tarball
> > > >
> > > > The tarball has all of the files needed for the Node Managers and
> > > Resource
> > > > Managers. The following shows how to create the tarball and place it
> in
> > > > HDFS:
> > > > cd ~
> > > > sudo cp -rp /opt/hadoop-2.7.0 .
> > > > sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> > > > sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> > > > hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
> > > >
> > > >
> > > > What I ended up doing... since I am running the resourcemanager
> > (myriad)
> > > in
> > > > marathon, is I created two tarballs. One is my hadoop-2.7.0-RM.tar.gz
> > > which
> > > > has the all the xml files still in the tar ball for shipping to
> > marathon.
> > > > Then other is hadoop-2.7.0-NM.tar.gz which per the instructions
> removes
> > > the
> > > > *.xml files from the /etc/hadoop/ directory.
> > > >
> > > >
> > > > I guess... my logic is that myriad creates the conf directory for the
> > > > nodemanagers... but then I thought, and I overthinking something? Am
> I
> > > > missing something? Could that be factoring into what I am doing here?
> > > >
> > > >
> > > > Obviously my first steps are to add the extra yarn-site.xml entries,
> > but
> > > in
> > > > this current setup, they are only going into the resource manager
> > > yarn-site
> > > > as the the node-managers don't have a yarn-site in their directories.
> > > Am I
> > > > looking at this correctly? Perhaps we could rethink the removal
> > process
> > > of
> > > > the XML files in the tarball to allow this to work correctly with a
> > > single
> > > > tarball?
> > > >
> > > > If I am missing something here, please advise!
> > > >
> > > >
> > > > John
> > > >
> > >
> >
>
--
Ruth Harris
Sr. Technical Writer, MapR
Re: Question about the Wiki Instructions on yarn-site.xml
Posted by Darin Johnson <db...@gmail.com>.
John,
Understood I don't think making the tempdir be setup that way is ideal.
We've had issues with other frameworks in the past.
Darin
On Sep 9, 2015 11:48 AM, "John Omernik" <jo...@omernik.com> wrote:
> Well at this point my biggest issue the root user stuff in the other thread
> and figuring out how to get it to work without making my slave's mesos temp
> only writable by root (is there a work around? And is this a best practice
> anyhow? what are the down stream effects of this etc)
>
> On Wed, Sep 9, 2015 at 10:45 AM, Darin Johnson <db...@gmail.com>
> wrote:
>
> > Hey John I'm going to try to recreate issue using vanilla hadoop later
> > today. Any other settings I should know about?
> > Darin
> > On Sep 9, 2015 9:42 AM, "John Omernik" <jo...@omernik.com> wrote:
> >
> > > This was another "slipped in" question in my other thread, I am
> breaking
> > > out for specific instructions. Basically, I was struggling with with
> > some
> > > things in the wiki on this page:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> > >
> > > In step 5:
> > > Step 5: Configure YARN to use Myriad
> > >
> > > Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as
> > instructed
> > > in Sample: myriad-config-default.yml
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> > > >
> > > .
> > >
> > >
> > > Issue 1: It should link to the yarn-site.xml page, not hte
> > > myriad-config.default.yml page
> > >
> > > Issue 2:
> > > It has us put that information in the yarn-site.xml This makes sense.
> > The
> > > resource manager needs to be aware of the myriad stuff.
> > >
> > > Then I go to create a tarball, (which I SHOULD be able to use for both
> > > resource manager and nodemanager... right?) However, the instructions
> > state
> > > to remove the *.xml files.
> > >
> > > Step 6: Create the Tarball
> > >
> > > The tarball has all of the files needed for the Node Managers and
> > Resource
> > > Managers. The following shows how to create the tarball and place it in
> > > HDFS:
> > > cd ~
> > > sudo cp -rp /opt/hadoop-2.7.0 .
> > > sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> > > sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> > > hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
> > >
> > >
> > > What I ended up doing... since I am running the resourcemanager
> (myriad)
> > in
> > > marathon, is I created two tarballs. One is my hadoop-2.7.0-RM.tar.gz
> > which
> > > has the all the xml files still in the tar ball for shipping to
> marathon.
> > > Then other is hadoop-2.7.0-NM.tar.gz which per the instructions removes
> > the
> > > *.xml files from the /etc/hadoop/ directory.
> > >
> > >
> > > I guess... my logic is that myriad creates the conf directory for the
> > > nodemanagers... but then I thought, and I overthinking something? Am I
> > > missing something? Could that be factoring into what I am doing here?
> > >
> > >
> > > Obviously my first steps are to add the extra yarn-site.xml entries,
> but
> > in
> > > this current setup, they are only going into the resource manager
> > yarn-site
> > > as the the node-managers don't have a yarn-site in their directories.
> > Am I
> > > looking at this correctly? Perhaps we could rethink the removal
> process
> > of
> > > the XML files in the tarball to allow this to work correctly with a
> > single
> > > tarball?
> > >
> > > If I am missing something here, please advise!
> > >
> > >
> > > John
> > >
> >
>
Re: Question about the Wiki Instructions on yarn-site.xml
Posted by John Omernik <jo...@omernik.com>.
Well at this point my biggest issue the root user stuff in the other thread
and figuring out how to get it to work without making my slave's mesos temp
only writable by root (is there a work around? And is this a best practice
anyhow? what are the down stream effects of this etc)
On Wed, Sep 9, 2015 at 10:45 AM, Darin Johnson <db...@gmail.com>
wrote:
> Hey John I'm going to try to recreate issue using vanilla hadoop later
> today. Any other settings I should know about?
> Darin
> On Sep 9, 2015 9:42 AM, "John Omernik" <jo...@omernik.com> wrote:
>
> > This was another "slipped in" question in my other thread, I am breaking
> > out for specific instructions. Basically, I was struggling with with
> some
> > things in the wiki on this page:
> >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> >
> > In step 5:
> > Step 5: Configure YARN to use Myriad
> >
> > Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as
> instructed
> > in Sample: myriad-config-default.yml
> > <
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> > >
> > .
> >
> >
> > Issue 1: It should link to the yarn-site.xml page, not hte
> > myriad-config.default.yml page
> >
> > Issue 2:
> > It has us put that information in the yarn-site.xml This makes sense.
> The
> > resource manager needs to be aware of the myriad stuff.
> >
> > Then I go to create a tarball, (which I SHOULD be able to use for both
> > resource manager and nodemanager... right?) However, the instructions
> state
> > to remove the *.xml files.
> >
> > Step 6: Create the Tarball
> >
> > The tarball has all of the files needed for the Node Managers and
> Resource
> > Managers. The following shows how to create the tarball and place it in
> > HDFS:
> > cd ~
> > sudo cp -rp /opt/hadoop-2.7.0 .
> > sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> > sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> > hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
> >
> >
> > What I ended up doing... since I am running the resourcemanager (myriad)
> in
> > marathon, is I created two tarballs. One is my hadoop-2.7.0-RM.tar.gz
> which
> > has the all the xml files still in the tar ball for shipping to marathon.
> > Then other is hadoop-2.7.0-NM.tar.gz which per the instructions removes
> the
> > *.xml files from the /etc/hadoop/ directory.
> >
> >
> > I guess... my logic is that myriad creates the conf directory for the
> > nodemanagers... but then I thought, and I overthinking something? Am I
> > missing something? Could that be factoring into what I am doing here?
> >
> >
> > Obviously my first steps are to add the extra yarn-site.xml entries, but
> in
> > this current setup, they are only going into the resource manager
> yarn-site
> > as the the node-managers don't have a yarn-site in their directories.
> Am I
> > looking at this correctly? Perhaps we could rethink the removal process
> of
> > the XML files in the tarball to allow this to work correctly with a
> single
> > tarball?
> >
> > If I am missing something here, please advise!
> >
> >
> > John
> >
>
Re: Question about the Wiki Instructions on yarn-site.xml
Posted by Darin Johnson <db...@gmail.com>.
Hey John I'm going to try to recreate issue using vanilla hadoop later
today. Any other settings I should know about?
Darin
On Sep 9, 2015 9:42 AM, "John Omernik" <jo...@omernik.com> wrote:
> This was another "slipped in" question in my other thread, I am breaking
> out for specific instructions. Basically, I was struggling with with some
> things in the wiki on this page:
>
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
>
> In step 5:
> Step 5: Configure YARN to use Myriad
>
> Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as instructed
> in Sample: myriad-config-default.yml
> <
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> >
> .
>
>
> Issue 1: It should link to the yarn-site.xml page, not hte
> myriad-config.default.yml page
>
> Issue 2:
> It has us put that information in the yarn-site.xml This makes sense. The
> resource manager needs to be aware of the myriad stuff.
>
> Then I go to create a tarball, (which I SHOULD be able to use for both
> resource manager and nodemanager... right?) However, the instructions state
> to remove the *.xml files.
>
> Step 6: Create the Tarball
>
> The tarball has all of the files needed for the Node Managers and Resource
> Managers. The following shows how to create the tarball and place it in
> HDFS:
> cd ~
> sudo cp -rp /opt/hadoop-2.7.0 .
> sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
>
>
> What I ended up doing... since I am running the resourcemanager (myriad) in
> marathon, is I created two tarballs. One is my hadoop-2.7.0-RM.tar.gz which
> has the all the xml files still in the tar ball for shipping to marathon.
> Then other is hadoop-2.7.0-NM.tar.gz which per the instructions removes the
> *.xml files from the /etc/hadoop/ directory.
>
>
> I guess... my logic is that myriad creates the conf directory for the
> nodemanagers... but then I thought, and I overthinking something? Am I
> missing something? Could that be factoring into what I am doing here?
>
>
> Obviously my first steps are to add the extra yarn-site.xml entries, but in
> this current setup, they are only going into the resource manager yarn-site
> as the the node-managers don't have a yarn-site in their directories. Am I
> looking at this correctly? Perhaps we could rethink the removal process of
> the XML files in the tarball to allow this to work correctly with a single
> tarball?
>
> If I am missing something here, please advise!
>
>
> John
>