You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@myriad.apache.org by John Omernik <jo...@omernik.com> on 2015/08/19 15:28:11 UTC
Documentation Comments
Today, I will be playing the role of the fool/jester trying to get Myriad
running. Basically, since getting Myriad running with Santosh quite a while
ago, and now trying again with new versions of Hadoop, MapR, and Myriad, I
wanted to hit up the wiki (
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home) and outline
points that as a non-dev living the code, are unclear to someone trying to
utilize myriad or understand it's operation.
Obviously, some of my points can be answered with "look here in the code"
or look at this page, but I will try to outline my thought processes as I
reviewed the current docs. Sometimes the way I approached the problem led
me down a path of to a certain page, missing the answer in a different
page, and thus some cross linking could be helpful.
Please do not let my points be taken as anything other than a desire to
improve how accessible Myriad is to the community, this is not a critique
of the hard work everyone has done on the project. I also understand that
given the work load and other issues, that fixing these issues in
documentation may not be a priority. I am listing them out here, so that
those folks who are SMEs on various points may be able to quickly add stuff
and we'll organize it later.
*Remote Distribution: *
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution
This whole section could use some work from a standpoint of what runs where
and where that component gets its files. For example, I think it would
help people to understand that the whole tarball created in step 6 has all
the files for node managers and resource managers. Basically, everything
runs from there. Here is a small example I am currently working with:
Starting Myriad:
Option 1: Use Marathon (provide example json, here is mine)
{
"cmd": "env && export
YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos
&& hadoop-2.7.0/bin/yarn resourcemanager",
"uris": ["maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz"],
"cpus": 1.0,
"mem": 1024,
"id": "myriad",
"instances": 1,
"user": "mapr"
}
In this case, Marathon grabs the hadoop tarball and pulls it down, this
tarball also has the Myriad yml file. When it executes the resource
manager, it is brought up in Myriad and ready to run node managers by
pulling the tarball to the slave nodes and executing the nodemanager. (I
would imagine the work with history server etc would also use this
tarball?).
>From here it will us NMInstances to launch a node manager. (Note, this is
different from when I originally set things up... before, I could run the
resource manager/myriad without a nodemanager, now it seems it's required
based on the config in the src... could we expound on this in the docs
somewhere?)
Option 2: ???? (Are there other ways to launch the resource manager?)
Step 6: So something that is unclear to me is the handling of the
hadoop/yarn config files. In Step 6 on this page, there is "sudo rm hadoop-
2.5.0/etc/hadoop/*.xml" This doesn't makes sense to me. I actually ignored
this step. For me, if I remove these xml files, then there is no place to
get my files... I think? Since I am running the RM and NM from the same
tarball, and Myriad config is here, and my goal is to not have anything
installed on a node, where would I set yarn settings? This could be much
clearer to me, and probably others.
Step 2: Should we just be copying the Myriad files to
/share/hadoop/yarn/lib folder? Do we worry about potentially overwrites of
jars or version conflicts?
*Configuring Cgroups*
https://cwiki.apache.org/confluence/display/MYRIAD/Configuring+Cgroups
At some point a little bit more about why one would want CGroups and issues
that could occur with them. While many folks using Mesos/Myriad may
understand this, others may not, and it's a good way to help people think
positively about our project if we help educate them along the way.
Minor point on enabling CGroups. This is confusing given my questions in
remote distribution. in this it says I need to edit my yarn-site.xml, but
in remote distribution it says delete my hadoop xml files. We need to
address this conflict cause it can be confusing for a user coming onboard
Nitpick: Enabling cgroups for mess-slave - should be - Enabling cgroups for
mesos-slave
*Myriad Configuration Properties*:
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Configuration+Properties
Based on the conversation on list with Yuliya,
"Currently, this file is built into Myriad Scheduler jar. So, if you need
to modify some of the properties in this file, modify them before building
Myriad Scheduler."
isn't accurate any more, and we should address that.
The configuration file in the wiki is an old one, the nmInstances isn't in
it, (and see my question about that above).
Frameworks and usernames. I think the users that the framework runs as,
the actual node and resource managers, etc is confusing to a user (I am
very confused!) When I first got Myriad up I set my user under the
executor to be mapr, and then it appeared to work with impersonation from
queries etc. Now, I am trying the remote distribution and I have users set
in the config, potentially a user in my marathon json, and I am getting
errors on permissions of files when a node manager tries to start (a
separate issue I will post later). Basically, this is complex, and a page
describing out what needs to run where with which permissions and how that
interacts will be huge for people looking to put this into play.
*Example Yarn Site:*
https://cwiki.apache.org/confluence/display/MYRIAD/Example%3A+yarn-site.xml
This is helpful, but where does it go? Remember, the remote distribution
had us delete the yarn-site in the hadoop etc folder.
*Myriad Webapp *
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Webapp
This should be fleshed out a bit more. Also, it's in the
/myriad-scheduler/src/main/resources/webapp based on my git clone, but in
the wiki that's not listed. I had to dig for it.
Some questions here: could the webapp be built during the myriad building
process? Could it be then be packaged as tarball for execution either
manually via marathon or automatically in a container on mesos? I
understand this is a fresh piece of the puzzle, I am just thinking about
and verbalizing the "where" on this for the future
Those are the items that come to mind thus far. I hope the tone of my
email is correct, this is a great project, and I want others to try it as I
have.
John Omernik
Re: Documentation Comments
Posted by Adam Bordelon <ad...@mesosphere.io>.
Add/Edit permissions granted.
I don't think we have official policies, except that as an Apache project
we should try to keep comments/communications public.
It is a wiki, so we can always change/update it as much as we want later.
No official review process, but feel free to send drafts/questions out to
the dev list and then you/we can update pages when the content is ready.
You can also just add a new page/sections to the wiki and then ask the list
to review it.
If anybody disagrees with this proposal, please reply with your
objection/suggestion, otherwise I'll assume lazy consensus.
Like we did with Ruth, it'd be great if you could sign an ICLA
<https://www.apache.org/licenses/#clas> so that it's clear that you're
donating your written content to the Apache Software Foundation for the
project to use as it pleases.
On Wed, Aug 19, 2015 at 11:37 AM, John Omernik <jo...@omernik.com> wrote:
> Thanks Adam, I signed up with "mandoskippy".
>
> I am honored to help in this capacity, for updating etc, do we go through
> some kind of review? Is it better to ask questions on the dev list then
> update when consensus occurs? How about when I'd like to post a page and
> then have someone review the work? If I have a comment on the page, is that
> public or can I just send to author? Just curious on any guidelines I
> should be following in that regard.
>
> John
>
>
>
> On Wed, Aug 19, 2015 at 1:29 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
>
> > John, thanks a ton for your valuable feedback! We're glad to have your
> > perspective as a user of the project, and I'm ready+willing to give you
> > edit access to the wiki if you want to update it with your learnings,
> > elaborate anything that's unclear, or add a new "John's tips" page. Just
> > sign up for a wiki account, send me your accountId, and I'll grant you
> edit
> > access.
> > (I'll let others answer your specific questions)
> >
> > On Wed, Aug 19, 2015 at 6:28 AM, John Omernik <jo...@omernik.com> wrote:
> >
> > > Today, I will be playing the role of the fool/jester trying to get
> Myriad
> > > running. Basically, since getting Myriad running with Santosh quite a
> > while
> > > ago, and now trying again with new versions of Hadoop, MapR, and
> Myriad,
> > I
> > > wanted to hit up the wiki (
> > > https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home) and
> > > outline
> > > points that as a non-dev living the code, are unclear to someone trying
> > to
> > > utilize myriad or understand it's operation.
> > >
> > > Obviously, some of my points can be answered with "look here in the
> code"
> > > or look at this page, but I will try to outline my thought processes
> as I
> > > reviewed the current docs. Sometimes the way I approached the problem
> > led
> > > me down a path of to a certain page, missing the answer in a different
> > > page, and thus some cross linking could be helpful.
> > >
> > > Please do not let my points be taken as anything other than a desire to
> > > improve how accessible Myriad is to the community, this is not a
> critique
> > > of the hard work everyone has done on the project. I also understand
> > that
> > > given the work load and other issues, that fixing these issues in
> > > documentation may not be a priority. I am listing them out here, so
> that
> > > those folks who are SMEs on various points may be able to quickly add
> > stuff
> > > and we'll organize it later.
> > >
> > >
> > > *Remote Distribution: *
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution
> > >
> > > This whole section could use some work from a standpoint of what runs
> > where
> > > and where that component gets its files. For example, I think it would
> > > help people to understand that the whole tarball created in step 6 has
> > all
> > > the files for node managers and resource managers. Basically,
> everything
> > > runs from there. Here is a small example I am currently working with:
> > >
> > >
> > > Starting Myriad:
> > > Option 1: Use Marathon (provide example json, here is mine)
> > > {
> > > "cmd": "env && export
> > >
> > >
> >
> YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos
> > > && hadoop-2.7.0/bin/yarn resourcemanager",
> > > "uris": ["maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz"],
> > > "cpus": 1.0,
> > > "mem": 1024,
> > > "id": "myriad",
> > > "instances": 1,
> > > "user": "mapr"
> > > }
> > >
> > > In this case, Marathon grabs the hadoop tarball and pulls it down, this
> > > tarball also has the Myriad yml file. When it executes the resource
> > > manager, it is brought up in Myriad and ready to run node managers by
> > > pulling the tarball to the slave nodes and executing the nodemanager.
> (I
> > > would imagine the work with history server etc would also use this
> > > tarball?).
> > >
> > > From here it will us NMInstances to launch a node manager. (Note, this
> > is
> > > different from when I originally set things up... before, I could run
> the
> > > resource manager/myriad without a nodemanager, now it seems it's
> required
> > > based on the config in the src... could we expound on this in the docs
> > > somewhere?)
> > >
> > >
> > > Option 2: ???? (Are there other ways to launch the resource manager?)
> > >
> > > Step 6: So something that is unclear to me is the handling of the
> > > hadoop/yarn config files. In Step 6 on this page, there is "sudo rm
> > > hadoop-
> > > 2.5.0/etc/hadoop/*.xml" This doesn't makes sense to me. I actually
> > ignored
> > > this step. For me, if I remove these xml files, then there is no place
> > to
> > > get my files... I think? Since I am running the RM and NM from the same
> > > tarball, and Myriad config is here, and my goal is to not have anything
> > > installed on a node, where would I set yarn settings? This could be
> much
> > > clearer to me, and probably others.
> > >
> > > Step 2: Should we just be copying the Myriad files to
> > > /share/hadoop/yarn/lib folder? Do we worry about potentially overwrites
> > of
> > > jars or version conflicts?
> > >
> > > *Configuring Cgroups*
> > > https://cwiki.apache.org/confluence/display/MYRIAD/Configuring+Cgroups
> > > At some point a little bit more about why one would want CGroups and
> > issues
> > > that could occur with them. While many folks using Mesos/Myriad may
> > > understand this, others may not, and it's a good way to help people
> think
> > > positively about our project if we help educate them along the way.
> > >
> > > Minor point on enabling CGroups. This is confusing given my questions
> in
> > > remote distribution. in this it says I need to edit my yarn-site.xml,
> but
> > > in remote distribution it says delete my hadoop xml files. We need to
> > > address this conflict cause it can be confusing for a user coming
> onboard
> > >
> > > Nitpick: Enabling cgroups for mess-slave - should be - Enabling cgroups
> > for
> > > mesos-slave
> > >
> > > *Myriad Configuration Properties*:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Configuration+Properties
> > > Based on the conversation on list with Yuliya,
> > > "Currently, this file is built into Myriad Scheduler jar. So, if you
> need
> > > to modify some of the properties in this file, modify them before
> > building
> > > Myriad Scheduler."
> > > isn't accurate any more, and we should address that.
> > >
> > > The configuration file in the wiki is an old one, the nmInstances isn't
> > in
> > > it, (and see my question about that above).
> > >
> > > Frameworks and usernames. I think the users that the framework runs
> as,
> > > the actual node and resource managers, etc is confusing to a user (I am
> > > very confused!) When I first got Myriad up I set my user under the
> > > executor to be mapr, and then it appeared to work with impersonation
> from
> > > queries etc. Now, I am trying the remote distribution and I have users
> > set
> > > in the config, potentially a user in my marathon json, and I am getting
> > > errors on permissions of files when a node manager tries to start (a
> > > separate issue I will post later). Basically, this is complex, and a
> page
> > > describing out what needs to run where with which permissions and how
> > that
> > > interacts will be huge for people looking to put this into play.
> > >
> > > *Example Yarn Site:*
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Example%3A+yarn-site.xml
> > >
> > > This is helpful, but where does it go? Remember, the remote
> distribution
> > > had us delete the yarn-site in the hadoop etc folder.
> > >
> > > *Myriad Webapp *
> > > https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Webapp
> > >
> > > This should be fleshed out a bit more. Also, it's in the
> > > /myriad-scheduler/src/main/resources/webapp based on my git clone, but
> in
> > > the wiki that's not listed. I had to dig for it.
> > >
> > > Some questions here: could the webapp be built during the myriad
> > building
> > > process? Could it be then be packaged as tarball for execution either
> > > manually via marathon or automatically in a container on mesos? I
> > > understand this is a fresh piece of the puzzle, I am just thinking
> about
> > > and verbalizing the "where" on this for the future
> > >
> > >
> > >
> > > Those are the items that come to mind thus far. I hope the tone of my
> > > email is correct, this is a great project, and I want others to try it
> > as I
> > > have.
> > >
> > > John Omernik
> > >
> >
>
Re: Documentation Comments
Posted by Ruth Harris <rh...@maprtech.com>.
hi John,
I just got some content onto the wiki that wasn't there before. Right now,
I'm working on organizing the content better... your comments will help a
LOT.
Thanks, Ruth
On Wed, Aug 19, 2015 at 11:56 AM, John Omernik <jo...@omernik.com> wrote:
> Happy to sign the ICLA. Who do I send it to? Ruth, I defer to your writing
> skills and suggestions on how to help. Happy to help in the way that you'd
> find easiest.
>
>
> John
>
> On Wed, Aug 19, 2015 at 1:48 PM, Ruth Harris <rh...@maprtech.com> wrote:
>
> > hi John,
> >
> > Thank you for your feedback. I'm the assigned technical writer for the
> > Myriad project. I'll also be working on updating the information.
> >
> > I'm also ok with what Adam indicated: directly editing or adding a John's
> > page. But please be aware that I'll also be working on the content. Last
> > week I did some cleanup work on the original files in GitHub and then
> > brought them into the Wiki, although, I still have some more work in
> terms
> > of organizing and identifying holes.
> >
> > If you like, I can create a "John's comments" page and then work on
> > incorporating some of the obvious things that you mentioned.
> >
> > Thanks, Ruth
> >
> > Ruth Harris
> > Sr. Tech. Writer
> > rharris@mapr.com
> >
> > On Wed, Aug 19, 2015 at 11:37 AM, John Omernik <jo...@omernik.com> wrote:
> >
> > > Thanks Adam, I signed up with "mandoskippy".
> > >
> > > I am honored to help in this capacity, for updating etc, do we go
> through
> > > some kind of review? Is it better to ask questions on the dev list then
> > > update when consensus occurs? How about when I'd like to post a page
> and
> > > then have someone review the work? If I have a comment on the page, is
> > that
> > > public or can I just send to author? Just curious on any guidelines I
> > > should be following in that regard.
> > >
> > > John
> > >
> > >
> > >
> > > On Wed, Aug 19, 2015 at 1:29 PM, Adam Bordelon <ad...@mesosphere.io>
> > wrote:
> > >
> > > > John, thanks a ton for your valuable feedback! We're glad to have
> your
> > > > perspective as a user of the project, and I'm ready+willing to give
> you
> > > > edit access to the wiki if you want to update it with your learnings,
> > > > elaborate anything that's unclear, or add a new "John's tips" page.
> > Just
> > > > sign up for a wiki account, send me your accountId, and I'll grant
> you
> > > edit
> > > > access.
> > > > (I'll let others answer your specific questions)
> > > >
> > > > On Wed, Aug 19, 2015 at 6:28 AM, John Omernik <jo...@omernik.com>
> > wrote:
> > > >
> > > > > Today, I will be playing the role of the fool/jester trying to get
> > > Myriad
> > > > > running. Basically, since getting Myriad running with Santosh
> quite a
> > > > while
> > > > > ago, and now trying again with new versions of Hadoop, MapR, and
> > > Myriad,
> > > > I
> > > > > wanted to hit up the wiki (
> > > > > https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home)
> and
> > > > > outline
> > > > > points that as a non-dev living the code, are unclear to someone
> > trying
> > > > to
> > > > > utilize myriad or understand it's operation.
> > > > >
> > > > > Obviously, some of my points can be answered with "look here in the
> > > code"
> > > > > or look at this page, but I will try to outline my thought
> processes
> > > as I
> > > > > reviewed the current docs. Sometimes the way I approached the
> > problem
> > > > led
> > > > > me down a path of to a certain page, missing the answer in a
> > different
> > > > > page, and thus some cross linking could be helpful.
> > > > >
> > > > > Please do not let my points be taken as anything other than a
> desire
> > to
> > > > > improve how accessible Myriad is to the community, this is not a
> > > critique
> > > > > of the hard work everyone has done on the project. I also
> understand
> > > > that
> > > > > given the work load and other issues, that fixing these issues in
> > > > > documentation may not be a priority. I am listing them out here,
> so
> > > that
> > > > > those folks who are SMEs on various points may be able to quickly
> add
> > > > stuff
> > > > > and we'll organize it later.
> > > > >
> > > > >
> > > > > *Remote Distribution: *
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution
> > > > >
> > > > > This whole section could use some work from a standpoint of what
> runs
> > > > where
> > > > > and where that component gets its files. For example, I think it
> > would
> > > > > help people to understand that the whole tarball created in step 6
> > has
> > > > all
> > > > > the files for node managers and resource managers. Basically,
> > > everything
> > > > > runs from there. Here is a small example I am currently working
> with:
> > > > >
> > > > >
> > > > > Starting Myriad:
> > > > > Option 1: Use Marathon (provide example json, here is mine)
> > > > > {
> > > > > "cmd": "env && export
> > > > >
> > > > >
> > > >
> > >
> >
> YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos
> > > > > && hadoop-2.7.0/bin/yarn resourcemanager",
> > > > > "uris": ["maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz"],
> > > > > "cpus": 1.0,
> > > > > "mem": 1024,
> > > > > "id": "myriad",
> > > > > "instances": 1,
> > > > > "user": "mapr"
> > > > > }
> > > > >
> > > > > In this case, Marathon grabs the hadoop tarball and pulls it down,
> > this
> > > > > tarball also has the Myriad yml file. When it executes the resource
> > > > > manager, it is brought up in Myriad and ready to run node managers
> by
> > > > > pulling the tarball to the slave nodes and executing the
> nodemanager.
> > > (I
> > > > > would imagine the work with history server etc would also use this
> > > > > tarball?).
> > > > >
> > > > > From here it will us NMInstances to launch a node manager. (Note,
> > this
> > > > is
> > > > > different from when I originally set things up... before, I could
> run
> > > the
> > > > > resource manager/myriad without a nodemanager, now it seems it's
> > > required
> > > > > based on the config in the src... could we expound on this in the
> > docs
> > > > > somewhere?)
> > > > >
> > > > >
> > > > > Option 2: ???? (Are there other ways to launch the resource
> manager?)
> > > > >
> > > > > Step 6: So something that is unclear to me is the handling of the
> > > > > hadoop/yarn config files. In Step 6 on this page, there is "sudo
> rm
> > > > > hadoop-
> > > > > 2.5.0/etc/hadoop/*.xml" This doesn't makes sense to me. I actually
> > > > ignored
> > > > > this step. For me, if I remove these xml files, then there is no
> > place
> > > > to
> > > > > get my files... I think? Since I am running the RM and NM from the
> > same
> > > > > tarball, and Myriad config is here, and my goal is to not have
> > anything
> > > > > installed on a node, where would I set yarn settings? This could
> be
> > > much
> > > > > clearer to me, and probably others.
> > > > >
> > > > > Step 2: Should we just be copying the Myriad files to
> > > > > /share/hadoop/yarn/lib folder? Do we worry about potentially
> > overwrites
> > > > of
> > > > > jars or version conflicts?
> > > > >
> > > > > *Configuring Cgroups*
> > > > >
> > https://cwiki.apache.org/confluence/display/MYRIAD/Configuring+Cgroups
> > > > > At some point a little bit more about why one would want CGroups
> and
> > > > issues
> > > > > that could occur with them. While many folks using Mesos/Myriad may
> > > > > understand this, others may not, and it's a good way to help people
> > > think
> > > > > positively about our project if we help educate them along the way.
> > > > >
> > > > > Minor point on enabling CGroups. This is confusing given my
> questions
> > > in
> > > > > remote distribution. in this it says I need to edit my
> yarn-site.xml,
> > > but
> > > > > in remote distribution it says delete my hadoop xml files. We need
> to
> > > > > address this conflict cause it can be confusing for a user coming
> > > onboard
> > > > >
> > > > > Nitpick: Enabling cgroups for mess-slave - should be - Enabling
> > cgroups
> > > > for
> > > > > mesos-slave
> > > > >
> > > > > *Myriad Configuration Properties*:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Configuration+Properties
> > > > > Based on the conversation on list with Yuliya,
> > > > > "Currently, this file is built into Myriad Scheduler jar. So, if
> you
> > > need
> > > > > to modify some of the properties in this file, modify them before
> > > > building
> > > > > Myriad Scheduler."
> > > > > isn't accurate any more, and we should address that.
> > > > >
> > > > > The configuration file in the wiki is an old one, the nmInstances
> > isn't
> > > > in
> > > > > it, (and see my question about that above).
> > > > >
> > > > > Frameworks and usernames. I think the users that the framework
> runs
> > > as,
> > > > > the actual node and resource managers, etc is confusing to a user
> (I
> > am
> > > > > very confused!) When I first got Myriad up I set my user under the
> > > > > executor to be mapr, and then it appeared to work with
> impersonation
> > > from
> > > > > queries etc. Now, I am trying the remote distribution and I have
> > users
> > > > set
> > > > > in the config, potentially a user in my marathon json, and I am
> > getting
> > > > > errors on permissions of files when a node manager tries to start
> (a
> > > > > separate issue I will post later). Basically, this is complex, and
> a
> > > page
> > > > > describing out what needs to run where with which permissions and
> how
> > > > that
> > > > > interacts will be huge for people looking to put this into play.
> > > > >
> > > > > *Example Yarn Site:*
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Example%3A+yarn-site.xml
> > > > >
> > > > > This is helpful, but where does it go? Remember, the remote
> > > distribution
> > > > > had us delete the yarn-site in the hadoop etc folder.
> > > > >
> > > > > *Myriad Webapp *
> > > > > https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Webapp
> > > > >
> > > > > This should be fleshed out a bit more. Also, it's in the
> > > > > /myriad-scheduler/src/main/resources/webapp based on my git clone,
> > but
> > > in
> > > > > the wiki that's not listed. I had to dig for it.
> > > > >
> > > > > Some questions here: could the webapp be built during the myriad
> > > > building
> > > > > process? Could it be then be packaged as tarball for execution
> either
> > > > > manually via marathon or automatically in a container on mesos? I
> > > > > understand this is a fresh piece of the puzzle, I am just thinking
> > > about
> > > > > and verbalizing the "where" on this for the future
> > > > >
> > > > >
> > > > >
> > > > > Those are the items that come to mind thus far. I hope the tone of
> > my
> > > > > email is correct, this is a great project, and I want others to try
> > it
> > > > as I
> > > > > have.
> > > > >
> > > > > John Omernik
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Ruth Harris
> > Sr. Technical Writer, MapR
> >
>
--
Ruth Harris
Sr. Technical Writer, MapR
Re: Documentation Comments
Posted by Ken Sipe <ke...@mesosphere.io>.
john,
the details of where to send it on in the pdf that must be filled out and signed. the fastest way is to send the ICLA to secretary@apache.org <ma...@apache.org>
ken
> On Aug 19, 2015, at 1:56 PM, John Omernik <jo...@omernik.com> wrote:
>
> Happy to sign the ICLA. Who do I send it to? Ruth, I defer to your writing
> skills and suggestions on how to help. Happy to help in the way that you'd
> find easiest.
>
>
> John
>
> On Wed, Aug 19, 2015 at 1:48 PM, Ruth Harris <rh...@maprtech.com> wrote:
>
>> hi John,
>>
>> Thank you for your feedback. I'm the assigned technical writer for the
>> Myriad project. I'll also be working on updating the information.
>>
>> I'm also ok with what Adam indicated: directly editing or adding a John's
>> page. But please be aware that I'll also be working on the content. Last
>> week I did some cleanup work on the original files in GitHub and then
>> brought them into the Wiki, although, I still have some more work in terms
>> of organizing and identifying holes.
>>
>> If you like, I can create a "John's comments" page and then work on
>> incorporating some of the obvious things that you mentioned.
>>
>> Thanks, Ruth
>>
>> Ruth Harris
>> Sr. Tech. Writer
>> rharris@mapr.com
>>
>> On Wed, Aug 19, 2015 at 11:37 AM, John Omernik <jo...@omernik.com> wrote:
>>
>>> Thanks Adam, I signed up with "mandoskippy".
>>>
>>> I am honored to help in this capacity, for updating etc, do we go through
>>> some kind of review? Is it better to ask questions on the dev list then
>>> update when consensus occurs? How about when I'd like to post a page and
>>> then have someone review the work? If I have a comment on the page, is
>> that
>>> public or can I just send to author? Just curious on any guidelines I
>>> should be following in that regard.
>>>
>>> John
>>>
>>>
>>>
>>> On Wed, Aug 19, 2015 at 1:29 PM, Adam Bordelon <ad...@mesosphere.io>
>> wrote:
>>>
>>>> John, thanks a ton for your valuable feedback! We're glad to have your
>>>> perspective as a user of the project, and I'm ready+willing to give you
>>>> edit access to the wiki if you want to update it with your learnings,
>>>> elaborate anything that's unclear, or add a new "John's tips" page.
>> Just
>>>> sign up for a wiki account, send me your accountId, and I'll grant you
>>> edit
>>>> access.
>>>> (I'll let others answer your specific questions)
>>>>
>>>> On Wed, Aug 19, 2015 at 6:28 AM, John Omernik <jo...@omernik.com>
>> wrote:
>>>>
>>>>> Today, I will be playing the role of the fool/jester trying to get
>>> Myriad
>>>>> running. Basically, since getting Myriad running with Santosh quite a
>>>> while
>>>>> ago, and now trying again with new versions of Hadoop, MapR, and
>>> Myriad,
>>>> I
>>>>> wanted to hit up the wiki (
>>>>> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home) and
>>>>> outline
>>>>> points that as a non-dev living the code, are unclear to someone
>> trying
>>>> to
>>>>> utilize myriad or understand it's operation.
>>>>>
>>>>> Obviously, some of my points can be answered with "look here in the
>>> code"
>>>>> or look at this page, but I will try to outline my thought processes
>>> as I
>>>>> reviewed the current docs. Sometimes the way I approached the
>> problem
>>>> led
>>>>> me down a path of to a certain page, missing the answer in a
>> different
>>>>> page, and thus some cross linking could be helpful.
>>>>>
>>>>> Please do not let my points be taken as anything other than a desire
>> to
>>>>> improve how accessible Myriad is to the community, this is not a
>>> critique
>>>>> of the hard work everyone has done on the project. I also understand
>>>> that
>>>>> given the work load and other issues, that fixing these issues in
>>>>> documentation may not be a priority. I am listing them out here, so
>>> that
>>>>> those folks who are SMEs on various points may be able to quickly add
>>>> stuff
>>>>> and we'll organize it later.
>>>>>
>>>>>
>>>>> *Remote Distribution: *
>>>>>
>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution
>>>>>
>>>>> This whole section could use some work from a standpoint of what runs
>>>> where
>>>>> and where that component gets its files. For example, I think it
>> would
>>>>> help people to understand that the whole tarball created in step 6
>> has
>>>> all
>>>>> the files for node managers and resource managers. Basically,
>>> everything
>>>>> runs from there. Here is a small example I am currently working with:
>>>>>
>>>>>
>>>>> Starting Myriad:
>>>>> Option 1: Use Marathon (provide example json, here is mine)
>>>>> {
>>>>> "cmd": "env && export
>>>>>
>>>>>
>>>>
>>>
>> YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos
>>>>> && hadoop-2.7.0/bin/yarn resourcemanager",
>>>>> "uris": ["maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz"],
>>>>> "cpus": 1.0,
>>>>> "mem": 1024,
>>>>> "id": "myriad",
>>>>> "instances": 1,
>>>>> "user": "mapr"
>>>>> }
>>>>>
>>>>> In this case, Marathon grabs the hadoop tarball and pulls it down,
>> this
>>>>> tarball also has the Myriad yml file. When it executes the resource
>>>>> manager, it is brought up in Myriad and ready to run node managers by
>>>>> pulling the tarball to the slave nodes and executing the nodemanager.
>>> (I
>>>>> would imagine the work with history server etc would also use this
>>>>> tarball?).
>>>>>
>>>>> From here it will us NMInstances to launch a node manager. (Note,
>> this
>>>> is
>>>>> different from when I originally set things up... before, I could run
>>> the
>>>>> resource manager/myriad without a nodemanager, now it seems it's
>>> required
>>>>> based on the config in the src... could we expound on this in the
>> docs
>>>>> somewhere?)
>>>>>
>>>>>
>>>>> Option 2: ???? (Are there other ways to launch the resource manager?)
>>>>>
>>>>> Step 6: So something that is unclear to me is the handling of the
>>>>> hadoop/yarn config files. In Step 6 on this page, there is "sudo rm
>>>>> hadoop-
>>>>> 2.5.0/etc/hadoop/*.xml" This doesn't makes sense to me. I actually
>>>> ignored
>>>>> this step. For me, if I remove these xml files, then there is no
>> place
>>>> to
>>>>> get my files... I think? Since I am running the RM and NM from the
>> same
>>>>> tarball, and Myriad config is here, and my goal is to not have
>> anything
>>>>> installed on a node, where would I set yarn settings? This could be
>>> much
>>>>> clearer to me, and probably others.
>>>>>
>>>>> Step 2: Should we just be copying the Myriad files to
>>>>> /share/hadoop/yarn/lib folder? Do we worry about potentially
>> overwrites
>>>> of
>>>>> jars or version conflicts?
>>>>>
>>>>> *Configuring Cgroups*
>>>>>
>> https://cwiki.apache.org/confluence/display/MYRIAD/Configuring+Cgroups
>>>>> At some point a little bit more about why one would want CGroups and
>>>> issues
>>>>> that could occur with them. While many folks using Mesos/Myriad may
>>>>> understand this, others may not, and it's a good way to help people
>>> think
>>>>> positively about our project if we help educate them along the way.
>>>>>
>>>>> Minor point on enabling CGroups. This is confusing given my questions
>>> in
>>>>> remote distribution. in this it says I need to edit my yarn-site.xml,
>>> but
>>>>> in remote distribution it says delete my hadoop xml files. We need to
>>>>> address this conflict cause it can be confusing for a user coming
>>> onboard
>>>>>
>>>>> Nitpick: Enabling cgroups for mess-slave - should be - Enabling
>> cgroups
>>>> for
>>>>> mesos-slave
>>>>>
>>>>> *Myriad Configuration Properties*:
>>>>>
>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Configuration+Properties
>>>>> Based on the conversation on list with Yuliya,
>>>>> "Currently, this file is built into Myriad Scheduler jar. So, if you
>>> need
>>>>> to modify some of the properties in this file, modify them before
>>>> building
>>>>> Myriad Scheduler."
>>>>> isn't accurate any more, and we should address that.
>>>>>
>>>>> The configuration file in the wiki is an old one, the nmInstances
>> isn't
>>>> in
>>>>> it, (and see my question about that above).
>>>>>
>>>>> Frameworks and usernames. I think the users that the framework runs
>>> as,
>>>>> the actual node and resource managers, etc is confusing to a user (I
>> am
>>>>> very confused!) When I first got Myriad up I set my user under the
>>>>> executor to be mapr, and then it appeared to work with impersonation
>>> from
>>>>> queries etc. Now, I am trying the remote distribution and I have
>> users
>>>> set
>>>>> in the config, potentially a user in my marathon json, and I am
>> getting
>>>>> errors on permissions of files when a node manager tries to start (a
>>>>> separate issue I will post later). Basically, this is complex, and a
>>> page
>>>>> describing out what needs to run where with which permissions and how
>>>> that
>>>>> interacts will be huge for people looking to put this into play.
>>>>>
>>>>> *Example Yarn Site:*
>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/MYRIAD/Example%3A+yarn-site.xml
>>>>>
>>>>> This is helpful, but where does it go? Remember, the remote
>>> distribution
>>>>> had us delete the yarn-site in the hadoop etc folder.
>>>>>
>>>>> *Myriad Webapp *
>>>>> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Webapp
>>>>>
>>>>> This should be fleshed out a bit more. Also, it's in the
>>>>> /myriad-scheduler/src/main/resources/webapp based on my git clone,
>> but
>>> in
>>>>> the wiki that's not listed. I had to dig for it.
>>>>>
>>>>> Some questions here: could the webapp be built during the myriad
>>>> building
>>>>> process? Could it be then be packaged as tarball for execution either
>>>>> manually via marathon or automatically in a container on mesos? I
>>>>> understand this is a fresh piece of the puzzle, I am just thinking
>>> about
>>>>> and verbalizing the "where" on this for the future
>>>>>
>>>>>
>>>>>
>>>>> Those are the items that come to mind thus far. I hope the tone of
>> my
>>>>> email is correct, this is a great project, and I want others to try
>> it
>>>> as I
>>>>> have.
>>>>>
>>>>> John Omernik
>>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Ruth Harris
>> Sr. Technical Writer, MapR
>>
Re: Documentation Comments
Posted by John Omernik <jo...@omernik.com>.
Happy to sign the ICLA. Who do I send it to? Ruth, I defer to your writing
skills and suggestions on how to help. Happy to help in the way that you'd
find easiest.
John
On Wed, Aug 19, 2015 at 1:48 PM, Ruth Harris <rh...@maprtech.com> wrote:
> hi John,
>
> Thank you for your feedback. I'm the assigned technical writer for the
> Myriad project. I'll also be working on updating the information.
>
> I'm also ok with what Adam indicated: directly editing or adding a John's
> page. But please be aware that I'll also be working on the content. Last
> week I did some cleanup work on the original files in GitHub and then
> brought them into the Wiki, although, I still have some more work in terms
> of organizing and identifying holes.
>
> If you like, I can create a "John's comments" page and then work on
> incorporating some of the obvious things that you mentioned.
>
> Thanks, Ruth
>
> Ruth Harris
> Sr. Tech. Writer
> rharris@mapr.com
>
> On Wed, Aug 19, 2015 at 11:37 AM, John Omernik <jo...@omernik.com> wrote:
>
> > Thanks Adam, I signed up with "mandoskippy".
> >
> > I am honored to help in this capacity, for updating etc, do we go through
> > some kind of review? Is it better to ask questions on the dev list then
> > update when consensus occurs? How about when I'd like to post a page and
> > then have someone review the work? If I have a comment on the page, is
> that
> > public or can I just send to author? Just curious on any guidelines I
> > should be following in that regard.
> >
> > John
> >
> >
> >
> > On Wed, Aug 19, 2015 at 1:29 PM, Adam Bordelon <ad...@mesosphere.io>
> wrote:
> >
> > > John, thanks a ton for your valuable feedback! We're glad to have your
> > > perspective as a user of the project, and I'm ready+willing to give you
> > > edit access to the wiki if you want to update it with your learnings,
> > > elaborate anything that's unclear, or add a new "John's tips" page.
> Just
> > > sign up for a wiki account, send me your accountId, and I'll grant you
> > edit
> > > access.
> > > (I'll let others answer your specific questions)
> > >
> > > On Wed, Aug 19, 2015 at 6:28 AM, John Omernik <jo...@omernik.com>
> wrote:
> > >
> > > > Today, I will be playing the role of the fool/jester trying to get
> > Myriad
> > > > running. Basically, since getting Myriad running with Santosh quite a
> > > while
> > > > ago, and now trying again with new versions of Hadoop, MapR, and
> > Myriad,
> > > I
> > > > wanted to hit up the wiki (
> > > > https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home) and
> > > > outline
> > > > points that as a non-dev living the code, are unclear to someone
> trying
> > > to
> > > > utilize myriad or understand it's operation.
> > > >
> > > > Obviously, some of my points can be answered with "look here in the
> > code"
> > > > or look at this page, but I will try to outline my thought processes
> > as I
> > > > reviewed the current docs. Sometimes the way I approached the
> problem
> > > led
> > > > me down a path of to a certain page, missing the answer in a
> different
> > > > page, and thus some cross linking could be helpful.
> > > >
> > > > Please do not let my points be taken as anything other than a desire
> to
> > > > improve how accessible Myriad is to the community, this is not a
> > critique
> > > > of the hard work everyone has done on the project. I also understand
> > > that
> > > > given the work load and other issues, that fixing these issues in
> > > > documentation may not be a priority. I am listing them out here, so
> > that
> > > > those folks who are SMEs on various points may be able to quickly add
> > > stuff
> > > > and we'll organize it later.
> > > >
> > > >
> > > > *Remote Distribution: *
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution
> > > >
> > > > This whole section could use some work from a standpoint of what runs
> > > where
> > > > and where that component gets its files. For example, I think it
> would
> > > > help people to understand that the whole tarball created in step 6
> has
> > > all
> > > > the files for node managers and resource managers. Basically,
> > everything
> > > > runs from there. Here is a small example I am currently working with:
> > > >
> > > >
> > > > Starting Myriad:
> > > > Option 1: Use Marathon (provide example json, here is mine)
> > > > {
> > > > "cmd": "env && export
> > > >
> > > >
> > >
> >
> YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos
> > > > && hadoop-2.7.0/bin/yarn resourcemanager",
> > > > "uris": ["maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz"],
> > > > "cpus": 1.0,
> > > > "mem": 1024,
> > > > "id": "myriad",
> > > > "instances": 1,
> > > > "user": "mapr"
> > > > }
> > > >
> > > > In this case, Marathon grabs the hadoop tarball and pulls it down,
> this
> > > > tarball also has the Myriad yml file. When it executes the resource
> > > > manager, it is brought up in Myriad and ready to run node managers by
> > > > pulling the tarball to the slave nodes and executing the nodemanager.
> > (I
> > > > would imagine the work with history server etc would also use this
> > > > tarball?).
> > > >
> > > > From here it will us NMInstances to launch a node manager. (Note,
> this
> > > is
> > > > different from when I originally set things up... before, I could run
> > the
> > > > resource manager/myriad without a nodemanager, now it seems it's
> > required
> > > > based on the config in the src... could we expound on this in the
> docs
> > > > somewhere?)
> > > >
> > > >
> > > > Option 2: ???? (Are there other ways to launch the resource manager?)
> > > >
> > > > Step 6: So something that is unclear to me is the handling of the
> > > > hadoop/yarn config files. In Step 6 on this page, there is "sudo rm
> > > > hadoop-
> > > > 2.5.0/etc/hadoop/*.xml" This doesn't makes sense to me. I actually
> > > ignored
> > > > this step. For me, if I remove these xml files, then there is no
> place
> > > to
> > > > get my files... I think? Since I am running the RM and NM from the
> same
> > > > tarball, and Myriad config is here, and my goal is to not have
> anything
> > > > installed on a node, where would I set yarn settings? This could be
> > much
> > > > clearer to me, and probably others.
> > > >
> > > > Step 2: Should we just be copying the Myriad files to
> > > > /share/hadoop/yarn/lib folder? Do we worry about potentially
> overwrites
> > > of
> > > > jars or version conflicts?
> > > >
> > > > *Configuring Cgroups*
> > > >
> https://cwiki.apache.org/confluence/display/MYRIAD/Configuring+Cgroups
> > > > At some point a little bit more about why one would want CGroups and
> > > issues
> > > > that could occur with them. While many folks using Mesos/Myriad may
> > > > understand this, others may not, and it's a good way to help people
> > think
> > > > positively about our project if we help educate them along the way.
> > > >
> > > > Minor point on enabling CGroups. This is confusing given my questions
> > in
> > > > remote distribution. in this it says I need to edit my yarn-site.xml,
> > but
> > > > in remote distribution it says delete my hadoop xml files. We need to
> > > > address this conflict cause it can be confusing for a user coming
> > onboard
> > > >
> > > > Nitpick: Enabling cgroups for mess-slave - should be - Enabling
> cgroups
> > > for
> > > > mesos-slave
> > > >
> > > > *Myriad Configuration Properties*:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Configuration+Properties
> > > > Based on the conversation on list with Yuliya,
> > > > "Currently, this file is built into Myriad Scheduler jar. So, if you
> > need
> > > > to modify some of the properties in this file, modify them before
> > > building
> > > > Myriad Scheduler."
> > > > isn't accurate any more, and we should address that.
> > > >
> > > > The configuration file in the wiki is an old one, the nmInstances
> isn't
> > > in
> > > > it, (and see my question about that above).
> > > >
> > > > Frameworks and usernames. I think the users that the framework runs
> > as,
> > > > the actual node and resource managers, etc is confusing to a user (I
> am
> > > > very confused!) When I first got Myriad up I set my user under the
> > > > executor to be mapr, and then it appeared to work with impersonation
> > from
> > > > queries etc. Now, I am trying the remote distribution and I have
> users
> > > set
> > > > in the config, potentially a user in my marathon json, and I am
> getting
> > > > errors on permissions of files when a node manager tries to start (a
> > > > separate issue I will post later). Basically, this is complex, and a
> > page
> > > > describing out what needs to run where with which permissions and how
> > > that
> > > > interacts will be huge for people looking to put this into play.
> > > >
> > > > *Example Yarn Site:*
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Example%3A+yarn-site.xml
> > > >
> > > > This is helpful, but where does it go? Remember, the remote
> > distribution
> > > > had us delete the yarn-site in the hadoop etc folder.
> > > >
> > > > *Myriad Webapp *
> > > > https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Webapp
> > > >
> > > > This should be fleshed out a bit more. Also, it's in the
> > > > /myriad-scheduler/src/main/resources/webapp based on my git clone,
> but
> > in
> > > > the wiki that's not listed. I had to dig for it.
> > > >
> > > > Some questions here: could the webapp be built during the myriad
> > > building
> > > > process? Could it be then be packaged as tarball for execution either
> > > > manually via marathon or automatically in a container on mesos? I
> > > > understand this is a fresh piece of the puzzle, I am just thinking
> > about
> > > > and verbalizing the "where" on this for the future
> > > >
> > > >
> > > >
> > > > Those are the items that come to mind thus far. I hope the tone of
> my
> > > > email is correct, this is a great project, and I want others to try
> it
> > > as I
> > > > have.
> > > >
> > > > John Omernik
> > > >
> > >
> >
>
>
>
> --
> Ruth Harris
> Sr. Technical Writer, MapR
>
Re: Documentation Comments
Posted by Ruth Harris <rh...@maprtech.com>.
hi John,
Thank you for your feedback. I'm the assigned technical writer for the
Myriad project. I'll also be working on updating the information.
I'm also ok with what Adam indicated: directly editing or adding a John's
page. But please be aware that I'll also be working on the content. Last
week I did some cleanup work on the original files in GitHub and then
brought them into the Wiki, although, I still have some more work in terms
of organizing and identifying holes.
If you like, I can create a "John's comments" page and then work on
incorporating some of the obvious things that you mentioned.
Thanks, Ruth
Ruth Harris
Sr. Tech. Writer
rharris@mapr.com
On Wed, Aug 19, 2015 at 11:37 AM, John Omernik <jo...@omernik.com> wrote:
> Thanks Adam, I signed up with "mandoskippy".
>
> I am honored to help in this capacity, for updating etc, do we go through
> some kind of review? Is it better to ask questions on the dev list then
> update when consensus occurs? How about when I'd like to post a page and
> then have someone review the work? If I have a comment on the page, is that
> public or can I just send to author? Just curious on any guidelines I
> should be following in that regard.
>
> John
>
>
>
> On Wed, Aug 19, 2015 at 1:29 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
>
> > John, thanks a ton for your valuable feedback! We're glad to have your
> > perspective as a user of the project, and I'm ready+willing to give you
> > edit access to the wiki if you want to update it with your learnings,
> > elaborate anything that's unclear, or add a new "John's tips" page. Just
> > sign up for a wiki account, send me your accountId, and I'll grant you
> edit
> > access.
> > (I'll let others answer your specific questions)
> >
> > On Wed, Aug 19, 2015 at 6:28 AM, John Omernik <jo...@omernik.com> wrote:
> >
> > > Today, I will be playing the role of the fool/jester trying to get
> Myriad
> > > running. Basically, since getting Myriad running with Santosh quite a
> > while
> > > ago, and now trying again with new versions of Hadoop, MapR, and
> Myriad,
> > I
> > > wanted to hit up the wiki (
> > > https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home) and
> > > outline
> > > points that as a non-dev living the code, are unclear to someone trying
> > to
> > > utilize myriad or understand it's operation.
> > >
> > > Obviously, some of my points can be answered with "look here in the
> code"
> > > or look at this page, but I will try to outline my thought processes
> as I
> > > reviewed the current docs. Sometimes the way I approached the problem
> > led
> > > me down a path of to a certain page, missing the answer in a different
> > > page, and thus some cross linking could be helpful.
> > >
> > > Please do not let my points be taken as anything other than a desire to
> > > improve how accessible Myriad is to the community, this is not a
> critique
> > > of the hard work everyone has done on the project. I also understand
> > that
> > > given the work load and other issues, that fixing these issues in
> > > documentation may not be a priority. I am listing them out here, so
> that
> > > those folks who are SMEs on various points may be able to quickly add
> > stuff
> > > and we'll organize it later.
> > >
> > >
> > > *Remote Distribution: *
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution
> > >
> > > This whole section could use some work from a standpoint of what runs
> > where
> > > and where that component gets its files. For example, I think it would
> > > help people to understand that the whole tarball created in step 6 has
> > all
> > > the files for node managers and resource managers. Basically,
> everything
> > > runs from there. Here is a small example I am currently working with:
> > >
> > >
> > > Starting Myriad:
> > > Option 1: Use Marathon (provide example json, here is mine)
> > > {
> > > "cmd": "env && export
> > >
> > >
> >
> YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos
> > > && hadoop-2.7.0/bin/yarn resourcemanager",
> > > "uris": ["maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz"],
> > > "cpus": 1.0,
> > > "mem": 1024,
> > > "id": "myriad",
> > > "instances": 1,
> > > "user": "mapr"
> > > }
> > >
> > > In this case, Marathon grabs the hadoop tarball and pulls it down, this
> > > tarball also has the Myriad yml file. When it executes the resource
> > > manager, it is brought up in Myriad and ready to run node managers by
> > > pulling the tarball to the slave nodes and executing the nodemanager.
> (I
> > > would imagine the work with history server etc would also use this
> > > tarball?).
> > >
> > > From here it will us NMInstances to launch a node manager. (Note, this
> > is
> > > different from when I originally set things up... before, I could run
> the
> > > resource manager/myriad without a nodemanager, now it seems it's
> required
> > > based on the config in the src... could we expound on this in the docs
> > > somewhere?)
> > >
> > >
> > > Option 2: ???? (Are there other ways to launch the resource manager?)
> > >
> > > Step 6: So something that is unclear to me is the handling of the
> > > hadoop/yarn config files. In Step 6 on this page, there is "sudo rm
> > > hadoop-
> > > 2.5.0/etc/hadoop/*.xml" This doesn't makes sense to me. I actually
> > ignored
> > > this step. For me, if I remove these xml files, then there is no place
> > to
> > > get my files... I think? Since I am running the RM and NM from the same
> > > tarball, and Myriad config is here, and my goal is to not have anything
> > > installed on a node, where would I set yarn settings? This could be
> much
> > > clearer to me, and probably others.
> > >
> > > Step 2: Should we just be copying the Myriad files to
> > > /share/hadoop/yarn/lib folder? Do we worry about potentially overwrites
> > of
> > > jars or version conflicts?
> > >
> > > *Configuring Cgroups*
> > > https://cwiki.apache.org/confluence/display/MYRIAD/Configuring+Cgroups
> > > At some point a little bit more about why one would want CGroups and
> > issues
> > > that could occur with them. While many folks using Mesos/Myriad may
> > > understand this, others may not, and it's a good way to help people
> think
> > > positively about our project if we help educate them along the way.
> > >
> > > Minor point on enabling CGroups. This is confusing given my questions
> in
> > > remote distribution. in this it says I need to edit my yarn-site.xml,
> but
> > > in remote distribution it says delete my hadoop xml files. We need to
> > > address this conflict cause it can be confusing for a user coming
> onboard
> > >
> > > Nitpick: Enabling cgroups for mess-slave - should be - Enabling cgroups
> > for
> > > mesos-slave
> > >
> > > *Myriad Configuration Properties*:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Configuration+Properties
> > > Based on the conversation on list with Yuliya,
> > > "Currently, this file is built into Myriad Scheduler jar. So, if you
> need
> > > to modify some of the properties in this file, modify them before
> > building
> > > Myriad Scheduler."
> > > isn't accurate any more, and we should address that.
> > >
> > > The configuration file in the wiki is an old one, the nmInstances isn't
> > in
> > > it, (and see my question about that above).
> > >
> > > Frameworks and usernames. I think the users that the framework runs
> as,
> > > the actual node and resource managers, etc is confusing to a user (I am
> > > very confused!) When I first got Myriad up I set my user under the
> > > executor to be mapr, and then it appeared to work with impersonation
> from
> > > queries etc. Now, I am trying the remote distribution and I have users
> > set
> > > in the config, potentially a user in my marathon json, and I am getting
> > > errors on permissions of files when a node manager tries to start (a
> > > separate issue I will post later). Basically, this is complex, and a
> page
> > > describing out what needs to run where with which permissions and how
> > that
> > > interacts will be huge for people looking to put this into play.
> > >
> > > *Example Yarn Site:*
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Example%3A+yarn-site.xml
> > >
> > > This is helpful, but where does it go? Remember, the remote
> distribution
> > > had us delete the yarn-site in the hadoop etc folder.
> > >
> > > *Myriad Webapp *
> > > https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Webapp
> > >
> > > This should be fleshed out a bit more. Also, it's in the
> > > /myriad-scheduler/src/main/resources/webapp based on my git clone, but
> in
> > > the wiki that's not listed. I had to dig for it.
> > >
> > > Some questions here: could the webapp be built during the myriad
> > building
> > > process? Could it be then be packaged as tarball for execution either
> > > manually via marathon or automatically in a container on mesos? I
> > > understand this is a fresh piece of the puzzle, I am just thinking
> about
> > > and verbalizing the "where" on this for the future
> > >
> > >
> > >
> > > Those are the items that come to mind thus far. I hope the tone of my
> > > email is correct, this is a great project, and I want others to try it
> > as I
> > > have.
> > >
> > > John Omernik
> > >
> >
>
--
Ruth Harris
Sr. Technical Writer, MapR
Re: Documentation Comments
Posted by John Omernik <jo...@omernik.com>.
Thanks Adam, I signed up with "mandoskippy".
I am honored to help in this capacity, for updating etc, do we go through
some kind of review? Is it better to ask questions on the dev list then
update when consensus occurs? How about when I'd like to post a page and
then have someone review the work? If I have a comment on the page, is that
public or can I just send to author? Just curious on any guidelines I
should be following in that regard.
John
On Wed, Aug 19, 2015 at 1:29 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
> John, thanks a ton for your valuable feedback! We're glad to have your
> perspective as a user of the project, and I'm ready+willing to give you
> edit access to the wiki if you want to update it with your learnings,
> elaborate anything that's unclear, or add a new "John's tips" page. Just
> sign up for a wiki account, send me your accountId, and I'll grant you edit
> access.
> (I'll let others answer your specific questions)
>
> On Wed, Aug 19, 2015 at 6:28 AM, John Omernik <jo...@omernik.com> wrote:
>
> > Today, I will be playing the role of the fool/jester trying to get Myriad
> > running. Basically, since getting Myriad running with Santosh quite a
> while
> > ago, and now trying again with new versions of Hadoop, MapR, and Myriad,
> I
> > wanted to hit up the wiki (
> > https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home) and
> > outline
> > points that as a non-dev living the code, are unclear to someone trying
> to
> > utilize myriad or understand it's operation.
> >
> > Obviously, some of my points can be answered with "look here in the code"
> > or look at this page, but I will try to outline my thought processes as I
> > reviewed the current docs. Sometimes the way I approached the problem
> led
> > me down a path of to a certain page, missing the answer in a different
> > page, and thus some cross linking could be helpful.
> >
> > Please do not let my points be taken as anything other than a desire to
> > improve how accessible Myriad is to the community, this is not a critique
> > of the hard work everyone has done on the project. I also understand
> that
> > given the work load and other issues, that fixing these issues in
> > documentation may not be a priority. I am listing them out here, so that
> > those folks who are SMEs on various points may be able to quickly add
> stuff
> > and we'll organize it later.
> >
> >
> > *Remote Distribution: *
> >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution
> >
> > This whole section could use some work from a standpoint of what runs
> where
> > and where that component gets its files. For example, I think it would
> > help people to understand that the whole tarball created in step 6 has
> all
> > the files for node managers and resource managers. Basically, everything
> > runs from there. Here is a small example I am currently working with:
> >
> >
> > Starting Myriad:
> > Option 1: Use Marathon (provide example json, here is mine)
> > {
> > "cmd": "env && export
> >
> >
> YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos
> > && hadoop-2.7.0/bin/yarn resourcemanager",
> > "uris": ["maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz"],
> > "cpus": 1.0,
> > "mem": 1024,
> > "id": "myriad",
> > "instances": 1,
> > "user": "mapr"
> > }
> >
> > In this case, Marathon grabs the hadoop tarball and pulls it down, this
> > tarball also has the Myriad yml file. When it executes the resource
> > manager, it is brought up in Myriad and ready to run node managers by
> > pulling the tarball to the slave nodes and executing the nodemanager. (I
> > would imagine the work with history server etc would also use this
> > tarball?).
> >
> > From here it will us NMInstances to launch a node manager. (Note, this
> is
> > different from when I originally set things up... before, I could run the
> > resource manager/myriad without a nodemanager, now it seems it's required
> > based on the config in the src... could we expound on this in the docs
> > somewhere?)
> >
> >
> > Option 2: ???? (Are there other ways to launch the resource manager?)
> >
> > Step 6: So something that is unclear to me is the handling of the
> > hadoop/yarn config files. In Step 6 on this page, there is "sudo rm
> > hadoop-
> > 2.5.0/etc/hadoop/*.xml" This doesn't makes sense to me. I actually
> ignored
> > this step. For me, if I remove these xml files, then there is no place
> to
> > get my files... I think? Since I am running the RM and NM from the same
> > tarball, and Myriad config is here, and my goal is to not have anything
> > installed on a node, where would I set yarn settings? This could be much
> > clearer to me, and probably others.
> >
> > Step 2: Should we just be copying the Myriad files to
> > /share/hadoop/yarn/lib folder? Do we worry about potentially overwrites
> of
> > jars or version conflicts?
> >
> > *Configuring Cgroups*
> > https://cwiki.apache.org/confluence/display/MYRIAD/Configuring+Cgroups
> > At some point a little bit more about why one would want CGroups and
> issues
> > that could occur with them. While many folks using Mesos/Myriad may
> > understand this, others may not, and it's a good way to help people think
> > positively about our project if we help educate them along the way.
> >
> > Minor point on enabling CGroups. This is confusing given my questions in
> > remote distribution. in this it says I need to edit my yarn-site.xml, but
> > in remote distribution it says delete my hadoop xml files. We need to
> > address this conflict cause it can be confusing for a user coming onboard
> >
> > Nitpick: Enabling cgroups for mess-slave - should be - Enabling cgroups
> for
> > mesos-slave
> >
> > *Myriad Configuration Properties*:
> >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Configuration+Properties
> > Based on the conversation on list with Yuliya,
> > "Currently, this file is built into Myriad Scheduler jar. So, if you need
> > to modify some of the properties in this file, modify them before
> building
> > Myriad Scheduler."
> > isn't accurate any more, and we should address that.
> >
> > The configuration file in the wiki is an old one, the nmInstances isn't
> in
> > it, (and see my question about that above).
> >
> > Frameworks and usernames. I think the users that the framework runs as,
> > the actual node and resource managers, etc is confusing to a user (I am
> > very confused!) When I first got Myriad up I set my user under the
> > executor to be mapr, and then it appeared to work with impersonation from
> > queries etc. Now, I am trying the remote distribution and I have users
> set
> > in the config, potentially a user in my marathon json, and I am getting
> > errors on permissions of files when a node manager tries to start (a
> > separate issue I will post later). Basically, this is complex, and a page
> > describing out what needs to run where with which permissions and how
> that
> > interacts will be huge for people looking to put this into play.
> >
> > *Example Yarn Site:*
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Example%3A+yarn-site.xml
> >
> > This is helpful, but where does it go? Remember, the remote distribution
> > had us delete the yarn-site in the hadoop etc folder.
> >
> > *Myriad Webapp *
> > https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Webapp
> >
> > This should be fleshed out a bit more. Also, it's in the
> > /myriad-scheduler/src/main/resources/webapp based on my git clone, but in
> > the wiki that's not listed. I had to dig for it.
> >
> > Some questions here: could the webapp be built during the myriad
> building
> > process? Could it be then be packaged as tarball for execution either
> > manually via marathon or automatically in a container on mesos? I
> > understand this is a fresh piece of the puzzle, I am just thinking about
> > and verbalizing the "where" on this for the future
> >
> >
> >
> > Those are the items that come to mind thus far. I hope the tone of my
> > email is correct, this is a great project, and I want others to try it
> as I
> > have.
> >
> > John Omernik
> >
>
Re: Documentation Comments
Posted by Adam Bordelon <ad...@mesosphere.io>.
John, thanks a ton for your valuable feedback! We're glad to have your
perspective as a user of the project, and I'm ready+willing to give you
edit access to the wiki if you want to update it with your learnings,
elaborate anything that's unclear, or add a new "John's tips" page. Just
sign up for a wiki account, send me your accountId, and I'll grant you edit
access.
(I'll let others answer your specific questions)
On Wed, Aug 19, 2015 at 6:28 AM, John Omernik <jo...@omernik.com> wrote:
> Today, I will be playing the role of the fool/jester trying to get Myriad
> running. Basically, since getting Myriad running with Santosh quite a while
> ago, and now trying again with new versions of Hadoop, MapR, and Myriad, I
> wanted to hit up the wiki (
> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home) and
> outline
> points that as a non-dev living the code, are unclear to someone trying to
> utilize myriad or understand it's operation.
>
> Obviously, some of my points can be answered with "look here in the code"
> or look at this page, but I will try to outline my thought processes as I
> reviewed the current docs. Sometimes the way I approached the problem led
> me down a path of to a certain page, missing the answer in a different
> page, and thus some cross linking could be helpful.
>
> Please do not let my points be taken as anything other than a desire to
> improve how accessible Myriad is to the community, this is not a critique
> of the hard work everyone has done on the project. I also understand that
> given the work load and other issues, that fixing these issues in
> documentation may not be a priority. I am listing them out here, so that
> those folks who are SMEs on various points may be able to quickly add stuff
> and we'll organize it later.
>
>
> *Remote Distribution: *
>
> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution
>
> This whole section could use some work from a standpoint of what runs where
> and where that component gets its files. For example, I think it would
> help people to understand that the whole tarball created in step 6 has all
> the files for node managers and resource managers. Basically, everything
> runs from there. Here is a small example I am currently working with:
>
>
> Starting Myriad:
> Option 1: Use Marathon (provide example json, here is mine)
> {
> "cmd": "env && export
>
> YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos
> && hadoop-2.7.0/bin/yarn resourcemanager",
> "uris": ["maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz"],
> "cpus": 1.0,
> "mem": 1024,
> "id": "myriad",
> "instances": 1,
> "user": "mapr"
> }
>
> In this case, Marathon grabs the hadoop tarball and pulls it down, this
> tarball also has the Myriad yml file. When it executes the resource
> manager, it is brought up in Myriad and ready to run node managers by
> pulling the tarball to the slave nodes and executing the nodemanager. (I
> would imagine the work with history server etc would also use this
> tarball?).
>
> From here it will us NMInstances to launch a node manager. (Note, this is
> different from when I originally set things up... before, I could run the
> resource manager/myriad without a nodemanager, now it seems it's required
> based on the config in the src... could we expound on this in the docs
> somewhere?)
>
>
> Option 2: ???? (Are there other ways to launch the resource manager?)
>
> Step 6: So something that is unclear to me is the handling of the
> hadoop/yarn config files. In Step 6 on this page, there is "sudo rm
> hadoop-
> 2.5.0/etc/hadoop/*.xml" This doesn't makes sense to me. I actually ignored
> this step. For me, if I remove these xml files, then there is no place to
> get my files... I think? Since I am running the RM and NM from the same
> tarball, and Myriad config is here, and my goal is to not have anything
> installed on a node, where would I set yarn settings? This could be much
> clearer to me, and probably others.
>
> Step 2: Should we just be copying the Myriad files to
> /share/hadoop/yarn/lib folder? Do we worry about potentially overwrites of
> jars or version conflicts?
>
> *Configuring Cgroups*
> https://cwiki.apache.org/confluence/display/MYRIAD/Configuring+Cgroups
> At some point a little bit more about why one would want CGroups and issues
> that could occur with them. While many folks using Mesos/Myriad may
> understand this, others may not, and it's a good way to help people think
> positively about our project if we help educate them along the way.
>
> Minor point on enabling CGroups. This is confusing given my questions in
> remote distribution. in this it says I need to edit my yarn-site.xml, but
> in remote distribution it says delete my hadoop xml files. We need to
> address this conflict cause it can be confusing for a user coming onboard
>
> Nitpick: Enabling cgroups for mess-slave - should be - Enabling cgroups for
> mesos-slave
>
> *Myriad Configuration Properties*:
>
> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Configuration+Properties
> Based on the conversation on list with Yuliya,
> "Currently, this file is built into Myriad Scheduler jar. So, if you need
> to modify some of the properties in this file, modify them before building
> Myriad Scheduler."
> isn't accurate any more, and we should address that.
>
> The configuration file in the wiki is an old one, the nmInstances isn't in
> it, (and see my question about that above).
>
> Frameworks and usernames. I think the users that the framework runs as,
> the actual node and resource managers, etc is confusing to a user (I am
> very confused!) When I first got Myriad up I set my user under the
> executor to be mapr, and then it appeared to work with impersonation from
> queries etc. Now, I am trying the remote distribution and I have users set
> in the config, potentially a user in my marathon json, and I am getting
> errors on permissions of files when a node manager tries to start (a
> separate issue I will post later). Basically, this is complex, and a page
> describing out what needs to run where with which permissions and how that
> interacts will be huge for people looking to put this into play.
>
> *Example Yarn Site:*
> https://cwiki.apache.org/confluence/display/MYRIAD/Example%3A+yarn-site.xml
>
> This is helpful, but where does it go? Remember, the remote distribution
> had us delete the yarn-site in the hadoop etc folder.
>
> *Myriad Webapp *
> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Webapp
>
> This should be fleshed out a bit more. Also, it's in the
> /myriad-scheduler/src/main/resources/webapp based on my git clone, but in
> the wiki that's not listed. I had to dig for it.
>
> Some questions here: could the webapp be built during the myriad building
> process? Could it be then be packaged as tarball for execution either
> manually via marathon or automatically in a container on mesos? I
> understand this is a fresh piece of the puzzle, I am just thinking about
> and verbalizing the "where" on this for the future
>
>
>
> Those are the items that come to mind thus far. I hope the tone of my
> email is correct, this is a great project, and I want others to try it as I
> have.
>
> John Omernik
>