You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bigtop.apache.org by David Fryer <df...@gmail.com> on 2014/08/07 15:23:41 UTC
Bigtop Presentation
Hi Bigtop!
I think I'm going to do a web presentation on setting up a bare-metal
hadoop cluster. The presentation slides that I will be using can be found
here:
https://docs.google.com/presentation/d/1cEkq2x-Q_n7nbFCttHIn3luoOkz3UMlEEtbudea9l1Y/edit?usp=sharing
If anyone has any edits or comments, I'd be happy to hear your feedback.
Thank You,
David Fryer
Re: Bigtop Presentation
Posted by Jay Vyas <ja...@gmail.com>.
sounds like good feedback.
- Given the thread below ..... I'd also suggest
1) spend some more on the way puppet works and dive into site.pp files, maybe diagram the core components.
2) Maybe even spend a minimal 1 - 2 slides on how puppet works? I wrote a blogpost recently about how I hack around debugging it. Maybe you could use that info indirectly in your talk http://jayunit100.blogspot.com/2014/07/a-few-tips-for-chasing-puppet-errors.html?m=1 ...
> On Aug 7, 2014, at 5:17 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
>> On Thu, Aug 07, 2014 at 02:55PM, David Fryer wrote:
>> Thanks for the feedback!
>> Does running puppet apply that way eliminate the need to make a site.csv?
>
> No, site.csv is still required. But it will allow you to set all the nodes
> uniformly per the specs from the site.csv. It goes without saying that
> site.csv needs to be available to each puppet execution (e.g. on every node).
>
>> What does bigtop_toolchain::packages include?
>
> It sets all needed packages for bigtop development, testing and such. The
> package sets are defined within the manifest - just look at it.
>
> Bigtop packages are explicitly declaring their runtime dependency. Which
> implies that everything should be already installed by running yum install
> (with or without Puppet). Is it your experience that running a Hadoop cluster
> requires something different from what bigtop packages are pulling in?
> If so - I'd be great to hear of it and fix it if there's a bug somewhere.
>
>> On slide #4 when writing the site.csv, I included the # in there to point
>> out that that was a note as opposed to something to be included, although
>> that is fairly unclear. What is a better way to make that obvious?
>
> It's more UX point, I guess. People tend to copy-paste code snippets and try
> to use them directly. In this case - '#' might be a cause of a later
> non-descriptive Puppet error ;) May be just make a comment outside of the code
> block?
>
>> Outside of that, I think I've fixed everything else.
>
> Thanks!
> Cos
>
>>> On Thu, Aug 7, 2014 at 2:38 PM, Konstantin Boudnik <co...@apache.org> wrote:
>>>
>>> Thanks for putting this together, David! Let's move into the coming Bigtop
>>> book (at some point) ;)
>>>
>>> Here's a suggestion:
>>> - the slides imply the use of the Bigtop master branch
>>> - which makes the sequence of the steps in slide #3 suboptimal. What I'd
>>> recommend to be done - and it will be less error-prone - is:
>>> 1) install puppet
>>> 2) clone bigtop and cd into the workspace
>>> 3) run
>>> puppet apply --modulepath=. -e "include bigtop_toolchain::packages"
>>>
>>> Slide #4
>>> step 1)
>>> IIRC '#' in the csv file will break Puppet execution
>>> jdk_package_name,<your java version> - it should be java package
>>> name, rather than just a version
>>> step 2) you don't need to make /data be owned by hdfs - only
>>> directoies /data/{1..4}
>>> step 3) needs to mention that deployment step has to be run on _each_
>>> node.
>>>
>>> As a practical note, I'd recommend to stay away from using 'jps' command as
>>> it's known to have funny issues. 'ps aux | grep java' is usually a
>>> bulletproof
>>> way to go.
>>>
>>> Hope it helps,
>>> cos
>>>
>>>> On Thu, Aug 07, 2014 at 09:23AM, David Fryer wrote:
>>>> Hi Bigtop!
>>>> I think I'm going to do a web presentation on setting up a bare-metal
>>>> hadoop cluster. The presentation slides that I will be using can be found
>>>> here:
>>> https://docs.google.com/presentation/d/1cEkq2x-Q_n7nbFCttHIn3luoOkz3UMlEEtbudea9l1Y/edit?usp=sharing
>>>> If anyone has any edits or comments, I'd be happy to hear your feedback.
>>>> Thank You,
>>>> David Fryer
>>>
Re: Bigtop Presentation
Posted by Konstantin Boudnik <co...@apache.org>.
On Thu, Aug 07, 2014 at 02:55PM, David Fryer wrote:
> Thanks for the feedback!
> Does running puppet apply that way eliminate the need to make a site.csv?
No, site.csv is still required. But it will allow you to set all the nodes
uniformly per the specs from the site.csv. It goes without saying that
site.csv needs to be available to each puppet execution (e.g. on every node).
> What does bigtop_toolchain::packages include?
It sets all needed packages for bigtop development, testing and such. The
package sets are defined within the manifest - just look at it.
Bigtop packages are explicitly declaring their runtime dependency. Which
implies that everything should be already installed by running yum install
(with or without Puppet). Is it your experience that running a Hadoop cluster
requires something different from what bigtop packages are pulling in?
If so - I'd be great to hear of it and fix it if there's a bug somewhere.
> On slide #4 when writing the site.csv, I included the # in there to point
> out that that was a note as opposed to something to be included, although
> that is fairly unclear. What is a better way to make that obvious?
It's more UX point, I guess. People tend to copy-paste code snippets and try
to use them directly. In this case - '#' might be a cause of a later
non-descriptive Puppet error ;) May be just make a comment outside of the code
block?
> Outside of that, I think I've fixed everything else.
Thanks!
Cos
> On Thu, Aug 7, 2014 at 2:38 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > Thanks for putting this together, David! Let's move into the coming Bigtop
> > book (at some point) ;)
> >
> > Here's a suggestion:
> > - the slides imply the use of the Bigtop master branch
> > - which makes the sequence of the steps in slide #3 suboptimal. What I'd
> > recommend to be done - and it will be less error-prone - is:
> > 1) install puppet
> > 2) clone bigtop and cd into the workspace
> > 3) run
> > puppet apply --modulepath=. -e "include bigtop_toolchain::packages"
> >
> > Slide #4
> > step 1)
> > IIRC '#' in the csv file will break Puppet execution
> > jdk_package_name,<your java version> - it should be java package
> > name, rather than just a version
> > step 2) you don't need to make /data be owned by hdfs - only
> > directoies /data/{1..4}
> > step 3) needs to mention that deployment step has to be run on _each_
> > node.
> >
> > As a practical note, I'd recommend to stay away from using 'jps' command as
> > it's known to have funny issues. 'ps aux | grep java' is usually a
> > bulletproof
> > way to go.
> >
> > Hope it helps,
> > cos
> >
> > On Thu, Aug 07, 2014 at 09:23AM, David Fryer wrote:
> > > Hi Bigtop!
> > > I think I'm going to do a web presentation on setting up a bare-metal
> > > hadoop cluster. The presentation slides that I will be using can be found
> > > here:
> > >
> > https://docs.google.com/presentation/d/1cEkq2x-Q_n7nbFCttHIn3luoOkz3UMlEEtbudea9l1Y/edit?usp=sharing
> > > If anyone has any edits or comments, I'd be happy to hear your feedback.
> > > Thank You,
> > > David Fryer
> >
Re: Bigtop Presentation
Posted by David Fryer <df...@gmail.com>.
Thanks for the feedback!
Does running puppet apply that way eliminate the need to make a site.csv?
What does bigtop_toolchain::packages include?
On slide #4 when writing the site.csv, I included the # in there to point
out that that was a note as opposed to something to be included, although
that is fairly unclear. What is a better way to make that obvious?
Outside of that, I think I've fixed everything else.
On Thu, Aug 7, 2014 at 2:38 PM, Konstantin Boudnik <co...@apache.org> wrote:
> Thanks for putting this together, David! Let's move into the coming Bigtop
> book (at some point) ;)
>
> Here's a suggestion:
> - the slides imply the use of the Bigtop master branch
> - which makes the sequence of the steps in slide #3 suboptimal. What I'd
> recommend to be done - and it will be less error-prone - is:
> 1) install puppet
> 2) clone bigtop and cd into the workspace
> 3) run
> puppet apply --modulepath=. -e "include bigtop_toolchain::packages"
>
> Slide #4
> step 1)
> IIRC '#' in the csv file will break Puppet execution
> jdk_package_name,<your java version> - it should be java package
> name, rather than just a version
> step 2) you don't need to make /data be owned by hdfs - only
> directoies /data/{1..4}
> step 3) needs to mention that deployment step has to be run on _each_
> node.
>
> As a practical note, I'd recommend to stay away from using 'jps' command as
> it's known to have funny issues. 'ps aux | grep java' is usually a
> bulletproof
> way to go.
>
> Hope it helps,
> cos
>
> On Thu, Aug 07, 2014 at 09:23AM, David Fryer wrote:
> > Hi Bigtop!
> > I think I'm going to do a web presentation on setting up a bare-metal
> > hadoop cluster. The presentation slides that I will be using can be found
> > here:
> >
> https://docs.google.com/presentation/d/1cEkq2x-Q_n7nbFCttHIn3luoOkz3UMlEEtbudea9l1Y/edit?usp=sharing
> > If anyone has any edits or comments, I'd be happy to hear your feedback.
> > Thank You,
> > David Fryer
>
Re: Bigtop Presentation
Posted by Konstantin Boudnik <co...@apache.org>.
Thanks for putting this together, David! Let's move into the coming Bigtop
book (at some point) ;)
Here's a suggestion:
- the slides imply the use of the Bigtop master branch
- which makes the sequence of the steps in slide #3 suboptimal. What I'd
recommend to be done - and it will be less error-prone - is:
1) install puppet
2) clone bigtop and cd into the workspace
3) run
puppet apply --modulepath=. -e "include bigtop_toolchain::packages"
Slide #4
step 1)
IIRC '#' in the csv file will break Puppet execution
jdk_package_name,<your java version> - it should be java package name, rather than just a version
step 2) you don't need to make /data be owned by hdfs - only directoies /data/{1..4}
step 3) needs to mention that deployment step has to be run on _each_ node.
As a practical note, I'd recommend to stay away from using 'jps' command as
it's known to have funny issues. 'ps aux | grep java' is usually a bulletproof
way to go.
Hope it helps,
cos
On Thu, Aug 07, 2014 at 09:23AM, David Fryer wrote:
> Hi Bigtop!
> I think I'm going to do a web presentation on setting up a bare-metal
> hadoop cluster. The presentation slides that I will be using can be found
> here:
> https://docs.google.com/presentation/d/1cEkq2x-Q_n7nbFCttHIn3luoOkz3UMlEEtbudea9l1Y/edit?usp=sharing
> If anyone has any edits or comments, I'd be happy to hear your feedback.
> Thank You,
> David Fryer
Re: Bigtop Presentation
Posted by Konstantin Boudnik <co...@apache.org>.
On Thu, Aug 07, 2014 at 10:40AM, David Fryer wrote:
> Mark, will this be fixed when Bigtop 0.8.0 goes out?
If you use master's recipes - it is there already.
> On Thu, Aug 7, 2014 at 10:36 AM, Mark Grover <ma...@apache.org> wrote:
>
> > I agree with Sean, good stuff!
> >
> > As for mapreduce_shuffle, we did make a similar change in BIGTOP-1118.
> > Have you take a look at that?
> >
> >
> > On Thu, Aug 7, 2014 at 7:31 AM, David Fryer <df...@gmail.com> wrote:
> >
> >> Thanks for the feedback! I'll make the change ASAP.
> >>
> >> As for the mapreduce edit, I haven't made a JIRA for it, but I believe
> >> that as of Hadoop 2.2, the default is correct. I'll double-check that, and
> >> if needed, will open a JIRA for it.
> >>
> >> Sent from my Mobile Device
> >> On Aug 7, 2014 10:28 AM, "Sean Mackrory" <ma...@gmail.com> wrote:
> >>
> >>> Great to see this being done - the project could really use nicer
> >>> documentation for users. Regarding the mapreduce_shuffle ->
> >>> mapreduce.shuffle issue - have you created or seen a JIRA for that? Haven't
> >>> run into it myself but we should get that fixed...
> >>>
> >>> One nitpick: when invoking scripts in /etc/init.d, it's usually
> >>> recommended to invoke them using the 'service' command. It cleans out the
> >>> environment to make sure the scripts are invoked in a nice predictable way.
> >>> It also looks a little more polished :)
> >>>
> >>> e.g
> >>>
> >>> /etc/init.d/hadoop-hdfs-namenode init -> service hadoop-hdfs-namenode
> >>> init
> >>>
> >>>
> >>> On Thu, Aug 7, 2014 at 7:23 AM, David Fryer <df...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi Bigtop!
> >>>> I think I'm going to do a web presentation on setting up a bare-metal
> >>>> hadoop cluster. The presentation slides that I will be using can be found
> >>>> here:
> >>>>
> >>>> https://docs.google.com/presentation/d/1cEkq2x-Q_n7nbFCttHIn3luoOkz3UMlEEtbudea9l1Y/edit?usp=sharing
> >>>> If anyone has any edits or comments, I'd be happy to hear your feedback.
> >>>> Thank You,
> >>>> David Fryer
> >>>>
> >>>
> >>>
> >
Re: Bigtop Presentation
Posted by Sean Mackrory <ma...@gmail.com>.
Looks like it!
On Thu, Aug 7, 2014 at 8:40 AM, David Fryer <df...@gmail.com> wrote:
> Mark, will this be fixed when Bigtop 0.8.0 goes out?
>
>
> On Thu, Aug 7, 2014 at 10:36 AM, Mark Grover <ma...@apache.org> wrote:
>
>> I agree with Sean, good stuff!
>>
>> As for mapreduce_shuffle, we did make a similar change in BIGTOP-1118.
>> Have you take a look at that?
>>
>>
>> On Thu, Aug 7, 2014 at 7:31 AM, David Fryer <df...@gmail.com> wrote:
>>
>>> Thanks for the feedback! I'll make the change ASAP.
>>>
>>> As for the mapreduce edit, I haven't made a JIRA for it, but I believe
>>> that as of Hadoop 2.2, the default is correct. I'll double-check that, and
>>> if needed, will open a JIRA for it.
>>>
>>> Sent from my Mobile Device
>>> On Aug 7, 2014 10:28 AM, "Sean Mackrory" <ma...@gmail.com> wrote:
>>>
>>>> Great to see this being done - the project could really use nicer
>>>> documentation for users. Regarding the mapreduce_shuffle ->
>>>> mapreduce.shuffle issue - have you created or seen a JIRA for that? Haven't
>>>> run into it myself but we should get that fixed...
>>>>
>>>> One nitpick: when invoking scripts in /etc/init.d, it's usually
>>>> recommended to invoke them using the 'service' command. It cleans out the
>>>> environment to make sure the scripts are invoked in a nice predictable way.
>>>> It also looks a little more polished :)
>>>>
>>>> e.g
>>>>
>>>> /etc/init.d/hadoop-hdfs-namenode init -> service hadoop-hdfs-namenode
>>>> init
>>>>
>>>>
>>>> On Thu, Aug 7, 2014 at 7:23 AM, David Fryer <df...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Bigtop!
>>>>> I think I'm going to do a web presentation on setting up a bare-metal
>>>>> hadoop cluster. The presentation slides that I will be using can be found
>>>>> here:
>>>>>
>>>>> https://docs.google.com/presentation/d/1cEkq2x-Q_n7nbFCttHIn3luoOkz3UMlEEtbudea9l1Y/edit?usp=sharing
>>>>> If anyone has any edits or comments, I'd be happy to hear your
>>>>> feedback.
>>>>> Thank You,
>>>>> David Fryer
>>>>>
>>>>
>>>>
>>
>
Re: Bigtop Presentation
Posted by David Fryer <df...@gmail.com>.
Great!
Sent from my Mobile Device
On Aug 7, 2014 10:43 AM, "Mark Grover" <ma...@apache.org> wrote:
> Yes, it's already been committed to the master branch (and hence 0.8).
>
>
> On Thu, Aug 7, 2014 at 7:40 AM, David Fryer <df...@gmail.com> wrote:
>
>> Mark, will this be fixed when Bigtop 0.8.0 goes out?
>>
>>
>> On Thu, Aug 7, 2014 at 10:36 AM, Mark Grover <ma...@apache.org> wrote:
>>
>>> I agree with Sean, good stuff!
>>>
>>> As for mapreduce_shuffle, we did make a similar change in BIGTOP-1118.
>>> Have you take a look at that?
>>>
>>>
>>> On Thu, Aug 7, 2014 at 7:31 AM, David Fryer <df...@gmail.com>
>>> wrote:
>>>
>>>> Thanks for the feedback! I'll make the change ASAP.
>>>>
>>>> As for the mapreduce edit, I haven't made a JIRA for it, but I believe
>>>> that as of Hadoop 2.2, the default is correct. I'll double-check that, and
>>>> if needed, will open a JIRA for it.
>>>>
>>>> Sent from my Mobile Device
>>>> On Aug 7, 2014 10:28 AM, "Sean Mackrory" <ma...@gmail.com> wrote:
>>>>
>>>>> Great to see this being done - the project could really use nicer
>>>>> documentation for users. Regarding the mapreduce_shuffle ->
>>>>> mapreduce.shuffle issue - have you created or seen a JIRA for that? Haven't
>>>>> run into it myself but we should get that fixed...
>>>>>
>>>>> One nitpick: when invoking scripts in /etc/init.d, it's usually
>>>>> recommended to invoke them using the 'service' command. It cleans out the
>>>>> environment to make sure the scripts are invoked in a nice predictable way.
>>>>> It also looks a little more polished :)
>>>>>
>>>>> e.g
>>>>>
>>>>> /etc/init.d/hadoop-hdfs-namenode init -> service hadoop-hdfs-namenode
>>>>> init
>>>>>
>>>>>
>>>>> On Thu, Aug 7, 2014 at 7:23 AM, David Fryer <df...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Bigtop!
>>>>>> I think I'm going to do a web presentation on setting up a bare-metal
>>>>>> hadoop cluster. The presentation slides that I will be using can be found
>>>>>> here:
>>>>>>
>>>>>> https://docs.google.com/presentation/d/1cEkq2x-Q_n7nbFCttHIn3luoOkz3UMlEEtbudea9l1Y/edit?usp=sharing
>>>>>> If anyone has any edits or comments, I'd be happy to hear your
>>>>>> feedback.
>>>>>> Thank You,
>>>>>> David Fryer
>>>>>>
>>>>>
>>>>>
>>>
>>
>
Re: Bigtop Presentation
Posted by Mark Grover <ma...@apache.org>.
Yes, it's already been committed to the master branch (and hence 0.8).
On Thu, Aug 7, 2014 at 7:40 AM, David Fryer <df...@gmail.com> wrote:
> Mark, will this be fixed when Bigtop 0.8.0 goes out?
>
>
> On Thu, Aug 7, 2014 at 10:36 AM, Mark Grover <ma...@apache.org> wrote:
>
>> I agree with Sean, good stuff!
>>
>> As for mapreduce_shuffle, we did make a similar change in BIGTOP-1118.
>> Have you take a look at that?
>>
>>
>> On Thu, Aug 7, 2014 at 7:31 AM, David Fryer <df...@gmail.com> wrote:
>>
>>> Thanks for the feedback! I'll make the change ASAP.
>>>
>>> As for the mapreduce edit, I haven't made a JIRA for it, but I believe
>>> that as of Hadoop 2.2, the default is correct. I'll double-check that, and
>>> if needed, will open a JIRA for it.
>>>
>>> Sent from my Mobile Device
>>> On Aug 7, 2014 10:28 AM, "Sean Mackrory" <ma...@gmail.com> wrote:
>>>
>>>> Great to see this being done - the project could really use nicer
>>>> documentation for users. Regarding the mapreduce_shuffle ->
>>>> mapreduce.shuffle issue - have you created or seen a JIRA for that? Haven't
>>>> run into it myself but we should get that fixed...
>>>>
>>>> One nitpick: when invoking scripts in /etc/init.d, it's usually
>>>> recommended to invoke them using the 'service' command. It cleans out the
>>>> environment to make sure the scripts are invoked in a nice predictable way.
>>>> It also looks a little more polished :)
>>>>
>>>> e.g
>>>>
>>>> /etc/init.d/hadoop-hdfs-namenode init -> service hadoop-hdfs-namenode
>>>> init
>>>>
>>>>
>>>> On Thu, Aug 7, 2014 at 7:23 AM, David Fryer <df...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Bigtop!
>>>>> I think I'm going to do a web presentation on setting up a bare-metal
>>>>> hadoop cluster. The presentation slides that I will be using can be found
>>>>> here:
>>>>>
>>>>> https://docs.google.com/presentation/d/1cEkq2x-Q_n7nbFCttHIn3luoOkz3UMlEEtbudea9l1Y/edit?usp=sharing
>>>>> If anyone has any edits or comments, I'd be happy to hear your
>>>>> feedback.
>>>>> Thank You,
>>>>> David Fryer
>>>>>
>>>>
>>>>
>>
>
Re: Bigtop Presentation
Posted by David Fryer <df...@gmail.com>.
Mark, will this be fixed when Bigtop 0.8.0 goes out?
On Thu, Aug 7, 2014 at 10:36 AM, Mark Grover <ma...@apache.org> wrote:
> I agree with Sean, good stuff!
>
> As for mapreduce_shuffle, we did make a similar change in BIGTOP-1118.
> Have you take a look at that?
>
>
> On Thu, Aug 7, 2014 at 7:31 AM, David Fryer <df...@gmail.com> wrote:
>
>> Thanks for the feedback! I'll make the change ASAP.
>>
>> As for the mapreduce edit, I haven't made a JIRA for it, but I believe
>> that as of Hadoop 2.2, the default is correct. I'll double-check that, and
>> if needed, will open a JIRA for it.
>>
>> Sent from my Mobile Device
>> On Aug 7, 2014 10:28 AM, "Sean Mackrory" <ma...@gmail.com> wrote:
>>
>>> Great to see this being done - the project could really use nicer
>>> documentation for users. Regarding the mapreduce_shuffle ->
>>> mapreduce.shuffle issue - have you created or seen a JIRA for that? Haven't
>>> run into it myself but we should get that fixed...
>>>
>>> One nitpick: when invoking scripts in /etc/init.d, it's usually
>>> recommended to invoke them using the 'service' command. It cleans out the
>>> environment to make sure the scripts are invoked in a nice predictable way.
>>> It also looks a little more polished :)
>>>
>>> e.g
>>>
>>> /etc/init.d/hadoop-hdfs-namenode init -> service hadoop-hdfs-namenode
>>> init
>>>
>>>
>>> On Thu, Aug 7, 2014 at 7:23 AM, David Fryer <df...@gmail.com>
>>> wrote:
>>>
>>>> Hi Bigtop!
>>>> I think I'm going to do a web presentation on setting up a bare-metal
>>>> hadoop cluster. The presentation slides that I will be using can be found
>>>> here:
>>>>
>>>> https://docs.google.com/presentation/d/1cEkq2x-Q_n7nbFCttHIn3luoOkz3UMlEEtbudea9l1Y/edit?usp=sharing
>>>> If anyone has any edits or comments, I'd be happy to hear your feedback.
>>>> Thank You,
>>>> David Fryer
>>>>
>>>
>>>
>
Re: Bigtop Presentation
Posted by Mark Grover <ma...@apache.org>.
I agree with Sean, good stuff!
As for mapreduce_shuffle, we did make a similar change in BIGTOP-1118. Have
you take a look at that?
On Thu, Aug 7, 2014 at 7:31 AM, David Fryer <df...@gmail.com> wrote:
> Thanks for the feedback! I'll make the change ASAP.
>
> As for the mapreduce edit, I haven't made a JIRA for it, but I believe
> that as of Hadoop 2.2, the default is correct. I'll double-check that, and
> if needed, will open a JIRA for it.
>
> Sent from my Mobile Device
> On Aug 7, 2014 10:28 AM, "Sean Mackrory" <ma...@gmail.com> wrote:
>
>> Great to see this being done - the project could really use nicer
>> documentation for users. Regarding the mapreduce_shuffle ->
>> mapreduce.shuffle issue - have you created or seen a JIRA for that? Haven't
>> run into it myself but we should get that fixed...
>>
>> One nitpick: when invoking scripts in /etc/init.d, it's usually
>> recommended to invoke them using the 'service' command. It cleans out the
>> environment to make sure the scripts are invoked in a nice predictable way.
>> It also looks a little more polished :)
>>
>> e.g
>>
>> /etc/init.d/hadoop-hdfs-namenode init -> service hadoop-hdfs-namenode init
>>
>>
>> On Thu, Aug 7, 2014 at 7:23 AM, David Fryer <df...@gmail.com> wrote:
>>
>>> Hi Bigtop!
>>> I think I'm going to do a web presentation on setting up a bare-metal
>>> hadoop cluster. The presentation slides that I will be using can be found
>>> here:
>>>
>>> https://docs.google.com/presentation/d/1cEkq2x-Q_n7nbFCttHIn3luoOkz3UMlEEtbudea9l1Y/edit?usp=sharing
>>> If anyone has any edits or comments, I'd be happy to hear your feedback.
>>> Thank You,
>>> David Fryer
>>>
>>
>>
Re: Bigtop Presentation
Posted by David Fryer <df...@gmail.com>.
Thanks for the feedback! I'll make the change ASAP.
As for the mapreduce edit, I haven't made a JIRA for it, but I believe that
as of Hadoop 2.2, the default is correct. I'll double-check that, and if
needed, will open a JIRA for it.
Sent from my Mobile Device
On Aug 7, 2014 10:28 AM, "Sean Mackrory" <ma...@gmail.com> wrote:
> Great to see this being done - the project could really use nicer
> documentation for users. Regarding the mapreduce_shuffle ->
> mapreduce.shuffle issue - have you created or seen a JIRA for that? Haven't
> run into it myself but we should get that fixed...
>
> One nitpick: when invoking scripts in /etc/init.d, it's usually
> recommended to invoke them using the 'service' command. It cleans out the
> environment to make sure the scripts are invoked in a nice predictable way.
> It also looks a little more polished :)
>
> e.g
>
> /etc/init.d/hadoop-hdfs-namenode init -> service hadoop-hdfs-namenode init
>
>
> On Thu, Aug 7, 2014 at 7:23 AM, David Fryer <df...@gmail.com> wrote:
>
>> Hi Bigtop!
>> I think I'm going to do a web presentation on setting up a bare-metal
>> hadoop cluster. The presentation slides that I will be using can be found
>> here:
>>
>> https://docs.google.com/presentation/d/1cEkq2x-Q_n7nbFCttHIn3luoOkz3UMlEEtbudea9l1Y/edit?usp=sharing
>> If anyone has any edits or comments, I'd be happy to hear your feedback.
>> Thank You,
>> David Fryer
>>
>
>
Re: Bigtop Presentation
Posted by Jay Vyas <ja...@gmail.com>.
Yes : great start Dave , here are two things you should update:
1) please specify the in the title that this is deploying bigtop 0.7.0... on the custom slides that this change is only necessary for deploying 0.7.0 rpms. The puppet recipes on head target for 0'8.0, but you are using 0.7.0 rpms, which require "mapreduce.shuffle" hence the need for that mapreduce_shuffle parameter change .
2) Can you also include debugging info about how to cat the yarn binary logs for when containers fail?
> On Aug 7, 2014, at 10:27 AM, Sean Mackrory <ma...@gmail.com> wrote:
>
> Great to see this being done - the project could really use nicer documentation for users. Regarding the mapreduce_shuffle -> mapreduce.shuffle issue - have you created or seen a JIRA for that? Haven't run into it myself but we should get that fixed...
>
> One nitpick: when invoking scripts in /etc/init.d, it's usually recommended to invoke them using the 'service' command. It cleans out the environment to make sure the scripts are invoked in a nice predictable way. It also looks a little more polished :)
>
> e.g
>
> /etc/init.d/hadoop-hdfs-namenode init -> service hadoop-hdfs-namenode init
>
>
>> On Thu, Aug 7, 2014 at 7:23 AM, David Fryer <df...@gmail.com> wrote:
>> Hi Bigtop!
>> I think I'm going to do a web presentation on setting up a bare-metal hadoop cluster. The presentation slides that I will be using can be found here:
>> https://docs.google.com/presentation/d/1cEkq2x-Q_n7nbFCttHIn3luoOkz3UMlEEtbudea9l1Y/edit?usp=sharing
>> If anyone has any edits or comments, I'd be happy to hear your feedback.
>> Thank You,
>> David Fryer
>
Re: Bigtop Presentation
Posted by Sean Mackrory <ma...@gmail.com>.
Great to see this being done - the project could really use nicer
documentation for users. Regarding the mapreduce_shuffle ->
mapreduce.shuffle issue - have you created or seen a JIRA for that? Haven't
run into it myself but we should get that fixed...
One nitpick: when invoking scripts in /etc/init.d, it's usually recommended
to invoke them using the 'service' command. It cleans out the environment
to make sure the scripts are invoked in a nice predictable way. It also
looks a little more polished :)
e.g
/etc/init.d/hadoop-hdfs-namenode init -> service hadoop-hdfs-namenode init
On Thu, Aug 7, 2014 at 7:23 AM, David Fryer <df...@gmail.com> wrote:
> Hi Bigtop!
> I think I'm going to do a web presentation on setting up a bare-metal
> hadoop cluster. The presentation slides that I will be using can be found
> here:
>
> https://docs.google.com/presentation/d/1cEkq2x-Q_n7nbFCttHIn3luoOkz3UMlEEtbudea9l1Y/edit?usp=sharing
> If anyone has any edits or comments, I'd be happy to hear your feedback.
> Thank You,
> David Fryer
>