You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Chris MacKenzie <st...@chrismackenziephotography.co.uk> on 2014/07/23 11:47:59 UTC
How to set up the conf folder
Hi,
Can anyone shed some light on this for me. Every time I attempt to set up
the conf directory, I run into a whole load of errors and ssh issues which
I don¹t see when my config files are in etc/hadoop
I want to understand how to use the conf directory. My ultimate goal is to
use symbolic links
I have a running cluster based version of hadoop-2.4.1. I start and stop
the cluster from the RM with:
./hadoop-2.4.1/sbin/start-dfs.sh
./hadoop-2.4.1/sbin/start-yarn.sh
My understanding is that to use the conf directory my settings should be
was follows:
Settings:
bash-profile has $HADOOP_CONF_DIR =
/scratch/extra/cm469/hadoop-2.4.1/etc/hadoop
Hadoop-env.sh has export
HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop/conf²}
conf/core-site.xml
conf/hdfs-site.xml
conf/mapred-site.xml
conf/yarn-site.xml
conf/capacity-scheduler.xml
etc/hadoop/hadoop-env.sh
etc/hadoop/slaves
etc/hadoop/masters
Thanks in advance,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
Re: How to set up the conf folder
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi Ravindra,
Thanks for replying, it’s much appreciated.
That’s always been the case with my setup:
export HADOOP_PREFIX="/scratch/extra/cm469/hadoop-2.4.1"
export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
I think my issue is that I have not set yarn-env.sh up correctly. TBH I
didn’t know it existed. I plan to get round to trying that at some point
in the near future.
Many thanks,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
From: Ravindra <ra...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, 28 July 2014 13:09
To: <us...@hadoop.apache.org>
Subject: Re: How to set up the conf folder
Hi,
Could you try putting this in .bash_profile
export HADOOP_CONF_DIR=/scratch/extra/cm469/hadoop-2.4.1/etc/hadoop/
Regards,
Ravindra
On Wed, Jul 23, 2014 at 3:17 PM, Chris MacKenzie
<st...@chrismackenziephotography.co.uk> wrote:
Hi,
Can anyone shed some light on this for me. Every time I attempt to set up
the conf directory, I run into a whole load of errors and ssh issues which
I don¹t see when my config files are in etc/hadoop
I want to understand how to use the conf directory. My ultimate goal is to
use symbolic links
I have a running cluster based version of hadoop-2.4.1. I start and stop
the cluster from the RM with:
./hadoop-2.4.1/sbin/start-dfs.sh
./hadoop-2.4.1/sbin/start-yarn.sh
My understanding is that to use the conf directory my settings should be
was follows:
Settings:
bash-profile has $HADOOP_CONF_DIR =
/scratch/extra/cm469/hadoop-2.4.1/etc/hadoop
Hadoop-env.sh has export
HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop/conf²}
conf/core-site.xml
conf/hdfs-site.xml
conf/mapred-site.xml
conf/yarn-site.xml
conf/capacity-scheduler.xml
etc/hadoop/hadoop-env.sh
etc/hadoop/slaves
etc/hadoop/masters
Thanks in advance,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk>
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
Re: How to set up the conf folder
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi Ravindra,
Thanks for replying, it’s much appreciated.
That’s always been the case with my setup:
export HADOOP_PREFIX="/scratch/extra/cm469/hadoop-2.4.1"
export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
I think my issue is that I have not set yarn-env.sh up correctly. TBH I
didn’t know it existed. I plan to get round to trying that at some point
in the near future.
Many thanks,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
From: Ravindra <ra...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, 28 July 2014 13:09
To: <us...@hadoop.apache.org>
Subject: Re: How to set up the conf folder
Hi,
Could you try putting this in .bash_profile
export HADOOP_CONF_DIR=/scratch/extra/cm469/hadoop-2.4.1/etc/hadoop/
Regards,
Ravindra
On Wed, Jul 23, 2014 at 3:17 PM, Chris MacKenzie
<st...@chrismackenziephotography.co.uk> wrote:
Hi,
Can anyone shed some light on this for me. Every time I attempt to set up
the conf directory, I run into a whole load of errors and ssh issues which
I don¹t see when my config files are in etc/hadoop
I want to understand how to use the conf directory. My ultimate goal is to
use symbolic links
I have a running cluster based version of hadoop-2.4.1. I start and stop
the cluster from the RM with:
./hadoop-2.4.1/sbin/start-dfs.sh
./hadoop-2.4.1/sbin/start-yarn.sh
My understanding is that to use the conf directory my settings should be
was follows:
Settings:
bash-profile has $HADOOP_CONF_DIR =
/scratch/extra/cm469/hadoop-2.4.1/etc/hadoop
Hadoop-env.sh has export
HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop/conf²}
conf/core-site.xml
conf/hdfs-site.xml
conf/mapred-site.xml
conf/yarn-site.xml
conf/capacity-scheduler.xml
etc/hadoop/hadoop-env.sh
etc/hadoop/slaves
etc/hadoop/masters
Thanks in advance,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk>
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
Re: How to set up the conf folder
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi Ravindra,
Thanks for replying, it’s much appreciated.
That’s always been the case with my setup:
export HADOOP_PREFIX="/scratch/extra/cm469/hadoop-2.4.1"
export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
I think my issue is that I have not set yarn-env.sh up correctly. TBH I
didn’t know it existed. I plan to get round to trying that at some point
in the near future.
Many thanks,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
From: Ravindra <ra...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, 28 July 2014 13:09
To: <us...@hadoop.apache.org>
Subject: Re: How to set up the conf folder
Hi,
Could you try putting this in .bash_profile
export HADOOP_CONF_DIR=/scratch/extra/cm469/hadoop-2.4.1/etc/hadoop/
Regards,
Ravindra
On Wed, Jul 23, 2014 at 3:17 PM, Chris MacKenzie
<st...@chrismackenziephotography.co.uk> wrote:
Hi,
Can anyone shed some light on this for me. Every time I attempt to set up
the conf directory, I run into a whole load of errors and ssh issues which
I don¹t see when my config files are in etc/hadoop
I want to understand how to use the conf directory. My ultimate goal is to
use symbolic links
I have a running cluster based version of hadoop-2.4.1. I start and stop
the cluster from the RM with:
./hadoop-2.4.1/sbin/start-dfs.sh
./hadoop-2.4.1/sbin/start-yarn.sh
My understanding is that to use the conf directory my settings should be
was follows:
Settings:
bash-profile has $HADOOP_CONF_DIR =
/scratch/extra/cm469/hadoop-2.4.1/etc/hadoop
Hadoop-env.sh has export
HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop/conf²}
conf/core-site.xml
conf/hdfs-site.xml
conf/mapred-site.xml
conf/yarn-site.xml
conf/capacity-scheduler.xml
etc/hadoop/hadoop-env.sh
etc/hadoop/slaves
etc/hadoop/masters
Thanks in advance,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk>
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
Re: How to set up the conf folder
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi Ravindra,
Thanks for replying, it’s much appreciated.
That’s always been the case with my setup:
export HADOOP_PREFIX="/scratch/extra/cm469/hadoop-2.4.1"
export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
I think my issue is that I have not set yarn-env.sh up correctly. TBH I
didn’t know it existed. I plan to get round to trying that at some point
in the near future.
Many thanks,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
From: Ravindra <ra...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, 28 July 2014 13:09
To: <us...@hadoop.apache.org>
Subject: Re: How to set up the conf folder
Hi,
Could you try putting this in .bash_profile
export HADOOP_CONF_DIR=/scratch/extra/cm469/hadoop-2.4.1/etc/hadoop/
Regards,
Ravindra
On Wed, Jul 23, 2014 at 3:17 PM, Chris MacKenzie
<st...@chrismackenziephotography.co.uk> wrote:
Hi,
Can anyone shed some light on this for me. Every time I attempt to set up
the conf directory, I run into a whole load of errors and ssh issues which
I don¹t see when my config files are in etc/hadoop
I want to understand how to use the conf directory. My ultimate goal is to
use symbolic links
I have a running cluster based version of hadoop-2.4.1. I start and stop
the cluster from the RM with:
./hadoop-2.4.1/sbin/start-dfs.sh
./hadoop-2.4.1/sbin/start-yarn.sh
My understanding is that to use the conf directory my settings should be
was follows:
Settings:
bash-profile has $HADOOP_CONF_DIR =
/scratch/extra/cm469/hadoop-2.4.1/etc/hadoop
Hadoop-env.sh has export
HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop/conf²}
conf/core-site.xml
conf/hdfs-site.xml
conf/mapred-site.xml
conf/yarn-site.xml
conf/capacity-scheduler.xml
etc/hadoop/hadoop-env.sh
etc/hadoop/slaves
etc/hadoop/masters
Thanks in advance,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk>
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
Re: How to set up the conf folder
Posted by Ravindra <ra...@gmail.com>.
Hi,
Could you try putting this in .bash_profile
export HADOOP_CONF_DIR=/scratch/extra/cm469/hadoop-2.4.1/etc/hadoop/
Regards,
Ravindra
On Wed, Jul 23, 2014 at 3:17 PM, Chris MacKenzie <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> Can anyone shed some light on this for me. Every time I attempt to set up
> the conf directory, I run into a whole load of errors and ssh issues which
> I don¹t see when my config files are in etc/hadoop
>
> I want to understand how to use the conf directory. My ultimate goal is to
> use symbolic links
>
> I have a running cluster based version of hadoop-2.4.1. I start and stop
> the cluster from the RM with:
>
> ./hadoop-2.4.1/sbin/start-dfs.sh
> ./hadoop-2.4.1/sbin/start-yarn.sh
>
> My understanding is that to use the conf directory my settings should be
> was follows:
>
>
> Settings:
> bash-profile has $HADOOP_CONF_DIR =
> /scratch/extra/cm469/hadoop-2.4.1/etc/hadoop
> Hadoop-env.sh has export
> HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop/conf²}
>
> conf/core-site.xml
> conf/hdfs-site.xml
>
> conf/mapred-site.xml
>
> conf/yarn-site.xml
>
> conf/capacity-scheduler.xml
>
>
> etc/hadoop/hadoop-env.sh
>
>
> etc/hadoop/slaves
>
> etc/hadoop/masters
>
>
>
> Thanks in advance,
>
> Chris MacKenzie
> telephone: 0131 332 6967
> email: studio@chrismackenziephotography.co.uk
> corporate: www.chrismackenziephotography.co.uk
> <http://www.chrismackenziephotography.co.uk/>
> <http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
> <http://www.linkedin.com/in/chrismackenziephotography/>
>
>
>
Re: How to set up the conf folder
Posted by Ravindra <ra...@gmail.com>.
Hi,
Could you try putting this in .bash_profile
export HADOOP_CONF_DIR=/scratch/extra/cm469/hadoop-2.4.1/etc/hadoop/
Regards,
Ravindra
On Wed, Jul 23, 2014 at 3:17 PM, Chris MacKenzie <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> Can anyone shed some light on this for me. Every time I attempt to set up
> the conf directory, I run into a whole load of errors and ssh issues which
> I don¹t see when my config files are in etc/hadoop
>
> I want to understand how to use the conf directory. My ultimate goal is to
> use symbolic links
>
> I have a running cluster based version of hadoop-2.4.1. I start and stop
> the cluster from the RM with:
>
> ./hadoop-2.4.1/sbin/start-dfs.sh
> ./hadoop-2.4.1/sbin/start-yarn.sh
>
> My understanding is that to use the conf directory my settings should be
> was follows:
>
>
> Settings:
> bash-profile has $HADOOP_CONF_DIR =
> /scratch/extra/cm469/hadoop-2.4.1/etc/hadoop
> Hadoop-env.sh has export
> HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop/conf²}
>
> conf/core-site.xml
> conf/hdfs-site.xml
>
> conf/mapred-site.xml
>
> conf/yarn-site.xml
>
> conf/capacity-scheduler.xml
>
>
> etc/hadoop/hadoop-env.sh
>
>
> etc/hadoop/slaves
>
> etc/hadoop/masters
>
>
>
> Thanks in advance,
>
> Chris MacKenzie
> telephone: 0131 332 6967
> email: studio@chrismackenziephotography.co.uk
> corporate: www.chrismackenziephotography.co.uk
> <http://www.chrismackenziephotography.co.uk/>
> <http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
> <http://www.linkedin.com/in/chrismackenziephotography/>
>
>
>
Re: How to set up the conf folder
Posted by Ravindra <ra...@gmail.com>.
Hi,
Could you try putting this in .bash_profile
export HADOOP_CONF_DIR=/scratch/extra/cm469/hadoop-2.4.1/etc/hadoop/
Regards,
Ravindra
On Wed, Jul 23, 2014 at 3:17 PM, Chris MacKenzie <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> Can anyone shed some light on this for me. Every time I attempt to set up
> the conf directory, I run into a whole load of errors and ssh issues which
> I don¹t see when my config files are in etc/hadoop
>
> I want to understand how to use the conf directory. My ultimate goal is to
> use symbolic links
>
> I have a running cluster based version of hadoop-2.4.1. I start and stop
> the cluster from the RM with:
>
> ./hadoop-2.4.1/sbin/start-dfs.sh
> ./hadoop-2.4.1/sbin/start-yarn.sh
>
> My understanding is that to use the conf directory my settings should be
> was follows:
>
>
> Settings:
> bash-profile has $HADOOP_CONF_DIR =
> /scratch/extra/cm469/hadoop-2.4.1/etc/hadoop
> Hadoop-env.sh has export
> HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop/conf²}
>
> conf/core-site.xml
> conf/hdfs-site.xml
>
> conf/mapred-site.xml
>
> conf/yarn-site.xml
>
> conf/capacity-scheduler.xml
>
>
> etc/hadoop/hadoop-env.sh
>
>
> etc/hadoop/slaves
>
> etc/hadoop/masters
>
>
>
> Thanks in advance,
>
> Chris MacKenzie
> telephone: 0131 332 6967
> email: studio@chrismackenziephotography.co.uk
> corporate: www.chrismackenziephotography.co.uk
> <http://www.chrismackenziephotography.co.uk/>
> <http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
> <http://www.linkedin.com/in/chrismackenziephotography/>
>
>
>
Re: How to set up the conf folder
Posted by Ravindra <ra...@gmail.com>.
Hi,
Could you try putting this in .bash_profile
export HADOOP_CONF_DIR=/scratch/extra/cm469/hadoop-2.4.1/etc/hadoop/
Regards,
Ravindra
On Wed, Jul 23, 2014 at 3:17 PM, Chris MacKenzie <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> Can anyone shed some light on this for me. Every time I attempt to set up
> the conf directory, I run into a whole load of errors and ssh issues which
> I don¹t see when my config files are in etc/hadoop
>
> I want to understand how to use the conf directory. My ultimate goal is to
> use symbolic links
>
> I have a running cluster based version of hadoop-2.4.1. I start and stop
> the cluster from the RM with:
>
> ./hadoop-2.4.1/sbin/start-dfs.sh
> ./hadoop-2.4.1/sbin/start-yarn.sh
>
> My understanding is that to use the conf directory my settings should be
> was follows:
>
>
> Settings:
> bash-profile has $HADOOP_CONF_DIR =
> /scratch/extra/cm469/hadoop-2.4.1/etc/hadoop
> Hadoop-env.sh has export
> HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop/conf²}
>
> conf/core-site.xml
> conf/hdfs-site.xml
>
> conf/mapred-site.xml
>
> conf/yarn-site.xml
>
> conf/capacity-scheduler.xml
>
>
> etc/hadoop/hadoop-env.sh
>
>
> etc/hadoop/slaves
>
> etc/hadoop/masters
>
>
>
> Thanks in advance,
>
> Chris MacKenzie
> telephone: 0131 332 6967
> email: studio@chrismackenziephotography.co.uk
> corporate: www.chrismackenziephotography.co.uk
> <http://www.chrismackenziephotography.co.uk/>
> <http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
> <http://www.linkedin.com/in/chrismackenziephotography/>
>
>
>