You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Gregorio ." <de...@outlook.it> on 2016/03/20 17:34:43 UTC
Possible path bug
Hello,
I'm new and I'm try for the first time to use hadoop (version 2.6.4). I downloaded the binaries from the official site (http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.6.4/hadoop-2.6.4.tar.gz).
I noticied something strange. In the hadoop folder, inside sbin there is the start-dfs.sh script. I changed it putting an echo at the beginning of the line 60.
When I run the start-dfs.sh I saw this:
$ hadoop-2.6.4/sbin/start-dfs.sh
Starting namenodes on [localhost]
/home/myuser/Desktop/Download/hadoop-2.6.4/sbin/hadoop-daemons.sh --config /home/myuser/Desktop/Download/hadoop-2.6.4/etc/hadoop --hostnames localhost --script /home/myuser/Desktop/Download/hadoop-2.6.4/sbin/hdfs start namenode
Starting secondary namenodes [0.0.0.0]
but in the sbin folder there isn't hdfs, because it is in the bin folder. I found the error comes from the line 25:
bin=`dirname "${BASH_SOURCE-$0}"`
where $BASH_SOURCE is hadoop-2.6.4/sbin/start-dfs.sh
is this a bug? or am I doing some kind of mistake?
I add here another little question:
if I try to do:
$ bin/hdfs start namenode
I got:
Could not find or load main class start
but if I try to do:
$ bin/hdfs namenode
seems start correctly. Is this the expected behaviour?
because I saw "bin/hdfs start namenode" is also used by start-dfs.sh
Thank you all,
Gregorio
RE: Possible path bug
Posted by "Gregorio ." <de...@outlook.it>.
Understood, thank you!
Gregorio
> Date: Sun, 20 Mar 2016 17:22:53 -0700
> Subject: Re: Possible path bug
> From: eyang@apache.org
> To: common-dev@hadoop.apache.org
>
> Hi Gregorio,
>
> ${BASH_SOURCE-$0} is not a bug. That is a short hand notation for locating
> the script's directory. There are some legacy reasons for this arrangement.
>
> $bin in hadoop-daemon[s].sh are designed to run in both development and
> binary deployment environment. Where $bin is hadoop source bin in
> development, and $bin is $HADOOP_PREFIX/sbin in binary deployed environment.
>
> hadoop-daemons.sh is the top layer of ssh to multiple nodes to start
> demonized processes.
>
> hadoop-daemon.sh is invoked by hadoop-daemons.sh to actually demonize the
> Hadoop services. hadoop-daemon.sh resolves the location of hdfs script
> base on environment variable in line 152-164.
>
> hdfs scripts are designed to run in foreground for user interaction, debug
> purpose or having some layer of provisioning system to run Hadoop processes
> and attach hadoop process as spawning child process.
>
> It is easy to misinterpret $bin variable due to the context switch in
> development environment and binary deployed environment.
> The development environment arrangement was for Hadoop prior to
> mavenization. I doubt running bin/* from hadoop source is still supported.
>
> There are still traces of hdfs script that utilize $bin variable directly
> without detecting if the script is running in source directory or binary
> deployment layout. Those are bugs that could get new developers into
> trouble. My recommendation is to avoid running the scripts from source to
> avoid the bugs introduced in HADOOP-7106. In HADOOP-6255, system admin
> related script were moved in $HADOOP_PREFIX/sbin instead of
> $HADOOP_PREFIX/bin. This change can be confusing, when developers used to
> do things one way.
>
> Hope this answers helps.
>
> regards,
> Eric
>
> On Sun, Mar 20, 2016 at 9:34 AM, Gregorio . <de...@outlook.it> wrote:
>
> > Hello,
> >
> > I'm new and I'm try for the first time to use hadoop (version 2.6.4). I
> > downloaded the binaries from the official site (
> > http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.6.4/hadoop-2.6.4.tar.gz
> > ).
> >
> > I noticied something strange. In the hadoop folder, inside sbin there is
> > the start-dfs.sh script. I changed it putting an echo at the beginning of
> > the line 60.
> >
> > When I run the start-dfs.sh I saw this:
> >
> > $ hadoop-2.6.4/sbin/start-dfs.sh
> >
> > Starting namenodes on [localhost]
> > /home/myuser/Desktop/Download/hadoop-2.6.4/sbin/hadoop-daemons.sh --config
> > /home/myuser/Desktop/Download/hadoop-2.6.4/etc/hadoop --hostnames localhost
> > --script /home/myuser/Desktop/Download/hadoop-2.6.4/sbin/hdfs start namenode
> > Starting secondary namenodes [0.0.0.0]
> >
> > but in the sbin folder there isn't hdfs, because it is in the bin folder.
> > I found the error comes from the line 25:
> >
> > bin=`dirname "${BASH_SOURCE-$0}"`
> >
> > where $BASH_SOURCE is hadoop-2.6.4/sbin/start-dfs.sh
> >
> > is this a bug? or am I doing some kind of mistake?
> >
> > I add here another little question:
> > if I try to do:
> >
> > $ bin/hdfs start namenode
> >
> > I got:
> >
> > Could not find or load main class start
> >
> > but if I try to do:
> >
> > $ bin/hdfs namenode
> >
> > seems start correctly. Is this the expected behaviour?
> > because I saw "bin/hdfs start namenode" is also used by start-dfs.sh
> >
> > Thank you all,
> > Gregorio
> >
Re: Possible path bug
Posted by Eric Yang <ey...@apache.org>.
Hi Gregorio,
${BASH_SOURCE-$0} is not a bug. That is a short hand notation for locating
the script's directory. There are some legacy reasons for this arrangement.
$bin in hadoop-daemon[s].sh are designed to run in both development and
binary deployment environment. Where $bin is hadoop source bin in
development, and $bin is $HADOOP_PREFIX/sbin in binary deployed environment.
hadoop-daemons.sh is the top layer of ssh to multiple nodes to start
demonized processes.
hadoop-daemon.sh is invoked by hadoop-daemons.sh to actually demonize the
Hadoop services. hadoop-daemon.sh resolves the location of hdfs script
base on environment variable in line 152-164.
hdfs scripts are designed to run in foreground for user interaction, debug
purpose or having some layer of provisioning system to run Hadoop processes
and attach hadoop process as spawning child process.
It is easy to misinterpret $bin variable due to the context switch in
development environment and binary deployed environment.
The development environment arrangement was for Hadoop prior to
mavenization. I doubt running bin/* from hadoop source is still supported.
There are still traces of hdfs script that utilize $bin variable directly
without detecting if the script is running in source directory or binary
deployment layout. Those are bugs that could get new developers into
trouble. My recommendation is to avoid running the scripts from source to
avoid the bugs introduced in HADOOP-7106. In HADOOP-6255, system admin
related script were moved in $HADOOP_PREFIX/sbin instead of
$HADOOP_PREFIX/bin. This change can be confusing, when developers used to
do things one way.
Hope this answers helps.
regards,
Eric
On Sun, Mar 20, 2016 at 9:34 AM, Gregorio . <de...@outlook.it> wrote:
> Hello,
>
> I'm new and I'm try for the first time to use hadoop (version 2.6.4). I
> downloaded the binaries from the official site (
> http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.6.4/hadoop-2.6.4.tar.gz
> ).
>
> I noticied something strange. In the hadoop folder, inside sbin there is
> the start-dfs.sh script. I changed it putting an echo at the beginning of
> the line 60.
>
> When I run the start-dfs.sh I saw this:
>
> $ hadoop-2.6.4/sbin/start-dfs.sh
>
> Starting namenodes on [localhost]
> /home/myuser/Desktop/Download/hadoop-2.6.4/sbin/hadoop-daemons.sh --config
> /home/myuser/Desktop/Download/hadoop-2.6.4/etc/hadoop --hostnames localhost
> --script /home/myuser/Desktop/Download/hadoop-2.6.4/sbin/hdfs start namenode
> Starting secondary namenodes [0.0.0.0]
>
> but in the sbin folder there isn't hdfs, because it is in the bin folder.
> I found the error comes from the line 25:
>
> bin=`dirname "${BASH_SOURCE-$0}"`
>
> where $BASH_SOURCE is hadoop-2.6.4/sbin/start-dfs.sh
>
> is this a bug? or am I doing some kind of mistake?
>
> I add here another little question:
> if I try to do:
>
> $ bin/hdfs start namenode
>
> I got:
>
> Could not find or load main class start
>
> but if I try to do:
>
> $ bin/hdfs namenode
>
> seems start correctly. Is this the expected behaviour?
> because I saw "bin/hdfs start namenode" is also used by start-dfs.sh
>
> Thank you all,
> Gregorio
>