You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bigtop.apache.org by "David Liu (gmail dev)" <em...@gmail.com> on 2012/03/20 19:03:58 UTC
bigtop ubuntu build wiki page added
I have added a build page for Ubuntu:
https://cwiki.apache.org/confluence/display/BIGTOP/Building+Bigtop+on+Ubuntu
Hope it is useful for other users.
David
Re: bigtop ubuntu build wiki page added
Posted by "David Liu (gmail dev)" <em...@gmail.com>.
While I put together the wiki page, I was trying to make all the steps
automated with the least human intervention, and noting places where human
involvement is needed.
The good thing is that other than devscripts and sun jdk, all the other
packages can be installed silently.
And for JDK, OpenSDK works great and can be installed in silence.
For devscripts, there is a bug outstanding with Unbuntu:
https://bugs.launchpad.net/ubuntu/+source/devscripts/+bug/612566
The other annoying thing about devscripts is that it hangs if you install
it in a window with less than 43 lines, as the curse-based interface
doesn't fit. We can make this easier by opening an xterm with the
specified window size, I guess, it could be a problem with pbuild if we
try to create the 'super-package'.
So, for now, we can try just make the devscripts install the manual first
step or the last step until the bug is addressed, and automate the most we
can with least manual steps.
Another way is to create custom Amazon AMIs with all the dependencies we
need.
Regards,
David
I tried a bit with automating the whole thing with some response file.
On 3/23/12 1:41 AM, "James Page" <ja...@canonical.com> wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA256
>
>On 23/03/12 04:45, Roman Shaposhnik wrote:
>> Great stuff! Although it would be really nice to package all the
>> dependencies we have to install by hand (even if it is just a
>> tarball-in-disguise type of a package)
>>
>> Also, I'm not quite sure whether there's any kind of tool we can
>> use to automatically install all the build-time dependencies.
>>
>> James, what do you guys use for Ubuntu to automatically pull the
>> build-time deps specified in source packages?
>
>This would be quite simple if ALL build-time deps where packaged - but
>they are not :-).
>
>For distro development the source package is typically built first and
>then the binary packages are build using a tool such as sbuild or
>pbuilder (see [0]).
>
>I have experimented with this a bit for bigtop - however the schroots
>need to be configured with appropriate JAVA_HOME, JAVA5_HOME and
>FORREST_HOME and MAVEN_HOME installs and configurations.
>
>sbuild will look at the build-deps for a package and then install them
>before it tries to build the package itself.
>
>There are a few non-schroot options as well - but its less clean in
>the fact that you might have conflicting packages or already have some
>of them installed.
>
>mk-build-deps does a similar function but without schroot's - it will
>create a meta package and install it for all of the build-deps for a
>give debian/control file.
>
>I will try to find some time to investigate this in a bit more detail
>- - it should be possible to put together some instructions/s script to
>get this all setup for people to use.
>
>It would also help me as I generally run the latest development
>release - and bigtop currently only builds on 10.04 :-).
>
>Cross posting to -dev.
>
>Cheers
>
>James
>
>[0] https://wiki.ubuntu.com/SecurityTeam/BuildEnvironment
>
>- --
>James Page
>Software Engineer, Ubuntu Server Team
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG v1.4.11 (GNU/Linux)
>Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
>iQIcBAEBCAAGBQJPbEVTAAoJEL/srsug59jDDBwP/ib8MtZvpESgFhfrXfHIBlAY
>i8hQiPMoPOR/07aR9ck6svS/68DJqrkavok10xulrT4PYpT0NopYkCM3FDYrlRiE
>6KSSshRKIEDxUc7Kv3NtGmBFKa8P6A+y1T9b/cq8ootLynYjUB7aaDsNuDOSD9wG
>7TDHmMYCZ1x5W39q+gZQrqiZays8eZBAlHs66ag+nWU4mvtMGnTUNVJJMUiK9+HK
>H1mRjt4fklEDT8Zif2A1nCQ9/L6eU5zQRjkyloa4rH+Tlh3F1ODaHlEB9YSeKqNl
>vmRvlNJkmGVVw0LxdjzvtxqdzrQhHZjo+PmaNRQprem7vGxREpmwb6gW6LEO7UG3
>CWP7U4zrBZfnLNxvbj8jx+XJfrEneRxxob2LsjW+ItmN9MKJHY1mTkaxP75MmsLk
>xTi8VNCzOJj8yLgBtRKKmDk2L0p+Peb3KWU9FNyeK4wuLE7xVqftkHIRUEAFxX10
>I5WszxUXKmZnaWQgcOwBsQIxmy6yZwnPiZ1w6WKuzHcLvUC/V6KHI2JucBUBOHMU
>rHfxYoBshHIYu52nmk/6CK77vsZxguRcKWU+EjaVWl2dDIdxyXiLkHVcqe1Jirh0
>btwOaA09EqOFYY4klHwXP+bipjrRkbXTbBXCDlc4jBT1Qq8a3yKiq3Ijvm9UkdrD
>k2WYuJMbV2I3vnEHHmWL
>=0jqT
>-----END PGP SIGNATURE-----
Re: bigtop ubuntu build wiki page added
Posted by "David Liu (gmail dev)" <em...@gmail.com>.
While I put together the wiki page, I was trying to make all the steps
automated with the least human intervention, and noting places where human
involvement is needed.
The good thing is that other than devscripts and sun jdk, all the other
packages can be installed silently.
And for JDK, OpenSDK works great and can be installed in silence.
For devscripts, there is a bug outstanding with Unbuntu:
https://bugs.launchpad.net/ubuntu/+source/devscripts/+bug/612566
The other annoying thing about devscripts is that it hangs if you install
it in a window with less than 43 lines, as the curse-based interface
doesn't fit. We can make this easier by opening an xterm with the
specified window size, I guess, it could be a problem with pbuild if we
try to create the 'super-package'.
So, for now, we can try just make the devscripts install the manual first
step or the last step until the bug is addressed, and automate the most we
can with least manual steps.
Another way is to create custom Amazon AMIs with all the dependencies we
need.
Regards,
David
I tried a bit with automating the whole thing with some response file.
On 3/23/12 1:41 AM, "James Page" <ja...@canonical.com> wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA256
>
>On 23/03/12 04:45, Roman Shaposhnik wrote:
>> Great stuff! Although it would be really nice to package all the
>> dependencies we have to install by hand (even if it is just a
>> tarball-in-disguise type of a package)
>>
>> Also, I'm not quite sure whether there's any kind of tool we can
>> use to automatically install all the build-time dependencies.
>>
>> James, what do you guys use for Ubuntu to automatically pull the
>> build-time deps specified in source packages?
>
>This would be quite simple if ALL build-time deps where packaged - but
>they are not :-).
>
>For distro development the source package is typically built first and
>then the binary packages are build using a tool such as sbuild or
>pbuilder (see [0]).
>
>I have experimented with this a bit for bigtop - however the schroots
>need to be configured with appropriate JAVA_HOME, JAVA5_HOME and
>FORREST_HOME and MAVEN_HOME installs and configurations.
>
>sbuild will look at the build-deps for a package and then install them
>before it tries to build the package itself.
>
>There are a few non-schroot options as well - but its less clean in
>the fact that you might have conflicting packages or already have some
>of them installed.
>
>mk-build-deps does a similar function but without schroot's - it will
>create a meta package and install it for all of the build-deps for a
>give debian/control file.
>
>I will try to find some time to investigate this in a bit more detail
>- - it should be possible to put together some instructions/s script to
>get this all setup for people to use.
>
>It would also help me as I generally run the latest development
>release - and bigtop currently only builds on 10.04 :-).
>
>Cross posting to -dev.
>
>Cheers
>
>James
>
>[0] https://wiki.ubuntu.com/SecurityTeam/BuildEnvironment
>
>- --
>James Page
>Software Engineer, Ubuntu Server Team
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG v1.4.11 (GNU/Linux)
>Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
>iQIcBAEBCAAGBQJPbEVTAAoJEL/srsug59jDDBwP/ib8MtZvpESgFhfrXfHIBlAY
>i8hQiPMoPOR/07aR9ck6svS/68DJqrkavok10xulrT4PYpT0NopYkCM3FDYrlRiE
>6KSSshRKIEDxUc7Kv3NtGmBFKa8P6A+y1T9b/cq8ootLynYjUB7aaDsNuDOSD9wG
>7TDHmMYCZ1x5W39q+gZQrqiZays8eZBAlHs66ag+nWU4mvtMGnTUNVJJMUiK9+HK
>H1mRjt4fklEDT8Zif2A1nCQ9/L6eU5zQRjkyloa4rH+Tlh3F1ODaHlEB9YSeKqNl
>vmRvlNJkmGVVw0LxdjzvtxqdzrQhHZjo+PmaNRQprem7vGxREpmwb6gW6LEO7UG3
>CWP7U4zrBZfnLNxvbj8jx+XJfrEneRxxob2LsjW+ItmN9MKJHY1mTkaxP75MmsLk
>xTi8VNCzOJj8yLgBtRKKmDk2L0p+Peb3KWU9FNyeK4wuLE7xVqftkHIRUEAFxX10
>I5WszxUXKmZnaWQgcOwBsQIxmy6yZwnPiZ1w6WKuzHcLvUC/V6KHI2JucBUBOHMU
>rHfxYoBshHIYu52nmk/6CK77vsZxguRcKWU+EjaVWl2dDIdxyXiLkHVcqe1Jirh0
>btwOaA09EqOFYY4klHwXP+bipjrRkbXTbBXCDlc4jBT1Qq8a3yKiq3Ijvm9UkdrD
>k2WYuJMbV2I3vnEHHmWL
>=0jqT
>-----END PGP SIGNATURE-----
Re: bigtop ubuntu build wiki page added
Posted by James Page <ja...@canonical.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 23/03/12 04:45, Roman Shaposhnik wrote:
> Great stuff! Although it would be really nice to package all the
> dependencies we have to install by hand (even if it is just a
> tarball-in-disguise type of a package)
>
> Also, I'm not quite sure whether there's any kind of tool we can
> use to automatically install all the build-time dependencies.
>
> James, what do you guys use for Ubuntu to automatically pull the
> build-time deps specified in source packages?
This would be quite simple if ALL build-time deps where packaged - but
they are not :-).
For distro development the source package is typically built first and
then the binary packages are build using a tool such as sbuild or
pbuilder (see [0]).
I have experimented with this a bit for bigtop - however the schroots
need to be configured with appropriate JAVA_HOME, JAVA5_HOME and
FORREST_HOME and MAVEN_HOME installs and configurations.
sbuild will look at the build-deps for a package and then install them
before it tries to build the package itself.
There are a few non-schroot options as well - but its less clean in
the fact that you might have conflicting packages or already have some
of them installed.
mk-build-deps does a similar function but without schroot's - it will
create a meta package and install it for all of the build-deps for a
give debian/control file.
I will try to find some time to investigate this in a bit more detail
- - it should be possible to put together some instructions/s script to
get this all setup for people to use.
It would also help me as I generally run the latest development
release - and bigtop currently only builds on 10.04 :-).
Cross posting to -dev.
Cheers
James
[0] https://wiki.ubuntu.com/SecurityTeam/BuildEnvironment
- --
James Page
Software Engineer, Ubuntu Server Team
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQIcBAEBCAAGBQJPbEVTAAoJEL/srsug59jDDBwP/ib8MtZvpESgFhfrXfHIBlAY
i8hQiPMoPOR/07aR9ck6svS/68DJqrkavok10xulrT4PYpT0NopYkCM3FDYrlRiE
6KSSshRKIEDxUc7Kv3NtGmBFKa8P6A+y1T9b/cq8ootLynYjUB7aaDsNuDOSD9wG
7TDHmMYCZ1x5W39q+gZQrqiZays8eZBAlHs66ag+nWU4mvtMGnTUNVJJMUiK9+HK
H1mRjt4fklEDT8Zif2A1nCQ9/L6eU5zQRjkyloa4rH+Tlh3F1ODaHlEB9YSeKqNl
vmRvlNJkmGVVw0LxdjzvtxqdzrQhHZjo+PmaNRQprem7vGxREpmwb6gW6LEO7UG3
CWP7U4zrBZfnLNxvbj8jx+XJfrEneRxxob2LsjW+ItmN9MKJHY1mTkaxP75MmsLk
xTi8VNCzOJj8yLgBtRKKmDk2L0p+Peb3KWU9FNyeK4wuLE7xVqftkHIRUEAFxX10
I5WszxUXKmZnaWQgcOwBsQIxmy6yZwnPiZ1w6WKuzHcLvUC/V6KHI2JucBUBOHMU
rHfxYoBshHIYu52nmk/6CK77vsZxguRcKWU+EjaVWl2dDIdxyXiLkHVcqe1Jirh0
btwOaA09EqOFYY4klHwXP+bipjrRkbXTbBXCDlc4jBT1Qq8a3yKiq3Ijvm9UkdrD
k2WYuJMbV2I3vnEHHmWL
=0jqT
-----END PGP SIGNATURE-----
Re: bigtop ubuntu build wiki page added
Posted by James Page <ja...@canonical.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 23/03/12 04:45, Roman Shaposhnik wrote:
> Great stuff! Although it would be really nice to package all the
> dependencies we have to install by hand (even if it is just a
> tarball-in-disguise type of a package)
>
> Also, I'm not quite sure whether there's any kind of tool we can
> use to automatically install all the build-time dependencies.
>
> James, what do you guys use for Ubuntu to automatically pull the
> build-time deps specified in source packages?
This would be quite simple if ALL build-time deps where packaged - but
they are not :-).
For distro development the source package is typically built first and
then the binary packages are build using a tool such as sbuild or
pbuilder (see [0]).
I have experimented with this a bit for bigtop - however the schroots
need to be configured with appropriate JAVA_HOME, JAVA5_HOME and
FORREST_HOME and MAVEN_HOME installs and configurations.
sbuild will look at the build-deps for a package and then install them
before it tries to build the package itself.
There are a few non-schroot options as well - but its less clean in
the fact that you might have conflicting packages or already have some
of them installed.
mk-build-deps does a similar function but without schroot's - it will
create a meta package and install it for all of the build-deps for a
give debian/control file.
I will try to find some time to investigate this in a bit more detail
- - it should be possible to put together some instructions/s script to
get this all setup for people to use.
It would also help me as I generally run the latest development
release - and bigtop currently only builds on 10.04 :-).
Cross posting to -dev.
Cheers
James
[0] https://wiki.ubuntu.com/SecurityTeam/BuildEnvironment
- --
James Page
Software Engineer, Ubuntu Server Team
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQIcBAEBCAAGBQJPbEVTAAoJEL/srsug59jDDBwP/ib8MtZvpESgFhfrXfHIBlAY
i8hQiPMoPOR/07aR9ck6svS/68DJqrkavok10xulrT4PYpT0NopYkCM3FDYrlRiE
6KSSshRKIEDxUc7Kv3NtGmBFKa8P6A+y1T9b/cq8ootLynYjUB7aaDsNuDOSD9wG
7TDHmMYCZ1x5W39q+gZQrqiZays8eZBAlHs66ag+nWU4mvtMGnTUNVJJMUiK9+HK
H1mRjt4fklEDT8Zif2A1nCQ9/L6eU5zQRjkyloa4rH+Tlh3F1ODaHlEB9YSeKqNl
vmRvlNJkmGVVw0LxdjzvtxqdzrQhHZjo+PmaNRQprem7vGxREpmwb6gW6LEO7UG3
CWP7U4zrBZfnLNxvbj8jx+XJfrEneRxxob2LsjW+ItmN9MKJHY1mTkaxP75MmsLk
xTi8VNCzOJj8yLgBtRKKmDk2L0p+Peb3KWU9FNyeK4wuLE7xVqftkHIRUEAFxX10
I5WszxUXKmZnaWQgcOwBsQIxmy6yZwnPiZ1w6WKuzHcLvUC/V6KHI2JucBUBOHMU
rHfxYoBshHIYu52nmk/6CK77vsZxguRcKWU+EjaVWl2dDIdxyXiLkHVcqe1Jirh0
btwOaA09EqOFYY4klHwXP+bipjrRkbXTbBXCDlc4jBT1Qq8a3yKiq3Ijvm9UkdrD
k2WYuJMbV2I3vnEHHmWL
=0jqT
-----END PGP SIGNATURE-----
Re: bigtop ubuntu build wiki page added
Posted by Roman Shaposhnik <rv...@apache.org>.
On Tue, Mar 20, 2012 at 11:03 AM, David Liu (gmail dev)
<em...@gmail.com> wrote:
> I have added a build page for
> Ubuntu: https://cwiki.apache.org/confluence/display/BIGTOP/Building+Bigtop+on+Ubuntu
>
> Hope it is useful for other users.
Great stuff! Although it would be really nice to package all the dependencies
we have to install by hand (even if it is just a tarball-in-disguise
type of a package)
Also, I'm not quite sure whether there's any kind of tool we can use
to automatically
install all the build-time dependencies.
James, what do you guys use for Ubuntu to automatically pull the build-time deps
specified in source packages?
Thanks,
Roman.
Re: bigtop ubuntu build wiki page added
Posted by James Page <ja...@canonical.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 23/03/12 04:47, Roman Shaposhnik wrote:
>> Hi James,
>>>
>>> The use of /usr/lib/jvm/default-java works as long as the
>>> JAVA_HOME is set. Only tried on Unbuntu though. I also tried
>>> OpenJDK 6 in addition to Sun JDK 6. Installing OpenJDK 6 is
>>> much easier (no need to wget/tar/change defaults). I will
>>> probably file a bug in Hbase, as its jvm detection code
>>> doesn't include /usr/lib/jvm/default-java, along with many
>>> others that bigtop's detection did as Bruno pointed out
>>> earlier.
>>>
>>> I tried /etc/init.d/hbase-master start with JAVA_HOME set in
>>> /etc/default/bigtop-utils. It doesn't seem to work. I added env
>>> cmd to the start up script, it cannot find JAVA_HOME (see
>>> below). Maybe I am using the wrong command here to start the
>>> services? Can you point me how startup scripts make use of
>>> /etc/default/bigtop-tuils?
> This is really weird and concerning. Can you try adding set -x
> right on the second line after #!/bin/bash in both scripts
> /etc/init.d/hbase-master and /usr/lib/hbase/bin/hbase and then post
> the output?
I think this is a bug - the hbase init script does not use
bigtop-detect-java like the hadoop one does.
It needs a:
# Autodetect JAVA_HOME if not defined
if [ -e /usr/libexec/bigtop-detect-javahome ]; then
. /usr/libexec/bigtop-detect-javahome
elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then
. /usr/lib/bigtop-utils/bigtop-detect-javahome
fi
Block upfront.
- --
James Page
Software Engineer, Ubuntu Server Team
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQIcBAEBCAAGBQJPbENdAAoJEL/srsug59jD4FkP/2PN5DR6wIWFyQa4kBDmp/+z
9eIQUXiH5JhL45HLwqenH6gQSlbVUtAvTpkIXOyPONYDYIlgaPNNzvMOrz/CCzQb
Ri2TMC8RvpbcX4AXoex6TF75EBoflxrRgL+5rDABiZ4Lmfey1+Fml63xjRNomaR1
uzOZaHc1wfM6ghWKygqqKvm0kOO1meF52ES8+8Gb+BHSyYfVluGEwWaQ97mxsHCt
nSMU/5rm12MgjQx1Spfzw3OkbCk6RuTRtJHYam+4Z/6LQiUfFgHw1XPwm6rMhFGA
oN2j45gP0QTq5h/JWD//i6d6OPUf+UeyG30WtdncGBTKgyNg7myPHI4bzvSAckx8
83nzltxqCiYSCetMuGUXEWKG0qRjUpINZOnhAWnQliVh3j8slvMFDNEnbnLOnBWV
tJ4jXA1K9P9mFb777tClVrC4MNmqLTX9txYTiFkC+W1M/EuqZJLs+cG6Su8OemyL
8lX+HTzjnpkgU9gwWTxhZtNQS+sz0Om0hKL0Wkm0/awLMxleUohTPem62+C3Ahnh
Ei8AlGRqQV04bg5N+q2T3OBQxp2DlHaoHaBmNKCpZVcXZT70YhCs1+9oaIOh0R5f
RPD1g3OC340zUAd1zmQ8A5VBoh1M51RaUbfrEem5WZI/dXCCTzHt0s7kcStxwrZC
s6ILc327zEzm2jOrJpOx
=63NL
-----END PGP SIGNATURE-----
Re: bigtop ubuntu build wiki page added
Posted by "David Liu (gmail dev)" <em...@gmail.com>.
ubuntu@ubuntu:/etc/default$ sudo /etc/init.d/hbase-master start
+ set -e
+ . /etc/default/hadoop
++ export HADOOP_HOME_WARN_SUPPRESS=true
++ HADOOP_HOME_WARN_SUPPRESS=true
++ export HADOOP_HOME=/usr/lib/hadoop
++ HADOOP_HOME=/usr/lib/hadoop
++ export HADOOP_NAMENODE_USER=hdfs
++ HADOOP_NAMENODE_USER=hdfs
++ export HADOOP_SECONDARYNAMENODE_USER=hdfs
++ HADOOP_SECONDARYNAMENODE_USER=hdfs
++ export HADOOP_DATANODE_USER=hdfs
++ HADOOP_DATANODE_USER=hdfs
++ export HADOOP_JOBTRACKER_USER=mapred
++ HADOOP_JOBTRACKER_USER=mapred
++ export HADOOP_TASKTRACKER_USER=mapred
++ HADOOP_TASKTRACKER_USER=mapred
++ export HADOOP_IDENT_STRING=hadoop
++ HADOOP_IDENT_STRING=hadoop
++ export HADOOP_PID_DIR=/var/run/hadoop
++ HADOOP_PID_DIR=/var/run/hadoop
++ export HADOOP_LOG_DIR=/var/log/hadoop
++ HADOOP_LOG_DIR=/var/log/hadoop
+ export HBASE_HOME=/usr/lib/hbase
+ HBASE_HOME=/usr/lib/hbase
+ export HBASE_PID_DIR=/var/run/hbase
+ HBASE_PID_DIR=/var/run/hbase
+ '[' -f /etc/default/hbase ']'
+ . /etc/default/hbase
++ export HBASE_PID_DIR=/var/run/hbase
++ HBASE_PID_DIR=/var/run/hbase
++ export HBASE_LOG_DIR=/var/log/hbase
++ HBASE_LOG_DIR=/var/log/hbase
++ export HBASE_IDENT_STRING=hbase
++ HBASE_IDENT_STRING=hbase
+ '[' -f /usr/lib/hbase/bin/hbase-config.sh ']'
+ . /usr/lib/hbase/bin/hbase-config.sh
++ env <<<<<<---------------- added to check current env
HADOOP_LOG_DIR=/var/log/hadoop
HADOOP_IDENT_STRING=hadoop
SHELL=/bin/bash
TERM=xterm
HADOOP_HOME=/usr/lib/hadoop
HBASE_LOG_DIR=/var/log/hbase
HADOOP_DATANODE_USER=hdfs
HADOOP_PID_DIR=/var/run/hadoop
USER=root
HBASE_HOME=/usr/lib/hbase
SUDO_USER=ubuntu
SUDO_UID=1001
USERNAME=root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R
6/bin
PWD=/etc/default
LANG=en_US.UTF-8
HADOOP_NAMENODE_USER=hdfs
HBASE_IDENT_STRING=hbase
HADOOP_JOBTRACKER_USER=mapred
SHLVL=1
SUDO_COMMAND=/etc/init.d/hbase-master start
HOME=/home/ubuntu
HADOOP_SECONDARYNAMENODE_USER=hdfs
LOGNAME=root
HADOOP_HOME_WARN_SUPPRESS=true
SUDO_GID=1001
DISPLAY=:0.0
HBASE_PID_DIR=/var/run/hbase
HADOOP_TASKTRACKER_USER=mapred
COLORTERM=gnome-terminal
XAUTHORITY=/var/run/gdm/auth-for-ubuntu-XKYG8D/database
_=/usr/bin/env
++ this=/usr/lib/hbase/bin/hbase-config.sh
++ '[' -h /usr/lib/hbase/bin/hbase-config.sh ']'
+++ dirname /usr/lib/hbase/bin/hbase-config.sh
++ bin=/usr/lib/hbase/bin
+++ basename /usr/lib/hbase/bin/hbase-config.sh
++ script=hbase-config.sh
+++ cd /usr/lib/hbase/bin
+++ pwd
++ bin=/usr/lib/hbase/bin
++ this=/usr/lib/hbase/bin/hbase-config.sh
++ '[' -z /usr/lib/hbase ']'
++ '[' 1 -gt 1 ']'
++ HBASE_CONF_DIR=/usr/lib/hbase/conf
++ HBASE_REGIONSERVERS=/usr/lib/hbase/conf/regionservers
++ HBASE_BACKUP_MASTERS=/usr/lib/hbase/conf/backup-masters
++ '[' -f /usr/lib/hbase/conf/hbase-env.sh ']'
++ . /usr/lib/hbase/conf/hbase-env.sh
+++ export HBASE_OPTS=-XX:+UseConcMarkSweepGC
+++ HBASE_OPTS=-XX:+UseConcMarkSweepGC
++ '[' -z '' ']'
++ for candidate in /usr/lib/jvm/java-6-sun
'/usr/lib/jvm/java-1.6.0-sun-1.6.0.*/jre'
'/usr/lib/jvm/java-1.6.0-sun-1.6.0.*' /usr/lib/j2sdk1.6-sun
'/usr/java/jdk1.6*' '/usr/java/jre1.6*' /Library/Java/Home
++ '[' -e /usr/lib/jvm/java-6-sun/bin/java ']'
++ for candidate in /usr/lib/jvm/java-6-sun
'/usr/lib/jvm/java-1.6.0-sun-1.6.0.*/jre'
'/usr/lib/jvm/java-1.6.0-sun-1.6.0.*' /usr/lib/j2sdk1.6-sun
'/usr/java/jdk1.6*' '/usr/java/jre1.6*' /Library/Java/Home
++ '[' -e '/usr/lib/jvm/java-1.6.0-sun-1.6.0.*/jre/bin/java' ']'
++ for candidate in /usr/lib/jvm/java-6-sun
'/usr/lib/jvm/java-1.6.0-sun-1.6.0.*/jre'
'/usr/lib/jvm/java-1.6.0-sun-1.6.0.*' /usr/lib/j2sdk1.6-sun
'/usr/java/jdk1.6*' '/usr/java/jre1.6*' /Library/Java/Home
++ '[' -e '/usr/lib/jvm/java-1.6.0-sun-1.6.0.*/bin/java' ']'
++ for candidate in /usr/lib/jvm/java-6-sun
'/usr/lib/jvm/java-1.6.0-sun-1.6.0.*/jre'
'/usr/lib/jvm/java-1.6.0-sun-1.6.0.*' /usr/lib/j2sdk1.6-sun
'/usr/java/jdk1.6*' '/usr/java/jre1.6*' /Library/Java/Home
++ '[' -e /usr/lib/j2sdk1.6-sun/bin/java ']'
++ for candidate in /usr/lib/jvm/java-6-sun
'/usr/lib/jvm/java-1.6.0-sun-1.6.0.*/jre'
'/usr/lib/jvm/java-1.6.0-sun-1.6.0.*' /usr/lib/j2sdk1.6-sun
'/usr/java/jdk1.6*' '/usr/java/jre1.6*' /Library/Java/Home
++ '[' -e '/usr/java/jdk1.6*/bin/java' ']'
++ for candidate in /usr/lib/jvm/java-6-sun
'/usr/lib/jvm/java-1.6.0-sun-1.6.0.*/jre'
'/usr/lib/jvm/java-1.6.0-sun-1.6.0.*' /usr/lib/j2sdk1.6-sun
'/usr/java/jdk1.6*' '/usr/java/jre1.6*' /Library/Java/Home
++ '[' -e '/usr/java/jre1.6*/bin/java' ']'
++ for candidate in /usr/lib/jvm/java-6-sun
'/usr/lib/jvm/java-1.6.0-sun-1.6.0.*/jre'
'/usr/lib/jvm/java-1.6.0-sun-1.6.0.*' /usr/lib/j2sdk1.6-sun
'/usr/java/jdk1.6*' '/usr/java/jre1.6*' /Library/Java/Home
++ '[' -e /Library/Java/Home/bin/java ']'
++ '[' -z '' ']'
++ cat
+======================================================================+
| Error: JAVA_HOME is not set and Java could not be found |
+----------------------------------------------------------------------+
| Please download the latest Sun JDK from the Sun Java web site |
| > http://java.sun.com/javase/downloads/ < |
| |
| HBase requires Java 1.6 or later. |
| NOTE: This script will find Sun Java whether you install using the |
| binary or the RPM based installer. |
+======================================================================+
++ exit 1
ubuntu@ubuntu:/etc/default$
On 3/22/12 8:47 PM, "Roman Shaposhnik" <rv...@apache.org> wrote:
>This is really weird and concerning. Can you try adding
> set -x
>right on the second line after #!/bin/bash in both scripts
>/etc/init.d/hbase-master and /usr/lib/hbase/bin/hbase
>and then post the output?
>
Re: bigtop ubuntu build wiki page added
Posted by Roman Shaposhnik <rv...@apache.org>.
On Thu, Mar 22, 2012 at 12:32 PM, David Liu (gmail dev)
<em...@gmail.com> wrote:
> Hi James,
>
> The use of /usr/lib/jvm/default-java works as long as the JAVA_HOME is
> set. Only tried on Unbuntu though.
> I also tried OpenJDK 6 in addition to Sun JDK 6. Installing OpenJDK 6 is
> much easier (no need to wget/tar/change defaults).
> I will probably file a bug in Hbase, as its jvm detection code doesn't
> include /usr/lib/jvm/default-java, along with many others that bigtop's
> detection did as Bruno pointed out earlier.
>
> I tried /etc/init.d/hbase-master start with JAVA_HOME set in
> /etc/default/bigtop-utils. It doesn't seem to work. I added env cmd to the
> start up script, it cannot find JAVA_HOME (see below). Maybe I am using
> the wrong command here to start the services? Can you point me how startup
> scripts make use of /etc/default/bigtop-tuils?
This is really weird and concerning. Can you try adding
set -x
right on the second line after #!/bin/bash in both scripts
/etc/init.d/hbase-master and /usr/lib/hbase/bin/hbase
and then post the output?
Thanks,
Roman.
Re: bigtop ubuntu build wiki page added
Posted by "David Liu (gmail dev)" <em...@gmail.com>.
Hi James,
The use of /usr/lib/jvm/default-java works as long as the JAVA_HOME is
set. Only tried on Unbuntu though.
I also tried OpenJDK 6 in addition to Sun JDK 6. Installing OpenJDK 6 is
much easier (no need to wget/tar/change defaults).
I will probably file a bug in Hbase, as its jvm detection code doesn't
include /usr/lib/jvm/default-java, along with many others that bigtop's
detection did as Bruno pointed out earlier.
I tried /etc/init.d/hbase-master start with JAVA_HOME set in
/etc/default/bigtop-utils. It doesn't seem to work. I added env cmd to the
start up script, it cannot find JAVA_HOME (see below). Maybe I am using
the wrong command here to start the services? Can you point me how startup
scripts make use of /etc/default/bigtop-tuils?
Thanks,
David
ubuntu@ubuntu:~$ sudo /etc/init.d/hbase-master start
HADOOP_LOG_DIR=/var/log/hadoop
HADOOP_IDENT_STRING=hadoop
SHELL=/bin/bash
HADOOP_HOME=/usr/lib/hadoop
HBASE_LOG_DIR=/var/log/hbase
HADOOP_DATANODE_USER=hdfs
HADOOP_PID_DIR=/var/run/hadoop
USER=root
HBASE_HOME=/usr/lib/hbase
SUDO_USER=ubuntu
SUDO_UID=1001
USERNAME=root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R
6/bin
PWD=/home/ubuntu
HADOOP_NAMENODE_USER=hdfs
HBASE_IDENT_STRING=hbase
HADOOP_JOBTRACKER_USER=mapred
SHLVL=1
SUDO_COMMAND=/etc/init.d/hbase-master start
HOME=/home/ubuntu
HADOOP_SECONDARYNAMENODE_USER=hdfs
LOGNAME=root
HADOOP_HOME_WARN_SUPPRESS=true
SUDO_GID=1001
HBASE_PID_DIR=/var/run/hbase
HADOOP_TASKTRACKER_USER=mapred
_=/usr/bin/env
+======================================================================+
| Error: JAVA_HOME is not set and Java could not be found |
+----------------------------------------------------------------------+
| Please download the latest Sun JDK from the Sun Java web site |
| > http://java.sun.com/javase/downloads/ < |
| |
| HBase requires Java 1.6 or later. |
| NOTE: This script will find Sun Java whether you install using the |
| binary or the RPM based installer. |
+======================================================================+
ubuntu@ubuntu:~$
On 3/22/12 8:40 AM, "James Page" <ja...@canonical.com> wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA256
>
>Hi David
>
>On 21/03/12 07:17, David Liu (gmail dev) wrote:
>> I observed inconsistency of the configuration file used between
>> hadoop and hbase. Seems that when starting hadoop service using:
>> /etc/init.d/hadoop-xxx start /etc/hadoop/conf/hadoop-env.sh is
>> used.
>>
>> When starting hbase service /etc/init.d/hbase-xxx start
>> /usr/lib/hbase/conf/hbase-env.sh is used.
>>
>> This leads to an issue during deb install: when JAVA is not
>> installed at the directory that is one of on the detection list,
>> even if you set your JAVA_HOME in /etc/hbase/conf/hbase-env.sh, it
>> will still fail. You can try to change the
>> /usr/lib/hbase/conf/hbase-env.sh file. But it is overwritten to
>> what comes with the deb file during deb installation.
>>
>> Would appreciate if someone can confirm it is a bug or not. If so,
>> we probably should track it in JIRA for fix in hbase or in bigtop.
>Can you try setting JAVA_HOME in /etc/default/bigtop-utils? That
>should override the automated detection of Java in all bigtop packages
>and allow you to install Java in a different location.
>
>I was also really interested to see that you used
>/usr/lib/jvm/default-java:
>
>export JAVA_HOME="/usr/lib/jvm/default-java"
>export JAVA5_HOME="/usr/lib/jvm/default-java"
>
>Does this work? I have to admit that I have never actually tried that...
>
>If so that is great as it reduces the dependencies require to build on
>Ubuntu...
>
>Once bigtop switches to supporting the next LTS (Ubuntu Precise) we
>also have maven 3 packaged which then only leaves forrest as an
>unpackaged first level build dependency.
>
>Cheers
>
>James
>
>- -- James Page Ubuntu Core Developer
>
>
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG v1.4.11 (GNU/Linux)
>Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
>iQIcBAEBCAAGBQJPa1XtAAoJEL/srsug59jDNGsP/2FXFRAwOq2QxVOFqD3wSyk8
>mfh24xLRkFVwQswTTws24iYXu1nuviMkjgE9KAg2QNHgVLoWeOn57TNBy/0FzcCH
>FwBYqAINriaFP7gLP6iKaME/N2e6v1vFJ3xMV0c2szcK8ugpKZ/C5wPLWyDXlBjK
>JgTRlmKeGLkfoDF79rQrlk2p6aAHUrATsFJLwQz3Uxy3M50pT8v1KW3zB+2ntWVX
>w4qk9ARwAVZnk9Km8CSgkYpwhsoDyzx7Bnnf4NPxrgdmYp+TjJPiGlI7nDi0e/IQ
>2LQzvu2a2hjNKOEsQrD/TVsnuHU7S5Y2G9nZR9DU5nC232OQf+hbrcAc3EhR6mFv
>oRtfCLlhADQzaE6QVfrN+KjufL9As1emv7CJ1lJ3Ig5PyUdjyIciB0QOlDBiTI2g
>gFyG/qV84U9/PBBwHn2muCyHafdO54zwQW1h0HK0COwupQkE4LFSw2nYg4uGgEV7
>oEiC1nbdVfAMeCxGujxCTLZWIv9430jTioUlp12ZEeDlkVzLEADGxDMJNVL3zVcX
>S8c4K6huyABCGc3d7zEFjHTSk8e2N8uuask4Go+xG5Od6HOxahO3wnuxoTTaRB48
>eksDSZWBoizRF2GVMhS1D0FbRzihiY9syUFJ1pFUfpPHe8BtKkDlVFTnVr63JA3w
>0/Tx681GHlwlx0miXBbo
>=y0Vm
>-----END PGP SIGNATURE-----
Re: bigtop ubuntu build wiki page added
Posted by James Page <ja...@canonical.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi David
On 21/03/12 07:17, David Liu (gmail dev) wrote:
> I observed inconsistency of the configuration file used between
> hadoop and hbase. Seems that when starting hadoop service using:
> /etc/init.d/hadoop-xxx start /etc/hadoop/conf/hadoop-env.sh is
> used.
>
> When starting hbase service /etc/init.d/hbase-xxx start
> /usr/lib/hbase/conf/hbase-env.sh is used.
>
> This leads to an issue during deb install: when JAVA is not
> installed at the directory that is one of on the detection list,
> even if you set your JAVA_HOME in /etc/hbase/conf/hbase-env.sh, it
> will still fail. You can try to change the
> /usr/lib/hbase/conf/hbase-env.sh file. But it is overwritten to
> what comes with the deb file during deb installation.
>
> Would appreciate if someone can confirm it is a bug or not. If so,
> we probably should track it in JIRA for fix in hbase or in bigtop.
Can you try setting JAVA_HOME in /etc/default/bigtop-utils? That
should override the automated detection of Java in all bigtop packages
and allow you to install Java in a different location.
I was also really interested to see that you used
/usr/lib/jvm/default-java:
export JAVA_HOME="/usr/lib/jvm/default-java"
export JAVA5_HOME="/usr/lib/jvm/default-java"
Does this work? I have to admit that I have never actually tried that...
If so that is great as it reduces the dependencies require to build on
Ubuntu...
Once bigtop switches to supporting the next LTS (Ubuntu Precise) we
also have maven 3 packaged which then only leaves forrest as an
unpackaged first level build dependency.
Cheers
James
- -- James Page Ubuntu Core Developer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQIcBAEBCAAGBQJPa1XtAAoJEL/srsug59jDNGsP/2FXFRAwOq2QxVOFqD3wSyk8
mfh24xLRkFVwQswTTws24iYXu1nuviMkjgE9KAg2QNHgVLoWeOn57TNBy/0FzcCH
FwBYqAINriaFP7gLP6iKaME/N2e6v1vFJ3xMV0c2szcK8ugpKZ/C5wPLWyDXlBjK
JgTRlmKeGLkfoDF79rQrlk2p6aAHUrATsFJLwQz3Uxy3M50pT8v1KW3zB+2ntWVX
w4qk9ARwAVZnk9Km8CSgkYpwhsoDyzx7Bnnf4NPxrgdmYp+TjJPiGlI7nDi0e/IQ
2LQzvu2a2hjNKOEsQrD/TVsnuHU7S5Y2G9nZR9DU5nC232OQf+hbrcAc3EhR6mFv
oRtfCLlhADQzaE6QVfrN+KjufL9As1emv7CJ1lJ3Ig5PyUdjyIciB0QOlDBiTI2g
gFyG/qV84U9/PBBwHn2muCyHafdO54zwQW1h0HK0COwupQkE4LFSw2nYg4uGgEV7
oEiC1nbdVfAMeCxGujxCTLZWIv9430jTioUlp12ZEeDlkVzLEADGxDMJNVL3zVcX
S8c4K6huyABCGc3d7zEFjHTSk8e2N8uuask4Go+xG5Od6HOxahO3wnuxoTTaRB48
eksDSZWBoizRF2GVMhS1D0FbRzihiY9syUFJ1pFUfpPHe8BtKkDlVFTnVr63JA3w
0/Tx681GHlwlx0miXBbo
=y0Vm
-----END PGP SIGNATURE-----
Re: bigtop ubuntu build wiki page added
Posted by "David Liu (gmail dev)" <em...@gmail.com>.
I observed inconsistency of the configuration file used between hadoop and
hbase. Seems that when starting hadoop service using:
/etc/init.d/hadoop-xxx start
/etc/hadoop/conf/hadoop-env.sh is used.
When starting hbase service
/etc/init.d/hbase-xxx start
/usr/lib/hbase/conf/hbase-env.sh is used.
This leads to an issue during deb install: when JAVA is not installed at
the directory that is one of on the detection list, even if you set your
JAVA_HOME in /etc/hbase/conf/hbase-env.sh, it will still fail. You can try
to change the /usr/lib/hbase/conf/hbase-env.sh file. But it is overwritten
to what comes with the deb file during deb installation.
Would appreciate if someone can confirm it is a bug or not. If so, we
probably should track it in JIRA for fix in hbase or in bigtop.
Also, seems the list of java location used by hbase below
(/usr/lib/hbase/bin/hbase-conf.sh) is quite a bit shorter than the bigtop
detection list:
/usr/lib/jvm/java-6-sun \
/usr/lib/jvm/java-1.6.0-sun-1.6.0.*/jre \
/usr/lib/jvm/java-1.6.0-sun-1.6.0.* \
/usr/lib/j2sdk1.6-sun \
/usr/java/jdk1.6* \
/usr/java/jre1.6* \
/Library/Java/Home
Thanks,
David
On 3/20/12 2:09 PM, "Bruno Mahé" <bm...@apache.org> wrote:
>On 03/20/2012 03:05 PM, Bruno Mahé wrote:
>> On 03/20/2012 11:03 AM, David Liu (gmail dev) wrote:
>>> I have added a build page for
>>> Ubuntu:
>>>https://cwiki.apache.org/confluence/display/BIGTOP/Building+Bigtop+on+Ub
>>>untu
>>>
>>> Hope it is useful for other users.
>>>
>>> David
>> Great page!
>>
>> One note though:
>> When installing the packages, you shouldn't need to manually set
>> JAVA_HOME, unless you put it in a non-standard location.
>> All wrapper scripts will make use of bigtop-utils to auto-detect
>>JAVA_HOME.
>> See
>>
>>https://github.com/apache/bigtop/blob/trunk/bigtop-packages/src/common/bi
>>gtop-utils/bigtop-detect-javahome
>>
>>
>> Thanks,
>> Bruno
>
>Sorry, I forgot to add:
>
>* sudo hadoop namenode -format => you must format the namenode as the
>user hdfs. So you should type "sudo -u hdfs hadoop namenode -format"
>
>* /usr/lib/hadoop/bin/st{art,op}-all.sh scripts are not supported and
>will be removed soon (as in as soon as someone tackles it). Services
>should be started/stopped through the init scripts located in
>/etc/init.d/hadoop-*
Re: bigtop ubuntu build wiki page added
Posted by "David Liu (gmail dev)" <em...@gmail.com>.
Thanks Bruno for your feedbacks. I will update based on your suggestions.
For JAVA_HOME, I extracted my jdk to /usr/lib/jvm/jdk1.6.0_31, and it
happens not to be in your list of detection list:
/usr/lib/jvm/java-6-sun \
/usr/lib/jvm/java-1.6.0-sun-1.6.0.*/jre/ \
/usr/lib/jvm/java-1.6.0-sun-1.6.0.* \
/usr/lib/j2sdk1.6-sun \
/usr/java/jdk1.6* \
/usr/java/jre1.6* \
/Library/Java/Home \
/usr/java/default \
/usr/lib/jvm/default-java \
/usr/lib/jvm/java-openjdk \
/usr/lib/jvm/jre-openjdk \
/usr/lib/jvm/java-1.6.0-openjdk \
/usr/lib/jvm/java-1.6.0-openjdk-* \
/usr/lib/jvm/jre-1.6.0-openjdk*
I am not sure what is the best way in terms of documenting a fool-proof
steps. But for now I will say either one add the JAVA_HOME or keep your
java in a known list of directory (such as on Mac /Library/Java/Home or
Linux /usr/lib/jvm/default-java).
Thanks,
David
On 3/20/12 2:09 PM, "Bruno Mahé" <bm...@apache.org> wrote:
>On 03/20/2012 03:05 PM, Bruno Mahé wrote:
>> On 03/20/2012 11:03 AM, David Liu (gmail dev) wrote:
>>> I have added a build page for
>>> Ubuntu:
>>>https://cwiki.apache.org/confluence/display/BIGTOP/Building+Bigtop+on+Ub
>>>untu
>>>
>>> Hope it is useful for other users.
>>>
>>> David
>> Great page!
>>
>> One note though:
>> When installing the packages, you shouldn't need to manually set
>> JAVA_HOME, unless you put it in a non-standard location.
>> All wrapper scripts will make use of bigtop-utils to auto-detect
>>JAVA_HOME.
>> See
>>
>>https://github.com/apache/bigtop/blob/trunk/bigtop-packages/src/common/bi
>>gtop-utils/bigtop-detect-javahome
>>
>>
>> Thanks,
>> Bruno
>
>Sorry, I forgot to add:
>
>* sudo hadoop namenode -format => you must format the namenode as the
>user hdfs. So you should type "sudo -u hdfs hadoop namenode -format"
>
>* /usr/lib/hadoop/bin/st{art,op}-all.sh scripts are not supported and
>will be removed soon (as in as soon as someone tackles it). Services
>should be started/stopped through the init scripts located in
>/etc/init.d/hadoop-*
Re: bigtop ubuntu build wiki page added
Posted by Bruno Mahé <bm...@apache.org>.
On 03/20/2012 03:05 PM, Bruno Mahé wrote:
> On 03/20/2012 11:03 AM, David Liu (gmail dev) wrote:
>> I have added a build page for
>> Ubuntu: https://cwiki.apache.org/confluence/display/BIGTOP/Building+Bigtop+on+Ubuntu
>>
>> Hope it is useful for other users.
>>
>> David
> Great page!
>
> One note though:
> When installing the packages, you shouldn't need to manually set
> JAVA_HOME, unless you put it in a non-standard location.
> All wrapper scripts will make use of bigtop-utils to auto-detect JAVA_HOME.
> See
> https://github.com/apache/bigtop/blob/trunk/bigtop-packages/src/common/bigtop-utils/bigtop-detect-javahome
>
>
> Thanks,
> Bruno
Sorry, I forgot to add:
* sudo hadoop namenode -format => you must format the namenode as the
user hdfs. So you should type "sudo -u hdfs hadoop namenode -format"
* /usr/lib/hadoop/bin/st{art,op}-all.sh scripts are not supported and
will be removed soon (as in as soon as someone tackles it). Services
should be started/stopped through the init scripts located in
/etc/init.d/hadoop-*
Re: bigtop ubuntu build wiki page added
Posted by Bruno Mahé <bm...@apache.org>.
On 03/20/2012 11:03 AM, David Liu (gmail dev) wrote:
> I have added a build page for
> Ubuntu: https://cwiki.apache.org/confluence/display/BIGTOP/Building+Bigtop+on+Ubuntu
>
> Hope it is useful for other users.
>
> David
Great page!
One note though:
When installing the packages, you shouldn't need to manually set
JAVA_HOME, unless you put it in a non-standard location.
All wrapper scripts will make use of bigtop-utils to auto-detect JAVA_HOME.
See
https://github.com/apache/bigtop/blob/trunk/bigtop-packages/src/common/bigtop-utils/bigtop-detect-javahome
Thanks,
Bruno