You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Jinyan Xu <Ji...@exar.com> on 2011/12/08 02:39:03 UTC
how to integrate snappy into hadoop
Hi ,
Anyone else have the experience integrating snappy into hadoop ? help me with it....
I find google doesn't provide the hadoop-snappy now :
"Hadoop-snappy is integrated into Hadoop Common(JUN 2011).
Hadoop-Snappy can be used as an add-on for recent (released) versions of Hadoop that do not provide Snappy Codec support yet.
Hadoop-Snappy is being kept in synch with Hadoop Common. "
Thanks!
________________________________
The information and any attached documents contained in this message
may be confidential and/or legally privileged. The message is
intended solely for the addressee(s). If you are not the intended
recipient, you are hereby notified that any use, dissemination, or
reproduction is strictly prohibited and may be unlawful. If you are
not the intended recipient, please contact the sender immediately by
return e-mail and destroy all copies of the original message.
Re: how to integrate snappy into hadoop
Posted by Prashant Kommireddi <pr...@gmail.com>.
I have not tried it with HBase, and yes 0.20.2 is not compatible with it.
What is the error you receive when you try compiling Snappy? I don't think
compiling Snappy would be dependent on HBase.
2011/12/7 Jinyan Xu <Ji...@exar.com>
> Hi Prashant Kommireddi,
>
> Last week, I read build-hadoop-from-source and follow it, but I failed to
> compile hbase with "mvn compile -Dsnappy ". Did you install HBase 0.90.2?
> According build-hadoop-from-source HBase 0.90.2 is incompatible with
> Hadoop0.20.2-release.
>
>
> -----Original Message-----
> From: Prashant Kommireddi [mailto:prash1784@gmail.com]
> Sent: 2011年12月8日 10:13
> To: common-user@hadoop.apache.org
> Subject: Re: how to integrate snappy into hadoop
>
> I had to struggle a bit while building Snappy for Hadoop 0.20.2 on Ubuntu.
> However, I have now been able to install it on a 10 node cluster and it
> works great for map output compression. Please check these notes, may be it
> might help in addition to official Hadoop-Snappy notes.
>
> Tom White (amongst a few others) is the best person to answer this since he
> is working directly with this integration project.
>
> *Installing Maven 3*
> http://www.discursive.com/blog/4636
>
> *Snappy build*
> http://code.google.com/p/hadoop-snappy/
>
> *Creating the symbolic link for BUILD to pass:* THIS IS IMPORTANT
> sudo ln -s
>
> /home/pkommireddi/dev/tools/Linux/jdk/jdk1.6.0_21_x64/jre/lib/amd64/server/libjvm.so
> /usr/local/lib/
>
>
> *Additional build notes:*
> http://shanky.org/2011/10/17/build-hadoop-from-source/
>
> Install commands I issued on localhost, these were notes for myself. There
> are dependencies that Snappy build needs, and you might NOT need all of the
> commands that I have issues below. Please refer to official notes and
> install accordingly.
>
> pkommireddi@pkommireddi-wsl:~$ history | grep install
> 1678 sudo apt-get install python-software-properties
> 1681 sudo apt-get install maven
> 1750 make install
> 1751 ./configure && make && sudo make install
> 1752 sudo apt-get install zlibc zlib1g zlib1g-dev
> 1763 sudo apt-get install subversion
> 1776 ./configure && make && sudo make install
> 1824 sudo apt-get install zlibc zlib1g zlib1g-dev
> 1846 make install
> 1847 sudo make install
> 1858 make installcheck
> 1861 sudo make install
> 1862 sudo make installcheck
> 1904 sudo apt-get install libtool
> 1907 sudo apt-get install automake
>
> Thanks,
> Prashant Kommireddi
>
> On Wed, Dec 7, 2011 at 5:39 PM, Jinyan Xu <Ji...@exar.com> wrote:
>
> > Hi ,
> >
> >
> > Anyone else have the experience integrating snappy into hadoop ? help me
> > with it....
> >
> > I find google doesn't provide the hadoop-snappy now :
> >
> > "Hadoop-snappy is integrated into Hadoop Common(JUN 2011).
> >
> > Hadoop-Snappy can be used as an add-on for recent (released) versions of
> > Hadoop that do not provide Snappy Codec support yet.
> >
> > Hadoop-Snappy is being kept in synch with Hadoop Common. "
> >
> >
> >
> >
> >
> > Thanks!
> >
> >
> > ________________________________
> > The information and any attached documents contained in this message
> > may be confidential and/or legally privileged. The message is
> > intended solely for the addressee(s). If you are not the intended
> > recipient, you are hereby notified that any use, dissemination, or
> > reproduction is strictly prohibited and may be unlawful. If you are
> > not the intended recipient, please contact the sender immediately by
> > return e-mail and destroy all copies of the original message.
> >
>
> The information and any attached documents contained in this message
> may be confidential and/or legally privileged. The message is
> intended solely for the addressee(s). If you are not the intended
> recipient, you are hereby notified that any use, dissemination, or
> reproduction is strictly prohibited and may be unlawful. If you are
> not the intended recipient, please contact the sender immediately by
> return e-mail and destroy all copies of the original message.
>
RE: how to integrate snappy into hadoop
Posted by Jinyan Xu <Ji...@exar.com>.
Hi Prashant Kommireddi,
Last week, I read build-hadoop-from-source and follow it, but I failed to compile hbase with "mvn compile -Dsnappy ". Did you install HBase 0.90.2?
According build-hadoop-from-source HBase 0.90.2 is incompatible with Hadoop0.20.2-release.
-----Original Message-----
From: Prashant Kommireddi [mailto:prash1784@gmail.com]
Sent: 2011年12月8日 10:13
To: common-user@hadoop.apache.org
Subject: Re: how to integrate snappy into hadoop
I had to struggle a bit while building Snappy for Hadoop 0.20.2 on Ubuntu.
However, I have now been able to install it on a 10 node cluster and it
works great for map output compression. Please check these notes, may be it
might help in addition to official Hadoop-Snappy notes.
Tom White (amongst a few others) is the best person to answer this since he
is working directly with this integration project.
*Installing Maven 3*
http://www.discursive.com/blog/4636
*Snappy build*
http://code.google.com/p/hadoop-snappy/
*Creating the symbolic link for BUILD to pass:* THIS IS IMPORTANT
sudo ln -s
/home/pkommireddi/dev/tools/Linux/jdk/jdk1.6.0_21_x64/jre/lib/amd64/server/libjvm.so
/usr/local/lib/
*Additional build notes:*
http://shanky.org/2011/10/17/build-hadoop-from-source/
Install commands I issued on localhost, these were notes for myself. There
are dependencies that Snappy build needs, and you might NOT need all of the
commands that I have issues below. Please refer to official notes and
install accordingly.
pkommireddi@pkommireddi-wsl:~$ history | grep install
1678 sudo apt-get install python-software-properties
1681 sudo apt-get install maven
1750 make install
1751 ./configure && make && sudo make install
1752 sudo apt-get install zlibc zlib1g zlib1g-dev
1763 sudo apt-get install subversion
1776 ./configure && make && sudo make install
1824 sudo apt-get install zlibc zlib1g zlib1g-dev
1846 make install
1847 sudo make install
1858 make installcheck
1861 sudo make install
1862 sudo make installcheck
1904 sudo apt-get install libtool
1907 sudo apt-get install automake
Thanks,
Prashant Kommireddi
On Wed, Dec 7, 2011 at 5:39 PM, Jinyan Xu <Ji...@exar.com> wrote:
> Hi ,
>
>
> Anyone else have the experience integrating snappy into hadoop ? help me
> with it....
>
> I find google doesn't provide the hadoop-snappy now :
>
> "Hadoop-snappy is integrated into Hadoop Common(JUN 2011).
>
> Hadoop-Snappy can be used as an add-on for recent (released) versions of
> Hadoop that do not provide Snappy Codec support yet.
>
> Hadoop-Snappy is being kept in synch with Hadoop Common. "
>
>
>
>
>
> Thanks!
>
>
> ________________________________
> The information and any attached documents contained in this message
> may be confidential and/or legally privileged. The message is
> intended solely for the addressee(s). If you are not the intended
> recipient, you are hereby notified that any use, dissemination, or
> reproduction is strictly prohibited and may be unlawful. If you are
> not the intended recipient, please contact the sender immediately by
> return e-mail and destroy all copies of the original message.
>
The information and any attached documents contained in this message
may be confidential and/or legally privileged. The message is
intended solely for the addressee(s). If you are not the intended
recipient, you are hereby notified that any use, dissemination, or
reproduction is strictly prohibited and may be unlawful. If you are
not the intended recipient, please contact the sender immediately by
return e-mail and destroy all copies of the original message.
Re: how to integrate snappy into hadoop
Posted by Prashant Kommireddi <pr...@gmail.com>.
I had to struggle a bit while building Snappy for Hadoop 0.20.2 on Ubuntu.
However, I have now been able to install it on a 10 node cluster and it
works great for map output compression. Please check these notes, may be it
might help in addition to official Hadoop-Snappy notes.
Tom White (amongst a few others) is the best person to answer this since he
is working directly with this integration project.
*Installing Maven 3*
http://www.discursive.com/blog/4636
*Snappy build*
http://code.google.com/p/hadoop-snappy/
*Creating the symbolic link for BUILD to pass:* THIS IS IMPORTANT
sudo ln -s
/home/pkommireddi/dev/tools/Linux/jdk/jdk1.6.0_21_x64/jre/lib/amd64/server/libjvm.so
/usr/local/lib/
*Additional build notes:*
http://shanky.org/2011/10/17/build-hadoop-from-source/
Install commands I issued on localhost, these were notes for myself. There
are dependencies that Snappy build needs, and you might NOT need all of the
commands that I have issues below. Please refer to official notes and
install accordingly.
pkommireddi@pkommireddi-wsl:~$ history | grep install
1678 sudo apt-get install python-software-properties
1681 sudo apt-get install maven
1750 make install
1751 ./configure && make && sudo make install
1752 sudo apt-get install zlibc zlib1g zlib1g-dev
1763 sudo apt-get install subversion
1776 ./configure && make && sudo make install
1824 sudo apt-get install zlibc zlib1g zlib1g-dev
1846 make install
1847 sudo make install
1858 make installcheck
1861 sudo make install
1862 sudo make installcheck
1904 sudo apt-get install libtool
1907 sudo apt-get install automake
Thanks,
Prashant Kommireddi
On Wed, Dec 7, 2011 at 5:39 PM, Jinyan Xu <Ji...@exar.com> wrote:
> Hi ,
>
>
> Anyone else have the experience integrating snappy into hadoop ? help me
> with it....
>
> I find google doesn't provide the hadoop-snappy now :
>
> "Hadoop-snappy is integrated into Hadoop Common(JUN 2011).
>
> Hadoop-Snappy can be used as an add-on for recent (released) versions of
> Hadoop that do not provide Snappy Codec support yet.
>
> Hadoop-Snappy is being kept in synch with Hadoop Common. "
>
>
>
>
>
> Thanks!
>
>
> ________________________________
> The information and any attached documents contained in this message
> may be confidential and/or legally privileged. The message is
> intended solely for the addressee(s). If you are not the intended
> recipient, you are hereby notified that any use, dissemination, or
> reproduction is strictly prohibited and may be unlawful. If you are
> not the intended recipient, please contact the sender immediately by
> return e-mail and destroy all copies of the original message.
>