You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ted Yu <yu...@gmail.com> on 2010/01/07 23:17:25 UTC
isSplitable() deprecated
According to:
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
isSplitable() is deprecated.
Which method should I use to replace it ?
Thanks
Re: isSplitable() deprecated
Posted by Edward Capriolo <ed...@gmail.com>.
When you are seeing errors early in the configure phase it is possible
that you do not have a complete development environment. gcc alone is
not enough.
Give this a whirl:
yum groupinstall 'Development Tools'
This is going to install many things but at least you wont be chasing
down dependencies 1 by 1.
Edward
On Fri, Jan 15, 2010 at 4:17 PM, Todd Lipcon <to...@cloudera.com> wrote:
> Are you starting from a clean tarball of the lzo stuff?
>
> Can you make sure your /tmp/ partition isn't full?
>
> There should be a config.log file hanging around somewhere also - perhaps
> this would be helpful.
>
> -Todd
>
> On Fri, Jan 15, 2010 at 1:14 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Todd:
>> Thanks for the continued support.
>>
>> I installed lzo-devel:
>> [root@tyu-linux batchclient]# rpm -ivh
>> /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm
>> warning: /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm: Header V3 DSA
>> signature: NOKEY, key ID 217521f6
>> Preparing... ###########################################
>> [100%]
>> 1:lzo-devel ###########################################
>> [100%]
>>
>> But I still see:
>> [exec] Try `/usr/bin/install --help' for more information.
>> [exec] checking for gcc... gcc
>> [exec] checking for C compiler default output file name...
>> [exec] configure: error: C compiler cannot create executables
>>
>>
>> On Fri, Jan 15, 2010 at 12:09 PM, Todd Lipcon <to...@cloudera.com> wrote:
>>
>> > Hi Ted,
>> >
>> > Did you also install liblzo-devel?
>> >
>> > Here are the packages I install for LZO:
>> >
>> > lzo-2.02-2.el5.1
>> > lzo-devel-2.02-2.el5.1
>> >
>> > -Todd
>> >
>> > On Tue, Jan 12, 2010 at 4:28 PM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> > > Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and latest
>> version
>> > > Nothing to do
>> > >
>> > > [root@tyu-linux batchclient]# yum install gcc-c++
>> > > Loaded plugins: fastestmirror
>> > > Loading mirror speeds from cached hostfile
>> > > * base: mirrors.kernel.org
>> > > * updates: updates.interworx.info
>> > > * addons: mirrors.kernel.org
>> > > * extras: mirrors.netdna.com
>> > > Setting up Install Process
>> > > Parsing package install arguments
>> > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and latest
>> > > version
>> > >
>> > >
>> > > On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <to...@cloudera.com>
>> wrote:
>> > >
>> > > > Make sure you have gcc, g++ installed?
>> > > >
>> > > > -Todd
>> > > >
>> > > > On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <yu...@gmail.com>
>> wrote:
>> > > >
>> > > > > I installed
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
>> > > > > .
>> > > > >
>> > > > > [root@tyu-linux software]# rpm -e lzo
>> > > > > [root@tyu-linux software]# rpm -ivh
>> > > ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
>> > > > > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header V3 DSA
>> > > > signature:
>> > > > > NOKEY, key ID 217521f6
>> > > > > Preparing...
>> > ###########################################
>> > > > > [100%]
>> > > > > 1:lzo
>> > ###########################################
>> > > > > [100%]
>> > > > >
>> > > > > [rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
>> > > > > lrwxrwxrwx 1 root root 16 Jan 12 10:29 /usr/lib64/liblzo2.so.2
>> ->
>> > > > > liblzo2.so.2.0.0
>> > > > > -rwxr-xr-x 1 root root 123408 May 6 2007
>> > /usr/lib64/liblzo2.so.2.0.0
>> > > > >
>> > > > > I still see this error:
>> > > > > [exec] configure: error: C compiler cannot create executables
>> > > > > [exec] See `config.log' for more details.
>> > > > >
>> > > > > BUILD FAILED
>> > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
>> > returned:
>> > > > 77
>> > > > >
>> > > > >
>> > > > > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <to...@cloudera.com>
>> > > wrote:
>> > > > >
>> > > > > > Hi Ted,
>> > > > > >
>> > > > > > Please make sure you have version 2.02 of liblzo installed.
>> There's
>> > > an
>> > > > > > earlier version in RHEL that doesn't fail properly at the
>> configure
>> > > > stage
>> > > > > > and I've seen it produce the error you pasted below.
>> > > > > >
>> > > > > > -Todd
>> > > > > >
>> > > > > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <yu...@gmail.com>
>> > > wrote:
>> > > > > >
>> > > > > > > I followed
>> > > http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
>> > > > > > >
>> > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and
>> > > latest
>> > > > > > > version
>> > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37
>> EDT
>> > > > 2009
>> > > > > > > x86_64 x86_64 x86_64 GNU/Linux
>> > > > > > >
>> > > > > > > For 32 bit build, I got:
>> > > > > > > [exec] checking for C compiler default output file name...
>> > > > > > > [exec] configure: error: C compiler cannot create
>> executables
>> > > > > > > [exec] See `config.log' for more details.
>> > > > > > >
>> > > > > > > BUILD FAILED
>> > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
>> > > > returned:
>> > > > > > 77
>> > > > > > >
>> > > > > > > For 64 bit build, I got:
>> > > > > > > [exec] gcc -DHAVE_CONFIG_H -I.
>> > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native -I./impl
>> > > > > > > -I/usr/java/default/include -I/usr/java/default/include/linux
>> > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
>> > > > > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -m64
>> -MT
>> > > > > > > impl/lzo/LzoCompressor.lo -MD -MP -MF
>> > > > impl/lzo/.deps/LzoCompressor.Tpo
>> > > > > -c
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
>> > > > > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
>> > > > > > > [exec]
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
>> > > > > > > In function
>> > > 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
>> > > > > > > [exec]
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
>> > > > > > > error: expected expression before ',' token
>> > > > > > > [exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1
>> > > > > > >
>> > > > > > > BUILD FAILED
>> > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251: exec
>> > > > returned:
>> > > > > 2
>> > > > > > >
>> > > > > > > Has anybody seen the above ?
>> > > > > > >
>> > > > > > > Thanks
>> > > > > > >
>> > > > > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <
>> todd@cloudera.com>
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Ted,
>> > > > > > > >
>> > > > > > > > You need to install liblzo from EPEL:
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
>> > > > > > > >
>> > > > > > > > -Todd
>> > > > > > > >
>> > > > > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <yuzhihong@gmail.com
>> >
>> > > > wrote:
>> > > > > > > >
>> > > > > > > > > Can someone tell me how I can install liblzo ?
>> > > > > > > > >
>> > > > > > > > > [root@tyu-linux lzo-2.03]# uname -a
>> > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14
>> 06:36:37
>> > > EDT
>> > > > > > 2009
>> > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
>> > > > > > > > > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
>> > > > > > > > > Loaded plugins: fastestmirror
>> > > > > > > > > Loading mirror speeds from cached hostfile
>> > > > > > > > > * base: mirrors.usc.edu
>> > > > > > > > > * updates: mirror.san.fastserv.com
>> > > > > > > > > * addons: centos.promopeddler.com
>> > > > > > > > > * extras: mirrors.versaweb.com
>> > > > > > > > > Setting up Install Process
>> > > > > > > > > Parsing package install arguments
>> > > > > > > > > No package liblzo-devel available.
>> > > > > > > > > Nothing to do
>> > > > > > > > >
>> > > > > > > > > Thanks
>> > > > > > > > >
>> > > > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <
>> > > kuosenhao@gmail.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Ted,
>> > > > > > > > > >
>> > > > > > > > > > You may want to consider LZO compression, which allows
>> > > > splitting
>> > > > > > for
>> > > > > > > a
>> > > > > > > > > > comporessed file for Map jobs. On the other hand, gzip
>> is
>> > > not
>> > > > > > > > > splittable.
>> > > > > > > > > >
>> > > > > > > > > > Check out these links.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
>> > > > > > > > > > http://wiki.apache.org/hadoop/UsingLzoCompression
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <
>> > yuzhihong@gmail.com>
>> > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > The input file is in .gz format
>> > > > > > > > > > > FYI
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <
>> > > yuzhihong@gmail.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > My current project processes input file of size
>> > 333302161
>> > > > > > bytes.
>> > > > > > > > > > > > What I plan to do is to split the file into equal
>> size
>> > > > pieces
>> > > > > > > (and
>> > > > > > > > on
>> > > > > > > > > > > blank
>> > > > > > > > > > > > line boundary) to improve performance.
>> > > > > > > > > > > >
>> > > > > > > > > > > > I found 12 classes in 0.20.1 source code which
>> > implement
>> > > > > > > > InputSplit.
>> > > > > > > > > > > >
>> > > > > > > > > > > > If someone has written code similar to what I plan to
>> > do,
>> > > > > > please
>> > > > > > > > > share
>> > > > > > > > > > > some
>> > > > > > > > > > > > hint.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <
>> > > > > > > amogh@yahoo-inc.com
>> > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > >> Hi,
>> > > > > > > > > > > >> The deprecation is due to the new evolving mapreduce
>> (
>> > > > > > > > > o.a.h.mapreduce
>> > > > > > > > > > )
>> > > > > > > > > > > >> APIs. Old APIs are supported for available
>> > > distributions.
>> > > > > The
>> > > > > > > > > > equivalent
>> > > > > > > > > > > of
>> > > > > > > > > > > >> TextInputFormat is available in new API :
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Thanks,
>> > > > > > > > > > > >> Amogh
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com>
>> > > wrote:
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> According to:
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> isSplitable() is deprecated.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Which method should I use to replace it ?
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Thanks
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: isSplitable() deprecated
Posted by Kevin Weil <ke...@gmail.com>.
Ted, you can use lzop 'myfile.txt' to produce 'myfile.txt.lzo'. The -d
option will decompress. You can also write data with the LzopOutputStream.
Kevin
On Fri, Jan 15, 2010 at 3:05 PM, Ted Yu <yu...@gmail.com> wrote:
> I was able to compile in 64 bit mode.
>
> One minor thing:
> I noticed that small files may get bigger LZ-compressed:
> [rialto@tyu-linux kevinweil-hadoop-lzo-916aeae]$ ls -l src/test/data/
> total 940
> -rw-rw-r-- 1 rialto rialto 588895 Jan 11 06:58 100000.txt
> -rw-rw-r-- 1 rialto rialto 346982 Jan 11 06:58 100000.txt.lzo
> -rw-rw-r-- 1 rialto rialto 292 Jan 11 06:58 100.txt
> -rw-rw-r-- 1 rialto rialto 353 Jan 11 06:58 100.txt.lzo
>
> BTW which Linux command do I use to produce .lzo file ?
>
> Thanks
>
> On Fri, Jan 15, 2010 at 2:33 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > But, I bet if you try the same with -m32, it won't work. You need the
> > 32-bit
> > toolchain if you want to compile the 32-bit libraries. I don't know
> exactly
> > how to go about setting this up. But you're better off with 64-bit JDK
> > anyway these days. So, switch your JAVA_HOME to point to a 64-bit JDK and
> > try again.
> >
> > -Todd
> >
> > On Fri, Jan 15, 2010 at 2:16 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > [rialto@tyu-linux kevinweil-hadoop-lzo-916aeae]$ ./a.out
> > > Hello World!
> > > [rialto@tyu-linux kevinweil-hadoop-lzo-916aeae]$ gcc -v
> > > Using built-in specs.
> > > Target: x86_64-redhat-linux
> > > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> > > --infodir=/usr/share/info --enable-shared --enable-threads=posix
> > > --enable-checking=release --with-system-zlib --enable-__cxa_atexit
> > > --disable-libunwind-exceptions --enable-libgcj-multifile
> > > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
> > > --enable-java-awt=gtk
> > > --disable-dssi --enable-plugin
> > > --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre
> > --with-cpu=generic
> > > --host=x86_64-redhat-linux
> > > Thread model: posix
> > > gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
> > >
> > >
> > > On Fri, Jan 15, 2010 at 1:47 PM, Alex Kozlov <al...@cloudera.com>
> > wrote:
> > >
> > > > Looks like your compiler is misconfigured. Can you try and compile a
> > > > simple
> > > > "Hello World!" program:
> > > >
> > > > [alexvk@monster01 ~]$ cat Hello.c
> > > > #include <stdio.h>
> > > > int main() {
> > > > printf("Hello World!\n");
> > > > }
> > > > [alexvk@monster01 ~]$ gcc Hello.c
> > > > [alexvk@monster01 ~]$ ./a.out
> > > > Hello World!
> > > > [alexvk@monster01 ~]$ gcc -v
> > > > Using built-in specs.
> > > > Target: x86_64-redhat-linux
> > > > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> > > > --infodir=/usr/share/info --enable-shared --enable-threads=posix
> > > > --enable-checking=release --with-system-zlib --enable-__cxa_atexit
> > > > --disable-libunwind-exceptions --enable-libgcj-multifile
> > > > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
> > > > --enable-java-awt=gtk
> > > > --disable-dssi --enable-plugin
> > > > --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre
> > > --with-cpu=generic
> > > > --host=x86_64-redhat-linux
> > > > Thread model: posix
> > > > gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
> > > >
> > > > Alex
> > > >
> > > > On Fri, Jan 15, 2010 at 1:34 PM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > I tried to set LD_LIBRARY_PATH to point to where crt1.o resides:
> > > > > export LD_LIBRARY_PATH=/usr/lib64
> > > > >
> > > > > Still the same issue.
> > > > > This is snippet of config.log:
> > > > >
> > > > > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> > > > > --infodir=/usr/share/info --enable-shared --enable-threads=posix
> > > > > --enable-checking=release --with-system-zlib --enable-__cxa_atexit
> > > > > --disable-libunwind-exceptions --enable-libgcj-multifile
> > > > > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
> > > > > --enable-java-awt=gtk
> > > > > --disable-dssi --enable-plugin
> > > > > --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre
> > > > --with-cpu=generic
> > > > > --host=x86_64-redhat-linux
> > > > > Thread model: posix
> > > > > gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
> > > > > configure:2781: $? = 0
> > > > > configure:2788: gcc -V >&5
> > > > > gcc: '-V' option must have argument
> > > > > configure:2791: $? = 1
> > > > > configure:2814: checking for C compiler default output file name
> > > > > configure:2841: gcc -m32 conftest.c >&5
> > > > > /usr/bin/ld: crt1.o: No such file: No such file or directory
> > > > > collect2: ld returned 1 exit status
> > > > > configure:2844: $? = 1
> > > > > configure:2882: result:
> > > > > configure: failed program was:
> > > > > | /* confdefs.h. */
> > > > > | #define PACKAGE_NAME "gpl-compression"
> > > > > | #define PACKAGE_TARNAME "gpl-compression"
> > > > > | #define PACKAGE_VERSION "0.1.0"
> > > > > | #define PACKAGE_STRING "gpl-compression 0.1.0"
> > > > > | #define PACKAGE_BUGREPORT "omalley@apache.org"
> > > > > | #define PACKAGE "gpl-compression"
> > > > > | #define VERSION "0.1.0"
> > > > > | #define _GNU_SOURCE 1
> > > > > | /* end confdefs.h. */
> > > > > |
> > > > > | int
> > > > > | main ()
> > > > > | {
> > > > > |
> > > > > | ;
> > > > > | return 0;
> > > > > | }
> > > > > configure:2889: error: C compiler cannot create executables
> > > > > See `config.log' for more details.
> > > > >
> > > > >
> > > > > On Fri, Jan 15, 2010 at 1:17 PM, Todd Lipcon <to...@cloudera.com>
> > > wrote:
> > > > >
> > > > > > Are you starting from a clean tarball of the lzo stuff?
> > > > > >
> > > > > > Can you make sure your /tmp/ partition isn't full?
> > > > > >
> > > > > > There should be a config.log file hanging around somewhere also -
> > > > perhaps
> > > > > > this would be helpful.
> > > > > >
> > > > > > -Todd
> > > > > >
> > > > > > On Fri, Jan 15, 2010 at 1:14 PM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Todd:
> > > > > > > Thanks for the continued support.
> > > > > > >
> > > > > > > I installed lzo-devel:
> > > > > > > [root@tyu-linux batchclient]# rpm -ivh
> > > > > > > /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm
> > > > > > > warning: /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm:
> Header
> > > V3
> > > > > DSA
> > > > > > > signature: NOKEY, key ID 217521f6
> > > > > > > Preparing...
> > > > ###########################################
> > > > > > > [100%]
> > > > > > > 1:lzo-devel
> > > > ###########################################
> > > > > > > [100%]
> > > > > > >
> > > > > > > But I still see:
> > > > > > > [exec] Try `/usr/bin/install --help' for more information.
> > > > > > > [exec] checking for gcc... gcc
> > > > > > > [exec] checking for C compiler default output file name...
> > > > > > > [exec] configure: error: C compiler cannot create
> executables
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jan 15, 2010 at 12:09 PM, Todd Lipcon <
> todd@cloudera.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Ted,
> > > > > > > >
> > > > > > > > Did you also install liblzo-devel?
> > > > > > > >
> > > > > > > > Here are the packages I install for LZO:
> > > > > > > >
> > > > > > > > lzo-2.02-2.el5.1
> > > > > > > > lzo-devel-2.02-2.el5.1
> > > > > > > >
> > > > > > > > -Todd
> > > > > > > >
> > > > > > > > On Tue, Jan 12, 2010 at 4:28 PM, Ted Yu <yuzhihong@gmail.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and
> > > latest
> > > > > > > version
> > > > > > > > > Nothing to do
> > > > > > > > >
> > > > > > > > > [root@tyu-linux batchclient]# yum install gcc-c++
> > > > > > > > > Loaded plugins: fastestmirror
> > > > > > > > > Loading mirror speeds from cached hostfile
> > > > > > > > > * base: mirrors.kernel.org
> > > > > > > > > * updates: updates.interworx.info
> > > > > > > > > * addons: mirrors.kernel.org
> > > > > > > > > * extras: mirrors.netdna.com
> > > > > > > > > Setting up Install Process
> > > > > > > > > Parsing package install arguments
> > > > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed
> and
> > > > > latest
> > > > > > > > > version
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <
> > > todd@cloudera.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Make sure you have gcc, g++ installed?
> > > > > > > > > >
> > > > > > > > > > -Todd
> > > > > > > > > >
> > > > > > > > > > On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <
> > > yuzhihong@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I installed
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
> > > > > > > > > > > .
> > > > > > > > > > >
> > > > > > > > > > > [root@tyu-linux software]# rpm -e lzo
> > > > > > > > > > > [root@tyu-linux software]# rpm -ivh
> > > > > > > > > ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
> > > > > > > > > > > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm:
> Header
> > > V3
> > > > > DSA
> > > > > > > > > > signature:
> > > > > > > > > > > NOKEY, key ID 217521f6
> > > > > > > > > > > Preparing...
> > > > > > > > ###########################################
> > > > > > > > > > > [100%]
> > > > > > > > > > > 1:lzo
> > > > > > > > ###########################################
> > > > > > > > > > > [100%]
> > > > > > > > > > >
> > > > > > > > > > > [rialto@tyu-linux java]$ ls -l
> /usr/lib64/liblzo2.so.2*
> > > > > > > > > > > lrwxrwxrwx 1 root root 16 Jan 12 10:29
> > > > > > /usr/lib64/liblzo2.so.2
> > > > > > > ->
> > > > > > > > > > > liblzo2.so.2.0.0
> > > > > > > > > > > -rwxr-xr-x 1 root root 123408 May 6 2007
> > > > > > > > /usr/lib64/liblzo2.so.2.0.0
> > > > > > > > > > >
> > > > > > > > > > > I still see this error:
> > > > > > > > > > > [exec] configure: error: C compiler cannot create
> > > > > > executables
> > > > > > > > > > > [exec] See `config.log' for more details.
> > > > > > > > > > >
> > > > > > > > > > > BUILD FAILED
> > > > > > > > > > >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243:
> > > exec
> > > > > > > > returned:
> > > > > > > > > > 77
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <
> > > > > todd@cloudera.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Ted,
> > > > > > > > > > > >
> > > > > > > > > > > > Please make sure you have version 2.02 of liblzo
> > > installed.
> > > > > > > There's
> > > > > > > > > an
> > > > > > > > > > > > earlier version in RHEL that doesn't fail properly at
> > the
> > > > > > > configure
> > > > > > > > > > stage
> > > > > > > > > > > > and I've seen it produce the error you pasted below.
> > > > > > > > > > > >
> > > > > > > > > > > > -Todd
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <
> > > > > yuzhihong@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I followed
> > > > > > > > > http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
> > > > > > > > > > > > >
> > > > > > > > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already
> > > installed
> > > > > and
> > > > > > > > > latest
> > > > > > > > > > > > > version
> > > > > > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul
> 14
> > > > > 06:36:37
> > > > > > > EDT
> > > > > > > > > > 2009
> > > > > > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > > > > > >
> > > > > > > > > > > > > For 32 bit build, I got:
> > > > > > > > > > > > > [exec] checking for C compiler default output
> > file
> > > > > > name...
> > > > > > > > > > > > > [exec] configure: error: C compiler cannot
> create
> > > > > > > executables
> > > > > > > > > > > > > [exec] See `config.log' for more details.
> > > > > > > > > > > > >
> > > > > > > > > > > > > BUILD FAILED
> > > > > > > > > > > > >
> > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243:
> > > > > exec
> > > > > > > > > > returned:
> > > > > > > > > > > > 77
> > > > > > > > > > > > >
> > > > > > > > > > > > > For 64 bit build, I got:
> > > > > > > > > > > > > [exec] gcc -DHAVE_CONFIG_H -I.
> > > > > > > > > > > > >
> > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native
> > > > > > -I./impl
> > > > > > > > > > > > > -I/usr/java/default/include
> > > > > -I/usr/java/default/include/linux
> > > > > > > > > > > > >
> > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> > > > > > > > > > > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2
> > > -m64
> > > > > -m64
> > > > > > > -MT
> > > > > > > > > > > > > impl/lzo/LzoCompressor.lo -MD -MP -MF
> > > > > > > > > > impl/lzo/.deps/LzoCompressor.Tpo
> > > > > > > > > > > -c
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> > > > > > > > > > > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> > > > > > > > > > > > > [exec]
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> > > > > > > > > > > > > In function
> > > > > > > > > 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> > > > > > > > > > > > > [exec]
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> > > > > > > > > > > > > error: expected expression before ',' token
> > > > > > > > > > > > > [exec] make: *** [impl/lzo/LzoCompressor.lo]
> > Error
> > > 1
> > > > > > > > > > > > >
> > > > > > > > > > > > > BUILD FAILED
> > > > > > > > > > > > >
> > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251:
> > > > > exec
> > > > > > > > > > returned:
> > > > > > > > > > > 2
> > > > > > > > > > > > >
> > > > > > > > > > > > > Has anybody seen the above ?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <
> > > > > > > todd@cloudera.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Ted,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You need to install liblzo from EPEL:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -Todd
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <
> > > > > > yuzhihong@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Can someone tell me how I can install liblzo ?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > [root@tyu-linux lzo-2.03]# uname -a
> > > > > > > > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue
> Jul
> > > 14
> > > > > > > 06:36:37
> > > > > > > > > EDT
> > > > > > > > > > > > 2009
> > > > > > > > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > > > > > > > > [root@tyu-linux lzo-2.03]# yum install
> > > liblzo-devel
> > > > > > > > > > > > > > > Loaded plugins: fastestmirror
> > > > > > > > > > > > > > > Loading mirror speeds from cached hostfile
> > > > > > > > > > > > > > > * base: mirrors.usc.edu
> > > > > > > > > > > > > > > * updates: mirror.san.fastserv.com
> > > > > > > > > > > > > > > * addons: centos.promopeddler.com
> > > > > > > > > > > > > > > * extras: mirrors.versaweb.com
> > > > > > > > > > > > > > > Setting up Install Process
> > > > > > > > > > > > > > > Parsing package install arguments
> > > > > > > > > > > > > > > No package liblzo-devel available.
> > > > > > > > > > > > > > > Nothing to do
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <
> > > > > > > > > kuosenhao@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Ted,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > You may want to consider LZO compression,
> which
> > > > > allows
> > > > > > > > > > splitting
> > > > > > > > > > > > for
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > comporessed file for Map jobs. On the other
> > > hand,
> > > > > gzip
> > > > > > > is
> > > > > > > > > not
> > > > > > > > > > > > > > > splittable.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Check out these links.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > > > > > > > > > > > > >
> > > http://wiki.apache.org/hadoop/UsingLzoCompression
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <
> > > > > > > > yuzhihong@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The input file is in .gz format
> > > > > > > > > > > > > > > > > FYI
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <
> > > > > > > > > yuzhihong@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > My current project processes input file
> of
> > > size
> > > > > > > > 333302161
> > > > > > > > > > > > bytes.
> > > > > > > > > > > > > > > > > > What I plan to do is to split the file
> into
> > > > equal
> > > > > > > size
> > > > > > > > > > pieces
> > > > > > > > > > > > > (and
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > blank
> > > > > > > > > > > > > > > > > > line boundary) to improve performance.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I found 12 classes in 0.20.1 source code
> > > which
> > > > > > > > implement
> > > > > > > > > > > > > > InputSplit.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > If someone has written code similar to
> what
> > I
> > > > > plan
> > > > > > to
> > > > > > > > do,
> > > > > > > > > > > > please
> > > > > > > > > > > > > > > share
> > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > hint.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh
> > Vasekar
> > > <
> > > > > > > > > > > > > amogh@yahoo-inc.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > > > > > > >> The deprecation is due to the new
> evolving
> > > > > > mapreduce
> > > > > > > (
> > > > > > > > > > > > > > > o.a.h.mapreduce
> > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > >> APIs. Old APIs are supported for
> available
> > > > > > > > > distributions.
> > > > > > > > > > > The
> > > > > > > > > > > > > > > > equivalent
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > >> TextInputFormat is available in new API
> :
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > >> Amogh
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <
> > > > > yuzhihong@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> According to:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> isSplitable() is deprecated.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Which method should I use to replace it
> ?
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Thanks
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Ted Yu <yu...@gmail.com>.
I was able to compile in 64 bit mode.
One minor thing:
I noticed that small files may get bigger LZ-compressed:
[rialto@tyu-linux kevinweil-hadoop-lzo-916aeae]$ ls -l src/test/data/
total 940
-rw-rw-r-- 1 rialto rialto 588895 Jan 11 06:58 100000.txt
-rw-rw-r-- 1 rialto rialto 346982 Jan 11 06:58 100000.txt.lzo
-rw-rw-r-- 1 rialto rialto 292 Jan 11 06:58 100.txt
-rw-rw-r-- 1 rialto rialto 353 Jan 11 06:58 100.txt.lzo
BTW which Linux command do I use to produce .lzo file ?
Thanks
On Fri, Jan 15, 2010 at 2:33 PM, Todd Lipcon <to...@cloudera.com> wrote:
> But, I bet if you try the same with -m32, it won't work. You need the
> 32-bit
> toolchain if you want to compile the 32-bit libraries. I don't know exactly
> how to go about setting this up. But you're better off with 64-bit JDK
> anyway these days. So, switch your JAVA_HOME to point to a 64-bit JDK and
> try again.
>
> -Todd
>
> On Fri, Jan 15, 2010 at 2:16 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > [rialto@tyu-linux kevinweil-hadoop-lzo-916aeae]$ ./a.out
> > Hello World!
> > [rialto@tyu-linux kevinweil-hadoop-lzo-916aeae]$ gcc -v
> > Using built-in specs.
> > Target: x86_64-redhat-linux
> > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> > --infodir=/usr/share/info --enable-shared --enable-threads=posix
> > --enable-checking=release --with-system-zlib --enable-__cxa_atexit
> > --disable-libunwind-exceptions --enable-libgcj-multifile
> > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
> > --enable-java-awt=gtk
> > --disable-dssi --enable-plugin
> > --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre
> --with-cpu=generic
> > --host=x86_64-redhat-linux
> > Thread model: posix
> > gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
> >
> >
> > On Fri, Jan 15, 2010 at 1:47 PM, Alex Kozlov <al...@cloudera.com>
> wrote:
> >
> > > Looks like your compiler is misconfigured. Can you try and compile a
> > > simple
> > > "Hello World!" program:
> > >
> > > [alexvk@monster01 ~]$ cat Hello.c
> > > #include <stdio.h>
> > > int main() {
> > > printf("Hello World!\n");
> > > }
> > > [alexvk@monster01 ~]$ gcc Hello.c
> > > [alexvk@monster01 ~]$ ./a.out
> > > Hello World!
> > > [alexvk@monster01 ~]$ gcc -v
> > > Using built-in specs.
> > > Target: x86_64-redhat-linux
> > > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> > > --infodir=/usr/share/info --enable-shared --enable-threads=posix
> > > --enable-checking=release --with-system-zlib --enable-__cxa_atexit
> > > --disable-libunwind-exceptions --enable-libgcj-multifile
> > > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
> > > --enable-java-awt=gtk
> > > --disable-dssi --enable-plugin
> > > --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre
> > --with-cpu=generic
> > > --host=x86_64-redhat-linux
> > > Thread model: posix
> > > gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
> > >
> > > Alex
> > >
> > > On Fri, Jan 15, 2010 at 1:34 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > I tried to set LD_LIBRARY_PATH to point to where crt1.o resides:
> > > > export LD_LIBRARY_PATH=/usr/lib64
> > > >
> > > > Still the same issue.
> > > > This is snippet of config.log:
> > > >
> > > > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> > > > --infodir=/usr/share/info --enable-shared --enable-threads=posix
> > > > --enable-checking=release --with-system-zlib --enable-__cxa_atexit
> > > > --disable-libunwind-exceptions --enable-libgcj-multifile
> > > > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
> > > > --enable-java-awt=gtk
> > > > --disable-dssi --enable-plugin
> > > > --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre
> > > --with-cpu=generic
> > > > --host=x86_64-redhat-linux
> > > > Thread model: posix
> > > > gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
> > > > configure:2781: $? = 0
> > > > configure:2788: gcc -V >&5
> > > > gcc: '-V' option must have argument
> > > > configure:2791: $? = 1
> > > > configure:2814: checking for C compiler default output file name
> > > > configure:2841: gcc -m32 conftest.c >&5
> > > > /usr/bin/ld: crt1.o: No such file: No such file or directory
> > > > collect2: ld returned 1 exit status
> > > > configure:2844: $? = 1
> > > > configure:2882: result:
> > > > configure: failed program was:
> > > > | /* confdefs.h. */
> > > > | #define PACKAGE_NAME "gpl-compression"
> > > > | #define PACKAGE_TARNAME "gpl-compression"
> > > > | #define PACKAGE_VERSION "0.1.0"
> > > > | #define PACKAGE_STRING "gpl-compression 0.1.0"
> > > > | #define PACKAGE_BUGREPORT "omalley@apache.org"
> > > > | #define PACKAGE "gpl-compression"
> > > > | #define VERSION "0.1.0"
> > > > | #define _GNU_SOURCE 1
> > > > | /* end confdefs.h. */
> > > > |
> > > > | int
> > > > | main ()
> > > > | {
> > > > |
> > > > | ;
> > > > | return 0;
> > > > | }
> > > > configure:2889: error: C compiler cannot create executables
> > > > See `config.log' for more details.
> > > >
> > > >
> > > > On Fri, Jan 15, 2010 at 1:17 PM, Todd Lipcon <to...@cloudera.com>
> > wrote:
> > > >
> > > > > Are you starting from a clean tarball of the lzo stuff?
> > > > >
> > > > > Can you make sure your /tmp/ partition isn't full?
> > > > >
> > > > > There should be a config.log file hanging around somewhere also -
> > > perhaps
> > > > > this would be helpful.
> > > > >
> > > > > -Todd
> > > > >
> > > > > On Fri, Jan 15, 2010 at 1:14 PM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > > > Todd:
> > > > > > Thanks for the continued support.
> > > > > >
> > > > > > I installed lzo-devel:
> > > > > > [root@tyu-linux batchclient]# rpm -ivh
> > > > > > /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm
> > > > > > warning: /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm: Header
> > V3
> > > > DSA
> > > > > > signature: NOKEY, key ID 217521f6
> > > > > > Preparing...
> > > ###########################################
> > > > > > [100%]
> > > > > > 1:lzo-devel
> > > ###########################################
> > > > > > [100%]
> > > > > >
> > > > > > But I still see:
> > > > > > [exec] Try `/usr/bin/install --help' for more information.
> > > > > > [exec] checking for gcc... gcc
> > > > > > [exec] checking for C compiler default output file name...
> > > > > > [exec] configure: error: C compiler cannot create executables
> > > > > >
> > > > > >
> > > > > > On Fri, Jan 15, 2010 at 12:09 PM, Todd Lipcon <todd@cloudera.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Hi Ted,
> > > > > > >
> > > > > > > Did you also install liblzo-devel?
> > > > > > >
> > > > > > > Here are the packages I install for LZO:
> > > > > > >
> > > > > > > lzo-2.02-2.el5.1
> > > > > > > lzo-devel-2.02-2.el5.1
> > > > > > >
> > > > > > > -Todd
> > > > > > >
> > > > > > > On Tue, Jan 12, 2010 at 4:28 PM, Ted Yu <yu...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and
> > latest
> > > > > > version
> > > > > > > > Nothing to do
> > > > > > > >
> > > > > > > > [root@tyu-linux batchclient]# yum install gcc-c++
> > > > > > > > Loaded plugins: fastestmirror
> > > > > > > > Loading mirror speeds from cached hostfile
> > > > > > > > * base: mirrors.kernel.org
> > > > > > > > * updates: updates.interworx.info
> > > > > > > > * addons: mirrors.kernel.org
> > > > > > > > * extras: mirrors.netdna.com
> > > > > > > > Setting up Install Process
> > > > > > > > Parsing package install arguments
> > > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and
> > > > latest
> > > > > > > > version
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <
> > todd@cloudera.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Make sure you have gcc, g++ installed?
> > > > > > > > >
> > > > > > > > > -Todd
> > > > > > > > >
> > > > > > > > > On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <
> > yuzhihong@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I installed
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
> > > > > > > > > > .
> > > > > > > > > >
> > > > > > > > > > [root@tyu-linux software]# rpm -e lzo
> > > > > > > > > > [root@tyu-linux software]# rpm -ivh
> > > > > > > > ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
> > > > > > > > > > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header
> > V3
> > > > DSA
> > > > > > > > > signature:
> > > > > > > > > > NOKEY, key ID 217521f6
> > > > > > > > > > Preparing...
> > > > > > > ###########################################
> > > > > > > > > > [100%]
> > > > > > > > > > 1:lzo
> > > > > > > ###########################################
> > > > > > > > > > [100%]
> > > > > > > > > >
> > > > > > > > > > [rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
> > > > > > > > > > lrwxrwxrwx 1 root root 16 Jan 12 10:29
> > > > > /usr/lib64/liblzo2.so.2
> > > > > > ->
> > > > > > > > > > liblzo2.so.2.0.0
> > > > > > > > > > -rwxr-xr-x 1 root root 123408 May 6 2007
> > > > > > > /usr/lib64/liblzo2.so.2.0.0
> > > > > > > > > >
> > > > > > > > > > I still see this error:
> > > > > > > > > > [exec] configure: error: C compiler cannot create
> > > > > executables
> > > > > > > > > > [exec] See `config.log' for more details.
> > > > > > > > > >
> > > > > > > > > > BUILD FAILED
> > > > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243:
> > exec
> > > > > > > returned:
> > > > > > > > > 77
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <
> > > > todd@cloudera.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Ted,
> > > > > > > > > > >
> > > > > > > > > > > Please make sure you have version 2.02 of liblzo
> > installed.
> > > > > > There's
> > > > > > > > an
> > > > > > > > > > > earlier version in RHEL that doesn't fail properly at
> the
> > > > > > configure
> > > > > > > > > stage
> > > > > > > > > > > and I've seen it produce the error you pasted below.
> > > > > > > > > > >
> > > > > > > > > > > -Todd
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <
> > > > yuzhihong@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I followed
> > > > > > > > http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
> > > > > > > > > > > >
> > > > > > > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already
> > installed
> > > > and
> > > > > > > > latest
> > > > > > > > > > > > version
> > > > > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14
> > > > 06:36:37
> > > > > > EDT
> > > > > > > > > 2009
> > > > > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > > > > >
> > > > > > > > > > > > For 32 bit build, I got:
> > > > > > > > > > > > [exec] checking for C compiler default output
> file
> > > > > name...
> > > > > > > > > > > > [exec] configure: error: C compiler cannot create
> > > > > > executables
> > > > > > > > > > > > [exec] See `config.log' for more details.
> > > > > > > > > > > >
> > > > > > > > > > > > BUILD FAILED
> > > > > > > > > > > >
> > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243:
> > > > exec
> > > > > > > > > returned:
> > > > > > > > > > > 77
> > > > > > > > > > > >
> > > > > > > > > > > > For 64 bit build, I got:
> > > > > > > > > > > > [exec] gcc -DHAVE_CONFIG_H -I.
> > > > > > > > > > > >
> -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native
> > > > > -I./impl
> > > > > > > > > > > > -I/usr/java/default/include
> > > > -I/usr/java/default/include/linux
> > > > > > > > > > > >
> > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> > > > > > > > > > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2
> > -m64
> > > > -m64
> > > > > > -MT
> > > > > > > > > > > > impl/lzo/LzoCompressor.lo -MD -MP -MF
> > > > > > > > > impl/lzo/.deps/LzoCompressor.Tpo
> > > > > > > > > > -c
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> > > > > > > > > > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> > > > > > > > > > > > [exec]
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> > > > > > > > > > > > In function
> > > > > > > > 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> > > > > > > > > > > > [exec]
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> > > > > > > > > > > > error: expected expression before ',' token
> > > > > > > > > > > > [exec] make: *** [impl/lzo/LzoCompressor.lo]
> Error
> > 1
> > > > > > > > > > > >
> > > > > > > > > > > > BUILD FAILED
> > > > > > > > > > > >
> > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251:
> > > > exec
> > > > > > > > > returned:
> > > > > > > > > > 2
> > > > > > > > > > > >
> > > > > > > > > > > > Has anybody seen the above ?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <
> > > > > > todd@cloudera.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Ted,
> > > > > > > > > > > > >
> > > > > > > > > > > > > You need to install liblzo from EPEL:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> > > > > > > > > > > > >
> > > > > > > > > > > > > -Todd
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <
> > > > > yuzhihong@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Can someone tell me how I can install liblzo ?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [root@tyu-linux lzo-2.03]# uname -a
> > > > > > > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul
> > 14
> > > > > > 06:36:37
> > > > > > > > EDT
> > > > > > > > > > > 2009
> > > > > > > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > > > > > > > [root@tyu-linux lzo-2.03]# yum install
> > liblzo-devel
> > > > > > > > > > > > > > Loaded plugins: fastestmirror
> > > > > > > > > > > > > > Loading mirror speeds from cached hostfile
> > > > > > > > > > > > > > * base: mirrors.usc.edu
> > > > > > > > > > > > > > * updates: mirror.san.fastserv.com
> > > > > > > > > > > > > > * addons: centos.promopeddler.com
> > > > > > > > > > > > > > * extras: mirrors.versaweb.com
> > > > > > > > > > > > > > Setting up Install Process
> > > > > > > > > > > > > > Parsing package install arguments
> > > > > > > > > > > > > > No package liblzo-devel available.
> > > > > > > > > > > > > > Nothing to do
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <
> > > > > > > > kuosenhao@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Ted,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > You may want to consider LZO compression, which
> > > > allows
> > > > > > > > > splitting
> > > > > > > > > > > for
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > comporessed file for Map jobs. On the other
> > hand,
> > > > gzip
> > > > > > is
> > > > > > > > not
> > > > > > > > > > > > > > splittable.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Check out these links.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > > > > > > > > > > > >
> > http://wiki.apache.org/hadoop/UsingLzoCompression
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <
> > > > > > > yuzhihong@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The input file is in .gz format
> > > > > > > > > > > > > > > > FYI
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <
> > > > > > > > yuzhihong@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > My current project processes input file of
> > size
> > > > > > > 333302161
> > > > > > > > > > > bytes.
> > > > > > > > > > > > > > > > > What I plan to do is to split the file into
> > > equal
> > > > > > size
> > > > > > > > > pieces
> > > > > > > > > > > > (and
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > blank
> > > > > > > > > > > > > > > > > line boundary) to improve performance.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I found 12 classes in 0.20.1 source code
> > which
> > > > > > > implement
> > > > > > > > > > > > > InputSplit.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > If someone has written code similar to what
> I
> > > > plan
> > > > > to
> > > > > > > do,
> > > > > > > > > > > please
> > > > > > > > > > > > > > share
> > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > hint.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh
> Vasekar
> > <
> > > > > > > > > > > > amogh@yahoo-inc.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > > > > > >> The deprecation is due to the new evolving
> > > > > mapreduce
> > > > > > (
> > > > > > > > > > > > > > o.a.h.mapreduce
> > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > >> APIs. Old APIs are supported for available
> > > > > > > > distributions.
> > > > > > > > > > The
> > > > > > > > > > > > > > > equivalent
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > >> TextInputFormat is available in new API :
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > >> Amogh
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <
> > > > yuzhihong@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> According to:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> isSplitable() is deprecated.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Which method should I use to replace it ?
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Thanks
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Todd Lipcon <to...@cloudera.com>.
But, I bet if you try the same with -m32, it won't work. You need the 32-bit
toolchain if you want to compile the 32-bit libraries. I don't know exactly
how to go about setting this up. But you're better off with 64-bit JDK
anyway these days. So, switch your JAVA_HOME to point to a 64-bit JDK and
try again.
-Todd
On Fri, Jan 15, 2010 at 2:16 PM, Ted Yu <yu...@gmail.com> wrote:
> [rialto@tyu-linux kevinweil-hadoop-lzo-916aeae]$ ./a.out
> Hello World!
> [rialto@tyu-linux kevinweil-hadoop-lzo-916aeae]$ gcc -v
> Using built-in specs.
> Target: x86_64-redhat-linux
> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> --infodir=/usr/share/info --enable-shared --enable-threads=posix
> --enable-checking=release --with-system-zlib --enable-__cxa_atexit
> --disable-libunwind-exceptions --enable-libgcj-multifile
> --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
> --enable-java-awt=gtk
> --disable-dssi --enable-plugin
> --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic
> --host=x86_64-redhat-linux
> Thread model: posix
> gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
>
>
> On Fri, Jan 15, 2010 at 1:47 PM, Alex Kozlov <al...@cloudera.com> wrote:
>
> > Looks like your compiler is misconfigured. Can you try and compile a
> > simple
> > "Hello World!" program:
> >
> > [alexvk@monster01 ~]$ cat Hello.c
> > #include <stdio.h>
> > int main() {
> > printf("Hello World!\n");
> > }
> > [alexvk@monster01 ~]$ gcc Hello.c
> > [alexvk@monster01 ~]$ ./a.out
> > Hello World!
> > [alexvk@monster01 ~]$ gcc -v
> > Using built-in specs.
> > Target: x86_64-redhat-linux
> > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> > --infodir=/usr/share/info --enable-shared --enable-threads=posix
> > --enable-checking=release --with-system-zlib --enable-__cxa_atexit
> > --disable-libunwind-exceptions --enable-libgcj-multifile
> > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
> > --enable-java-awt=gtk
> > --disable-dssi --enable-plugin
> > --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre
> --with-cpu=generic
> > --host=x86_64-redhat-linux
> > Thread model: posix
> > gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
> >
> > Alex
> >
> > On Fri, Jan 15, 2010 at 1:34 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > I tried to set LD_LIBRARY_PATH to point to where crt1.o resides:
> > > export LD_LIBRARY_PATH=/usr/lib64
> > >
> > > Still the same issue.
> > > This is snippet of config.log:
> > >
> > > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> > > --infodir=/usr/share/info --enable-shared --enable-threads=posix
> > > --enable-checking=release --with-system-zlib --enable-__cxa_atexit
> > > --disable-libunwind-exceptions --enable-libgcj-multifile
> > > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
> > > --enable-java-awt=gtk
> > > --disable-dssi --enable-plugin
> > > --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre
> > --with-cpu=generic
> > > --host=x86_64-redhat-linux
> > > Thread model: posix
> > > gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
> > > configure:2781: $? = 0
> > > configure:2788: gcc -V >&5
> > > gcc: '-V' option must have argument
> > > configure:2791: $? = 1
> > > configure:2814: checking for C compiler default output file name
> > > configure:2841: gcc -m32 conftest.c >&5
> > > /usr/bin/ld: crt1.o: No such file: No such file or directory
> > > collect2: ld returned 1 exit status
> > > configure:2844: $? = 1
> > > configure:2882: result:
> > > configure: failed program was:
> > > | /* confdefs.h. */
> > > | #define PACKAGE_NAME "gpl-compression"
> > > | #define PACKAGE_TARNAME "gpl-compression"
> > > | #define PACKAGE_VERSION "0.1.0"
> > > | #define PACKAGE_STRING "gpl-compression 0.1.0"
> > > | #define PACKAGE_BUGREPORT "omalley@apache.org"
> > > | #define PACKAGE "gpl-compression"
> > > | #define VERSION "0.1.0"
> > > | #define _GNU_SOURCE 1
> > > | /* end confdefs.h. */
> > > |
> > > | int
> > > | main ()
> > > | {
> > > |
> > > | ;
> > > | return 0;
> > > | }
> > > configure:2889: error: C compiler cannot create executables
> > > See `config.log' for more details.
> > >
> > >
> > > On Fri, Jan 15, 2010 at 1:17 PM, Todd Lipcon <to...@cloudera.com>
> wrote:
> > >
> > > > Are you starting from a clean tarball of the lzo stuff?
> > > >
> > > > Can you make sure your /tmp/ partition isn't full?
> > > >
> > > > There should be a config.log file hanging around somewhere also -
> > perhaps
> > > > this would be helpful.
> > > >
> > > > -Todd
> > > >
> > > > On Fri, Jan 15, 2010 at 1:14 PM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > Todd:
> > > > > Thanks for the continued support.
> > > > >
> > > > > I installed lzo-devel:
> > > > > [root@tyu-linux batchclient]# rpm -ivh
> > > > > /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm
> > > > > warning: /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm: Header
> V3
> > > DSA
> > > > > signature: NOKEY, key ID 217521f6
> > > > > Preparing...
> > ###########################################
> > > > > [100%]
> > > > > 1:lzo-devel
> > ###########################################
> > > > > [100%]
> > > > >
> > > > > But I still see:
> > > > > [exec] Try `/usr/bin/install --help' for more information.
> > > > > [exec] checking for gcc... gcc
> > > > > [exec] checking for C compiler default output file name...
> > > > > [exec] configure: error: C compiler cannot create executables
> > > > >
> > > > >
> > > > > On Fri, Jan 15, 2010 at 12:09 PM, Todd Lipcon <to...@cloudera.com>
> > > wrote:
> > > > >
> > > > > > Hi Ted,
> > > > > >
> > > > > > Did you also install liblzo-devel?
> > > > > >
> > > > > > Here are the packages I install for LZO:
> > > > > >
> > > > > > lzo-2.02-2.el5.1
> > > > > > lzo-devel-2.02-2.el5.1
> > > > > >
> > > > > > -Todd
> > > > > >
> > > > > > On Tue, Jan 12, 2010 at 4:28 PM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and
> latest
> > > > > version
> > > > > > > Nothing to do
> > > > > > >
> > > > > > > [root@tyu-linux batchclient]# yum install gcc-c++
> > > > > > > Loaded plugins: fastestmirror
> > > > > > > Loading mirror speeds from cached hostfile
> > > > > > > * base: mirrors.kernel.org
> > > > > > > * updates: updates.interworx.info
> > > > > > > * addons: mirrors.kernel.org
> > > > > > > * extras: mirrors.netdna.com
> > > > > > > Setting up Install Process
> > > > > > > Parsing package install arguments
> > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and
> > > latest
> > > > > > > version
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <
> todd@cloudera.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Make sure you have gcc, g++ installed?
> > > > > > > >
> > > > > > > > -Todd
> > > > > > > >
> > > > > > > > On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <
> yuzhihong@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > I installed
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
> > > > > > > > > .
> > > > > > > > >
> > > > > > > > > [root@tyu-linux software]# rpm -e lzo
> > > > > > > > > [root@tyu-linux software]# rpm -ivh
> > > > > > > ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
> > > > > > > > > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header
> V3
> > > DSA
> > > > > > > > signature:
> > > > > > > > > NOKEY, key ID 217521f6
> > > > > > > > > Preparing...
> > > > > > ###########################################
> > > > > > > > > [100%]
> > > > > > > > > 1:lzo
> > > > > > ###########################################
> > > > > > > > > [100%]
> > > > > > > > >
> > > > > > > > > [rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
> > > > > > > > > lrwxrwxrwx 1 root root 16 Jan 12 10:29
> > > > /usr/lib64/liblzo2.so.2
> > > > > ->
> > > > > > > > > liblzo2.so.2.0.0
> > > > > > > > > -rwxr-xr-x 1 root root 123408 May 6 2007
> > > > > > /usr/lib64/liblzo2.so.2.0.0
> > > > > > > > >
> > > > > > > > > I still see this error:
> > > > > > > > > [exec] configure: error: C compiler cannot create
> > > > executables
> > > > > > > > > [exec] See `config.log' for more details.
> > > > > > > > >
> > > > > > > > > BUILD FAILED
> > > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243:
> exec
> > > > > > returned:
> > > > > > > > 77
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <
> > > todd@cloudera.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Ted,
> > > > > > > > > >
> > > > > > > > > > Please make sure you have version 2.02 of liblzo
> installed.
> > > > > There's
> > > > > > > an
> > > > > > > > > > earlier version in RHEL that doesn't fail properly at the
> > > > > configure
> > > > > > > > stage
> > > > > > > > > > and I've seen it produce the error you pasted below.
> > > > > > > > > >
> > > > > > > > > > -Todd
> > > > > > > > > >
> > > > > > > > > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <
> > > yuzhihong@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I followed
> > > > > > > http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
> > > > > > > > > > >
> > > > > > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already
> installed
> > > and
> > > > > > > latest
> > > > > > > > > > > version
> > > > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14
> > > 06:36:37
> > > > > EDT
> > > > > > > > 2009
> > > > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > > > >
> > > > > > > > > > > For 32 bit build, I got:
> > > > > > > > > > > [exec] checking for C compiler default output file
> > > > name...
> > > > > > > > > > > [exec] configure: error: C compiler cannot create
> > > > > executables
> > > > > > > > > > > [exec] See `config.log' for more details.
> > > > > > > > > > >
> > > > > > > > > > > BUILD FAILED
> > > > > > > > > > >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243:
> > > exec
> > > > > > > > returned:
> > > > > > > > > > 77
> > > > > > > > > > >
> > > > > > > > > > > For 64 bit build, I got:
> > > > > > > > > > > [exec] gcc -DHAVE_CONFIG_H -I.
> > > > > > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native
> > > > -I./impl
> > > > > > > > > > > -I/usr/java/default/include
> > > -I/usr/java/default/include/linux
> > > > > > > > > > >
> > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> > > > > > > > > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2
> -m64
> > > -m64
> > > > > -MT
> > > > > > > > > > > impl/lzo/LzoCompressor.lo -MD -MP -MF
> > > > > > > > impl/lzo/.deps/LzoCompressor.Tpo
> > > > > > > > > -c
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> > > > > > > > > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> > > > > > > > > > > [exec]
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> > > > > > > > > > > In function
> > > > > > > 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> > > > > > > > > > > [exec]
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> > > > > > > > > > > error: expected expression before ',' token
> > > > > > > > > > > [exec] make: *** [impl/lzo/LzoCompressor.lo] Error
> 1
> > > > > > > > > > >
> > > > > > > > > > > BUILD FAILED
> > > > > > > > > > >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251:
> > > exec
> > > > > > > > returned:
> > > > > > > > > 2
> > > > > > > > > > >
> > > > > > > > > > > Has anybody seen the above ?
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <
> > > > > todd@cloudera.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Ted,
> > > > > > > > > > > >
> > > > > > > > > > > > You need to install liblzo from EPEL:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> > > > > > > > > > > >
> > > > > > > > > > > > -Todd
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <
> > > > yuzhihong@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Can someone tell me how I can install liblzo ?
> > > > > > > > > > > > >
> > > > > > > > > > > > > [root@tyu-linux lzo-2.03]# uname -a
> > > > > > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul
> 14
> > > > > 06:36:37
> > > > > > > EDT
> > > > > > > > > > 2009
> > > > > > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > > > > > > [root@tyu-linux lzo-2.03]# yum install
> liblzo-devel
> > > > > > > > > > > > > Loaded plugins: fastestmirror
> > > > > > > > > > > > > Loading mirror speeds from cached hostfile
> > > > > > > > > > > > > * base: mirrors.usc.edu
> > > > > > > > > > > > > * updates: mirror.san.fastserv.com
> > > > > > > > > > > > > * addons: centos.promopeddler.com
> > > > > > > > > > > > > * extras: mirrors.versaweb.com
> > > > > > > > > > > > > Setting up Install Process
> > > > > > > > > > > > > Parsing package install arguments
> > > > > > > > > > > > > No package liblzo-devel available.
> > > > > > > > > > > > > Nothing to do
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <
> > > > > > > kuosenhao@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Ted,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You may want to consider LZO compression, which
> > > allows
> > > > > > > > splitting
> > > > > > > > > > for
> > > > > > > > > > > a
> > > > > > > > > > > > > > comporessed file for Map jobs. On the other
> hand,
> > > gzip
> > > > > is
> > > > > > > not
> > > > > > > > > > > > > splittable.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Check out these links.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > > > > > > > > > > >
> http://wiki.apache.org/hadoop/UsingLzoCompression
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <
> > > > > > yuzhihong@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The input file is in .gz format
> > > > > > > > > > > > > > > FYI
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <
> > > > > > > yuzhihong@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > My current project processes input file of
> size
> > > > > > 333302161
> > > > > > > > > > bytes.
> > > > > > > > > > > > > > > > What I plan to do is to split the file into
> > equal
> > > > > size
> > > > > > > > pieces
> > > > > > > > > > > (and
> > > > > > > > > > > > on
> > > > > > > > > > > > > > > blank
> > > > > > > > > > > > > > > > line boundary) to improve performance.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I found 12 classes in 0.20.1 source code
> which
> > > > > > implement
> > > > > > > > > > > > InputSplit.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > If someone has written code similar to what I
> > > plan
> > > > to
> > > > > > do,
> > > > > > > > > > please
> > > > > > > > > > > > > share
> > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > hint.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar
> <
> > > > > > > > > > > amogh@yahoo-inc.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > > > > >> The deprecation is due to the new evolving
> > > > mapreduce
> > > > > (
> > > > > > > > > > > > > o.a.h.mapreduce
> > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > >> APIs. Old APIs are supported for available
> > > > > > > distributions.
> > > > > > > > > The
> > > > > > > > > > > > > > equivalent
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > >> TextInputFormat is available in new API :
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > >> Amogh
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <
> > > yuzhihong@gmail.com>
> > > > > > > wrote:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> According to:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> isSplitable() is deprecated.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Which method should I use to replace it ?
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Thanks
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Ted Yu <yu...@gmail.com>.
[rialto@tyu-linux kevinweil-hadoop-lzo-916aeae]$ ./a.out
Hello World!
[rialto@tyu-linux kevinweil-hadoop-lzo-916aeae]$ gcc -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-libgcj-multifile
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk
--disable-dssi --enable-plugin
--with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic
--host=x86_64-redhat-linux
Thread model: posix
gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
On Fri, Jan 15, 2010 at 1:47 PM, Alex Kozlov <al...@cloudera.com> wrote:
> Looks like your compiler is misconfigured. Can you try and compile a
> simple
> "Hello World!" program:
>
> [alexvk@monster01 ~]$ cat Hello.c
> #include <stdio.h>
> int main() {
> printf("Hello World!\n");
> }
> [alexvk@monster01 ~]$ gcc Hello.c
> [alexvk@monster01 ~]$ ./a.out
> Hello World!
> [alexvk@monster01 ~]$ gcc -v
> Using built-in specs.
> Target: x86_64-redhat-linux
> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> --infodir=/usr/share/info --enable-shared --enable-threads=posix
> --enable-checking=release --with-system-zlib --enable-__cxa_atexit
> --disable-libunwind-exceptions --enable-libgcj-multifile
> --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
> --enable-java-awt=gtk
> --disable-dssi --enable-plugin
> --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic
> --host=x86_64-redhat-linux
> Thread model: posix
> gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
>
> Alex
>
> On Fri, Jan 15, 2010 at 1:34 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > I tried to set LD_LIBRARY_PATH to point to where crt1.o resides:
> > export LD_LIBRARY_PATH=/usr/lib64
> >
> > Still the same issue.
> > This is snippet of config.log:
> >
> > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> > --infodir=/usr/share/info --enable-shared --enable-threads=posix
> > --enable-checking=release --with-system-zlib --enable-__cxa_atexit
> > --disable-libunwind-exceptions --enable-libgcj-multifile
> > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
> > --enable-java-awt=gtk
> > --disable-dssi --enable-plugin
> > --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre
> --with-cpu=generic
> > --host=x86_64-redhat-linux
> > Thread model: posix
> > gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
> > configure:2781: $? = 0
> > configure:2788: gcc -V >&5
> > gcc: '-V' option must have argument
> > configure:2791: $? = 1
> > configure:2814: checking for C compiler default output file name
> > configure:2841: gcc -m32 conftest.c >&5
> > /usr/bin/ld: crt1.o: No such file: No such file or directory
> > collect2: ld returned 1 exit status
> > configure:2844: $? = 1
> > configure:2882: result:
> > configure: failed program was:
> > | /* confdefs.h. */
> > | #define PACKAGE_NAME "gpl-compression"
> > | #define PACKAGE_TARNAME "gpl-compression"
> > | #define PACKAGE_VERSION "0.1.0"
> > | #define PACKAGE_STRING "gpl-compression 0.1.0"
> > | #define PACKAGE_BUGREPORT "omalley@apache.org"
> > | #define PACKAGE "gpl-compression"
> > | #define VERSION "0.1.0"
> > | #define _GNU_SOURCE 1
> > | /* end confdefs.h. */
> > |
> > | int
> > | main ()
> > | {
> > |
> > | ;
> > | return 0;
> > | }
> > configure:2889: error: C compiler cannot create executables
> > See `config.log' for more details.
> >
> >
> > On Fri, Jan 15, 2010 at 1:17 PM, Todd Lipcon <to...@cloudera.com> wrote:
> >
> > > Are you starting from a clean tarball of the lzo stuff?
> > >
> > > Can you make sure your /tmp/ partition isn't full?
> > >
> > > There should be a config.log file hanging around somewhere also -
> perhaps
> > > this would be helpful.
> > >
> > > -Todd
> > >
> > > On Fri, Jan 15, 2010 at 1:14 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > Todd:
> > > > Thanks for the continued support.
> > > >
> > > > I installed lzo-devel:
> > > > [root@tyu-linux batchclient]# rpm -ivh
> > > > /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm
> > > > warning: /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm: Header V3
> > DSA
> > > > signature: NOKEY, key ID 217521f6
> > > > Preparing...
> ###########################################
> > > > [100%]
> > > > 1:lzo-devel
> ###########################################
> > > > [100%]
> > > >
> > > > But I still see:
> > > > [exec] Try `/usr/bin/install --help' for more information.
> > > > [exec] checking for gcc... gcc
> > > > [exec] checking for C compiler default output file name...
> > > > [exec] configure: error: C compiler cannot create executables
> > > >
> > > >
> > > > On Fri, Jan 15, 2010 at 12:09 PM, Todd Lipcon <to...@cloudera.com>
> > wrote:
> > > >
> > > > > Hi Ted,
> > > > >
> > > > > Did you also install liblzo-devel?
> > > > >
> > > > > Here are the packages I install for LZO:
> > > > >
> > > > > lzo-2.02-2.el5.1
> > > > > lzo-devel-2.02-2.el5.1
> > > > >
> > > > > -Todd
> > > > >
> > > > > On Tue, Jan 12, 2010 at 4:28 PM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > > > Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and latest
> > > > version
> > > > > > Nothing to do
> > > > > >
> > > > > > [root@tyu-linux batchclient]# yum install gcc-c++
> > > > > > Loaded plugins: fastestmirror
> > > > > > Loading mirror speeds from cached hostfile
> > > > > > * base: mirrors.kernel.org
> > > > > > * updates: updates.interworx.info
> > > > > > * addons: mirrors.kernel.org
> > > > > > * extras: mirrors.netdna.com
> > > > > > Setting up Install Process
> > > > > > Parsing package install arguments
> > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and
> > latest
> > > > > > version
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <to...@cloudera.com>
> > > > wrote:
> > > > > >
> > > > > > > Make sure you have gcc, g++ installed?
> > > > > > >
> > > > > > > -Todd
> > > > > > >
> > > > > > > On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <yu...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > I installed
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
> > > > > > > > .
> > > > > > > >
> > > > > > > > [root@tyu-linux software]# rpm -e lzo
> > > > > > > > [root@tyu-linux software]# rpm -ivh
> > > > > > ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
> > > > > > > > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header V3
> > DSA
> > > > > > > signature:
> > > > > > > > NOKEY, key ID 217521f6
> > > > > > > > Preparing...
> > > > > ###########################################
> > > > > > > > [100%]
> > > > > > > > 1:lzo
> > > > > ###########################################
> > > > > > > > [100%]
> > > > > > > >
> > > > > > > > [rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
> > > > > > > > lrwxrwxrwx 1 root root 16 Jan 12 10:29
> > > /usr/lib64/liblzo2.so.2
> > > > ->
> > > > > > > > liblzo2.so.2.0.0
> > > > > > > > -rwxr-xr-x 1 root root 123408 May 6 2007
> > > > > /usr/lib64/liblzo2.so.2.0.0
> > > > > > > >
> > > > > > > > I still see this error:
> > > > > > > > [exec] configure: error: C compiler cannot create
> > > executables
> > > > > > > > [exec] See `config.log' for more details.
> > > > > > > >
> > > > > > > > BUILD FAILED
> > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
> > > > > returned:
> > > > > > > 77
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <
> > todd@cloudera.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Ted,
> > > > > > > > >
> > > > > > > > > Please make sure you have version 2.02 of liblzo installed.
> > > > There's
> > > > > > an
> > > > > > > > > earlier version in RHEL that doesn't fail properly at the
> > > > configure
> > > > > > > stage
> > > > > > > > > and I've seen it produce the error you pasted below.
> > > > > > > > >
> > > > > > > > > -Todd
> > > > > > > > >
> > > > > > > > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <
> > yuzhihong@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I followed
> > > > > > http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
> > > > > > > > > >
> > > > > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed
> > and
> > > > > > latest
> > > > > > > > > > version
> > > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14
> > 06:36:37
> > > > EDT
> > > > > > > 2009
> > > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > > >
> > > > > > > > > > For 32 bit build, I got:
> > > > > > > > > > [exec] checking for C compiler default output file
> > > name...
> > > > > > > > > > [exec] configure: error: C compiler cannot create
> > > > executables
> > > > > > > > > > [exec] See `config.log' for more details.
> > > > > > > > > >
> > > > > > > > > > BUILD FAILED
> > > > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243:
> > exec
> > > > > > > returned:
> > > > > > > > > 77
> > > > > > > > > >
> > > > > > > > > > For 64 bit build, I got:
> > > > > > > > > > [exec] gcc -DHAVE_CONFIG_H -I.
> > > > > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native
> > > -I./impl
> > > > > > > > > > -I/usr/java/default/include
> > -I/usr/java/default/include/linux
> > > > > > > > > >
> -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> > > > > > > > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64
> > -m64
> > > > -MT
> > > > > > > > > > impl/lzo/LzoCompressor.lo -MD -MP -MF
> > > > > > > impl/lzo/.deps/LzoCompressor.Tpo
> > > > > > > > -c
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> > > > > > > > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> > > > > > > > > > [exec]
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> > > > > > > > > > In function
> > > > > > 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> > > > > > > > > > [exec]
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> > > > > > > > > > error: expected expression before ',' token
> > > > > > > > > > [exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1
> > > > > > > > > >
> > > > > > > > > > BUILD FAILED
> > > > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251:
> > exec
> > > > > > > returned:
> > > > > > > > 2
> > > > > > > > > >
> > > > > > > > > > Has anybody seen the above ?
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <
> > > > todd@cloudera.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Ted,
> > > > > > > > > > >
> > > > > > > > > > > You need to install liblzo from EPEL:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> > > > > > > > > > >
> > > > > > > > > > > -Todd
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <
> > > yuzhihong@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Can someone tell me how I can install liblzo ?
> > > > > > > > > > > >
> > > > > > > > > > > > [root@tyu-linux lzo-2.03]# uname -a
> > > > > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14
> > > > 06:36:37
> > > > > > EDT
> > > > > > > > > 2009
> > > > > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > > > > > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
> > > > > > > > > > > > Loaded plugins: fastestmirror
> > > > > > > > > > > > Loading mirror speeds from cached hostfile
> > > > > > > > > > > > * base: mirrors.usc.edu
> > > > > > > > > > > > * updates: mirror.san.fastserv.com
> > > > > > > > > > > > * addons: centos.promopeddler.com
> > > > > > > > > > > > * extras: mirrors.versaweb.com
> > > > > > > > > > > > Setting up Install Process
> > > > > > > > > > > > Parsing package install arguments
> > > > > > > > > > > > No package liblzo-devel available.
> > > > > > > > > > > > Nothing to do
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <
> > > > > > kuosenhao@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Ted,
> > > > > > > > > > > > >
> > > > > > > > > > > > > You may want to consider LZO compression, which
> > allows
> > > > > > > splitting
> > > > > > > > > for
> > > > > > > > > > a
> > > > > > > > > > > > > comporessed file for Map jobs. On the other hand,
> > gzip
> > > > is
> > > > > > not
> > > > > > > > > > > > splittable.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Check out these links.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > > > > > > > > > > http://wiki.apache.org/hadoop/UsingLzoCompression
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <
> > > > > yuzhihong@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > The input file is in .gz format
> > > > > > > > > > > > > > FYI
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <
> > > > > > yuzhihong@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > My current project processes input file of size
> > > > > 333302161
> > > > > > > > > bytes.
> > > > > > > > > > > > > > > What I plan to do is to split the file into
> equal
> > > > size
> > > > > > > pieces
> > > > > > > > > > (and
> > > > > > > > > > > on
> > > > > > > > > > > > > > blank
> > > > > > > > > > > > > > > line boundary) to improve performance.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I found 12 classes in 0.20.1 source code which
> > > > > implement
> > > > > > > > > > > InputSplit.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > If someone has written code similar to what I
> > plan
> > > to
> > > > > do,
> > > > > > > > > please
> > > > > > > > > > > > share
> > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > hint.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <
> > > > > > > > > > amogh@yahoo-inc.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > > > >> The deprecation is due to the new evolving
> > > mapreduce
> > > > (
> > > > > > > > > > > > o.a.h.mapreduce
> > > > > > > > > > > > > )
> > > > > > > > > > > > > > >> APIs. Old APIs are supported for available
> > > > > > distributions.
> > > > > > > > The
> > > > > > > > > > > > > equivalent
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > >> TextInputFormat is available in new API :
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > >> Amogh
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <
> > yuzhihong@gmail.com>
> > > > > > wrote:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> According to:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> isSplitable() is deprecated.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Which method should I use to replace it ?
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Thanks
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Alex Kozlov <al...@cloudera.com>.
Looks like your compiler is misconfigured. Can you try and compile a simple
"Hello World!" program:
[alexvk@monster01 ~]$ cat Hello.c
#include <stdio.h>
int main() {
printf("Hello World!\n");
}
[alexvk@monster01 ~]$ gcc Hello.c
[alexvk@monster01 ~]$ ./a.out
Hello World!
[alexvk@monster01 ~]$ gcc -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-libgcj-multifile
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk
--disable-dssi --enable-plugin
--with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic
--host=x86_64-redhat-linux
Thread model: posix
gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
Alex
On Fri, Jan 15, 2010 at 1:34 PM, Ted Yu <yu...@gmail.com> wrote:
> I tried to set LD_LIBRARY_PATH to point to where crt1.o resides:
> export LD_LIBRARY_PATH=/usr/lib64
>
> Still the same issue.
> This is snippet of config.log:
>
> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> --infodir=/usr/share/info --enable-shared --enable-threads=posix
> --enable-checking=release --with-system-zlib --enable-__cxa_atexit
> --disable-libunwind-exceptions --enable-libgcj-multifile
> --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
> --enable-java-awt=gtk
> --disable-dssi --enable-plugin
> --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic
> --host=x86_64-redhat-linux
> Thread model: posix
> gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
> configure:2781: $? = 0
> configure:2788: gcc -V >&5
> gcc: '-V' option must have argument
> configure:2791: $? = 1
> configure:2814: checking for C compiler default output file name
> configure:2841: gcc -m32 conftest.c >&5
> /usr/bin/ld: crt1.o: No such file: No such file or directory
> collect2: ld returned 1 exit status
> configure:2844: $? = 1
> configure:2882: result:
> configure: failed program was:
> | /* confdefs.h. */
> | #define PACKAGE_NAME "gpl-compression"
> | #define PACKAGE_TARNAME "gpl-compression"
> | #define PACKAGE_VERSION "0.1.0"
> | #define PACKAGE_STRING "gpl-compression 0.1.0"
> | #define PACKAGE_BUGREPORT "omalley@apache.org"
> | #define PACKAGE "gpl-compression"
> | #define VERSION "0.1.0"
> | #define _GNU_SOURCE 1
> | /* end confdefs.h. */
> |
> | int
> | main ()
> | {
> |
> | ;
> | return 0;
> | }
> configure:2889: error: C compiler cannot create executables
> See `config.log' for more details.
>
>
> On Fri, Jan 15, 2010 at 1:17 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > Are you starting from a clean tarball of the lzo stuff?
> >
> > Can you make sure your /tmp/ partition isn't full?
> >
> > There should be a config.log file hanging around somewhere also - perhaps
> > this would be helpful.
> >
> > -Todd
> >
> > On Fri, Jan 15, 2010 at 1:14 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Todd:
> > > Thanks for the continued support.
> > >
> > > I installed lzo-devel:
> > > [root@tyu-linux batchclient]# rpm -ivh
> > > /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm
> > > warning: /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm: Header V3
> DSA
> > > signature: NOKEY, key ID 217521f6
> > > Preparing... ###########################################
> > > [100%]
> > > 1:lzo-devel ###########################################
> > > [100%]
> > >
> > > But I still see:
> > > [exec] Try `/usr/bin/install --help' for more information.
> > > [exec] checking for gcc... gcc
> > > [exec] checking for C compiler default output file name...
> > > [exec] configure: error: C compiler cannot create executables
> > >
> > >
> > > On Fri, Jan 15, 2010 at 12:09 PM, Todd Lipcon <to...@cloudera.com>
> wrote:
> > >
> > > > Hi Ted,
> > > >
> > > > Did you also install liblzo-devel?
> > > >
> > > > Here are the packages I install for LZO:
> > > >
> > > > lzo-2.02-2.el5.1
> > > > lzo-devel-2.02-2.el5.1
> > > >
> > > > -Todd
> > > >
> > > > On Tue, Jan 12, 2010 at 4:28 PM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and latest
> > > version
> > > > > Nothing to do
> > > > >
> > > > > [root@tyu-linux batchclient]# yum install gcc-c++
> > > > > Loaded plugins: fastestmirror
> > > > > Loading mirror speeds from cached hostfile
> > > > > * base: mirrors.kernel.org
> > > > > * updates: updates.interworx.info
> > > > > * addons: mirrors.kernel.org
> > > > > * extras: mirrors.netdna.com
> > > > > Setting up Install Process
> > > > > Parsing package install arguments
> > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and
> latest
> > > > > version
> > > > >
> > > > >
> > > > > On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <to...@cloudera.com>
> > > wrote:
> > > > >
> > > > > > Make sure you have gcc, g++ installed?
> > > > > >
> > > > > > -Todd
> > > > > >
> > > > > > On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <yu...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > I installed
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
> > > > > > > .
> > > > > > >
> > > > > > > [root@tyu-linux software]# rpm -e lzo
> > > > > > > [root@tyu-linux software]# rpm -ivh
> > > > > ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
> > > > > > > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header V3
> DSA
> > > > > > signature:
> > > > > > > NOKEY, key ID 217521f6
> > > > > > > Preparing...
> > > > ###########################################
> > > > > > > [100%]
> > > > > > > 1:lzo
> > > > ###########################################
> > > > > > > [100%]
> > > > > > >
> > > > > > > [rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
> > > > > > > lrwxrwxrwx 1 root root 16 Jan 12 10:29
> > /usr/lib64/liblzo2.so.2
> > > ->
> > > > > > > liblzo2.so.2.0.0
> > > > > > > -rwxr-xr-x 1 root root 123408 May 6 2007
> > > > /usr/lib64/liblzo2.so.2.0.0
> > > > > > >
> > > > > > > I still see this error:
> > > > > > > [exec] configure: error: C compiler cannot create
> > executables
> > > > > > > [exec] See `config.log' for more details.
> > > > > > >
> > > > > > > BUILD FAILED
> > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
> > > > returned:
> > > > > > 77
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <
> todd@cloudera.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Ted,
> > > > > > > >
> > > > > > > > Please make sure you have version 2.02 of liblzo installed.
> > > There's
> > > > > an
> > > > > > > > earlier version in RHEL that doesn't fail properly at the
> > > configure
> > > > > > stage
> > > > > > > > and I've seen it produce the error you pasted below.
> > > > > > > >
> > > > > > > > -Todd
> > > > > > > >
> > > > > > > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <
> yuzhihong@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > I followed
> > > > > http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
> > > > > > > > >
> > > > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed
> and
> > > > > latest
> > > > > > > > > version
> > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14
> 06:36:37
> > > EDT
> > > > > > 2009
> > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > >
> > > > > > > > > For 32 bit build, I got:
> > > > > > > > > [exec] checking for C compiler default output file
> > name...
> > > > > > > > > [exec] configure: error: C compiler cannot create
> > > executables
> > > > > > > > > [exec] See `config.log' for more details.
> > > > > > > > >
> > > > > > > > > BUILD FAILED
> > > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243:
> exec
> > > > > > returned:
> > > > > > > > 77
> > > > > > > > >
> > > > > > > > > For 64 bit build, I got:
> > > > > > > > > [exec] gcc -DHAVE_CONFIG_H -I.
> > > > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native
> > -I./impl
> > > > > > > > > -I/usr/java/default/include
> -I/usr/java/default/include/linux
> > > > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> > > > > > > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64
> -m64
> > > -MT
> > > > > > > > > impl/lzo/LzoCompressor.lo -MD -MP -MF
> > > > > > impl/lzo/.deps/LzoCompressor.Tpo
> > > > > > > -c
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> > > > > > > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> > > > > > > > > [exec]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> > > > > > > > > In function
> > > > > 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> > > > > > > > > [exec]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> > > > > > > > > error: expected expression before ',' token
> > > > > > > > > [exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1
> > > > > > > > >
> > > > > > > > > BUILD FAILED
> > > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251:
> exec
> > > > > > returned:
> > > > > > > 2
> > > > > > > > >
> > > > > > > > > Has anybody seen the above ?
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <
> > > todd@cloudera.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Ted,
> > > > > > > > > >
> > > > > > > > > > You need to install liblzo from EPEL:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> > > > > > > > > >
> > > > > > > > > > -Todd
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <
> > yuzhihong@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Can someone tell me how I can install liblzo ?
> > > > > > > > > > >
> > > > > > > > > > > [root@tyu-linux lzo-2.03]# uname -a
> > > > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14
> > > 06:36:37
> > > > > EDT
> > > > > > > > 2009
> > > > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > > > > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
> > > > > > > > > > > Loaded plugins: fastestmirror
> > > > > > > > > > > Loading mirror speeds from cached hostfile
> > > > > > > > > > > * base: mirrors.usc.edu
> > > > > > > > > > > * updates: mirror.san.fastserv.com
> > > > > > > > > > > * addons: centos.promopeddler.com
> > > > > > > > > > > * extras: mirrors.versaweb.com
> > > > > > > > > > > Setting up Install Process
> > > > > > > > > > > Parsing package install arguments
> > > > > > > > > > > No package liblzo-devel available.
> > > > > > > > > > > Nothing to do
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <
> > > > > kuosenhao@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Ted,
> > > > > > > > > > > >
> > > > > > > > > > > > You may want to consider LZO compression, which
> allows
> > > > > > splitting
> > > > > > > > for
> > > > > > > > > a
> > > > > > > > > > > > comporessed file for Map jobs. On the other hand,
> gzip
> > > is
> > > > > not
> > > > > > > > > > > splittable.
> > > > > > > > > > > >
> > > > > > > > > > > > Check out these links.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > > > > > > > > > http://wiki.apache.org/hadoop/UsingLzoCompression
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <
> > > > yuzhihong@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > The input file is in .gz format
> > > > > > > > > > > > > FYI
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <
> > > > > yuzhihong@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > My current project processes input file of size
> > > > 333302161
> > > > > > > > bytes.
> > > > > > > > > > > > > > What I plan to do is to split the file into equal
> > > size
> > > > > > pieces
> > > > > > > > > (and
> > > > > > > > > > on
> > > > > > > > > > > > > blank
> > > > > > > > > > > > > > line boundary) to improve performance.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I found 12 classes in 0.20.1 source code which
> > > > implement
> > > > > > > > > > InputSplit.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If someone has written code similar to what I
> plan
> > to
> > > > do,
> > > > > > > > please
> > > > > > > > > > > share
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > hint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <
> > > > > > > > > amogh@yahoo-inc.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > > >> The deprecation is due to the new evolving
> > mapreduce
> > > (
> > > > > > > > > > > o.a.h.mapreduce
> > > > > > > > > > > > )
> > > > > > > > > > > > > >> APIs. Old APIs are supported for available
> > > > > distributions.
> > > > > > > The
> > > > > > > > > > > > equivalent
> > > > > > > > > > > > > of
> > > > > > > > > > > > > >> TextInputFormat is available in new API :
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > >> Amogh
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <
> yuzhihong@gmail.com>
> > > > > wrote:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> According to:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> isSplitable() is deprecated.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Which method should I use to replace it ?
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Thanks
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Ted,
Looks like you may be running a 32-bit JDK, so it's trying to compile 32-bit
GPL compression libs. But, you're probably missing some parts of the 32-bit
toolchain.
-Todd
On Fri, Jan 15, 2010 at 1:34 PM, Ted Yu <yu...@gmail.com> wrote:
> I tried to set LD_LIBRARY_PATH to point to where crt1.o resides:
> export LD_LIBRARY_PATH=/usr/lib64
>
> Still the same issue.
> This is snippet of config.log:
>
> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> --infodir=/usr/share/info --enable-shared --enable-threads=posix
> --enable-checking=release --with-system-zlib --enable-__cxa_atexit
> --disable-libunwind-exceptions --enable-libgcj-multifile
> --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
> --enable-java-awt=gtk
> --disable-dssi --enable-plugin
> --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic
> --host=x86_64-redhat-linux
> Thread model: posix
> gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
> configure:2781: $? = 0
> configure:2788: gcc -V >&5
> gcc: '-V' option must have argument
> configure:2791: $? = 1
> configure:2814: checking for C compiler default output file name
> configure:2841: gcc -m32 conftest.c >&5
> /usr/bin/ld: crt1.o: No such file: No such file or directory
> collect2: ld returned 1 exit status
> configure:2844: $? = 1
> configure:2882: result:
> configure: failed program was:
> | /* confdefs.h. */
> | #define PACKAGE_NAME "gpl-compression"
> | #define PACKAGE_TARNAME "gpl-compression"
> | #define PACKAGE_VERSION "0.1.0"
> | #define PACKAGE_STRING "gpl-compression 0.1.0"
> | #define PACKAGE_BUGREPORT "omalley@apache.org"
> | #define PACKAGE "gpl-compression"
> | #define VERSION "0.1.0"
> | #define _GNU_SOURCE 1
> | /* end confdefs.h. */
> |
> | int
> | main ()
> | {
> |
> | ;
> | return 0;
> | }
> configure:2889: error: C compiler cannot create executables
> See `config.log' for more details.
>
>
> On Fri, Jan 15, 2010 at 1:17 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > Are you starting from a clean tarball of the lzo stuff?
> >
> > Can you make sure your /tmp/ partition isn't full?
> >
> > There should be a config.log file hanging around somewhere also - perhaps
> > this would be helpful.
> >
> > -Todd
> >
> > On Fri, Jan 15, 2010 at 1:14 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Todd:
> > > Thanks for the continued support.
> > >
> > > I installed lzo-devel:
> > > [root@tyu-linux batchclient]# rpm -ivh
> > > /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm
> > > warning: /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm: Header V3
> DSA
> > > signature: NOKEY, key ID 217521f6
> > > Preparing... ###########################################
> > > [100%]
> > > 1:lzo-devel ###########################################
> > > [100%]
> > >
> > > But I still see:
> > > [exec] Try `/usr/bin/install --help' for more information.
> > > [exec] checking for gcc... gcc
> > > [exec] checking for C compiler default output file name...
> > > [exec] configure: error: C compiler cannot create executables
> > >
> > >
> > > On Fri, Jan 15, 2010 at 12:09 PM, Todd Lipcon <to...@cloudera.com>
> wrote:
> > >
> > > > Hi Ted,
> > > >
> > > > Did you also install liblzo-devel?
> > > >
> > > > Here are the packages I install for LZO:
> > > >
> > > > lzo-2.02-2.el5.1
> > > > lzo-devel-2.02-2.el5.1
> > > >
> > > > -Todd
> > > >
> > > > On Tue, Jan 12, 2010 at 4:28 PM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and latest
> > > version
> > > > > Nothing to do
> > > > >
> > > > > [root@tyu-linux batchclient]# yum install gcc-c++
> > > > > Loaded plugins: fastestmirror
> > > > > Loading mirror speeds from cached hostfile
> > > > > * base: mirrors.kernel.org
> > > > > * updates: updates.interworx.info
> > > > > * addons: mirrors.kernel.org
> > > > > * extras: mirrors.netdna.com
> > > > > Setting up Install Process
> > > > > Parsing package install arguments
> > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and
> latest
> > > > > version
> > > > >
> > > > >
> > > > > On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <to...@cloudera.com>
> > > wrote:
> > > > >
> > > > > > Make sure you have gcc, g++ installed?
> > > > > >
> > > > > > -Todd
> > > > > >
> > > > > > On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <yu...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > I installed
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
> > > > > > > .
> > > > > > >
> > > > > > > [root@tyu-linux software]# rpm -e lzo
> > > > > > > [root@tyu-linux software]# rpm -ivh
> > > > > ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
> > > > > > > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header V3
> DSA
> > > > > > signature:
> > > > > > > NOKEY, key ID 217521f6
> > > > > > > Preparing...
> > > > ###########################################
> > > > > > > [100%]
> > > > > > > 1:lzo
> > > > ###########################################
> > > > > > > [100%]
> > > > > > >
> > > > > > > [rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
> > > > > > > lrwxrwxrwx 1 root root 16 Jan 12 10:29
> > /usr/lib64/liblzo2.so.2
> > > ->
> > > > > > > liblzo2.so.2.0.0
> > > > > > > -rwxr-xr-x 1 root root 123408 May 6 2007
> > > > /usr/lib64/liblzo2.so.2.0.0
> > > > > > >
> > > > > > > I still see this error:
> > > > > > > [exec] configure: error: C compiler cannot create
> > executables
> > > > > > > [exec] See `config.log' for more details.
> > > > > > >
> > > > > > > BUILD FAILED
> > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
> > > > returned:
> > > > > > 77
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <
> todd@cloudera.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Ted,
> > > > > > > >
> > > > > > > > Please make sure you have version 2.02 of liblzo installed.
> > > There's
> > > > > an
> > > > > > > > earlier version in RHEL that doesn't fail properly at the
> > > configure
> > > > > > stage
> > > > > > > > and I've seen it produce the error you pasted below.
> > > > > > > >
> > > > > > > > -Todd
> > > > > > > >
> > > > > > > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <
> yuzhihong@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > I followed
> > > > > http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
> > > > > > > > >
> > > > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed
> and
> > > > > latest
> > > > > > > > > version
> > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14
> 06:36:37
> > > EDT
> > > > > > 2009
> > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > >
> > > > > > > > > For 32 bit build, I got:
> > > > > > > > > [exec] checking for C compiler default output file
> > name...
> > > > > > > > > [exec] configure: error: C compiler cannot create
> > > executables
> > > > > > > > > [exec] See `config.log' for more details.
> > > > > > > > >
> > > > > > > > > BUILD FAILED
> > > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243:
> exec
> > > > > > returned:
> > > > > > > > 77
> > > > > > > > >
> > > > > > > > > For 64 bit build, I got:
> > > > > > > > > [exec] gcc -DHAVE_CONFIG_H -I.
> > > > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native
> > -I./impl
> > > > > > > > > -I/usr/java/default/include
> -I/usr/java/default/include/linux
> > > > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> > > > > > > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64
> -m64
> > > -MT
> > > > > > > > > impl/lzo/LzoCompressor.lo -MD -MP -MF
> > > > > > impl/lzo/.deps/LzoCompressor.Tpo
> > > > > > > -c
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> > > > > > > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> > > > > > > > > [exec]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> > > > > > > > > In function
> > > > > 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> > > > > > > > > [exec]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> > > > > > > > > error: expected expression before ',' token
> > > > > > > > > [exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1
> > > > > > > > >
> > > > > > > > > BUILD FAILED
> > > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251:
> exec
> > > > > > returned:
> > > > > > > 2
> > > > > > > > >
> > > > > > > > > Has anybody seen the above ?
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <
> > > todd@cloudera.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Ted,
> > > > > > > > > >
> > > > > > > > > > You need to install liblzo from EPEL:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> > > > > > > > > >
> > > > > > > > > > -Todd
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <
> > yuzhihong@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Can someone tell me how I can install liblzo ?
> > > > > > > > > > >
> > > > > > > > > > > [root@tyu-linux lzo-2.03]# uname -a
> > > > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14
> > > 06:36:37
> > > > > EDT
> > > > > > > > 2009
> > > > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > > > > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
> > > > > > > > > > > Loaded plugins: fastestmirror
> > > > > > > > > > > Loading mirror speeds from cached hostfile
> > > > > > > > > > > * base: mirrors.usc.edu
> > > > > > > > > > > * updates: mirror.san.fastserv.com
> > > > > > > > > > > * addons: centos.promopeddler.com
> > > > > > > > > > > * extras: mirrors.versaweb.com
> > > > > > > > > > > Setting up Install Process
> > > > > > > > > > > Parsing package install arguments
> > > > > > > > > > > No package liblzo-devel available.
> > > > > > > > > > > Nothing to do
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <
> > > > > kuosenhao@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Ted,
> > > > > > > > > > > >
> > > > > > > > > > > > You may want to consider LZO compression, which
> allows
> > > > > > splitting
> > > > > > > > for
> > > > > > > > > a
> > > > > > > > > > > > comporessed file for Map jobs. On the other hand,
> gzip
> > > is
> > > > > not
> > > > > > > > > > > splittable.
> > > > > > > > > > > >
> > > > > > > > > > > > Check out these links.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > > > > > > > > > http://wiki.apache.org/hadoop/UsingLzoCompression
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <
> > > > yuzhihong@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > The input file is in .gz format
> > > > > > > > > > > > > FYI
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <
> > > > > yuzhihong@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > My current project processes input file of size
> > > > 333302161
> > > > > > > > bytes.
> > > > > > > > > > > > > > What I plan to do is to split the file into equal
> > > size
> > > > > > pieces
> > > > > > > > > (and
> > > > > > > > > > on
> > > > > > > > > > > > > blank
> > > > > > > > > > > > > > line boundary) to improve performance.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I found 12 classes in 0.20.1 source code which
> > > > implement
> > > > > > > > > > InputSplit.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If someone has written code similar to what I
> plan
> > to
> > > > do,
> > > > > > > > please
> > > > > > > > > > > share
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > hint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <
> > > > > > > > > amogh@yahoo-inc.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > > >> The deprecation is due to the new evolving
> > mapreduce
> > > (
> > > > > > > > > > > o.a.h.mapreduce
> > > > > > > > > > > > )
> > > > > > > > > > > > > >> APIs. Old APIs are supported for available
> > > > > distributions.
> > > > > > > The
> > > > > > > > > > > > equivalent
> > > > > > > > > > > > > of
> > > > > > > > > > > > > >> TextInputFormat is available in new API :
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > >> Amogh
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <
> yuzhihong@gmail.com>
> > > > > wrote:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> According to:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> isSplitable() is deprecated.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Which method should I use to replace it ?
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Thanks
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Ted Yu <yu...@gmail.com>.
I tried to set LD_LIBRARY_PATH to point to where crt1.o resides:
export LD_LIBRARY_PATH=/usr/lib64
Still the same issue.
This is snippet of config.log:
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-libgcj-multifile
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk
--disable-dssi --enable-plugin
--with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic
--host=x86_64-redhat-linux
Thread model: posix
gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
configure:2781: $? = 0
configure:2788: gcc -V >&5
gcc: '-V' option must have argument
configure:2791: $? = 1
configure:2814: checking for C compiler default output file name
configure:2841: gcc -m32 conftest.c >&5
/usr/bin/ld: crt1.o: No such file: No such file or directory
collect2: ld returned 1 exit status
configure:2844: $? = 1
configure:2882: result:
configure: failed program was:
| /* confdefs.h. */
| #define PACKAGE_NAME "gpl-compression"
| #define PACKAGE_TARNAME "gpl-compression"
| #define PACKAGE_VERSION "0.1.0"
| #define PACKAGE_STRING "gpl-compression 0.1.0"
| #define PACKAGE_BUGREPORT "omalley@apache.org"
| #define PACKAGE "gpl-compression"
| #define VERSION "0.1.0"
| #define _GNU_SOURCE 1
| /* end confdefs.h. */
|
| int
| main ()
| {
|
| ;
| return 0;
| }
configure:2889: error: C compiler cannot create executables
See `config.log' for more details.
On Fri, Jan 15, 2010 at 1:17 PM, Todd Lipcon <to...@cloudera.com> wrote:
> Are you starting from a clean tarball of the lzo stuff?
>
> Can you make sure your /tmp/ partition isn't full?
>
> There should be a config.log file hanging around somewhere also - perhaps
> this would be helpful.
>
> -Todd
>
> On Fri, Jan 15, 2010 at 1:14 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Todd:
> > Thanks for the continued support.
> >
> > I installed lzo-devel:
> > [root@tyu-linux batchclient]# rpm -ivh
> > /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm
> > warning: /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm: Header V3 DSA
> > signature: NOKEY, key ID 217521f6
> > Preparing... ###########################################
> > [100%]
> > 1:lzo-devel ###########################################
> > [100%]
> >
> > But I still see:
> > [exec] Try `/usr/bin/install --help' for more information.
> > [exec] checking for gcc... gcc
> > [exec] checking for C compiler default output file name...
> > [exec] configure: error: C compiler cannot create executables
> >
> >
> > On Fri, Jan 15, 2010 at 12:09 PM, Todd Lipcon <to...@cloudera.com> wrote:
> >
> > > Hi Ted,
> > >
> > > Did you also install liblzo-devel?
> > >
> > > Here are the packages I install for LZO:
> > >
> > > lzo-2.02-2.el5.1
> > > lzo-devel-2.02-2.el5.1
> > >
> > > -Todd
> > >
> > > On Tue, Jan 12, 2010 at 4:28 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and latest
> > version
> > > > Nothing to do
> > > >
> > > > [root@tyu-linux batchclient]# yum install gcc-c++
> > > > Loaded plugins: fastestmirror
> > > > Loading mirror speeds from cached hostfile
> > > > * base: mirrors.kernel.org
> > > > * updates: updates.interworx.info
> > > > * addons: mirrors.kernel.org
> > > > * extras: mirrors.netdna.com
> > > > Setting up Install Process
> > > > Parsing package install arguments
> > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and latest
> > > > version
> > > >
> > > >
> > > > On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <to...@cloudera.com>
> > wrote:
> > > >
> > > > > Make sure you have gcc, g++ installed?
> > > > >
> > > > > -Todd
> > > > >
> > > > > On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > > >
> > > > > > I installed
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
> > > > > > .
> > > > > >
> > > > > > [root@tyu-linux software]# rpm -e lzo
> > > > > > [root@tyu-linux software]# rpm -ivh
> > > > ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
> > > > > > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header V3 DSA
> > > > > signature:
> > > > > > NOKEY, key ID 217521f6
> > > > > > Preparing...
> > > ###########################################
> > > > > > [100%]
> > > > > > 1:lzo
> > > ###########################################
> > > > > > [100%]
> > > > > >
> > > > > > [rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
> > > > > > lrwxrwxrwx 1 root root 16 Jan 12 10:29
> /usr/lib64/liblzo2.so.2
> > ->
> > > > > > liblzo2.so.2.0.0
> > > > > > -rwxr-xr-x 1 root root 123408 May 6 2007
> > > /usr/lib64/liblzo2.so.2.0.0
> > > > > >
> > > > > > I still see this error:
> > > > > > [exec] configure: error: C compiler cannot create
> executables
> > > > > > [exec] See `config.log' for more details.
> > > > > >
> > > > > > BUILD FAILED
> > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
> > > returned:
> > > > > 77
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <todd@cloudera.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Hi Ted,
> > > > > > >
> > > > > > > Please make sure you have version 2.02 of liblzo installed.
> > There's
> > > > an
> > > > > > > earlier version in RHEL that doesn't fail properly at the
> > configure
> > > > > stage
> > > > > > > and I've seen it produce the error you pasted below.
> > > > > > >
> > > > > > > -Todd
> > > > > > >
> > > > > > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <yu...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > I followed
> > > > http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
> > > > > > > >
> > > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and
> > > > latest
> > > > > > > > version
> > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37
> > EDT
> > > > > 2009
> > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > >
> > > > > > > > For 32 bit build, I got:
> > > > > > > > [exec] checking for C compiler default output file
> name...
> > > > > > > > [exec] configure: error: C compiler cannot create
> > executables
> > > > > > > > [exec] See `config.log' for more details.
> > > > > > > >
> > > > > > > > BUILD FAILED
> > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
> > > > > returned:
> > > > > > > 77
> > > > > > > >
> > > > > > > > For 64 bit build, I got:
> > > > > > > > [exec] gcc -DHAVE_CONFIG_H -I.
> > > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native
> -I./impl
> > > > > > > > -I/usr/java/default/include -I/usr/java/default/include/linux
> > > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> > > > > > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -m64
> > -MT
> > > > > > > > impl/lzo/LzoCompressor.lo -MD -MP -MF
> > > > > impl/lzo/.deps/LzoCompressor.Tpo
> > > > > > -c
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> > > > > > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> > > > > > > > [exec]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> > > > > > > > In function
> > > > 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> > > > > > > > [exec]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> > > > > > > > error: expected expression before ',' token
> > > > > > > > [exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1
> > > > > > > >
> > > > > > > > BUILD FAILED
> > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251: exec
> > > > > returned:
> > > > > > 2
> > > > > > > >
> > > > > > > > Has anybody seen the above ?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <
> > todd@cloudera.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Ted,
> > > > > > > > >
> > > > > > > > > You need to install liblzo from EPEL:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> > > > > > > > >
> > > > > > > > > -Todd
> > > > > > > > >
> > > > > > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <
> yuzhihong@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Can someone tell me how I can install liblzo ?
> > > > > > > > > >
> > > > > > > > > > [root@tyu-linux lzo-2.03]# uname -a
> > > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14
> > 06:36:37
> > > > EDT
> > > > > > > 2009
> > > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > > > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
> > > > > > > > > > Loaded plugins: fastestmirror
> > > > > > > > > > Loading mirror speeds from cached hostfile
> > > > > > > > > > * base: mirrors.usc.edu
> > > > > > > > > > * updates: mirror.san.fastserv.com
> > > > > > > > > > * addons: centos.promopeddler.com
> > > > > > > > > > * extras: mirrors.versaweb.com
> > > > > > > > > > Setting up Install Process
> > > > > > > > > > Parsing package install arguments
> > > > > > > > > > No package liblzo-devel available.
> > > > > > > > > > Nothing to do
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <
> > > > kuosenhao@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Ted,
> > > > > > > > > > >
> > > > > > > > > > > You may want to consider LZO compression, which allows
> > > > > splitting
> > > > > > > for
> > > > > > > > a
> > > > > > > > > > > comporessed file for Map jobs. On the other hand, gzip
> > is
> > > > not
> > > > > > > > > > splittable.
> > > > > > > > > > >
> > > > > > > > > > > Check out these links.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > > > > > > > > http://wiki.apache.org/hadoop/UsingLzoCompression
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <
> > > yuzhihong@gmail.com>
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > The input file is in .gz format
> > > > > > > > > > > > FYI
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <
> > > > yuzhihong@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > My current project processes input file of size
> > > 333302161
> > > > > > > bytes.
> > > > > > > > > > > > > What I plan to do is to split the file into equal
> > size
> > > > > pieces
> > > > > > > > (and
> > > > > > > > > on
> > > > > > > > > > > > blank
> > > > > > > > > > > > > line boundary) to improve performance.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I found 12 classes in 0.20.1 source code which
> > > implement
> > > > > > > > > InputSplit.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If someone has written code similar to what I plan
> to
> > > do,
> > > > > > > please
> > > > > > > > > > share
> > > > > > > > > > > > some
> > > > > > > > > > > > > hint.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <
> > > > > > > > amogh@yahoo-inc.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > >> The deprecation is due to the new evolving
> mapreduce
> > (
> > > > > > > > > > o.a.h.mapreduce
> > > > > > > > > > > )
> > > > > > > > > > > > >> APIs. Old APIs are supported for available
> > > > distributions.
> > > > > > The
> > > > > > > > > > > equivalent
> > > > > > > > > > > > of
> > > > > > > > > > > > >> TextInputFormat is available in new API :
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > >> Amogh
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com>
> > > > wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> According to:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> isSplitable() is deprecated.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Which method should I use to replace it ?
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Todd Lipcon <to...@cloudera.com>.
Are you starting from a clean tarball of the lzo stuff?
Can you make sure your /tmp/ partition isn't full?
There should be a config.log file hanging around somewhere also - perhaps
this would be helpful.
-Todd
On Fri, Jan 15, 2010 at 1:14 PM, Ted Yu <yu...@gmail.com> wrote:
> Todd:
> Thanks for the continued support.
>
> I installed lzo-devel:
> [root@tyu-linux batchclient]# rpm -ivh
> /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm
> warning: /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm: Header V3 DSA
> signature: NOKEY, key ID 217521f6
> Preparing... ###########################################
> [100%]
> 1:lzo-devel ###########################################
> [100%]
>
> But I still see:
> [exec] Try `/usr/bin/install --help' for more information.
> [exec] checking for gcc... gcc
> [exec] checking for C compiler default output file name...
> [exec] configure: error: C compiler cannot create executables
>
>
> On Fri, Jan 15, 2010 at 12:09 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > Hi Ted,
> >
> > Did you also install liblzo-devel?
> >
> > Here are the packages I install for LZO:
> >
> > lzo-2.02-2.el5.1
> > lzo-devel-2.02-2.el5.1
> >
> > -Todd
> >
> > On Tue, Jan 12, 2010 at 4:28 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and latest
> version
> > > Nothing to do
> > >
> > > [root@tyu-linux batchclient]# yum install gcc-c++
> > > Loaded plugins: fastestmirror
> > > Loading mirror speeds from cached hostfile
> > > * base: mirrors.kernel.org
> > > * updates: updates.interworx.info
> > > * addons: mirrors.kernel.org
> > > * extras: mirrors.netdna.com
> > > Setting up Install Process
> > > Parsing package install arguments
> > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and latest
> > > version
> > >
> > >
> > > On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <to...@cloudera.com>
> wrote:
> > >
> > > > Make sure you have gcc, g++ installed?
> > > >
> > > > -Todd
> > > >
> > > > On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > >
> > > > > I installed
> > > > >
> > > > >
> > > >
> > >
> >
> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
> > > > > .
> > > > >
> > > > > [root@tyu-linux software]# rpm -e lzo
> > > > > [root@tyu-linux software]# rpm -ivh
> > > ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
> > > > > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header V3 DSA
> > > > signature:
> > > > > NOKEY, key ID 217521f6
> > > > > Preparing...
> > ###########################################
> > > > > [100%]
> > > > > 1:lzo
> > ###########################################
> > > > > [100%]
> > > > >
> > > > > [rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
> > > > > lrwxrwxrwx 1 root root 16 Jan 12 10:29 /usr/lib64/liblzo2.so.2
> ->
> > > > > liblzo2.so.2.0.0
> > > > > -rwxr-xr-x 1 root root 123408 May 6 2007
> > /usr/lib64/liblzo2.so.2.0.0
> > > > >
> > > > > I still see this error:
> > > > > [exec] configure: error: C compiler cannot create executables
> > > > > [exec] See `config.log' for more details.
> > > > >
> > > > > BUILD FAILED
> > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
> > returned:
> > > > 77
> > > > >
> > > > >
> > > > > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <to...@cloudera.com>
> > > wrote:
> > > > >
> > > > > > Hi Ted,
> > > > > >
> > > > > > Please make sure you have version 2.02 of liblzo installed.
> There's
> > > an
> > > > > > earlier version in RHEL that doesn't fail properly at the
> configure
> > > > stage
> > > > > > and I've seen it produce the error you pasted below.
> > > > > >
> > > > > > -Todd
> > > > > >
> > > > > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <yu...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > I followed
> > > http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
> > > > > > >
> > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and
> > > latest
> > > > > > > version
> > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37
> EDT
> > > > 2009
> > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > >
> > > > > > > For 32 bit build, I got:
> > > > > > > [exec] checking for C compiler default output file name...
> > > > > > > [exec] configure: error: C compiler cannot create
> executables
> > > > > > > [exec] See `config.log' for more details.
> > > > > > >
> > > > > > > BUILD FAILED
> > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
> > > > returned:
> > > > > > 77
> > > > > > >
> > > > > > > For 64 bit build, I got:
> > > > > > > [exec] gcc -DHAVE_CONFIG_H -I.
> > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native -I./impl
> > > > > > > -I/usr/java/default/include -I/usr/java/default/include/linux
> > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> > > > > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -m64
> -MT
> > > > > > > impl/lzo/LzoCompressor.lo -MD -MP -MF
> > > > impl/lzo/.deps/LzoCompressor.Tpo
> > > > > -c
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> > > > > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> > > > > > > [exec]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> > > > > > > In function
> > > 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> > > > > > > [exec]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> > > > > > > error: expected expression before ',' token
> > > > > > > [exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1
> > > > > > >
> > > > > > > BUILD FAILED
> > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251: exec
> > > > returned:
> > > > > 2
> > > > > > >
> > > > > > > Has anybody seen the above ?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <
> todd@cloudera.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Ted,
> > > > > > > >
> > > > > > > > You need to install liblzo from EPEL:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> > > > > > > >
> > > > > > > > -Todd
> > > > > > > >
> > > > > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <yuzhihong@gmail.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Can someone tell me how I can install liblzo ?
> > > > > > > > >
> > > > > > > > > [root@tyu-linux lzo-2.03]# uname -a
> > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14
> 06:36:37
> > > EDT
> > > > > > 2009
> > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
> > > > > > > > > Loaded plugins: fastestmirror
> > > > > > > > > Loading mirror speeds from cached hostfile
> > > > > > > > > * base: mirrors.usc.edu
> > > > > > > > > * updates: mirror.san.fastserv.com
> > > > > > > > > * addons: centos.promopeddler.com
> > > > > > > > > * extras: mirrors.versaweb.com
> > > > > > > > > Setting up Install Process
> > > > > > > > > Parsing package install arguments
> > > > > > > > > No package liblzo-devel available.
> > > > > > > > > Nothing to do
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <
> > > kuosenhao@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Ted,
> > > > > > > > > >
> > > > > > > > > > You may want to consider LZO compression, which allows
> > > > splitting
> > > > > > for
> > > > > > > a
> > > > > > > > > > comporessed file for Map jobs. On the other hand, gzip
> is
> > > not
> > > > > > > > > splittable.
> > > > > > > > > >
> > > > > > > > > > Check out these links.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > > > > > > > http://wiki.apache.org/hadoop/UsingLzoCompression
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <
> > yuzhihong@gmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > The input file is in .gz format
> > > > > > > > > > > FYI
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <
> > > yuzhihong@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > My current project processes input file of size
> > 333302161
> > > > > > bytes.
> > > > > > > > > > > > What I plan to do is to split the file into equal
> size
> > > > pieces
> > > > > > > (and
> > > > > > > > on
> > > > > > > > > > > blank
> > > > > > > > > > > > line boundary) to improve performance.
> > > > > > > > > > > >
> > > > > > > > > > > > I found 12 classes in 0.20.1 source code which
> > implement
> > > > > > > > InputSplit.
> > > > > > > > > > > >
> > > > > > > > > > > > If someone has written code similar to what I plan to
> > do,
> > > > > > please
> > > > > > > > > share
> > > > > > > > > > > some
> > > > > > > > > > > > hint.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <
> > > > > > > amogh@yahoo-inc.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Hi,
> > > > > > > > > > > >> The deprecation is due to the new evolving mapreduce
> (
> > > > > > > > > o.a.h.mapreduce
> > > > > > > > > > )
> > > > > > > > > > > >> APIs. Old APIs are supported for available
> > > distributions.
> > > > > The
> > > > > > > > > > equivalent
> > > > > > > > > > > of
> > > > > > > > > > > >> TextInputFormat is available in new API :
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > > > > > > > >>
> > > > > > > > > > > >> Thanks,
> > > > > > > > > > > >> Amogh
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com>
> > > wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> According to:
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > > > > > > > >>
> > > > > > > > > > > >> isSplitable() is deprecated.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Which method should I use to replace it ?
> > > > > > > > > > > >>
> > > > > > > > > > > >> Thanks
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Ted Yu <yu...@gmail.com>.
Todd:
Thanks for the continued support.
I installed lzo-devel:
[root@tyu-linux batchclient]# rpm -ivh
/opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm
warning: /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm: Header V3 DSA
signature: NOKEY, key ID 217521f6
Preparing... ###########################################
[100%]
1:lzo-devel ###########################################
[100%]
But I still see:
[exec] Try `/usr/bin/install --help' for more information.
[exec] checking for gcc... gcc
[exec] checking for C compiler default output file name...
[exec] configure: error: C compiler cannot create executables
On Fri, Jan 15, 2010 at 12:09 PM, Todd Lipcon <to...@cloudera.com> wrote:
> Hi Ted,
>
> Did you also install liblzo-devel?
>
> Here are the packages I install for LZO:
>
> lzo-2.02-2.el5.1
> lzo-devel-2.02-2.el5.1
>
> -Todd
>
> On Tue, Jan 12, 2010 at 4:28 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and latest version
> > Nothing to do
> >
> > [root@tyu-linux batchclient]# yum install gcc-c++
> > Loaded plugins: fastestmirror
> > Loading mirror speeds from cached hostfile
> > * base: mirrors.kernel.org
> > * updates: updates.interworx.info
> > * addons: mirrors.kernel.org
> > * extras: mirrors.netdna.com
> > Setting up Install Process
> > Parsing package install arguments
> > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and latest
> > version
> >
> >
> > On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <to...@cloudera.com> wrote:
> >
> > > Make sure you have gcc, g++ installed?
> > >
> > > -Todd
> > >
> > > On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > I installed
> > > >
> > > >
> > >
> >
> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
> > > > .
> > > >
> > > > [root@tyu-linux software]# rpm -e lzo
> > > > [root@tyu-linux software]# rpm -ivh
> > ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
> > > > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header V3 DSA
> > > signature:
> > > > NOKEY, key ID 217521f6
> > > > Preparing...
> ###########################################
> > > > [100%]
> > > > 1:lzo
> ###########################################
> > > > [100%]
> > > >
> > > > [rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
> > > > lrwxrwxrwx 1 root root 16 Jan 12 10:29 /usr/lib64/liblzo2.so.2 ->
> > > > liblzo2.so.2.0.0
> > > > -rwxr-xr-x 1 root root 123408 May 6 2007
> /usr/lib64/liblzo2.so.2.0.0
> > > >
> > > > I still see this error:
> > > > [exec] configure: error: C compiler cannot create executables
> > > > [exec] See `config.log' for more details.
> > > >
> > > > BUILD FAILED
> > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
> returned:
> > > 77
> > > >
> > > >
> > > > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <to...@cloudera.com>
> > wrote:
> > > >
> > > > > Hi Ted,
> > > > >
> > > > > Please make sure you have version 2.02 of liblzo installed. There's
> > an
> > > > > earlier version in RHEL that doesn't fail properly at the configure
> > > stage
> > > > > and I've seen it produce the error you pasted below.
> > > > >
> > > > > -Todd
> > > > >
> > > > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > > >
> > > > > > I followed
> > http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
> > > > > >
> > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and
> > latest
> > > > > > version
> > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT
> > > 2009
> > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > >
> > > > > > For 32 bit build, I got:
> > > > > > [exec] checking for C compiler default output file name...
> > > > > > [exec] configure: error: C compiler cannot create executables
> > > > > > [exec] See `config.log' for more details.
> > > > > >
> > > > > > BUILD FAILED
> > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
> > > returned:
> > > > > 77
> > > > > >
> > > > > > For 64 bit build, I got:
> > > > > > [exec] gcc -DHAVE_CONFIG_H -I.
> > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native -I./impl
> > > > > > -I/usr/java/default/include -I/usr/java/default/include/linux
> > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> > > > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -m64 -MT
> > > > > > impl/lzo/LzoCompressor.lo -MD -MP -MF
> > > impl/lzo/.deps/LzoCompressor.Tpo
> > > > -c
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> > > > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> > > > > > [exec]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> > > > > > In function
> > 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> > > > > > [exec]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> > > > > > error: expected expression before ',' token
> > > > > > [exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1
> > > > > >
> > > > > > BUILD FAILED
> > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251: exec
> > > returned:
> > > > 2
> > > > > >
> > > > > > Has anybody seen the above ?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <to...@cloudera.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi Ted,
> > > > > > >
> > > > > > > You need to install liblzo from EPEL:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> > > > > > >
> > > > > > > -Todd
> > > > > > >
> > > > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <yu...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Can someone tell me how I can install liblzo ?
> > > > > > > >
> > > > > > > > [root@tyu-linux lzo-2.03]# uname -a
> > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37
> > EDT
> > > > > 2009
> > > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
> > > > > > > > Loaded plugins: fastestmirror
> > > > > > > > Loading mirror speeds from cached hostfile
> > > > > > > > * base: mirrors.usc.edu
> > > > > > > > * updates: mirror.san.fastserv.com
> > > > > > > > * addons: centos.promopeddler.com
> > > > > > > > * extras: mirrors.versaweb.com
> > > > > > > > Setting up Install Process
> > > > > > > > Parsing package install arguments
> > > > > > > > No package liblzo-devel available.
> > > > > > > > Nothing to do
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <
> > kuosenhao@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Ted,
> > > > > > > > >
> > > > > > > > > You may want to consider LZO compression, which allows
> > > splitting
> > > > > for
> > > > > > a
> > > > > > > > > comporessed file for Map jobs. On the other hand, gzip is
> > not
> > > > > > > > splittable.
> > > > > > > > >
> > > > > > > > > Check out these links.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > > > > > > http://wiki.apache.org/hadoop/UsingLzoCompression
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <
> yuzhihong@gmail.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > The input file is in .gz format
> > > > > > > > > > FYI
> > > > > > > > > >
> > > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <
> > yuzhihong@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > My current project processes input file of size
> 333302161
> > > > > bytes.
> > > > > > > > > > > What I plan to do is to split the file into equal size
> > > pieces
> > > > > > (and
> > > > > > > on
> > > > > > > > > > blank
> > > > > > > > > > > line boundary) to improve performance.
> > > > > > > > > > >
> > > > > > > > > > > I found 12 classes in 0.20.1 source code which
> implement
> > > > > > > InputSplit.
> > > > > > > > > > >
> > > > > > > > > > > If someone has written code similar to what I plan to
> do,
> > > > > please
> > > > > > > > share
> > > > > > > > > > some
> > > > > > > > > > > hint.
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <
> > > > > > amogh@yahoo-inc.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi,
> > > > > > > > > > >> The deprecation is due to the new evolving mapreduce (
> > > > > > > > o.a.h.mapreduce
> > > > > > > > > )
> > > > > > > > > > >> APIs. Old APIs are supported for available
> > distributions.
> > > > The
> > > > > > > > > equivalent
> > > > > > > > > > of
> > > > > > > > > > >> TextInputFormat is available in new API :
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >> Amogh
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com>
> > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> According to:
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > > > > > > >>
> > > > > > > > > > >> isSplitable() is deprecated.
> > > > > > > > > > >>
> > > > > > > > > > >> Which method should I use to replace it ?
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Ted,
Did you also install liblzo-devel?
Here are the packages I install for LZO:
lzo-2.02-2.el5.1
lzo-devel-2.02-2.el5.1
-Todd
On Tue, Jan 12, 2010 at 4:28 PM, Ted Yu <yu...@gmail.com> wrote:
> Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and latest version
> Nothing to do
>
> [root@tyu-linux batchclient]# yum install gcc-c++
> Loaded plugins: fastestmirror
> Loading mirror speeds from cached hostfile
> * base: mirrors.kernel.org
> * updates: updates.interworx.info
> * addons: mirrors.kernel.org
> * extras: mirrors.netdna.com
> Setting up Install Process
> Parsing package install arguments
> Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and latest
> version
>
>
> On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > Make sure you have gcc, g++ installed?
> >
> > -Todd
> >
> > On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > I installed
> > >
> > >
> >
> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
> > > .
> > >
> > > [root@tyu-linux software]# rpm -e lzo
> > > [root@tyu-linux software]# rpm -ivh
> ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
> > > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header V3 DSA
> > signature:
> > > NOKEY, key ID 217521f6
> > > Preparing... ###########################################
> > > [100%]
> > > 1:lzo ###########################################
> > > [100%]
> > >
> > > [rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
> > > lrwxrwxrwx 1 root root 16 Jan 12 10:29 /usr/lib64/liblzo2.so.2 ->
> > > liblzo2.so.2.0.0
> > > -rwxr-xr-x 1 root root 123408 May 6 2007 /usr/lib64/liblzo2.so.2.0.0
> > >
> > > I still see this error:
> > > [exec] configure: error: C compiler cannot create executables
> > > [exec] See `config.log' for more details.
> > >
> > > BUILD FAILED
> > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec returned:
> > 77
> > >
> > >
> > > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <to...@cloudera.com>
> wrote:
> > >
> > > > Hi Ted,
> > > >
> > > > Please make sure you have version 2.02 of liblzo installed. There's
> an
> > > > earlier version in RHEL that doesn't fail properly at the configure
> > stage
> > > > and I've seen it produce the error you pasted below.
> > > >
> > > > -Todd
> > > >
> > > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > >
> > > > > I followed
> http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
> > > > >
> > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and
> latest
> > > > > version
> > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT
> > 2009
> > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > >
> > > > > For 32 bit build, I got:
> > > > > [exec] checking for C compiler default output file name...
> > > > > [exec] configure: error: C compiler cannot create executables
> > > > > [exec] See `config.log' for more details.
> > > > >
> > > > > BUILD FAILED
> > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
> > returned:
> > > > 77
> > > > >
> > > > > For 64 bit build, I got:
> > > > > [exec] gcc -DHAVE_CONFIG_H -I.
> > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native -I./impl
> > > > > -I/usr/java/default/include -I/usr/java/default/include/linux
> > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> > > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -m64 -MT
> > > > > impl/lzo/LzoCompressor.lo -MD -MP -MF
> > impl/lzo/.deps/LzoCompressor.Tpo
> > > -c
> > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> > > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> > > > > [exec]
> > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> > > > > In function
> 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> > > > > [exec]
> > > > >
> > > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> > > > > error: expected expression before ',' token
> > > > > [exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1
> > > > >
> > > > > BUILD FAILED
> > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251: exec
> > returned:
> > > 2
> > > > >
> > > > > Has anybody seen the above ?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <to...@cloudera.com>
> > > wrote:
> > > > >
> > > > > > Hi Ted,
> > > > > >
> > > > > > You need to install liblzo from EPEL:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> > > > > >
> > > > > > -Todd
> > > > > >
> > > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Can someone tell me how I can install liblzo ?
> > > > > > >
> > > > > > > [root@tyu-linux lzo-2.03]# uname -a
> > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37
> EDT
> > > > 2009
> > > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
> > > > > > > Loaded plugins: fastestmirror
> > > > > > > Loading mirror speeds from cached hostfile
> > > > > > > * base: mirrors.usc.edu
> > > > > > > * updates: mirror.san.fastserv.com
> > > > > > > * addons: centos.promopeddler.com
> > > > > > > * extras: mirrors.versaweb.com
> > > > > > > Setting up Install Process
> > > > > > > Parsing package install arguments
> > > > > > > No package liblzo-devel available.
> > > > > > > Nothing to do
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <
> kuosenhao@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Ted,
> > > > > > > >
> > > > > > > > You may want to consider LZO compression, which allows
> > splitting
> > > > for
> > > > > a
> > > > > > > > comporessed file for Map jobs. On the other hand, gzip is
> not
> > > > > > > splittable.
> > > > > > > >
> > > > > > > > Check out these links.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > > > > > http://wiki.apache.org/hadoop/UsingLzoCompression
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <yu...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > The input file is in .gz format
> > > > > > > > > FYI
> > > > > > > > >
> > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <
> yuzhihong@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > My current project processes input file of size 333302161
> > > > bytes.
> > > > > > > > > > What I plan to do is to split the file into equal size
> > pieces
> > > > > (and
> > > > > > on
> > > > > > > > > blank
> > > > > > > > > > line boundary) to improve performance.
> > > > > > > > > >
> > > > > > > > > > I found 12 classes in 0.20.1 source code which implement
> > > > > > InputSplit.
> > > > > > > > > >
> > > > > > > > > > If someone has written code similar to what I plan to do,
> > > > please
> > > > > > > share
> > > > > > > > > some
> > > > > > > > > > hint.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <
> > > > > amogh@yahoo-inc.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi,
> > > > > > > > > >> The deprecation is due to the new evolving mapreduce (
> > > > > > > o.a.h.mapreduce
> > > > > > > > )
> > > > > > > > > >> APIs. Old APIs are supported for available
> distributions.
> > > The
> > > > > > > > equivalent
> > > > > > > > > of
> > > > > > > > > >> TextInputFormat is available in new API :
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >> Amogh
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com>
> wrote:
> > > > > > > > > >>
> > > > > > > > > >> According to:
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > > > > > >>
> > > > > > > > > >> isSplitable() is deprecated.
> > > > > > > > > >>
> > > > > > > > > >> Which method should I use to replace it ?
> > > > > > > > > >>
> > > > > > > > > >> Thanks
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Ted Yu <yu...@gmail.com>.
Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and latest version
Nothing to do
[root@tyu-linux batchclient]# yum install gcc-c++
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.kernel.org
* updates: updates.interworx.info
* addons: mirrors.kernel.org
* extras: mirrors.netdna.com
Setting up Install Process
Parsing package install arguments
Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and latest version
On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <to...@cloudera.com> wrote:
> Make sure you have gcc, g++ installed?
>
> -Todd
>
> On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > I installed
> >
> >
> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
> > .
> >
> > [root@tyu-linux software]# rpm -e lzo
> > [root@tyu-linux software]# rpm -ivh ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
> > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header V3 DSA
> signature:
> > NOKEY, key ID 217521f6
> > Preparing... ###########################################
> > [100%]
> > 1:lzo ###########################################
> > [100%]
> >
> > [rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
> > lrwxrwxrwx 1 root root 16 Jan 12 10:29 /usr/lib64/liblzo2.so.2 ->
> > liblzo2.so.2.0.0
> > -rwxr-xr-x 1 root root 123408 May 6 2007 /usr/lib64/liblzo2.so.2.0.0
> >
> > I still see this error:
> > [exec] configure: error: C compiler cannot create executables
> > [exec] See `config.log' for more details.
> >
> > BUILD FAILED
> > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec returned:
> 77
> >
> >
> > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <to...@cloudera.com> wrote:
> >
> > > Hi Ted,
> > >
> > > Please make sure you have version 2.02 of liblzo installed. There's an
> > > earlier version in RHEL that doesn't fail properly at the configure
> stage
> > > and I've seen it produce the error you pasted below.
> > >
> > > -Todd
> > >
> > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > I followed http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
> > > >
> > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and latest
> > > > version
> > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT
> 2009
> > > > x86_64 x86_64 x86_64 GNU/Linux
> > > >
> > > > For 32 bit build, I got:
> > > > [exec] checking for C compiler default output file name...
> > > > [exec] configure: error: C compiler cannot create executables
> > > > [exec] See `config.log' for more details.
> > > >
> > > > BUILD FAILED
> > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec
> returned:
> > > 77
> > > >
> > > > For 64 bit build, I got:
> > > > [exec] gcc -DHAVE_CONFIG_H -I.
> > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native -I./impl
> > > > -I/usr/java/default/include -I/usr/java/default/include/linux
> > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -m64 -MT
> > > > impl/lzo/LzoCompressor.lo -MD -MP -MF
> impl/lzo/.deps/LzoCompressor.Tpo
> > -c
> > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> > > > [exec]
> > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> > > > In function 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> > > > [exec]
> > > >
> > > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> > > > error: expected expression before ',' token
> > > > [exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1
> > > >
> > > > BUILD FAILED
> > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251: exec
> returned:
> > 2
> > > >
> > > > Has anybody seen the above ?
> > > >
> > > > Thanks
> > > >
> > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <to...@cloudera.com>
> > wrote:
> > > >
> > > > > Hi Ted,
> > > > >
> > > > > You need to install liblzo from EPEL:
> > > > >
> > > > >
> > > >
> > >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> > > > >
> > > > > -Todd
> > > > >
> > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > > > Can someone tell me how I can install liblzo ?
> > > > > >
> > > > > > [root@tyu-linux lzo-2.03]# uname -a
> > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT
> > > 2009
> > > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
> > > > > > Loaded plugins: fastestmirror
> > > > > > Loading mirror speeds from cached hostfile
> > > > > > * base: mirrors.usc.edu
> > > > > > * updates: mirror.san.fastserv.com
> > > > > > * addons: centos.promopeddler.com
> > > > > > * extras: mirrors.versaweb.com
> > > > > > Setting up Install Process
> > > > > > Parsing package install arguments
> > > > > > No package liblzo-devel available.
> > > > > > Nothing to do
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <kuosenhao@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Ted,
> > > > > > >
> > > > > > > You may want to consider LZO compression, which allows
> splitting
> > > for
> > > > a
> > > > > > > comporessed file for Map jobs. On the other hand, gzip is not
> > > > > > splittable.
> > > > > > >
> > > > > > > Check out these links.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > > > > http://wiki.apache.org/hadoop/UsingLzoCompression
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <yu...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > The input file is in .gz format
> > > > > > > > FYI
> > > > > > > >
> > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <yuzhihong@gmail.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > My current project processes input file of size 333302161
> > > bytes.
> > > > > > > > > What I plan to do is to split the file into equal size
> pieces
> > > > (and
> > > > > on
> > > > > > > > blank
> > > > > > > > > line boundary) to improve performance.
> > > > > > > > >
> > > > > > > > > I found 12 classes in 0.20.1 source code which implement
> > > > > InputSplit.
> > > > > > > > >
> > > > > > > > > If someone has written code similar to what I plan to do,
> > > please
> > > > > > share
> > > > > > > > some
> > > > > > > > > hint.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <
> > > > amogh@yahoo-inc.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi,
> > > > > > > > >> The deprecation is due to the new evolving mapreduce (
> > > > > > o.a.h.mapreduce
> > > > > > > )
> > > > > > > > >> APIs. Old APIs are supported for available distributions.
> > The
> > > > > > > equivalent
> > > > > > > > of
> > > > > > > > >> TextInputFormat is available in new API :
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> Amogh
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com> wrote:
> > > > > > > > >>
> > > > > > > > >> According to:
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > > > > >>
> > > > > > > > >> isSplitable() is deprecated.
> > > > > > > > >>
> > > > > > > > >> Which method should I use to replace it ?
> > > > > > > > >>
> > > > > > > > >> Thanks
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Todd Lipcon <to...@cloudera.com>.
Make sure you have gcc, g++ installed?
-Todd
On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <yu...@gmail.com> wrote:
> I installed
>
> ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday
> .
>
> [root@tyu-linux software]# rpm -e lzo
> [root@tyu-linux software]# rpm -ivh ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
> warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header V3 DSA signature:
> NOKEY, key ID 217521f6
> Preparing... ###########################################
> [100%]
> 1:lzo ###########################################
> [100%]
>
> [rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
> lrwxrwxrwx 1 root root 16 Jan 12 10:29 /usr/lib64/liblzo2.so.2 ->
> liblzo2.so.2.0.0
> -rwxr-xr-x 1 root root 123408 May 6 2007 /usr/lib64/liblzo2.so.2.0.0
>
> I still see this error:
> [exec] configure: error: C compiler cannot create executables
> [exec] See `config.log' for more details.
>
> BUILD FAILED
> /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec returned: 77
>
>
> On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > Hi Ted,
> >
> > Please make sure you have version 2.02 of liblzo installed. There's an
> > earlier version in RHEL that doesn't fail properly at the configure stage
> > and I've seen it produce the error you pasted below.
> >
> > -Todd
> >
> > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > I followed http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
> > >
> > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and latest
> > > version
> > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT 2009
> > > x86_64 x86_64 x86_64 GNU/Linux
> > >
> > > For 32 bit build, I got:
> > > [exec] checking for C compiler default output file name...
> > > [exec] configure: error: C compiler cannot create executables
> > > [exec] See `config.log' for more details.
> > >
> > > BUILD FAILED
> > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec returned:
> > 77
> > >
> > > For 64 bit build, I got:
> > > [exec] gcc -DHAVE_CONFIG_H -I.
> > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native -I./impl
> > > -I/usr/java/default/include -I/usr/java/default/include/linux
> > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -m64 -MT
> > > impl/lzo/LzoCompressor.lo -MD -MP -MF impl/lzo/.deps/LzoCompressor.Tpo
> -c
> > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> > > [exec]
> > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> > > In function 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> > > [exec]
> > >
> > >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> > > error: expected expression before ',' token
> > > [exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1
> > >
> > > BUILD FAILED
> > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251: exec returned:
> 2
> > >
> > > Has anybody seen the above ?
> > >
> > > Thanks
> > >
> > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <to...@cloudera.com>
> wrote:
> > >
> > > > Hi Ted,
> > > >
> > > > You need to install liblzo from EPEL:
> > > >
> > > >
> > >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> > > >
> > > > -Todd
> > > >
> > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > Can someone tell me how I can install liblzo ?
> > > > >
> > > > > [root@tyu-linux lzo-2.03]# uname -a
> > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT
> > 2009
> > > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
> > > > > Loaded plugins: fastestmirror
> > > > > Loading mirror speeds from cached hostfile
> > > > > * base: mirrors.usc.edu
> > > > > * updates: mirror.san.fastserv.com
> > > > > * addons: centos.promopeddler.com
> > > > > * extras: mirrors.versaweb.com
> > > > > Setting up Install Process
> > > > > Parsing package install arguments
> > > > > No package liblzo-devel available.
> > > > > Nothing to do
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <ku...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Ted,
> > > > > >
> > > > > > You may want to consider LZO compression, which allows splitting
> > for
> > > a
> > > > > > comporessed file for Map jobs. On the other hand, gzip is not
> > > > > splittable.
> > > > > >
> > > > > > Check out these links.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > > > http://wiki.apache.org/hadoop/UsingLzoCompression
> > > > > >
> > > > > >
> > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > The input file is in .gz format
> > > > > > > FYI
> > > > > > >
> > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <yu...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > My current project processes input file of size 333302161
> > bytes.
> > > > > > > > What I plan to do is to split the file into equal size pieces
> > > (and
> > > > on
> > > > > > > blank
> > > > > > > > line boundary) to improve performance.
> > > > > > > >
> > > > > > > > I found 12 classes in 0.20.1 source code which implement
> > > > InputSplit.
> > > > > > > >
> > > > > > > > If someone has written code similar to what I plan to do,
> > please
> > > > > share
> > > > > > > some
> > > > > > > > hint.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <
> > > amogh@yahoo-inc.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi,
> > > > > > > >> The deprecation is due to the new evolving mapreduce (
> > > > > o.a.h.mapreduce
> > > > > > )
> > > > > > > >> APIs. Old APIs are supported for available distributions.
> The
> > > > > > equivalent
> > > > > > > of
> > > > > > > >> TextInputFormat is available in new API :
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> Amogh
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com> wrote:
> > > > > > > >>
> > > > > > > >> According to:
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > > > >>
> > > > > > > >> isSplitable() is deprecated.
> > > > > > > >>
> > > > > > > >> Which method should I use to replace it ?
> > > > > > > >>
> > > > > > > >> Thanks
> > > > > > > >>
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Ted Yu <yu...@gmail.com>.
I installed
ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday.
[root@tyu-linux software]# rpm -e lzo
[root@tyu-linux software]# rpm -ivh ~rialto/lzo-2.02-2.el5.1.x86_64.rpm
warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header V3 DSA signature:
NOKEY, key ID 217521f6
Preparing... ###########################################
[100%]
1:lzo ###########################################
[100%]
[rialto@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2*
lrwxrwxrwx 1 root root 16 Jan 12 10:29 /usr/lib64/liblzo2.so.2 ->
liblzo2.so.2.0.0
-rwxr-xr-x 1 root root 123408 May 6 2007 /usr/lib64/liblzo2.so.2.0.0
I still see this error:
[exec] configure: error: C compiler cannot create executables
[exec] See `config.log' for more details.
BUILD FAILED
/home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec returned: 77
On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon <to...@cloudera.com> wrote:
> Hi Ted,
>
> Please make sure you have version 2.02 of liblzo installed. There's an
> earlier version in RHEL that doesn't fail properly at the configure stage
> and I've seen it produce the error you pasted below.
>
> -Todd
>
> On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > I followed http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
> >
> > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and latest
> > version
> > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT 2009
> > x86_64 x86_64 x86_64 GNU/Linux
> >
> > For 32 bit build, I got:
> > [exec] checking for C compiler default output file name...
> > [exec] configure: error: C compiler cannot create executables
> > [exec] See `config.log' for more details.
> >
> > BUILD FAILED
> > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec returned:
> 77
> >
> > For 64 bit build, I got:
> > [exec] gcc -DHAVE_CONFIG_H -I.
> > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native -I./impl
> > -I/usr/java/default/include -I/usr/java/default/include/linux
> > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -m64 -MT
> > impl/lzo/LzoCompressor.lo -MD -MP -MF impl/lzo/.deps/LzoCompressor.Tpo -c
> >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> > [exec]
> >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> > In function 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> > [exec]
> >
> >
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> > error: expected expression before ',' token
> > [exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1
> >
> > BUILD FAILED
> > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251: exec returned: 2
> >
> > Has anybody seen the above ?
> >
> > Thanks
> >
> > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <to...@cloudera.com> wrote:
> >
> > > Hi Ted,
> > >
> > > You need to install liblzo from EPEL:
> > >
> > >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> > >
> > > -Todd
> > >
> > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > Can someone tell me how I can install liblzo ?
> > > >
> > > > [root@tyu-linux lzo-2.03]# uname -a
> > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT
> 2009
> > > > x86_64 x86_64 x86_64 GNU/Linux
> > > > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
> > > > Loaded plugins: fastestmirror
> > > > Loading mirror speeds from cached hostfile
> > > > * base: mirrors.usc.edu
> > > > * updates: mirror.san.fastserv.com
> > > > * addons: centos.promopeddler.com
> > > > * extras: mirrors.versaweb.com
> > > > Setting up Install Process
> > > > Parsing package install arguments
> > > > No package liblzo-devel available.
> > > > Nothing to do
> > > >
> > > > Thanks
> > > >
> > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <ku...@gmail.com>
> > wrote:
> > > >
> > > > > Ted,
> > > > >
> > > > > You may want to consider LZO compression, which allows splitting
> for
> > a
> > > > > comporessed file for Map jobs. On the other hand, gzip is not
> > > > splittable.
> > > > >
> > > > > Check out these links.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > > http://wiki.apache.org/hadoop/UsingLzoCompression
> > > > >
> > > > >
> > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > > > The input file is in .gz format
> > > > > > FYI
> > > > > >
> > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > My current project processes input file of size 333302161
> bytes.
> > > > > > > What I plan to do is to split the file into equal size pieces
> > (and
> > > on
> > > > > > blank
> > > > > > > line boundary) to improve performance.
> > > > > > >
> > > > > > > I found 12 classes in 0.20.1 source code which implement
> > > InputSplit.
> > > > > > >
> > > > > > > If someone has written code similar to what I plan to do,
> please
> > > > share
> > > > > > some
> > > > > > > hint.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <
> > amogh@yahoo-inc.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > >> Hi,
> > > > > > >> The deprecation is due to the new evolving mapreduce (
> > > > o.a.h.mapreduce
> > > > > )
> > > > > > >> APIs. Old APIs are supported for available distributions. The
> > > > > equivalent
> > > > > > of
> > > > > > >> TextInputFormat is available in new API :
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Amogh
> > > > > > >>
> > > > > > >>
> > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com> wrote:
> > > > > > >>
> > > > > > >> According to:
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > > >>
> > > > > > >> isSplitable() is deprecated.
> > > > > > >>
> > > > > > >> Which method should I use to replace it ?
> > > > > > >>
> > > > > > >> Thanks
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Ted,
Please make sure you have version 2.02 of liblzo installed. There's an
earlier version in RHEL that doesn't fail properly at the configure stage
and I've seen it produce the error you pasted below.
-Todd
On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <yu...@gmail.com> wrote:
> I followed http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
>
> Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and latest
> version
> Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT 2009
> x86_64 x86_64 x86_64 GNU/Linux
>
> For 32 bit build, I got:
> [exec] checking for C compiler default output file name...
> [exec] configure: error: C compiler cannot create executables
> [exec] See `config.log' for more details.
>
> BUILD FAILED
> /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec returned: 77
>
> For 64 bit build, I got:
> [exec] gcc -DHAVE_CONFIG_H -I.
> -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native -I./impl
> -I/usr/java/default/include -I/usr/java/default/include/linux
> -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
> -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -m64 -MT
> impl/lzo/LzoCompressor.lo -MD -MP -MF impl/lzo/.deps/LzoCompressor.Tpo -c
>
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
> -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
> [exec]
>
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
> In function 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
> [exec]
>
> /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
> error: expected expression before ',' token
> [exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1
>
> BUILD FAILED
> /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251: exec returned: 2
>
> Has anybody seen the above ?
>
> Thanks
>
> On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > Hi Ted,
> >
> > You need to install liblzo from EPEL:
> >
> >
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
> >
> > -Todd
> >
> > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Can someone tell me how I can install liblzo ?
> > >
> > > [root@tyu-linux lzo-2.03]# uname -a
> > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT 2009
> > > x86_64 x86_64 x86_64 GNU/Linux
> > > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
> > > Loaded plugins: fastestmirror
> > > Loading mirror speeds from cached hostfile
> > > * base: mirrors.usc.edu
> > > * updates: mirror.san.fastserv.com
> > > * addons: centos.promopeddler.com
> > > * extras: mirrors.versaweb.com
> > > Setting up Install Process
> > > Parsing package install arguments
> > > No package liblzo-devel available.
> > > Nothing to do
> > >
> > > Thanks
> > >
> > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <ku...@gmail.com>
> wrote:
> > >
> > > > Ted,
> > > >
> > > > You may want to consider LZO compression, which allows splitting for
> a
> > > > comporessed file for Map jobs. On the other hand, gzip is not
> > > splittable.
> > > >
> > > > Check out these links.
> > > >
> > > >
> > > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > > http://wiki.apache.org/hadoop/UsingLzoCompression
> > > >
> > > >
> > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > The input file is in .gz format
> > > > > FYI
> > > > >
> > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > > > My current project processes input file of size 333302161 bytes.
> > > > > > What I plan to do is to split the file into equal size pieces
> (and
> > on
> > > > > blank
> > > > > > line boundary) to improve performance.
> > > > > >
> > > > > > I found 12 classes in 0.20.1 source code which implement
> > InputSplit.
> > > > > >
> > > > > > If someone has written code similar to what I plan to do, please
> > > share
> > > > > some
> > > > > > hint.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > >
> > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <
> amogh@yahoo-inc.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >> The deprecation is due to the new evolving mapreduce (
> > > o.a.h.mapreduce
> > > > )
> > > > > >> APIs. Old APIs are supported for available distributions. The
> > > > equivalent
> > > > > of
> > > > > >> TextInputFormat is available in new API :
> > > > > >>
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Amogh
> > > > > >>
> > > > > >>
> > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com> wrote:
> > > > > >>
> > > > > >> According to:
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > > >>
> > > > > >> isSplitable() is deprecated.
> > > > > >>
> > > > > >> Which method should I use to replace it ?
> > > > > >>
> > > > > >> Thanks
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Ted Yu <yu...@gmail.com>.
I followed http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and latest version
Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT 2009
x86_64 x86_64 x86_64 GNU/Linux
For 32 bit build, I got:
[exec] checking for C compiler default output file name...
[exec] configure: error: C compiler cannot create executables
[exec] See `config.log' for more details.
BUILD FAILED
/home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec returned: 77
For 64 bit build, I got:
[exec] gcc -DHAVE_CONFIG_H -I.
-I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native -I./impl
-I/usr/java/default/include -I/usr/java/default/include/linux
-I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl
-Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -m64 -MT
impl/lzo/LzoCompressor.lo -MD -MP -MF impl/lzo/.deps/LzoCompressor.Tpo -c
/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c
-fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
[exec]
/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:
In function 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs':
[exec]
/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123:
error: expected expression before ',' token
[exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1
BUILD FAILED
/home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251: exec returned: 2
Has anybody seen the above ?
Thanks
On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <to...@cloudera.com> wrote:
> Hi Ted,
>
> You need to install liblzo from EPEL:
>
> http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
>
> -Todd
>
> On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Can someone tell me how I can install liblzo ?
> >
> > [root@tyu-linux lzo-2.03]# uname -a
> > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT 2009
> > x86_64 x86_64 x86_64 GNU/Linux
> > [root@tyu-linux lzo-2.03]# yum install liblzo-devel
> > Loaded plugins: fastestmirror
> > Loading mirror speeds from cached hostfile
> > * base: mirrors.usc.edu
> > * updates: mirror.san.fastserv.com
> > * addons: centos.promopeddler.com
> > * extras: mirrors.versaweb.com
> > Setting up Install Process
> > Parsing package install arguments
> > No package liblzo-devel available.
> > Nothing to do
> >
> > Thanks
> >
> > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <ku...@gmail.com> wrote:
> >
> > > Ted,
> > >
> > > You may want to consider LZO compression, which allows splitting for a
> > > comporessed file for Map jobs. On the other hand, gzip is not
> > splittable.
> > >
> > > Check out these links.
> > >
> > >
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > > http://wiki.apache.org/hadoop/UsingLzoCompression
> > >
> > >
> > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > The input file is in .gz format
> > > > FYI
> > > >
> > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > My current project processes input file of size 333302161 bytes.
> > > > > What I plan to do is to split the file into equal size pieces (and
> on
> > > > blank
> > > > > line boundary) to improve performance.
> > > > >
> > > > > I found 12 classes in 0.20.1 source code which implement
> InputSplit.
> > > > >
> > > > > If someone has written code similar to what I plan to do, please
> > share
> > > > some
> > > > > hint.
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <amogh@yahoo-inc.com
> >
> > > > wrote:
> > > > >
> > > > >> Hi,
> > > > >> The deprecation is due to the new evolving mapreduce (
> > o.a.h.mapreduce
> > > )
> > > > >> APIs. Old APIs are supported for available distributions. The
> > > equivalent
> > > > of
> > > > >> TextInputFormat is available in new API :
> > > > >>
> > > > >>
> > > > >>
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > > >>
> > > > >> Thanks,
> > > > >> Amogh
> > > > >>
> > > > >>
> > > > >> On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com> wrote:
> > > > >>
> > > > >> According to:
> > > > >>
> > > > >>
> > > >
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > > >>
> > > > >> isSplitable() is deprecated.
> > > > >>
> > > > >> Which method should I use to replace it ?
> > > > >>
> > > > >> Thanks
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Ted,
You need to install liblzo from EPEL:
http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html
-Todd
On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <yu...@gmail.com> wrote:
> Can someone tell me how I can install liblzo ?
>
> [root@tyu-linux lzo-2.03]# uname -a
> Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT 2009
> x86_64 x86_64 x86_64 GNU/Linux
> [root@tyu-linux lzo-2.03]# yum install liblzo-devel
> Loaded plugins: fastestmirror
> Loading mirror speeds from cached hostfile
> * base: mirrors.usc.edu
> * updates: mirror.san.fastserv.com
> * addons: centos.promopeddler.com
> * extras: mirrors.versaweb.com
> Setting up Install Process
> Parsing package install arguments
> No package liblzo-devel available.
> Nothing to do
>
> Thanks
>
> On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <ku...@gmail.com> wrote:
>
> > Ted,
> >
> > You may want to consider LZO compression, which allows splitting for a
> > comporessed file for Map jobs. On the other hand, gzip is not
> splittable.
> >
> > Check out these links.
> >
> >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> > http://wiki.apache.org/hadoop/UsingLzoCompression
> >
> >
> > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > The input file is in .gz format
> > > FYI
> > >
> > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > My current project processes input file of size 333302161 bytes.
> > > > What I plan to do is to split the file into equal size pieces (and on
> > > blank
> > > > line boundary) to improve performance.
> > > >
> > > > I found 12 classes in 0.20.1 source code which implement InputSplit.
> > > >
> > > > If someone has written code similar to what I plan to do, please
> share
> > > some
> > > > hint.
> > > >
> > > > Thanks
> > > >
> > > >
> > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <am...@yahoo-inc.com>
> > > wrote:
> > > >
> > > >> Hi,
> > > >> The deprecation is due to the new evolving mapreduce (
> o.a.h.mapreduce
> > )
> > > >> APIs. Old APIs are supported for available distributions. The
> > equivalent
> > > of
> > > >> TextInputFormat is available in new API :
> > > >>
> > > >>
> > > >>
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > > >>
> > > >> Thanks,
> > > >> Amogh
> > > >>
> > > >>
> > > >> On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com> wrote:
> > > >>
> > > >> According to:
> > > >>
> > > >>
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > > >>
> > > >> isSplitable() is deprecated.
> > > >>
> > > >> Which method should I use to replace it ?
> > > >>
> > > >> Thanks
> > > >>
> > > >>
> > > >
> > >
> >
>
Re: isSplitable() deprecated
Posted by Ted Yu <yu...@gmail.com>.
Can someone tell me how I can install liblzo ?
[root@tyu-linux lzo-2.03]# uname -a
Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT 2009
x86_64 x86_64 x86_64 GNU/Linux
[root@tyu-linux lzo-2.03]# yum install liblzo-devel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.usc.edu
* updates: mirror.san.fastserv.com
* addons: centos.promopeddler.com
* extras: mirrors.versaweb.com
Setting up Install Process
Parsing package install arguments
No package liblzo-devel available.
Nothing to do
Thanks
On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <ku...@gmail.com> wrote:
> Ted,
>
> You may want to consider LZO compression, which allows splitting for a
> comporessed file for Map jobs. On the other hand, gzip is not splittable.
>
> Check out these links.
>
>
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
> http://wiki.apache.org/hadoop/UsingLzoCompression
>
>
> On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > The input file is in .gz format
> > FYI
> >
> > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > My current project processes input file of size 333302161 bytes.
> > > What I plan to do is to split the file into equal size pieces (and on
> > blank
> > > line boundary) to improve performance.
> > >
> > > I found 12 classes in 0.20.1 source code which implement InputSplit.
> > >
> > > If someone has written code similar to what I plan to do, please share
> > some
> > > hint.
> > >
> > > Thanks
> > >
> > >
> > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <am...@yahoo-inc.com>
> > wrote:
> > >
> > >> Hi,
> > >> The deprecation is due to the new evolving mapreduce ( o.a.h.mapreduce
> )
> > >> APIs. Old APIs are supported for available distributions. The
> equivalent
> > of
> > >> TextInputFormat is available in new API :
> > >>
> > >>
> > >>
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> > >>
> > >> Thanks,
> > >> Amogh
> > >>
> > >>
> > >> On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com> wrote:
> > >>
> > >> According to:
> > >>
> > >>
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> > >>
> > >> isSplitable() is deprecated.
> > >>
> > >> Which method should I use to replace it ?
> > >>
> > >> Thanks
> > >>
> > >>
> > >
> >
>
Re: isSplitable() deprecated
Posted by Steve Kuo <ku...@gmail.com>.
Ted,
You may want to consider LZO compression, which allows splitting for a
comporessed file for Map jobs. On the other hand, gzip is not splittable.
Check out these links.
http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/
http://wiki.apache.org/hadoop/UsingLzoCompression
On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <yu...@gmail.com> wrote:
> The input file is in .gz format
> FYI
>
> On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > My current project processes input file of size 333302161 bytes.
> > What I plan to do is to split the file into equal size pieces (and on
> blank
> > line boundary) to improve performance.
> >
> > I found 12 classes in 0.20.1 source code which implement InputSplit.
> >
> > If someone has written code similar to what I plan to do, please share
> some
> > hint.
> >
> > Thanks
> >
> >
> > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <am...@yahoo-inc.com>
> wrote:
> >
> >> Hi,
> >> The deprecation is due to the new evolving mapreduce ( o.a.h.mapreduce )
> >> APIs. Old APIs are supported for available distributions. The equivalent
> of
> >> TextInputFormat is available in new API :
> >>
> >>
> >>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
> >>
> >> Thanks,
> >> Amogh
> >>
> >>
> >> On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com> wrote:
> >>
> >> According to:
> >>
> >>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
> >>
> >> isSplitable() is deprecated.
> >>
> >> Which method should I use to replace it ?
> >>
> >> Thanks
> >>
> >>
> >
>
Re: isSplitable() deprecated
Posted by Ted Yu <yu...@gmail.com>.
The input file is in .gz format
FYI
On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <yu...@gmail.com> wrote:
> My current project processes input file of size 333302161 bytes.
> What I plan to do is to split the file into equal size pieces (and on blank
> line boundary) to improve performance.
>
> I found 12 classes in 0.20.1 source code which implement InputSplit.
>
> If someone has written code similar to what I plan to do, please share some
> hint.
>
> Thanks
>
>
> On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <am...@yahoo-inc.com> wrote:
>
>> Hi,
>> The deprecation is due to the new evolving mapreduce ( o.a.h.mapreduce )
>> APIs. Old APIs are supported for available distributions. The equivalent of
>> TextInputFormat is available in new API :
>>
>>
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
>>
>> Thanks,
>> Amogh
>>
>>
>> On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com> wrote:
>>
>> According to:
>>
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
>>
>> isSplitable() is deprecated.
>>
>> Which method should I use to replace it ?
>>
>> Thanks
>>
>>
>
Re: isSplitable() deprecated
Posted by Ted Yu <yu...@gmail.com>.
My current project processes input file of size 333302161 bytes.
What I plan to do is to split the file into equal size pieces (and on blank
line boundary) to improve performance.
I found 12 classes in 0.20.1 source code which implement InputSplit.
If someone has written code similar to what I plan to do, please share some
hint.
Thanks
On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar <am...@yahoo-inc.com> wrote:
> Hi,
> The deprecation is due to the new evolving mapreduce ( o.a.h.mapreduce )
> APIs. Old APIs are supported for available distributions. The equivalent of
> TextInputFormat is available in new API :
>
>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
>
> Thanks,
> Amogh
>
>
> On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com> wrote:
>
> According to:
>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
>
> isSplitable() is deprecated.
>
> Which method should I use to replace it ?
>
> Thanks
>
>
Re: isSplitable() deprecated
Posted by Amogh Vasekar <am...@yahoo-inc.com>.
Hi,
The deprecation is due to the new evolving mapreduce ( o.a.h.mapreduce ) APIs. Old APIs are supported for available distributions. The equivalent of TextInputFormat is available in new API :
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
Thanks,
Amogh
On 1/8/10 3:47 AM, "Ted Yu" <yu...@gmail.com> wrote:
According to:
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29
isSplitable() is deprecated.
Which method should I use to replace it ?
Thanks