You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Ronald Petty <ro...@gmail.com> on 2011/12/02 00:55:07 UTC
Snow Leopard Compilation Help
Hello,
I am new to Hadoop development and seem to be stuck on building with Snow
Leopard. Here is what is going on:
1. svn checkout
http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk
2. wget http://protobuf.googlecode.com/files/protobuf-2.4.1.tar.gz
3. tar ... proto...gz; cd proto...
4. ./configure --prefix=/hadoop/contribute/protobuf/;make;make install
5. export PATH=/hadoop/contribute/protobuf/bin/:$PATH
6. cd hadoop-trunk
7. mvn clean
8. mvn install -Dmaven.test.skip.exec=true
9. mvn assembly:assembly -Pnative
10. Error
[INFO] --- make-maven-plugin:1.0-beta-1:make-install (compile) @
hadoop-common ---
[INFO] /bin/sh ./libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I.
-I/Library/Java/Home/include
-I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/src
-I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/javah
-I/usr/local/include -g -Wall -fPIC -O2 -m64 -g -O2 -MT ZlibCompressor.lo
-MD -MP -MF .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f
'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo
'./'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
[INFO] libtool: compile: gcc -DHAVE_CONFIG_H -I.
-I/Library/Java/Home/include
-I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/src
-I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/javah
-I/usr/local/include -g -Wall -fPIC -O2 -m64 -g -O2 -MT ZlibCompressor.lo
-MD -MP -MF .deps/ZlibCompressor.Tpo -c
src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c -fno-common -DPIC
-o .libs/ZlibCompressor.o
[INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c: In function
'Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_initIDs':
[INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c:71: error:
'libnotfound' undeclared (first use in this function)
[INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c:71: error:
(Each undeclared identifier is reported only once
[INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c:71: error:
for each function it appears in.)
[INFO] make: *** [ZlibCompressor.lo] Error 1
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ FAILURE [46.914s]
[INFO] Apache Hadoop Project POM ......................... SKIPPED
[INFO] Apache Hadoop Annotations ......................... SKIPPED
....
I looked around and found this
http://wiki.apache.org/hadoop/UsingLzoCompression. I tried to mess with
lzo via MacPort. Seems to be there, but I am not certain where to go from
here.
Also, how do you search the mail archives (
http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/)?
Thanks for the help.
Kindest regards.
Ron
RE: Compressor setInput input permanence
Posted by Tim Broberg <Ti...@exar.com>.
Thanks for the response, Todd.
I'll crawl the trunk for compressor / decompressor references today. If you think of any other versions that should be scanned, please chime in.
- Tim.
________________________________________
From: Todd Lipcon [todd@cloudera.com]
Sent: Sunday, December 04, 2011 10:51 PM
To: common-dev@hadoop.apache.org; Tim Broberg
Subject: Re: Compressor setInput input permanence
Hi Tim,
My guess is that this contract isn't explicitly documented anywhere.
But the good news is that the set of implementors and users of this
API is fairly well contained.
I'd propose you do the following:
- Look for any dependent projects which use the Compressor API
directly. I know HBase does. I believe Avro does. Hive and Pig might.
Accumulo probably does. A google code search or github search for
"import org.apache.hadoop.io.compress" would probably give a pretty
exhaustive list.
- Throughout those, look at see if they all maintain the buffer
between setInput and compress.
- If so, file a JIRA to document this as part of the compression API
javadoc, and then we'll be more explicit about it from now on?
-Todd
On Sat, Dec 3, 2011 at 12:18 AM, Tim Broberg <tb...@yahoo.com> wrote:
> The question is, how long can a Compressor count on the user buffer to stick around after a call to setInput()?
>
> The Compressor object has a method, setInput whose inputs are an array reference, an offset and a length.
>
> I would expect that this input would no longer be guaranteed to persist after the setInput call returns.
>
> ...but in ZlibCompressor and SnappyCompressor, when there is no buffer room for len bytes, the Compressor makes a copy of the reference, offset, and length, clears the needsInput condition, and returns waiting for a call to compress() to unload the buffers through the compressor. The Compressor implementations count on the data to persist after setInput returns until compress() is called.
>
> So, the data persist after the call. Does all such data persist?
>
> In theory, could a Compressor avoid a copy by just collecting references to each input user buffer passed in and then sending all these references to the compression library when compress() is called?
>
> ...or do these user buffers get reused before that time?
>
> By keeping references to these buffers, am I preventing them from getting garbage collected and potentially soaking up large amounts of memory?
>
> Where is the persistence of the contents of these user buffers supposed to be documented?
>
> TIA,
> - Tim.
--
Todd Lipcon
Software Engineer, Cloudera
The information and any attached documents contained in this message
may be confidential and/or legally privileged. The message is
intended solely for the addressee(s). If you are not the intended
recipient, you are hereby notified that any use, dissemination, or
reproduction is strictly prohibited and may be unlawful. If you are
not the intended recipient, please contact the sender immediately by
return e-mail and destroy all copies of the original message.
Re: Compressor setInput input permanence
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Tim,
My guess is that this contract isn't explicitly documented anywhere.
But the good news is that the set of implementors and users of this
API is fairly well contained.
I'd propose you do the following:
- Look for any dependent projects which use the Compressor API
directly. I know HBase does. I believe Avro does. Hive and Pig might.
Accumulo probably does. A google code search or github search for
"import org.apache.hadoop.io.compress" would probably give a pretty
exhaustive list.
- Throughout those, look at see if they all maintain the buffer
between setInput and compress.
- If so, file a JIRA to document this as part of the compression API
javadoc, and then we'll be more explicit about it from now on?
-Todd
On Sat, Dec 3, 2011 at 12:18 AM, Tim Broberg <tb...@yahoo.com> wrote:
> The question is, how long can a Compressor count on the user buffer to stick around after a call to setInput()?
>
> The Compressor object has a method, setInput whose inputs are an array reference, an offset and a length.
>
> I would expect that this input would no longer be guaranteed to persist after the setInput call returns.
>
> ...but in ZlibCompressor and SnappyCompressor, when there is no buffer room for len bytes, the Compressor makes a copy of the reference, offset, and length, clears the needsInput condition, and returns waiting for a call to compress() to unload the buffers through the compressor. The Compressor implementations count on the data to persist after setInput returns until compress() is called.
>
> So, the data persist after the call. Does all such data persist?
>
> In theory, could a Compressor avoid a copy by just collecting references to each input user buffer passed in and then sending all these references to the compression library when compress() is called?
>
> ...or do these user buffers get reused before that time?
>
> By keeping references to these buffers, am I preventing them from getting garbage collected and potentially soaking up large amounts of memory?
>
> Where is the persistence of the contents of these user buffers supposed to be documented?
>
> TIA,
> - Tim.
--
Todd Lipcon
Software Engineer, Cloudera
Compressor setInput input permanence
Posted by Tim Broberg <tb...@yahoo.com>.
The question is, how long can a Compressor count on the user buffer to stick around after a call to setInput()?
The Compressor object has a method, setInput whose inputs are an array reference, an offset and a length.
I would expect that this input would no longer be guaranteed to persist after the setInput call returns.
...but in ZlibCompressor and SnappyCompressor, when there is no buffer room for len bytes, the Compressor makes a copy of the reference, offset, and length, clears the needsInput condition, and returns waiting for a call to compress() to unload the buffers through the compressor. The Compressor implementations count on the data to persist after setInput returns until compress() is called.
So, the data persist after the call. Does all such data persist?
In theory, could a Compressor avoid a copy by just collecting references to each input user buffer passed in and then sending all these references to the compression library when compress() is called?
...or do these user buffers get reused before that time?
By keeping references to these buffers, am I preventing them from getting garbage collected and potentially soaking up large amounts of memory?
Where is the persistence of the contents of these user buffers supposed to be documented?
TIA,
- Tim.
Re: Snow Leopard Compilation Help
Posted by Mi...@emc.com.
Ronald,
Please take a look at https://issues.apache.org/jira/browse/HADOOP-7147,
and https://issues.apache.org/jira/browse/HADOOP-7824
- milind
On 12/1/11 5:31 PM, "Ronald Petty" <ro...@gmail.com> wrote:
>Alejandro,
>
>I suppose I will give it a go since that is the computer I have. I tried
>searching on JIRA for issues mac related but its hard for me to tell which
>ones might be related or not. Should I just figure it out and email the
>list with my fix (if I find one?)
>
>Ron
>
>On Thu, Dec 1, 2011 at 7:23 PM, Alejandro Abdelnur
><tu...@cloudera.com>wrote:
>
>> Ron,
>>
>> Hadoop native currently does not compile in Mac OS X. There have been
>>some
>> JIRAs to fix that, but nobody took on them.
>>
>> Thanks.
>>
>> Alejandro
>>
>> On Thu, Dec 1, 2011 at 3:55 PM, Ronald Petty <ro...@gmail.com>
>> wrote:
>>
>> > Hello,
>> >
>> > I am new to Hadoop development and seem to be stuck on building with
>>Snow
>> > Leopard. Here is what is going on:
>> >
>> > 1. svn checkout
>> > http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk
>> > 2. wget http://protobuf.googlecode.com/files/protobuf-2.4.1.tar.gz
>> > 3. tar ... proto...gz; cd proto...
>> > 4. ./configure --prefix=/hadoop/contribute/protobuf/;make;make
>>install
>> > 5. export PATH=/hadoop/contribute/protobuf/bin/:$PATH
>> > 6. cd hadoop-trunk
>> > 7. mvn clean
>> > 8. mvn install -Dmaven.test.skip.exec=true
>> > 9. mvn assembly:assembly -Pnative
>> > 10. Error
>> >
>> > [INFO] --- make-maven-plugin:1.0-beta-1:make-install (compile) @
>> > hadoop-common ---
>> > [INFO] /bin/sh ./libtool --tag=CC --mode=compile gcc
>>-DHAVE_CONFIG_H
>> -I.
>> > -I/Library/Java/Home/include
>> >
>> >
>>
>>-I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/tar
>>get/native/src
>> >
>> >
>>
>>-I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/tar
>>get/native/javah
>> > -I/usr/local/include -g -Wall -fPIC -O2 -m64 -g -O2 -MT
>>ZlibCompressor.lo
>> > -MD -MP -MF .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f
>> > 'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo
>> > './'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>> > [INFO] libtool: compile: gcc -DHAVE_CONFIG_H -I.
>> > -I/Library/Java/Home/include
>> >
>> >
>>
>>-I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/tar
>>get/native/src
>> >
>> >
>>
>>-I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/tar
>>get/native/javah
>> > -I/usr/local/include -g -Wall -fPIC -O2 -m64 -g -O2 -MT
>>ZlibCompressor.lo
>> > -MD -MP -MF .deps/ZlibCompressor.Tpo -c
>> > src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c -fno-common
>> -DPIC
>> > -o .libs/ZlibCompressor.o
>> > [INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c: In
>> function
>> > 'Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_initIDs':
>> > [INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c:71:
>>error:
>> > 'libnotfound' undeclared (first use in this function)
>> > [INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c:71:
>>error:
>> > (Each undeclared identifier is reported only once
>> > [INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c:71:
>>error:
>> > for each function it appears in.)
>> > [INFO] make: *** [ZlibCompressor.lo] Error 1
>> > [INFO]
>> >
>>------------------------------------------------------------------------
>> > [INFO] Reactor Summary:
>> > [INFO]
>> > [INFO] Apache Hadoop Main ................................ FAILURE
>> > [46.914s]
>> > [INFO] Apache Hadoop Project POM ......................... SKIPPED
>> > [INFO] Apache Hadoop Annotations ......................... SKIPPED
>> >
>> > ....
>> >
>> > I looked around and found this
>> > http://wiki.apache.org/hadoop/UsingLzoCompression. I tried to mess
>>with
>> > lzo via MacPort. Seems to be there, but I am not certain where to go
>> from
>> > here.
>> >
>> > Also, how do you search the mail archives (
>> > http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/)?
>> >
>> > Thanks for the help.
>> >
>> > Kindest regards.
>> >
>> > Ron
>> >
>>
Re: Snow Leopard Compilation Help
Posted by Ronald Petty <ro...@gmail.com>.
Alejandro,
I suppose I will give it a go since that is the computer I have. I tried
searching on JIRA for issues mac related but its hard for me to tell which
ones might be related or not. Should I just figure it out and email the
list with my fix (if I find one?)
Ron
On Thu, Dec 1, 2011 at 7:23 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Ron,
>
> Hadoop native currently does not compile in Mac OS X. There have been some
> JIRAs to fix that, but nobody took on them.
>
> Thanks.
>
> Alejandro
>
> On Thu, Dec 1, 2011 at 3:55 PM, Ronald Petty <ro...@gmail.com>
> wrote:
>
> > Hello,
> >
> > I am new to Hadoop development and seem to be stuck on building with Snow
> > Leopard. Here is what is going on:
> >
> > 1. svn checkout
> > http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk
> > 2. wget http://protobuf.googlecode.com/files/protobuf-2.4.1.tar.gz
> > 3. tar ... proto...gz; cd proto...
> > 4. ./configure --prefix=/hadoop/contribute/protobuf/;make;make install
> > 5. export PATH=/hadoop/contribute/protobuf/bin/:$PATH
> > 6. cd hadoop-trunk
> > 7. mvn clean
> > 8. mvn install -Dmaven.test.skip.exec=true
> > 9. mvn assembly:assembly -Pnative
> > 10. Error
> >
> > [INFO] --- make-maven-plugin:1.0-beta-1:make-install (compile) @
> > hadoop-common ---
> > [INFO] /bin/sh ./libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H
> -I.
> > -I/Library/Java/Home/include
> >
> >
> -I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/src
> >
> >
> -I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/javah
> > -I/usr/local/include -g -Wall -fPIC -O2 -m64 -g -O2 -MT ZlibCompressor.lo
> > -MD -MP -MF .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f
> > 'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo
> > './'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
> > [INFO] libtool: compile: gcc -DHAVE_CONFIG_H -I.
> > -I/Library/Java/Home/include
> >
> >
> -I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/src
> >
> >
> -I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/javah
> > -I/usr/local/include -g -Wall -fPIC -O2 -m64 -g -O2 -MT ZlibCompressor.lo
> > -MD -MP -MF .deps/ZlibCompressor.Tpo -c
> > src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c -fno-common
> -DPIC
> > -o .libs/ZlibCompressor.o
> > [INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c: In
> function
> > 'Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_initIDs':
> > [INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c:71: error:
> > 'libnotfound' undeclared (first use in this function)
> > [INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c:71: error:
> > (Each undeclared identifier is reported only once
> > [INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c:71: error:
> > for each function it appears in.)
> > [INFO] make: *** [ZlibCompressor.lo] Error 1
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] Reactor Summary:
> > [INFO]
> > [INFO] Apache Hadoop Main ................................ FAILURE
> > [46.914s]
> > [INFO] Apache Hadoop Project POM ......................... SKIPPED
> > [INFO] Apache Hadoop Annotations ......................... SKIPPED
> >
> > ....
> >
> > I looked around and found this
> > http://wiki.apache.org/hadoop/UsingLzoCompression. I tried to mess with
> > lzo via MacPort. Seems to be there, but I am not certain where to go
> from
> > here.
> >
> > Also, how do you search the mail archives (
> > http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/)?
> >
> > Thanks for the help.
> >
> > Kindest regards.
> >
> > Ron
> >
>
Re: Snow Leopard Compilation Help
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Ron,
Hadoop native currently does not compile in Mac OS X. There have been some
JIRAs to fix that, but nobody took on them.
Thanks.
Alejandro
On Thu, Dec 1, 2011 at 3:55 PM, Ronald Petty <ro...@gmail.com> wrote:
> Hello,
>
> I am new to Hadoop development and seem to be stuck on building with Snow
> Leopard. Here is what is going on:
>
> 1. svn checkout
> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk
> 2. wget http://protobuf.googlecode.com/files/protobuf-2.4.1.tar.gz
> 3. tar ... proto...gz; cd proto...
> 4. ./configure --prefix=/hadoop/contribute/protobuf/;make;make install
> 5. export PATH=/hadoop/contribute/protobuf/bin/:$PATH
> 6. cd hadoop-trunk
> 7. mvn clean
> 8. mvn install -Dmaven.test.skip.exec=true
> 9. mvn assembly:assembly -Pnative
> 10. Error
>
> [INFO] --- make-maven-plugin:1.0-beta-1:make-install (compile) @
> hadoop-common ---
> [INFO] /bin/sh ./libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I.
> -I/Library/Java/Home/include
>
> -I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/src
>
> -I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/javah
> -I/usr/local/include -g -Wall -fPIC -O2 -m64 -g -O2 -MT ZlibCompressor.lo
> -MD -MP -MF .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f
> 'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo
> './'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
> [INFO] libtool: compile: gcc -DHAVE_CONFIG_H -I.
> -I/Library/Java/Home/include
>
> -I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/src
>
> -I/hadoop/contribute/hadoop-trunk/hadoop-common-project/hadoop-common/target/native/javah
> -I/usr/local/include -g -Wall -fPIC -O2 -m64 -g -O2 -MT ZlibCompressor.lo
> -MD -MP -MF .deps/ZlibCompressor.Tpo -c
> src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c -fno-common -DPIC
> -o .libs/ZlibCompressor.o
> [INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c: In function
> 'Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_initIDs':
> [INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c:71: error:
> 'libnotfound' undeclared (first use in this function)
> [INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c:71: error:
> (Each undeclared identifier is reported only once
> [INFO] src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c:71: error:
> for each function it appears in.)
> [INFO] make: *** [ZlibCompressor.lo] Error 1
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Hadoop Main ................................ FAILURE
> [46.914s]
> [INFO] Apache Hadoop Project POM ......................... SKIPPED
> [INFO] Apache Hadoop Annotations ......................... SKIPPED
>
> ....
>
> I looked around and found this
> http://wiki.apache.org/hadoop/UsingLzoCompression. I tried to mess with
> lzo via MacPort. Seems to be there, but I am not certain where to go from
> here.
>
> Also, how do you search the mail archives (
> http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/)?
>
> Thanks for the help.
>
> Kindest regards.
>
> Ron
>