Hadoop native compression libs [FreeBSD-specific] - Revisited

[Sean Dean <se...@rogers.com>]

I recently received a private email from a Nutch user who had troubles compiling the latest Hadoop native compression libraries due to the recent addition of lzo2 compression support. It seems my previous method of compiling for FreeBSD is somewhat outdated now, but the good news is there is a solution and it allows us (FreeBSD users) to use lzo2 support!
 
Many of the same procedures will need to be done, since it still includes zlib support. For the purpose of saving some space in this email you can refer to this site for the archived message, including instructions on what needs to be done for the first half of the operation: http://www.mail-archive.com/nutch-user@lucene.apache.org/msg06839.html.
 
We will need to install another port, this includes the library and headers for lzo2 support on FreeBSD:
 
cd /usr/ports/archivers/lzo2/ && make install clean
 
After this is installed we need to copy the installed headers and library files into other directories:
 
cp -R /usr/local/include/lzo /usr/include/
cp /usr/local/lib/liblzo2.* /usr/lib/
 
Now its time to run through all steps in my previous email (the one linked above). You will need to copy one more file in addition to all the others, just run this command:
 
cp src/org_apache_hadoop.h src/org/apache/hadoop/io/compress/lzo/
 
Now I haven't actually been able to test these new Hadoop native libraries in production, but I'm personally expecting them to work fine. If you find any problems, either during compilation or run-time use then let me know and we can work out the bugs.
 
 
Sean Dean