You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Kris Jirapinyo <kj...@biz360.com> on 2009/06/03 20:11:17 UTC
Fastlz coming?
Hi all,
In the remove lzo JIRA ticket
https://issues.apache.org/jira/browse/HADOOP-4874 Tatu mentioned he was
going to port fastlz from C to Java and provide a patch. Has there been any
updates on that? Or is anyone working on any additional custom compression
codecs?
Thanks,
Kris J.
Re: Fastlz coming?
Posted by Hong Tang <ht...@yahoo-inc.com>.
Using com.hadoop.compression.lzo.LzoCodec is not much different from
using other codecs: adding the hadoop-gpl-compression-0.1.0-dev.jar in
your classpath, and add the path to the native library
libgplcompression.so in system property java.library.path.
Hope this helps, Hong
On Jun 4, 2009, at 10:29 AM, Kris Jirapinyo wrote:
> Is there any documentation on that site on how we can use lzo? I
> don't see
> any entries on the wiki page of the project. I see an entry on the
> Hadoop
> wiki (http://wiki.apache.org/hadoop/UsingLzoCompression) but seems
> like
> that's more oriented towards HBase. I am on hadoop 0.19.1.
>
> Thanks,
> Kris J.
>
> On Thu, Jun 4, 2009 at 3:02 AM, Johan Oskarsson <jo...@oskarsson.nu>
> wrote:
>
>> We're using Lzo still, works great for those big log files:
>> http://code.google.com/p/hadoop-gpl-compression/
>>
>> /Johan
>>
>> Kris Jirapinyo wrote:
>>> Hi all,
>>> In the remove lzo JIRA ticket
>>> https://issues.apache.org/jira/browse/HADOOP-4874 Tatu mentioned
>>> he was
>>> going to port fastlz from C to Java and provide a patch. Has
>>> there been
>> any
>>> updates on that? Or is anyone working on any additional custom
>> compression
>>> codecs?
>>>
>>> Thanks,
>>> Kris J.
>>>
>>
>>
Re: Fastlz coming?
Posted by Owen O'Malley <om...@apache.org>.
On Jun 4, 2009, at 11:19 AM, Kris Jirapinyo wrote:
> Hopefully we can have a new page on the hadoop wiki on how to
> use custom compression so that people won't have to go search
> through the
> threads to find the answer in the future.
Yes, it would be extremely useful if you could start a wiki page with
your discoveries. Since you had to discover it, it will be clear to
you what needs to be documented.
On a similar note, I always find that I write the best comments in
code after I leave a section for 6 months and have to figure out what
I meant when I wrote it. Otherwise, it is too obvious. *smile*
-- Owen
Re: Fastlz coming?
Posted by Kris Jirapinyo <kr...@biz360.com>.
Thanks Matt. Hopefully we can have a new page on the hadoop wiki on how to
use custom compression so that people won't have to go search through the
threads to find the answer in the future.
On Thu, Jun 4, 2009 at 10:33 AM, Matt Massie <ma...@cloudera.com> wrote:
> Kris-
>
> You might take a look at some of the previous lzo threads on this list for
> help.
>
> See:
> http://www.mail-archive.com/search?q=lzo&l=core-user%40hadoop.apache.org
>
> -Matt
>
>
> On Jun 4, 2009, at 10:29 AM, Kris Jirapinyo wrote:
>
> Is there any documentation on that site on how we can use lzo? I don't
>> see
>> any entries on the wiki page of the project. I see an entry on the Hadoop
>> wiki (http://wiki.apache.org/hadoop/UsingLzoCompression) but seems like
>> that's more oriented towards HBase. I am on hadoop 0.19.1.
>>
>> Thanks,
>> Kris J.
>>
>> On Thu, Jun 4, 2009 at 3:02 AM, Johan Oskarsson <jo...@oskarsson.nu>
>> wrote:
>>
>> We're using Lzo still, works great for those big log files:
>>> http://code.google.com/p/hadoop-gpl-compression/
>>>
>>> /Johan
>>>
>>> Kris Jirapinyo wrote:
>>>
>>>> Hi all,
>>>> In the remove lzo JIRA ticket
>>>> https://issues.apache.org/jira/browse/HADOOP-4874 Tatu mentioned he was
>>>> going to port fastlz from C to Java and provide a patch. Has there been
>>>>
>>> any
>>>
>>>> updates on that? Or is anyone working on any additional custom
>>>>
>>> compression
>>>
>>>> codecs?
>>>>
>>>> Thanks,
>>>> Kris J.
>>>>
>>>>
>>>
>>>
>
Re: Fastlz coming?
Posted by Matt Massie <ma...@cloudera.com>.
Kris-
You might take a look at some of the previous lzo threads on this list
for help.
See: http://www.mail-archive.com/search?q=lzo&l=core-user%40hadoop.apache.org
-Matt
On Jun 4, 2009, at 10:29 AM, Kris Jirapinyo wrote:
> Is there any documentation on that site on how we can use lzo? I
> don't see
> any entries on the wiki page of the project. I see an entry on the
> Hadoop
> wiki (http://wiki.apache.org/hadoop/UsingLzoCompression) but seems
> like
> that's more oriented towards HBase. I am on hadoop 0.19.1.
>
> Thanks,
> Kris J.
>
> On Thu, Jun 4, 2009 at 3:02 AM, Johan Oskarsson <jo...@oskarsson.nu>
> wrote:
>
>> We're using Lzo still, works great for those big log files:
>> http://code.google.com/p/hadoop-gpl-compression/
>>
>> /Johan
>>
>> Kris Jirapinyo wrote:
>>> Hi all,
>>> In the remove lzo JIRA ticket
>>> https://issues.apache.org/jira/browse/HADOOP-4874 Tatu mentioned
>>> he was
>>> going to port fastlz from C to Java and provide a patch. Has
>>> there been
>> any
>>> updates on that? Or is anyone working on any additional custom
>> compression
>>> codecs?
>>>
>>> Thanks,
>>> Kris J.
>>>
>>
>>
Re: Fastlz coming?
Posted by Kris Jirapinyo <kr...@biz360.com>.
Is there any documentation on that site on how we can use lzo? I don't see
any entries on the wiki page of the project. I see an entry on the Hadoop
wiki (http://wiki.apache.org/hadoop/UsingLzoCompression) but seems like
that's more oriented towards HBase. I am on hadoop 0.19.1.
Thanks,
Kris J.
On Thu, Jun 4, 2009 at 3:02 AM, Johan Oskarsson <jo...@oskarsson.nu> wrote:
> We're using Lzo still, works great for those big log files:
> http://code.google.com/p/hadoop-gpl-compression/
>
> /Johan
>
> Kris Jirapinyo wrote:
> > Hi all,
> > In the remove lzo JIRA ticket
> > https://issues.apache.org/jira/browse/HADOOP-4874 Tatu mentioned he was
> > going to port fastlz from C to Java and provide a patch. Has there been
> any
> > updates on that? Or is anyone working on any additional custom
> compression
> > codecs?
> >
> > Thanks,
> > Kris J.
> >
>
>
Re: Fastlz coming?
Posted by Johan Oskarsson <jo...@oskarsson.nu>.
We're using Lzo still, works great for those big log files:
http://code.google.com/p/hadoop-gpl-compression/
/Johan
Kris Jirapinyo wrote:
> Hi all,
> In the remove lzo JIRA ticket
> https://issues.apache.org/jira/browse/HADOOP-4874 Tatu mentioned he was
> going to port fastlz from C to Java and provide a patch. Has there been any
> updates on that? Or is anyone working on any additional custom compression
> codecs?
>
> Thanks,
> Kris J.
>