You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by sonali <so...@cybage.com> on 2010/07/15 12:45:13 UTC
LZO Compression
Hey all,
I am working on LZO compression for improving MapREduce performance,
I just want to know where can i find following files :
| 1. hadoop-lzo-lib.jar|
| 2. hadoop-lzo-native-libs
3. ||standard-hadoop-native-libs
Thanks and regards,
Sonali
|
"Legal Disclaimer: This electronic message and all contents contain information from Cybage Software Private Limited which may be privileged, confidential, or otherwise protected from disclosure. The information is intended to be for the addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use of the contents of this message is strictly prohibited. If you have received this electronic message in error please notify the sender by reply e-mail to and destroy the original message and all copies. Cybage has taken every reasonable precaution to minimize the risk of malicious content in the mail, but is not liable for any damage you may sustain as a result of any malicious content in this e-mail. You should carry out your own malicious content checks before opening the e-mail or attachment."
www.cybage.com
Re: Preferred Java version
Posted by Eric Sammer <es...@cloudera.com>.
Raymond:
Both _16 and _20 are heavily used in production and seem to be very stable.
I don't know that we have enough information about _21 yet to say how that
does but it's worth testing if you have some extra time.
On Fri, Jul 30, 2010 at 10:54 PM, Otis Gospodnetic <
otis_gospodnetic@yahoo.com> wrote:
> I don't have the answer, but I know there is a _21 update available, and
> includes some HotSpot fixes.
>
>
> Otis----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Hadoop ecosystem search :: http://search-hadoop.com/
>
>
>
> ----- Original Message ----
> > From: Raymond Jennings III <ra...@yahoo.com>
> > To: common-user@hadoop.apache.org
> > Sent: Fri, July 16, 2010 12:44:03 PM
> > Subject: Preferred Java version
> >
> > Is 1.6.0_17 or 1.6.0_20 preferred as the JRE for hadoop? Thank you.
> >
> >
> >
> >
> >
>
--
Eric Sammer
twitter: esammer
data: www.cloudera.com
Re: Preferred Java version
Posted by Otis Gospodnetic <ot...@yahoo.com>.
I don't have the answer, but I know there is a _21 update available, and
includes some HotSpot fixes.
Otis----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Hadoop ecosystem search :: http://search-hadoop.com/
----- Original Message ----
> From: Raymond Jennings III <ra...@yahoo.com>
> To: common-user@hadoop.apache.org
> Sent: Fri, July 16, 2010 12:44:03 PM
> Subject: Preferred Java version
>
> Is 1.6.0_17 or 1.6.0_20 preferred as the JRE for hadoop? Thank you.
>
>
>
>
>
Preferred Java version
Posted by Raymond Jennings III <ra...@yahoo.com>.
Is 1.6.0_17 or 1.6.0_20 preferred as the JRE for hadoop? Thank you.
Re: WritableComparable question
Posted by Owen O'Malley <om...@apache.org>.
On Jul 19, 2010, at 2:15 PM, Raymond Jennings III wrote:
> The only way I could fix this was to re-initialize my vectors in the
> "public
> void readFields(DataInput in)" method. This does not seem like I
> should have to
> do this or do I ???
Yes, readFields has to clear the data structures. MapReduce reuses
objects in the loops.
-- Owen
Re: WritableComparable question
Posted by James Seigel <ja...@tynt.com>.
It has to reuse or the object creation would be killer!
Cheers
James
Sent from my mobile. Please excuse the typos.
On 2010-07-19, at 3:16 PM, Raymond Jennings III <ra...@yahoo.com> wrote:
> I am trying to convert some MR programs that use Text only to instead use some
> custom classes. One of my classes has a Vector type and I found that the vector
> grows with each call to my reducer such that the last call to the reducer has
> every value within the Vector from all of the classes that use this vector
> type. Does MR reuse objects?
>
> The only way I could fix this was to re-initialize my vectors in the "public
> void readFields(DataInput in)" method. This does not seem like I should have to
> do this or do I ???
>
> Thanks,
> Ray
>
>
>
>
WritableComparable question
Posted by Raymond Jennings III <ra...@yahoo.com>.
I am trying to convert some MR programs that use Text only to instead use some
custom classes. One of my classes has a Vector type and I found that the vector
grows with each call to my reducer such that the last call to the reducer has
every value within the Vector from all of the classes that use this vector
type. Does MR reuse objects?
The only way I could fix this was to re-initialize my vectors in the "public
void readFields(DataInput in)" method. This does not seem like I should have to
do this or do I ???
Thanks,
Ray
Is hdfs reliable? Very odd error
Posted by Raymond Jennings III <ra...@yahoo.com>.
I copied a 230GB file into my hadoop cluster. After my MR job kept failing I
tracked down the error to one line of formatted text.
I copied the file back out of hdfs and when I compare it to the original file
there are about 20 bytes on one line (out of 230GB) that are different.
Is there no CRC or checksum done when copying files into hdfs?
(Just to be clear, I copied the original file out of hdfs - not the output of my
MR job.)
Re: LZO Compression
Posted by Harsh J <qw...@gmail.com>.
On Thu, Jul 15, 2010 at 4:15 PM, sonali <so...@cybage.com> wrote:
> Hey all,
> I am working on LZO compression for improving MapREduce performance,
> I just want to know where can i find following files :
> | 1. hadoop-lzo-lib.jar|
> | 2. hadoop-lzo-native-libs
These don't come with Apache Hadoop anymore. Refer to
http://github.com/kevinweil/hadoop-lzo (README file) for a clean guide
on setting up LZO.
>
> 3. ||standard-hadoop-native-libs
These exist at an appropriate (architecture dependent) subdirectory
under $HADOOP_HOME/libs/native/
>
>
> Thanks and regards,
> Sonali
>
> |
>
>
>
>
>
> "Legal Disclaimer: This electronic message and all contents contain
> information from Cybage Software Private Limited which may be privileged,
> confidential, or otherwise protected from disclosure. The information is
> intended to be for the addressee(s) only. If you are not an addressee, any
> disclosure, copy, distribution, or use of the contents of this message is
> strictly prohibited. If you have received this electronic message in error
> please notify the sender by reply e-mail to and destroy the original message
> and all copies. Cybage has taken every reasonable precaution to minimize the
> risk of malicious content in the mail, but is not liable for any damage you
> may sustain as a result of any malicious content in this e-mail. You should
> carry out your own malicious content checks before opening the e-mail or
> attachment." www.cybage.com
>
>
--
Harsh J
www.harshj.com