Hadoop / Hive Error when trying to load data to table that had Lzo compression in it.

[pradeep T <it...@gmail.com>]

Hi All,

SET hive.exec.compress.output=true;
SET io.seqfile.compression.type=BLOCK;
SET mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec;
set mapred.job.priority=VERY_HIGH;
set mapred.job.name=loading data from YYY to XXXXXX;

insert overwrite table XXXXXXXXX partition (pixeldate, pixelhr, pixelmin,
reportts, clustertype, logStatus, reportgroup, requesttype, clicktype,
plugintype) select key, ipaddress, timezone,  url, responsecode,
responsesize, referrer, useragent, decodedurl, domain , specificdomain,
sourceurl, destinationurl, destinationdomain, cid, cpc, rpc,
geocountry,geoDefault,userid, sessionid, requestid, pagetypeid, articleid,
keywords, responsetime, widgetid, pagetype, sourceIP, geocity, isnewuser,
errormessage, logValue, contractstatus, pagetypederived, domainid, noofads,
noofrelatedposts, adopt, addomain, prid, topposition, leftposition,
requestTimestamp, occurenceFreq, suspectStatus, isbillable, platform,
to_date(logdate) pixeldate, hour(logdate) as pixelhr, (hour(logdate) * 60)
+ (floor(minute(logdate) / 15) * 15) as pixelmin, reportts, clustertype,
logStatus,  reportgroup, requesttype, clicktype, plugintype from YYYYYYY
where reportts=20130619042507 and clustertype = 'DEFAULT'  and
reportdate='2013-06-19' and logstatus in('VALID','Invalid','Error')


When I run this query. I am seeing this error in the tasktracker log.


In task tracker System Logs we are seeing the following errors.

In ERRORS :

We see "*Error:
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.getRemaining(J)I*"

*syslog logs*

2013-07-15 05:39:41,239 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library
2013-07-15 05:39:41,241 ERROR org.apache.hadoop.io.nativeio.NativeIO:
*Unable to initialize NativeIO libraries*
java.lang.NoSuchFieldError: workaroundNonThreadSafePasswdCalls
	at org.apache.hadoop.io.nativeio.NativeIO.initNative(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO.<clinit>(NativeIO.java:89)
	at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:653)
	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)


Can someone please throw light in to it.


Some added Info :

I have upgraded Hadoop from .20.2 to 1.1.2

and hive from .7.0 to .11.

After Upgrade I faced lot of Jar file issues and cleared each with lot
of search in the internet.

But this haunting us for few days. Any help would be grateful.


Thanks,

Pradeep

Re: Hadoop / Hive Error when trying to load data to table that had Lzo compression in it.

[李洪忠 <lh...@hotmail.com>]

you need to recompile the hadoop-lzo jar and so from 0.20.x to 1.x.
于 2013/7/15 18:55, pradeep T 写道:
> After Upgrade I faced lot of Jar file issues and cleared each with lot of search in the internet.

Re: Hadoop / Hive Error when trying to load data to table that had Lzo compression in it.

[李洪忠 <lh...@hotmail.com>]

you need to recompile the hadoop-lzo jar and so from 0.20.x to 1.x.
于 2013/7/15 18:55, pradeep T 写道:
> After Upgrade I faced lot of Jar file issues and cleared each with lot of search in the internet.

Re: Hadoop / Hive Error when trying to load data to table that had Lzo compression in it.

[李洪忠 <lh...@hotmail.com>]

you need to recompile the hadoop-lzo jar and so from 0.20.x to 1.x.
于 2013/7/15 18:55, pradeep T 写道:
> After Upgrade I faced lot of Jar file issues and cleared each with lot of search in the internet.

Re: Hadoop / Hive Error when trying to load data to table that had Lzo compression in it.

[李洪忠 <lh...@hotmail.com>]

you need to recompile the hadoop-lzo jar and so from 0.20.x to 1.x.
于 2013/7/15 18:55, pradeep T 写道:
> After Upgrade I faced lot of Jar file issues and cleared each with lot of search in the internet.