You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Christian Kunz <ck...@yahoo-inc.com> on 2008/11/01 04:01:10 UTC

Archives larger thatn 2^31 bytes in DistributedCache

In hadoop-0.17
we tried to use a 2.2GB archive and seemingly ran into
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6599383:

java.util.zip.ZipException: error in opening zip file
    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.(ZipFile.java:114)
    at java.util.zip.ZipFile.(ZipFile.java:131)
    at org.apache.hadoop.fs.FileUtil.unZip(FileUtil.java:421)
    at 
org.apache.hadoop.filecache.DistributedCache.localizeCache(DistributedCache.
java:338)
    at 
org.apache.hadoop.filecache.DistributedCache.getLocalCache(DistributedCache.
java:161)
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:137)

Any work around known for hadoop-0.17 (except multiple smaller archives)?

Is it correct to assume that in hadoop-0.18 this is no longer an issue when
using tar.gz?

Thanks,
Christian