You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2014/02/03 09:24:10 UTC
[jira] [Commented] (TEZ-789) Setting intermediate output
compression does not work
[ https://issues.apache.org/jira/browse/TEZ-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889301#comment-13889301 ]
Rohini Palaniswamy commented on TEZ-789:
----------------------------------------
Note: This was with TEZ-711-5.patch
Exception in the task logs:
java.io.IOException: Premature EOF from inputStream
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194) at org.apache.tez.runtime.library.common.shuffle.impl.Fetcher.shuffleToMemory(Fetcher.java:640)
at org.apache.tez.runtime.library.common.shuffle.impl.Fetcher.copyMapOutput(Fetcher.java:463) at org.apache.tez.runtime.library.common.shuffle.impl.Fetcher.copyFromHost(Fetcher.java:377)
at org.apache.tez.runtime.library.common.shuffle.impl.Fetcher.run(Fetcher.java:183)
2014-02-02 21:26:53,045 INFO [fetcher#5] org.apache.tez.runtime.library.common.shuffle.impl.Fetcher: fetcher#5 failed to read map headernull decomp: -1, -1
java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:183)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152)
at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:174)
at org.apache.http.impl.io.IdentityInputStream.read(IdentityInputStream.java:86)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:120)
at java.io.DataInputStream.readByte(DataInputStream.java:265)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
at org.apache.hadoop.io.WritableUtils.readStringSafely(WritableUtils.java:475)
at org.apache.tez.runtime.library.common.shuffle.impl.ShuffleHeader.readFields(ShuffleHeader.java:82)
at org.apache.tez.runtime.library.common.shuffle.impl.Fetcher.copyMapOutput(Fetcher.java:417)
at org.apache.tez.runtime.library.common.shuffle.impl.Fetcher.copyFromHost(Fetcher.java:377)
at org.apache.tez.runtime.library.common.shuffle.impl.Fetcher.run(Fetcher.java:183)
Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
at org.apache.tez.runtime.library.common.shuffle.impl.ShuffleScheduler.checkReducerHealth(ShuffleScheduler.java:284)
at org.apache.tez.runtime.library.common.shuffle.impl.ShuffleScheduler.copyFailed(ShuffleScheduler.java:206)
at org.apache.tez.runtime.library.common.shuffle.impl.Fetcher.copyFromHost(Fetcher.java:383)
at org.apache.tez.runtime.library.common.shuffle.impl.Fetcher.run(Fetcher.java:183)
> Setting intermediate output compression does not work
> -----------------------------------------------------
>
> Key: TEZ-789
> URL: https://issues.apache.org/jira/browse/TEZ-789
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.3.0
> Reporter: Rohini Palaniswamy
>
> Setting tez.runtime.intermediate-output.should-compress to true and tez.runtime.intermediate-output.compress.codec to com.hadoop.compression.lzo.LzoCodec did not work at all. Setting org.apache.hadoop.io.compress.DefaultCodec most tasks succeeded, but still some failed.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)