You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Siddharth Seth (JIRA)" <ji...@apache.org> on 2014/11/25 00:55:12 UTC
[jira] [Commented] (TEZ-1800) Integer overflow in
ExternalSorter.getInitialMemoryRequirement()
[ https://issues.apache.org/jira/browse/TEZ-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223787#comment-14223787 ]
Siddharth Seth commented on TEZ-1800:
-------------------------------------
+1.
> Integer overflow in ExternalSorter.getInitialMemoryRequirement()
> ----------------------------------------------------------------
>
> Key: TEZ-1800
> URL: https://issues.apache.org/jira/browse/TEZ-1800
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Attachments: TEZ-1800.1.patch
>
>
> When trying to implement support for tez.runtime.io.sort.mb > 2 GB in pipelinedsorter, I encountered integer overflow exception in ExternalSorter.
> Exception thrown is given below.
> {code}
> ], TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.IllegalArgumentException
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
> at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:158)
> at org.apache.tez.runtime.common.resources.MemoryDistributor.requestMemory(MemoryDistributor.java:97)
> at org.apache.tez.runtime.api.impl.TezTaskContextImpl.requestInitialMemory(TezTaskContextImpl.java:192)
> at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.initialize(OrderedPartitionedKVOutput.java:95)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.call(LogicalIOProcessorRuntimeTask.java:430)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.call(LogicalIOProcessorRuntimeTask.java:409)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Actual place where the integer overflow happens is in ExternalSorter.getInitialMemoryRequirement()
> {code}
> int initialMemRequestMb =
> conf.getInt(
> TezRuntimeConfiguration.TEZ_RUNTIME_IO_SORT_MB,
> TezRuntimeConfiguration.TEZ_RUNTIME_IO_SORT_MB_DEFAULT);
> long reqBytes = initialMemRequestMb << 20;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)