You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:22:07 UTC
[jira] [Updated] (SPARK-12083) java.lang.IllegalArgumentException:
requirement failed: Overflowed precision (q98)
[ https://issues.apache.org/jira/browse/SPARK-12083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-12083:
---------------------------------
Labels: bulk-closed perfomance (was: perfomance)
> java.lang.IllegalArgumentException: requirement failed: Overflowed precision (q98)
> ----------------------------------------------------------------------------------
>
> Key: SPARK-12083
> URL: https://issues.apache.org/jira/browse/SPARK-12083
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0
> Environment: CentOS release 6.6
> Reporter: Dileep Kumar
> Priority: Major
> Labels: bulk-closed, perfomance
>
> While running with 10 users we found that one of the executor randomly hangs during q98 execution. The behavior is random in way that it happens at different time but for the same query. Tried to get a stack trace but was not successful in generating the stack trace.
> Here is the last exception that I saw before the hang:
> java.lang.IllegalArgumentException: requirement failed: Overflowed precision
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.sql.types.Decimal.set(Decimal.scala:111)
> at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:335)
> at org.apache.spark.sql.types.Decimal.apply(Decimal.scala)
> at org.apache.spark.sql.catalyst.expressions.UnsafeRow.getDecimal(UnsafeRow.java:388)
> at org.apache.spark.sql.catalyst.expressions.JoinedRow.getDecimal(JoinedRow.scala:95)
> at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificMutableProjection.apply(Unknown Source)
> ===================================
> One of the other executor before this had the following exception:
> FetchFailed(BlockManagerId(10, d2412.halxg.cloudera.com, 45956), shuffleId=0, mapId=212, reduceId=492, message=
> org.apache.spark.shuffle.FetchFailedException: Failed to connect to d2412.halxg.cloudera.com/10.20.122.112:45956
> at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:321)
> at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:306)
> at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:51)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:173)
> at org.apache.spark.sql.execution.TungstenSort.org$apache$spark$sql$execution$TungstenSort$$executePartition$1(sort.scala:160)
> at org.apache.spark.sql.execution.TungstenSort$$anonfun$doExecute$4.apply(sort.scala:169)
> at org.apache.spark.sql.execution.TungstenSort$$anonfun$doExecute$4.apply(sort.scala:169)
> at org.apache.spark.rdd.MapPartitionsWithPreparationRDD.compute(MapPartitionsWithPreparationRDD.scala:64)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:88)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Failed to connect to d2412.halxg.cloudera.com/10.20.122.112:45956
> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:193)
> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:156)
> at org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:88)
> at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
> at org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43)
> at org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ... 3 more
> Caused by: java.net.ConnectException: Connection refused: d2412.halxg.cloudera.com/10.20.122.112:45956
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
> at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> ... 1 more
> )
> =============================
> During stack trace this is what I found:
> Deadlock Detection:
> No deadlocks found.
> Error accessing address 0x7f428f893d80
> sun.jvm.hotspot.debugger.UnmappedAddressException: 7f428f893d80
> at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
> at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
> at sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:217)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:482)
> at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
> at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
> at sun.jvm.hotspot.types.basic.BasicField.getAddress(BasicField.java:248)
> at sun.jvm.hotspot.types.basic.BasicAddressFieldWrapper.getValue(BasicAddressFieldWrapper.java:48)
> at sun.jvm.hotspot.runtime.Threads.first(Threads.java:145)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
> at sun.tools.jstack.JStack.main(JStack.java:102)
> =================
> For schema and query details look at the following JIRA:
> https://issues.apache.org/jira/browse/SPARK-11873
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org