You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:16:43 UTC
[jira] [Resolved] (SPARK-21863) while performing saveToMemSQL, get
Exception in thread "Thread-23" java.lang.AssertionError: assertion failed:
Task -1024 release lock on block rdd_5_2 more times than it acquired it
[ https://issues.apache.org/jira/browse/SPARK-21863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-21863.
----------------------------------
Resolution: Incomplete
> while performing saveToMemSQL, get Exception in thread "Thread-23" java.lang.AssertionError: assertion failed: Task -1024 release lock on block rdd_5_2 more times than it acquired it
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-21863
> URL: https://issues.apache.org/jira/browse/SPARK-21863
> Project: Spark
> Issue Type: Question
> Components: SQL
> Affects Versions: 2.0.1
> Reporter: Vidya
> Priority: Major
> Labels: bulk-closed
>
> I am running spark-submit command on local desktop and while trying to saving data I get _"Task -1024 release lock on block rdd_5_2 more times than it acquired it"_ exception. The data gets saved in the target table anyway. Wonder why its throws that exception.
> Here is snippet of code that throws exception
> {code:java}
> //reject duplicates
> val uniques = schemaEthnicities.filter("ETHNICITY_CD NOT IN (" + codes + ")")
> uniques.write.mode(SaveMode.Overwrite).format("com.memsql.spark.connector").save("CD_ETHNICITY_SPARK")
> {code}
> As a workaround I am creating a new dataframe and writing the dataframe to the table
> {code:java}
> //AGAIN when I try to invoke write method on uniques dataset - it throws exception
> // Task -release lock on block rdd more times than it acquired it spark, hence the workaround of creating DF
> val uniqueFrame = sparkSession.sqlContext.createDataFrame(uniques.collect())
> uniqueFrame.write.mode(SaveMode.Overwrite).format("com.memsql.spark.connector").save("CD_ETHNICITY_SPARK")
> {code}
> Exception Stacktrace:
> og4j:ERROR Either File or DatePattern options are not set for appender [RollingAppenderU].
> xception in thread "Thread-23" java.lang.AssertionError: assertion failed: Task -1024 release lock on block rdd_5_2 more times than it acquired it
> at scala.Predef$.assert(Predef.scala:170)
> at org.apache.spark.storage.BlockInfoManager.unlock(BlockInfoManager.scala:298)
> at org.apache.spark.storage.BlockManager.releaseLock(BlockManager.scala:630)
> at org.apache.spark.storage.BlockManager$$anonfun$1.apply$mcV$sp(BlockManager.scala:434)
> at org.apache.spark.util.CompletionIterator$$anon$1.completion(CompletionIterator.scala:46)
> at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:35)
> at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:461)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificColumnarIterator.hasNext(Unknown Source)
> at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
> at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at com.memsql.spark.connector.LoadDataStrategy$$anon$2.run(LoadDataStrategy.scala:52)
> at java.lang.Thread.run(Thread.java:745)
> xception in thread "Thread-24" java.lang.AssertionError: assertion failed: Task -1024 release lock on block rdd_5_1 more times than it acquired it
> at scala.Predef$.assert(Predef.scala:170)
> at org.apache.spark.storage.BlockInfoManager.unlock(BlockInfoManager.scala:298)
> at org.apache.spark.storage.BlockManager.releaseLock(BlockManager.scala:630)
> at org.apache.spark.storage.BlockManager$$anonfun$1.apply$mcV$sp(BlockManager.scala:434)
> at org.apache.spark.util.CompletionIterator$$anon$1.completion(CompletionIterator.scala:46)
> at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:35)
> at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:461)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificColumnarIterator.hasNext(Unknown Source)
> at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
> at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at com.memsql.spark.connector.LoadDataStrategy$$anon$2.run(LoadDataStrategy.scala:52)
> at java.lang.Thread.run(Thread.java:745)
> xception in thread "Thread-22" java.lang.AssertionError: assertion failed: Task -1024 release lock on block rdd_5_0 more times than it acquired it
> at scala.Predef$.assert(Predef.scala:170)
> at org.apache.spark.storage.BlockInfoManager.unlock(BlockInfoManager.scala:298)
> at org.apache.spark.storage.BlockManager.releaseLock(BlockManager.scala:630)
> at org.apache.spark.storage.BlockManager$$anonfun$1.apply$mcV$sp(BlockManager.scala:434)
> at org.apache.spark.util.CompletionIterator$$anon$1.completion(CompletionIterator.scala:46)
> at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:35)
> at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:461)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificColumnarIterator.hasNext(Unknown Source)
> at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
> at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at com.memsql.spark.connector.LoadDataStrategy$$anon$2.run(LoadDataStrategy.scala:52)
> at java.lang.Thread.run(Thread.java:745)
> 7/08/29 10:37:11 ERROR Executor: Exception in task 1.0 in stage 9.0 (TID 612)
> ava.lang.AssertionError: assertion failed
> at scala.Predef$.assert(Predef.scala:156)
> at org.apache.spark.storage.BlockInfo.checkInvariants(BlockInfoManager.scala:84)
> at org.apache.spark.storage.BlockInfo.readerCount_$eq(BlockInfoManager.scala:66)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2$$anonfun$apply$2.apply(BlockInfoManager.scala:362)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2$$anonfun$apply$2.apply(BlockInfoManager.scala:361)
> at scala.Option.foreach(Option.scala:257)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2.apply(BlockInfoManager.scala:361)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2.apply(BlockInfoManager.scala:356)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask(BlockInfoManager.scala:356)
> at org.apache.spark.storage.BlockManager.releaseAllLocksForTask(BlockManager.scala:646)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:281)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 7/08/29 10:37:11 ERROR Executor: Exception in task 2.0 in stage 9.0 (TID 613)
> ava.lang.AssertionError: assertion failed
> at scala.Predef$.assert(Predef.scala:156)
> at org.apache.spark.storage.BlockInfo.checkInvariants(BlockInfoManager.scala:84)
> at org.apache.spark.storage.BlockInfo.readerCount_$eq(BlockInfoManager.scala:66)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2$$anonfun$apply$2.apply(BlockInfoManager.scala:362)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2$$anonfun$apply$2.apply(BlockInfoManager.scala:361)
> at scala.Option.foreach(Option.scala:257)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2.apply(BlockInfoManager.scala:361)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2.apply(BlockInfoManager.scala:356)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask(BlockInfoManager.scala:356)
> at org.apache.spark.storage.BlockManager.releaseAllLocksForTask(BlockManager.scala:646)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:281)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 7/08/29 10:37:11 ERROR Executor: Exception in task 0.0 in stage 9.0 (TID 611)
> ava.lang.AssertionError: assertion failed
> at scala.Predef$.assert(Predef.scala:156)
> at org.apache.spark.storage.BlockInfo.checkInvariants(BlockInfoManager.scala:84)
> at org.apache.spark.storage.BlockInfo.readerCount_$eq(BlockInfoManager.scala:66)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2$$anonfun$apply$2.apply(BlockInfoManager.scala:362)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2$$anonfun$apply$2.apply(BlockInfoManager.scala:361)
> at scala.Option.foreach(Option.scala:257)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2.apply(BlockInfoManager.scala:361)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2.apply(BlockInfoManager.scala:356)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask(BlockInfoManager.scala:356)
> at org.apache.spark.storage.BlockManager.releaseAllLocksForTask(BlockManager.scala:646)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:281)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 7/08/29 10:37:11 ERROR TaskSetManager: Task 0 in stage 9.0 failed 1 times; aborting job
> xception in thread "main" com.memsql.spark.SaveToMemSQLException: SaveToMemSQLException: org.apache.spark.SparkException: Job aborted due to stage fail
> at scala.Predef$.assert(Predef.scala:156)
> at org.apache.spark.storage.BlockInfo.checkInvariants(BlockInfoManager.scala:84)
> at org.apache.spark.storage.BlockInfo.readerCount_$eq(BlockInfoManager.scala:66)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2$$anonfun$apply$2.apply(BlockInfoManager.scala:362)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2$$anonfun$apply$2.apply(BlockInfoManager.scala:361)
> at scala.Option.foreach(Option.scala:257)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2.apply(BlockInfoManager.scala:361)
> at org.apache.spark.storage.BlockInfoManager$$anonfun$releaseAllLocksForTask$2.apply(BlockInfoManager.scala:356)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask(BlockInfoManager.scala:356)
> at org.apache.spark.storage.BlockManager.releaseAllLocksForTask(BlockManager.scala:646)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:281)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> river stacktrace:
> at com.memsql.spark.connector.DataFrameFunctions.saveToMemSQL(DataFrameFunctions.scala:67)
> at com.memsql.spark.connector.MemSQLTableRelation.insert(MemSQLRelation.scala:185)
> at com.memsql.spark.connector.DefaultSource.createRelation(DefaultSource.scala:51)
> at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:442)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:194)
> at org.collegeboard.spark.lookup.EthnicityMain$.main(EthnicityMain.scala:100)
> at org.collegeboard.spark.lookup.EthnicityMain.main(EthnicityMain.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> 7/08/29 10:37:12 ERROR ShutdownHookManager: Exception while deleting Spark temp dir: C:\Users\vkalsi\AppData\Local\Temp\spark-aaf9be61-3400-48df-83e3-7
> ava.io.IOException: Failed to delete: C:\Users\vkalsi\AppData\Local\Temp\spark-aaf9be61-3400-48df-83e3-7129d4df5294\userFiles-342b7742-6a8e-413b-9540-f
> at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1006)
> at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:64)
> at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:61)
> at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
> at org.apache.spark.util.ShutdownHookManager$$anonfun$1.apply$mcV$sp(ShutdownHookManager.scala:61)
> at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:215)
> at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:187)
> at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:187)
> at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:187)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1877)
> at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:187)
> at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:187)
> at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:187)
> at scala.util.Try$.apply(Try.scala:192)
> at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:187)
> at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:177)
> at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
> 7/08/29 10:37:12 ERROR ShutdownHookManager: Exception while deleting Spark temp dir: C:\Users\vkalsi\AppData\Local\Temp\spark-aaf9be61-3400-48df-83e3-7
> ava.io.IOException: Failed to delete: C:\Users\vkalsi\AppData\Local\Temp\spark-aaf9be61-3400-48df-83e3-7129d4df5294
> at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1006)
> at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:64)
> at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:61)
> at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
> at org.apache.spark.util.ShutdownHookManager$$anonfun$1.apply$mcV$sp(ShutdownHookManager.scala:61)
> at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:215)
> at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:187)
> at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:187)
> at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:187)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1877)
> at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:187)
> at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:187)
> at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:187)
> at scala.util.Try$.apply(Try.scala:192)
> at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:187)
> at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:177)
> at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org