You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "lai fangmin (Jira)" <ji...@apache.org> on 2021/12/23 05:37:00 UTC
[jira] [Updated] (SPARK-37720) /1624610690382/state/0/8/1272.delta does not exist at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider.updateFromDeltaFile
[ https://issues.apache.org/jira/browse/SPARK-37720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
lai fangmin updated SPARK-37720:
--------------------------------
Summary: /1624610690382/state/0/8/1272.delta does not exist at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider.updateFromDeltaFile (was: /BMT1630977855165/CA1624610690382/state/0/8/1272.delta does not exist at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider.updateFromDeltaFile)
> /1624610690382/state/0/8/1272.delta does not exist at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider.updateFromDeltaFile
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-37720
> URL: https://issues.apache.org/jira/browse/SPARK-37720
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.4.5, 3.0.1, 3.1.2
> Environment: hadoop 2.8.5 spark 3.1.2 and spark2.4.5
> Reporter: lai fangmin
> Priority: Major
> Labels: issues
>
> hi,
> I using structured streaming and checkpoint. If anybody has the following errors happen occasionly? SPARK 3.1.2 and spark2.4.5,hadoop 2.8.5
> An error occurred while calling o126.getResult. : org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301) at org.apache.spark.security.SocketAuthServer.getResult(SocketAuthServer.scala:97) at org.apache.spark.security.SocketAuthServer.getResult(SocketAuthServer.scala:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 8 in stage 1.0 failed 4 times, most recent failure: Lost task 8.3 in stage 1.0 (TID 103) (SYSOPS00260773 executor 6): java.lang.IllegalStateException: Error reading delta file hdfs://24610690382/state/0/8/1272.delta of HDFSStateStoreProvider[id = (op=0,part=8),dir = hdfs://24610690382/state/0/8]: hdfs://1624610690382/state/0/8/1272.delta does not exist at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider.updateFromDeltaFile(HDFSBackedStateStoreProvider.scala:461) at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider.$anonfun$loadMap$4(HDFSBackedStateStoreProvider.scala:417) at scala.runtime.java8.JFunction1$mcVJ$sp.apply(JFunction1$mcVJ$sp.java:23) at scala.collection.immutable.NumericRange.foreach(NumericRange.scala:74) at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider.$anonfun$loadMap$2(HDFSBackedStateStoreProvider.scala:416) at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:597) at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider.loadMap(HDFSBackedStateStoreProvider.scala:389) at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider.getLoadedMapForStore(HDFSBackedStateStoreProvider.scala:236) at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider.getStore(HDFSBackedStateStoreProvider.scala:220) at org.apache.spark.sql.execution.streaming.state.StateStore$.get(StateStore.scala:469) at org.apache.spark.sql.execution.streaming.state.StateStoreRDD.compute(StateStoreRDD.scala:125) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.rdd.MapPartitionsRDD.c
> Any suggestions to fix?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org