You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yang Jie (Jira)" <ji...@apache.org> on 2023/01/19 08:21:00 UTC
[jira] [Updated] (SPARK-42112) Add null check before `ContinuousWriteRDD#compute` method close dataWriter
[ https://issues.apache.org/jira/browse/SPARK-42112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Jie updated SPARK-42112:
-----------------------------
Description:
Run
mvn clean test -pl sql/core -Dtest=none -DwildcardSuites=org.apache.spark.sql.streaming.continuous.ContinuousSuite -am
NPE exists in the test log
{code:java}
- repeatedly restart
...
16:07:39.891 ERROR org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD: Writer for partition 1 is aborting.
16:07:39.891 ERROR org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD: Writer for partition 1 aborted.
16:07:39.892 WARN org.apache.spark.util.Utils: Suppressing exception in finally: null
java.lang.NullPointerException
at org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD.$anonfun$compute$7(ContinuousWriteRDD.scala:91)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1558)
at org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD.compute(ContinuousWriteRDD.scala:91)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:139)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1502)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750) {code}
The test did not fail because Utils.tryWithSafeFinallyAndFailureCallbacks function suppressed Exception in finally block
> Add null check before `ContinuousWriteRDD#compute` method close dataWriter
> --------------------------------------------------------------------------
>
> Key: SPARK-42112
> URL: https://issues.apache.org/jira/browse/SPARK-42112
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Yang Jie
> Priority: Minor
>
> Run
>
> mvn clean test -pl sql/core -Dtest=none -DwildcardSuites=org.apache.spark.sql.streaming.continuous.ContinuousSuite -am
>
> NPE exists in the test log
>
> {code:java}
> - repeatedly restart
> ...
> 16:07:39.891 ERROR org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD: Writer for partition 1 is aborting.
> 16:07:39.891 ERROR org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD: Writer for partition 1 aborted.
> 16:07:39.892 WARN org.apache.spark.util.Utils: Suppressing exception in finally: null
> java.lang.NullPointerException
> at org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD.$anonfun$compute$7(ContinuousWriteRDD.scala:91)
> at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1558)
> at org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD.compute(ContinuousWriteRDD.scala:91)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
> at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
> at org.apache.spark.scheduler.Task.run(Task.scala:139)
> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1502)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750) {code}
>
>
> The test did not fail because Utils.tryWithSafeFinallyAndFailureCallbacks function suppressed Exception in finally block
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org