[spark] branch branch-3.3 updated: [MINOR][TESTS] Avoid NPE in an anonym SparkListener in DataFrameReaderWriterSuite

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
     new 9b0d49b02e9 [MINOR][TESTS] Avoid NPE in an anonym SparkListener in DataFrameReaderWriterSuite
9b0d49b02e9 is described below

commit 9b0d49b02e959cfa36dc3b0870f6381a730fed2e
Author: Kent Yao <ya...@apache.org>
AuthorDate: Tue Feb 21 21:07:52 2023 +0900

    [MINOR][TESTS] Avoid NPE in an anonym SparkListener in DataFrameReaderWriterSuite
    
    ### What changes were proposed in this pull request?
    
    Avoid the following NPE in an anonym SparkListener in DataFrameReaderWriterSuite, as job desc may be absent
    
    ```
    java.lang.NullPointerException
            at java.util.concurrent.ConcurrentLinkedQueue.checkNotNull(ConcurrentLinkedQueue.java:920)
            at java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue.java:327)
            at java.util.concurrent.ConcurrentLinkedQueue.add(ConcurrentLinkedQueue.java:297)
            at org.apache.spark.sql.test.DataFrameReaderWriterSuite$$anon$2.onJobStart(DataFrameReaderWriterSuite.scala:1151)
            at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37)
            at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
            at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
            at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
            at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
            at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
            at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
            at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
            at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
            at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
            at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
            at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
            at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1462)
            at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)
    
    ```
    
    ### Why are the changes needed?
    
    Test Improvement
    
    ### Does this PR introduce _any_ user-facing change?
    
    no
    
    ### How was this patch tested?
    
    existing tests
    
    Closes #40102 from yaooqinn/test-minor.
    
    Authored-by: Kent Yao <ya...@apache.org>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
    (cherry picked from commit 088ebdeea67dd509048a7559f1c92a3636e18ce6)
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 .../scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala   | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
index dabd9c001eb..c933ab50d21 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
@@ -1132,7 +1132,8 @@ class DataFrameReaderWriterSuite extends QueryTest with SharedSparkSession with
         val jobDescriptions = new ConcurrentLinkedQueue[String]()
         val jobListener = new SparkListener {
           override def onJobStart(jobStart: SparkListenerJobStart): Unit = {
-            jobDescriptions.add(jobStart.properties.getProperty(SparkContext.SPARK_JOB_DESCRIPTION))
+            val desc = jobStart.properties.getProperty(SparkContext.SPARK_JOB_DESCRIPTION)
+            if (desc != null) jobDescriptions.add(desc)
           }
         }
         sparkContext.addSparkListener(jobListener)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org