[GitHub] dongjoon-hyun opened a new pull request #23851: [SPARK-26950][SQL][TEST] Make RandomDataGenerator use Float.NaN or Double.NaN for all NaN values

[GitBox <gi...@apache.org>]

dongjoon-hyun opened a new pull request #23851: [SPARK-26950][SQL][TEST] Make RandomDataGenerator use Float.NaN or Double.NaN for all NaN values
URL: https://github.com/apache/spark/pull/23851
 
 
   ## What changes were proposed in this pull request?
   
   Apache Spark uses the predefined `Float.NaN` and `Double.NaN` for NaN values, but there exists more NaN values with different binary presentations.
   
   ```scala
   scala> java.nio.ByteBuffer.allocate(4).putFloat(Float.NaN).array
   res1: Array[Byte] = Array(127, -64, 0, 0)
   
   scala> val x = java.lang.Float.intBitsToFloat(-6966608)
   x: Float = NaN
   
   scala> java.nio.ByteBuffer.allocate(4).putFloat(x).array
   res2: Array[Byte] = Array(-1, -107, -78, -80)
   ```
   
   Since user can have these values, `RandomDataGenerator` generates these NaN values. However, this causes `checkEvaluationWithUnsafeProjection` failures due to the difference between `UnsafeRow` binary presentation. The following is the UT failure instance. This PR aims to fix this UT flakiness.
   
   - https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/102528/testReport/
   
   ## How was this patch tested?
   
   Pass the Jenkins with the newly added test cases.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org