[spark] branch master updated: [SPARK-26963][MLLIB] SizeEstimator can't make some JDK fields accessible in Java 9+

[sr...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new ab4e83a  [SPARK-26963][MLLIB] SizeEstimator can't make some JDK fields accessible in Java 9+
ab4e83a is described below

commit ab4e83aca7cecc6cce0f0c51f3674481b381ba34
Author: Sean Owen <se...@databricks.com>
AuthorDate: Sat Feb 23 11:01:47 2019 -0600

    [SPARK-26963][MLLIB] SizeEstimator can't make some JDK fields accessible in Java 9+
    
    ## What changes were proposed in this pull request?
    
    Don't use inaccessible fields in SizeEstimator, which comes up in Java 9+
    
    ## How was this patch tested?
    
    Manually ran tests with Java 11; it causes these tests that failed before to pass.
    This ought to pass on Java 8 as there's effectively no change for Java 8.
    
    Closes #23866 from srowen/SPARK-26963.
    
    Authored-by: Sean Owen <se...@databricks.com>
    Signed-off-by: Sean Owen <se...@databricks.com>
---
 .../main/scala/org/apache/spark/util/SizeEstimator.scala | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/util/SizeEstimator.scala b/core/src/main/scala/org/apache/spark/util/SizeEstimator.scala
index e12b6b7..4837b01 100644
--- a/core/src/main/scala/org/apache/spark/util/SizeEstimator.scala
+++ b/core/src/main/scala/org/apache/spark/util/SizeEstimator.scala
@@ -334,9 +334,21 @@ object SizeEstimator extends Logging {
         if (fieldClass.isPrimitive) {
           sizeCount(primitiveSize(fieldClass)) += 1
         } else {
-          field.setAccessible(true) // Enable future get()'s on this field
+          // Note: in Java 9+ this would be better with trySetAccessible and canAccess
+          try {
+            field.setAccessible(true) // Enable future get()'s on this field
+            pointerFields = field :: pointerFields
+          } catch {
+            // If the field isn't accessible, we can still record the pointer size
+            // but can't know more about the field, so ignore it
+            case _: SecurityException =>
+              // do nothing
+            // Java 9+ can throw InaccessibleObjectException but the class is Java 9+-only
+            case re: RuntimeException
+                if re.getClass.getSimpleName == "InaccessibleObjectException" =>
+              // do nothing
+          }
           sizeCount(pointerSize) += 1
-          pointerFields = field :: pointerFields
         }
       }
     }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org