[GitHub] spark pull request: [SPARK-14558][CORE] In ClosureCleaner, clean t...

[a-roberts <gi...@git.apache.org>]

Github user a-roberts commented on the pull request:

    https://github.com/apache/spark/pull/12327#issuecomment-222107940
  
    > cacheSize1 and cacheSize2 are both the size after cleaning. The difference is that, cacheSize1 is the size after cleaned the data with line object reference, cacheSize2 is the size after cleaned the data without line object reference.
    
    Looking for clarity here, is it true that clean "with the reference" should be bigger (cacheSize1) and clean "without the reference" should be smaller (cacheSize2)? 
    
    OpenJDK, cacheSize1: 180392, cacheSize2: 187896 (bigger without the line object reference)
    
    IBM JDK, cacheSize1: 354692, cacheSize2: 263800 (smaller without the line object reference)
    
    What exactly does "without line object reference" mean and should cacheSize1 be smaller or bigger than cacheSize2?
    
    I know the SizeEstimator overestimates for IBM Java so our cached footprint is much larger (handling this), so because of the larger difference we get this test failing, OpenJDK **fails** with Kryo and IBM **passes** with Kryo for this test.
    
    A better check would be to run with and without the closure cleaner change and to check the second result is less by the size of the line object, so based on our cacheSize2 being smaller (without the line object reference), I'm thinking that IBM Java functions as expected and OpenJDK doesn't - but this depends on my questions above, interested to hear what you think @cloud-fan


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org