[GitHub] [spark] viirya commented on pull request #31682: [WIP][SPARK-34545][SQL] Fix issues with valueCompare feature of pyrolite

[GitBox <gi...@apache.org>]

viirya commented on pull request #31682:
URL: https://github.com/apache/spark/pull/31682#issuecomment-788242330


   > > I think this is because Spark 2.4 uses pyrolite 4.13 where it looks the value comparison feature is 
   > > I think another option would be to disable the valueCompare explicitly.
   > 
   > Yea, I think so, too. Could you try `valueCompare=false` then check performance changes just in case?
   
   Looks like `valueCompare` as true actually isn't for performance gain. But it's also hard to tell because reduced pickle size may improve performance too. I guess performance shouldn't be an issue here.
   
   https://github.com/irmen/Pyrolite/blob/pyrolite-4.21/java/src/main/java/net/razorvine/pickle/Pickler.java#L86-L89
   
   ```java
   /**
    * When memoizing, compare objects by value. This saves pickle size, but can slow down pickling.
    * Also, it should only be used if the object graph is immutable. Unused if useMemo is false.
    */
    protected boolean valueCompare=true;
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org