[GitHub] [spark] viirya commented on issue #26105: [SPARK-26570][SQL] Reduce memory consumption when transforming very many filestatus in InMemoryFileIndex.bulkListLeafFiles

[GitBox <gi...@apache.org>]

viirya commented on issue #26105: [SPARK-26570][SQL] Reduce memory consumption when transforming very many filestatus in InMemoryFileIndex.bulkListLeafFiles
URL: https://github.com/apache/spark/pull/26105#issuecomment-541838731
 
 
   First from the stacktrace, I was thinking it is possible that the reason to user OOM is that SerializableFileStatus is too many. Then it is possible to cause OOM when transforming them back to Status. This PR was created for that.
   
   But from users latest posts, from jmap logs, looks like SerializableFileStatus do not hold too much memory. For now I am not sure SerializableFileStatus is the cause of the OOM, so close this first.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org