[GitHub] [incubator-livy] runzhiwang opened a new pull request #232: [LIVY-667][WIP] Collecting a part of partition to the driver by batch to avoid OOM

[GitBox <gi...@apache.org>]

runzhiwang opened a new pull request #232: [LIVY-667][WIP] Collecting a part of partition to the driver by batch to avoid OOM
URL: https://github.com/apache/incubator-livy/pull/232
 
 
   ## What changes were proposed in this pull request?
   
   Collecting a part of partition to the driver by batch to avoid OOM
   
   Background:
   1. When enable livy.server.thrift.incrementalCollect, thrift use toLocalIterator to load one partition at each time instead of the whole rdd to avoid OutOfMemory. However, if the largest partition is too big, the OutOfMemory still occurs.
   
   2. This PR collect a part of partition to the driver by batch at each time to avoid OOM.
   
   ## How was this patch tested?
   
    create a big size of data into one partition and query them all.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services