[GitHub] [incubator-hudi] v3nkatesh commented on a change in pull request #802: Cache RDD to avoid recomputing data ingestion and hbase puts actions

[GitBox <gi...@apache.org>]

v3nkatesh commented on a change in pull request #802: Cache RDD to avoid recomputing data ingestion and hbase puts actions
URL: https://github.com/apache/incubator-hudi/pull/802#discussion_r309889595
 
 

 ##########
 File path: hoodie-client/src/main/java/com/uber/hoodie/index/hbase/HBaseIndex.java
 ##########
 @@ -368,21 +381,13 @@ private static void sleepForTime(int sleepTimeMs) {
   public JavaRDD<WriteStatus> updateLocation(JavaRDD<WriteStatus> writeStatusRDD, JavaSparkContext jsc,
       HoodieTable<T> hoodieTable) {
     final HBaseIndexQPSResourceAllocator hBaseIndexQPSResourceAllocator = createQPSResourceAllocator(this.config);
-    JavaRDD<WriteStatus> writeStatusResultRDD;
     setPutBatchSize(writeStatusRDD, hBaseIndexQPSResourceAllocator, jsc);
-    logger.info("multiPutBatchSize: before puts" + multiPutBatchSize);
+    logger.info("multiPutBatchSize: before hbase puts" + multiPutBatchSize);
     JavaRDD<WriteStatus> writeStatusJavaRDD = writeStatusRDD.mapPartitionsWithIndex(
         updateLocationFunction(), true);
-    // Forcing a spark action so HBase puts are triggered before releasing resources
-    if (this.config.getHBaseIndexShouldComputeQPSDynamically()) {
-      logger.info("writestatus count: " + writeStatusJavaRDD.count());
-      writeStatusResultRDD = writeStatusRDD;
-    } else {
-      writeStatusResultRDD = writeStatusJavaRDD;
-    }
-    // Release QPS resources as HBAse puts are done at this point
-    hBaseIndexQPSResourceAllocator.releaseQPSResources();
-    return writeStatusResultRDD;
+    // caching the index updated status RDD
+    writeStatusJavaRDD = writeStatusJavaRDD.persist(config.getWriteStatusStorageLevel());
 
 Review comment:
   The problem is that, resources are acquired at spark driver level and so re-triggering might not acquire resources. I agree that after persisting, it might only be a small chance but to make best effort, releasing resources from close() at the end is safe, wdyt ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services