[I] Invoke preCombine in real time view by converting arrayWritable to Avro [hudi]

["hudi-bot (via GitHub)" <gi...@apache.org>]

hudi-bot opened a new issue, #14485:
URL: https://github.com/apache/hudi/issues/14485

   There are 2 issues with the realtime input format:
   
    
    # Delta records (updates) might not have the entire row change log, in such an update, we need to be able to call preCombine of the HoodieRecordPayload implementation so that we merge existing data from parquet (full row change log) with the new column being updated.
    # In case there is some custom computation of columns in a custom implementation of the HoodieRecordPayload, that will be missed in the realtime input format right now. We need to honor that by calling preCombine.
   
    
   
   Both of the above are use-cases for power users who implement their own custom record. Since this is not common, this is lower priority. 
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-152
   - Type: Bug
   
   
   ---
   
   
   ## Comments
   
   27/Jan/21 18:45;shivnarayan;[~nishith29]: are you referring to OverwriteNonDefaultsWithLatestAvroPayload ? ;;;
   
   ---
   
   17/May/21 00:42;nishith29;The parquet + log records are combined in `HoodieRealtimeCompactedRecordReader`. 
   
    
   
   Here, the following code is invoked
   
    
   {code:java}
   if (!deltaRecordMap.isEmpty()) {
     // TODO(VC): Right now, we assume all records in log, have a matching base record. (which
     // would be true until we have a way to index logs too)
     // return from delta records map if we have some match.
     String key = arrayWritable.get()[HoodieInputFormatUtils.HOODIE_RECORD_KEY_COL_POS].toString();
     if (deltaRecordMap.containsKey(key)) {
       // TODO(NA): Invoke preCombine here by converting arrayWritable to Avro. This is required since the
       // deltaRecord may not be a full record and needs values of columns from the parquet
       Option<GenericRecord> rec;
       rec = buildGenericRecordwithCustomPayload(deltaRecordMap.get(key)); <----- (1) latest record is retrieved from log files
       // If the record is not present, this is a delete record using an empty payload so skip this base record
       // and move to the next record
       while (!rec.isPresent()) {
         // if current parquet reader has no record, return false
         if (!this.parquetReader.next(aVoid, arrayWritable)) {
           return false;
         }
         String tempKey = arrayWritable.get()[HoodieInputFormatUtils.HOODIE_RECORD_KEY_COL_POS].toString();
         if (deltaRecordMap.containsKey(tempKey)) {
           rec = buildGenericRecordwithCustomPayload(deltaRecordMap.get(tempKey));
         } else {
           // need to return true, since now log file does not contain tempKey, but parquet file contains tempKey
           return true;
         }
       }
   
   Later 
   
   Writable[] originalValue = arrayWritable.get();
   try {
     // Sometime originalValue.length > replaceValue.length.
     // This can happen when hive query is looking for pseudo parquet columns like BLOCK_OFFSET_INSIDE_FILE
     System.arraycopy(replaceValue, 0, originalValue, 0,
         Math.min(originalValue.length, replaceValue.length)); <---- (2) We simply copy the data from the record from log to the original record from parquet
     arrayWritable.set(originalValue);
   }{code}
   Before calling (2), we should convert the record from parquet which is arrayWritable.get() to GenericRecord and then call preCombine() to honor the preCombine() between 2 record merging.
   
    
   
   [~shivnarayan] FYI
   
    ;;;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Invoke preCombine in real time view by converting arrayWritable to Avro [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan closed issue #14485: Invoke preCombine in real time view by converting arrayWritable to Avro
URL: https://github.com/apache/hudi/issues/14485


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Invoke preCombine in real time view by converting arrayWritable to Avro [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan commented on issue #14485:
URL: https://github.com/apache/hudi/issues/14485#issuecomment-4795170535

   Closing for inactivity. JIRA: [HUDI-152](https://issues.apache.org/jira/browse/HUDI-152). This issue has had no activity for over two years and the GitHub thread has no replies. If this is still relevant, please open a fresh issue with reproduction details on a recent release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org