[GitHub] [incubator-hudi] nsivabalan commented on issue #1073: [WIP] [HUDI-377] Adding Delete() support to DeltaStreamer

[GitBox <gi...@apache.org>]

nsivabalan commented on issue #1073: [WIP] [HUDI-377] Adding Delete() support to DeltaStreamer
URL: https://github.com/apache/incubator-hudi/pull/1073#issuecomment-562846781
 
 
   @bvaradar @vinothchandar : Current patch in its state is buggy. I synced up with Balaji on it and we came up with a solution. Here is the gist. 
   
   Without changing the source and target schema(to add delete marker) in DeltaStreamer configs, we can't get this done. I went ahead and made changes locally (to configs, schema, source records)and was able to get it working. But before I go ahead and fix all tests, I wanted to get consensus that this is a good from usability standpoint and right way to add delete support in DeltaStreamer. 
   
   - This will enforce all users of DeltaStreamer to change their schema to add this delete marker that we define("_hoodie_delete_marker"). 
   
   Here are two other options I could think of. 
   Option a. Introduce a new config for delete marker key. Hoodie internal classes will assume this field as delete marker (all the way to OverwriteWithLatestAvroPayload), and act accordingly. 
   two concerns here:
     1. Not sure if we can pass in this arg to  OverwriteWithLatestAvroPayload though or how much changes this would involve. 
     2. As of now, I intercept the schema in SchemaProvider and append the delete marker field to both source schema. Hope thats fine? 
     3. This should be fixable, but still figuring out. I found ways to define an optional field in json schema, but I am yet to figure out ways to add an optional field via code i.e. new Schema.Field(String name, Schema schema, String doc, JsonNode defaultValue)) which is what is required in SchemaRegistry. For now, have added as a mandatory field. So, if we couldn't add an optional field, then all serialized records will have this delete marker for non deleted records. 
   
   Option b. Introduce a predicate function that we can use while reading from source to determine whether the record is deleted or not. Will introduce an optional delete marker field(still need to figure out ways to add an optional field in code) in target schema(we can keep this our own key as "_hoodie_delete_marker") and proceed further. OverwriteWithLatestAvroPayload will detect it and will return Optional.empty(). And so at the storage, no records will have this optional field.
   
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services