You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "soumilshah1995 (via GitHub)" <gi...@apache.org> on 2024/01/15 21:22:20 UTC

[I] [SUPPORT] Hudi DeltaStreamer with Flattening Transformer [hudi]

soumilshah1995 opened a new issue, #10499:
URL: https://github.com/apache/hudi/issues/10499

   Hello, I'm currently experimenting with the Hudi delta streamer and working on creating part 12 of the delta streamer playlist. For the next video, my goal is to cover the Hudi SQL-based transformer and the flattening transformer.
   
   I've encountered a challenge with the flattening transformer. The data schema in the schema registry is structured as follows:
   
   ```
   {
     "type": "record",
     "name": "Order",
     "fields": [
       {
         "name": "order_id",
         "type": "string"
       },
       {
         "name": "name",
         "type": "string"
       },
       {
         "name": "order_value",
         "type": "string"
       },
       {
         "name": "priority",
         "type": "string"
       },
       {
         "name": "order_date",
         "type": "string"
       },
       {
         "name": "ts",
         "type": "string"
       },
       {
         "name": "customer",
         "type": {
           "type": "record",
           "name": "Customer",
           "fields": [
             {
               "name": "customer_id",
               "type": "string"
             }
           ]
         }
       }
     ]
   }
   ```
   Without the flattening transformer, everything works fine using the following configuration:
   
   ```
   --transformer-class org.apache.hudi.utilities.transform.SqlQueryBasedTransformer \
   hoodie.streamer.transformer.sql=SELECT * FROM <SRC> a
   
   ```
   
   Now, I'm attempting to implement the flattening transformer by following the guide [here](https://github.com/apache/hudi/pull/4097) and [official documentation](https://hudi.apache.org/docs/next/transforms/#flattening-transformer).
   
   Here is the configuration I've tried:
   
   ```
   --transformer-class org.apache.hudi.utilities.transform.FlatteningTransformer,org.apache.hudi.utilities.transform.SqlQueryBasedTransformer \
   
   ```
   And the SQL query variations:
   
   
   ```
   hoodie.streamer.transformer.sql=SELECT order_id, name, order_value, priority, order_date, ts, customer.customer_id FROM <SRC> a
   
   OR
   
   hoodie.streamer.transformer.sql=SELECT a.order_id, a.name, a.order_value, a.priority, a.order_date, a.ts, a.customer_id FROM <SRC> a
   
   
   OR
   hoodie.streamer.transformer.sql=SELECT a.order_id, a.name, a.order_value, a.priority, a.order_date, a.ts, a.customer_id FROM <SRC> a
   
   OR
   
   hoodie.streamer.transformer.sql=SELECT order_id, name, order_value, priority, order_date, ts, customer_id FROM <SRC> a
   
   
   ```
   
   However, it seems like the flattening transformer is not working, or I might be using it incorrectly. Any assistance or guidance on this matter would be greatly appreciated. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org