You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "furquan1993 (via GitHub)" <gi...@apache.org> on 2025/11/13 11:43:13 UTC

[I] Encoded value for the complex field [hudi]

furquan1993 opened a new issue, #14259:
URL: https://github.com/apache/hudi/issues/14259

   ### Bug Description
   
   **What happened:**
   when querying a hudi table with a complex data column, like array/map with incremental cdc, the resulted after row is having some encoded value for the complex field.
   
   for example. I have a column name policy_types which is an array of string. When running cdc query on the table its returning the below json. You can see policy_types is some kind of encoded object.
   
   **What you expected:**
   
   The actual value for this field. 
   for example: list of strings
   
   
   **Steps to reproduce:**
   1. Create a table with a column type as array of string
   2. Insert few records
   3. Run incremental CDC query to fetch before/after record.
   4. You will find the after to contain encoded value for array type column.
   
   ```
   {
     "_hoodie_record_key": "2",
     "_hoodie_partition_path": "",
     "policy_types": {
       "baseObject": "AAAAAAAAAAARAAAAWAAAABUAAABwAAAAAQAAAIgAAAAAAAAAkAAAAEgAAACQAAAAAgAAAAAAAAAMTQAAAAAAAGYAAAAAAAAABwAAANgAAAAgAAAA4AAAADIwMjUxMTEzMTExNzI3MzAzAAAAAAAAADIwMjUxMTEzMTExNzI3MzAzXzBfMQAAADIAAAAAAAAANDEyZjJmOTAtZmM0MS00ZDNjLThlMzMtZTI4N2JkYzUwMTY2LTBfMC0yNi0yOV8yMDI1MTExMzExMTcyNzMwMy5wYXJxdWV0UEVORElORwABAAAAAAAAAAAAAAAAAAAABAAAABgAAABBVVRPAAAAAA==",
       "baseOffset": 240,
       "sizeInBytes": 32
     },
     "_hoodie_file_name": "412f2f90-fc41-4d3c-8e33-e287bdc50166-0_0-26-29_20251113111727303.parquet",
     "_hoodie_commit_seqno": "20251113111727303_0_1",
     "order_id": 2,
     "order_customer_id": 102,
     "order_status": "PENDING",
     "_hoodie_commit_time": "20251113111727303",
     "order_date": 19724
   }
   ```
   
   
   ### Environment
   
   **Hudi version:**
   **Query engine:** (Spark/Flink/Trino etc)
   **Relevant configs:**
   
   
   ### Logs and Stack Trace
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Encoded value for the complex field [hudi]

Posted by "furquan1993 (via GitHub)" <gi...@apache.org>.

furquan1993 commented on issue #14259:
URL: https://github.com/apache/hudi/issues/14259#issuecomment-3575408949

   @cshuo I built your branch and it resolved the issue. However, there is still hudi metadata missing from the records in alternate fashion, as I have mentioned above.
   
   On further doing some hit & trial, i can see that when I am using `hoodie.datasource.write.operation` as `upsert`, then this happens, but for insert_overwrite/bulk_insert this is working fine. This can be unrelated, should I open another bug for that?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Encoded value for the complex field [hudi]

Posted by "furquan1993 (via GitHub)" <gi...@apache.org>.

furquan1993 commented on issue #14259:
URL: https://github.com/apache/hudi/issues/14259#issuecomment-3540255770

   Thanks for reproducing the issue. I understand you get the update results, this is also my finding if I update a record, that record turns out right, however, I am also trying to capture the inserts as I want to pass in that data to my sink. I am getting encoded results for those inserts. The before data shows NULL and after data shows encoded value for complex fields.
   Can you please also check if you can find the inserts correctly?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Encoded value for the complex field [hudi]

Posted by "furquan1993 (via GitHub)" <gi...@apache.org>.

furquan1993 commented on issue #14259:
URL: https://github.com/apache/hudi/issues/14259#issuecomment-3562602421

   Another observation is every even commit I am able to get proper row data, but for every odd commit i am getting encoded data. 
   cdc commit from -> to
   0 -> 1 | encoded data
   1 -> 2 | proper data
   2 -> 3 | encoded data
   3 -> 4 | proper data.
   
   attaching my code and sample cdc table output.
   
   [Main.java](https://github.com/user-attachments/files/23674647/Main.java)
   [output.csv](https://github.com/user-attachments/files/23674648/output.csv)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Encoded value for the complex field [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.

cshuo commented on issue #14259:
URL: https://github.com/apache/hudi/issues/14259#issuecomment-3578349132

   > I built your branch and it resolved the issue. However, there is still hudi metadata missing from the records in alternate fashion, as I have mentioned above.
   
   @furquan1993 Yeah, I believe this is another problem which is irrelevant to this issue. Let's open another issue to avoid mixing them together, so we can close this issue once the pr is landed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Encoded value for the complex field [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 closed issue #14259: Encoded value for the complex field
URL: https://github.com/apache/hudi/issues/14259


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Encoded value for the complex field [hudi]

Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.

deepakpanda93 commented on issue #14259:
URL: https://github.com/apache/hudi/issues/14259#issuecomment-3540706866

   Hello @furquan1993 
   
   I have tried adding some records and while capturing them it shows encoded values. 
   
   - It seems not to be a bug, Because for INSERT, there is no "before", so Hudi logs the entire new row as a binary-coded "insert payload". 
   - In DATA_BEFORE mode, INSERT has no before, so Hudi writes a compact binary payload.
   
   - The solution is to update this setting in writes:
   ```"hoodie.table.cdc.supplemental.logging.mode": "DATA_BEFORE_AFTER"```
   
   - In DATA_BEFORE_AFTER mode, Hudi logs a proper structured record for after even for INSERT.
   
   [HUDI_CDC_ENCODE_Update.txt](https://github.com/user-attachments/files/23578369/HUDI_CDC_ENCODE_Update.txt)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Encoded value for the complex field [hudi]

Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.

deepakpanda93 commented on issue #14259:
URL: https://github.com/apache/hudi/issues/14259#issuecomment-3562627782

   Thank you for the information @furquan1993 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Encoded value for the complex field [hudi]

Posted by "furquan1993 (via GitHub)" <gi...@apache.org>.

furquan1993 commented on issue #14259:
URL: https://github.com/apache/hudi/issues/14259#issuecomment-3574730883

   Thanks @cshuo, this looks like the issue. I will give it a try


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Encoded value for the complex field [hudi]

Posted by "furquan1993 (via GitHub)" <gi...@apache.org>.

furquan1993 commented on issue #14259:
URL: https://github.com/apache/hudi/issues/14259#issuecomment-3579393107

   @deepakpanda93 @cshuo Also, can we patch this fix on 1.0.2 version?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Encoded value for the complex field [hudi]

Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.

deepakpanda93 commented on issue #14259:
URL: https://github.com/apache/hudi/issues/14259#issuecomment-3532995341

   Hello @furquan1993 
   
   I have tried reproducing the above using Hudi-1.0.2 with Spark-3.5 and the mentioned configs, but could not reproduce. Its showing the expected output. Attached my sample code snippet and output.
   
   Please review and let me know if any changes in the sample code required.
   
   [Hudi_CDC_Encoded.txt](https://github.com/user-attachments/files/23548907/Hudi_CDC_Encoded.txt)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Encoded value for the complex field [hudi]

Posted by "furquan1993 (via GitHub)" <gi...@apache.org>.

furquan1993 commented on issue #14259:
URL: https://github.com/apache/hudi/issues/14259#issuecomment-3541587849

   Ok, now I am clear. So you are running incremental cdc starting from first commit. Which gives you only 1 record and that is fine. However, if you update your being_instant value to 0 (basically starting from the very beginning), you would get encoded value. My requirement is to also capture those 10 records as well which you are inserting first before inserting/updating any other record. 
   
   Inshort, I am getting correct values between 1st and 2nd commit. However, if I try and get values before 0th and 1st commit, I get encoded value. How can i resolve that. 
   
   JFYI, i am doing this in Java. I doubt that have anything to do with it though.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Encoded value for the complex field [hudi]

Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.

deepakpanda93 commented on issue #14259:
URL: https://github.com/apache/hudi/issues/14259#issuecomment-3540280196

   @furquan1993 Do you want me to run the example code for INSERT operation instead of UPSERT ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Encoded value for the complex field [hudi]

Posted by "furquan1993 (via GitHub)" <gi...@apache.org>.

furquan1993 commented on issue #14259:
URL: https://github.com/apache/hudi/issues/14259#issuecomment-3540403067

   Yes please
   
   On Mon, Nov 17, 2025 at 12:40 PM deepakpanda93 ***@***.***>
   wrote:
   
   > *deepakpanda93* left a comment (apache/hudi#14259)
   > <https://github.com/apache/hudi/issues/14259#issuecomment-3540280196>
   >
   > @furquan1993 <https://github.com/furquan1993> Do you want me to run the
   > example code for INSERT operation instead of UPSERT ?
   >
   > —
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/hudi/issues/14259#issuecomment-3540280196>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/ABJVRV5C47EPJI6KG5JHUAD35FYFVAVCNFSM6AAAAACL7XTL72VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKNBQGI4DAMJZGY>
   > .
   > You are receiving this because you were mentioned.Message ID:
   > ***@***.***>
   >
   
   
   -- 
   Furquan Ahmed
   Tech Lead
   8080430446
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Encoded value for the complex field [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.

cshuo commented on issue #14259:
URL: https://github.com/apache/hudi/issues/14259#issuecomment-3574622908

   @furquan1993 the problem is when the `before` or `after` fields of cdc record is constructed from spark `InternalRow`, it's converted by the `InternalRowToJsonStringConverter` from `InternalRow` to json string, but the converted currently do not support converting complex type fields into plain json string, so there can be some 'encoded' string.
   
   here is an fix for 1.1: https://github.com/apache/hudi/pull/14351, you can have a try if it solves your problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org