[I] Fix HoodieStreamer writing json value issue [hudi]

["hudi-bot (via GitHub)" <gi...@apache.org>]

hudi-bot opened a new issue, #14443:
URL: https://github.com/apache/hudi/issues/14443

   when value is a json string, HoodieDeltaStreamer will write it as a map string, which is inconsistent with the original data
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-6964
   - Type: Bug
   - Fix version(s):
     - 0.16.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Fix HoodieStreamer writing json value issue [hudi]

["rangareddy (via GitHub)" <gi...@apache.org>]

rangareddy commented on issue #14443:
URL: https://github.com/apache/hudi/issues/14443#issuecomment-4902420831

   **Sample Reproducible code:**
   
   ```scala
   import org.apache.avro.Schema
   // target schema: payload declared as a plain STRING
   val schema = new Schema.Parser().parse(
     """{"type":"record","name":"Event","fields":[
          {"name":"id","type":"string"},
          {"name":"payload","type":"string"}]}""")
   // source record: payload's value is a JSON object
   val json = """{"id":"1","payload":{"a":1,"b":"x","nested":{"k":"v"}}}"""
   val rec  = new org.apache.hudi.avro.MercifulJsonConverter().convert(json, schema)
   println("OUTPUT_payload = " + rec.get("payload"))
   ```
   
   **Hudi 0.15.0 Output:**
   
   `OUTPUT_payload = {a=1, b=x, nested={k=v}}`
   
   **Hudi 1.0.2 Output:**
   
   `OUTPUT_payload = {"a":1,"b":"x","nested":{"k":"v"}}`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Fix HoodieStreamer writing json value issue [hudi]

["rangareddy (via GitHub)" <gi...@apache.org>]

rangareddy commented on issue #14443:
URL: https://github.com/apache/hudi/issues/14443#issuecomment-4902430294

   This issue was reviewed as part of the JIRA-migrated backlog triage, and I reproduced it to confirm the fix.
   
   Root cause: when a source JSON field's value is a JSON object but the target schema types that field as `string`, `MercifulJsonConverter` (the JSON->Avro converter used by the Hudi streamer) wrote the value via the parsed object's `toString()`. That object is a Java `Map`, so `toString()` renders `{a=1, b=x}` instead of the original JSON `{"a":1,"b":"x"}`.
   
   This was fixed by #11825 (HUDI-8125, commit f64ff33), which changed the string handler to pass through String values unchanged and serialize non-String values with Jackson `ObjectMapper.writeValueAsString` (`StringProcessor` in hudi-common `org/apache/hudi/avro/MercifulJsonConverter.java`). First released in 1.0.0.
   
   Verified with a minimal `MercifulJsonConverter.convert()` test (field `payload` typed `string`, value `{"a":1,"b":"x","nested":{"k":"v"}}`):
   - Hudi 0.15.0 -> `payload = {a=1, b=x, nested={k=v}}`  (the bug)
   - Hudi 1.0.2 -> `payload = {"a":1,"b":"x","nested":{"k":"v"}}`  (fixed, round-trips the input)
   
   Note: this issue's fixVersion (0.16.0) was never released; the fix landed on the 1.0.x line via HUDI-8125.
   
   If you're still hitting this on 1.0.0+, please reopen with repro steps.
   
   Closing as fixed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Fix HoodieStreamer writing json value issue [hudi]

["rangareddy (via GitHub)" <gi...@apache.org>]

rangareddy closed issue #14443: Fix HoodieStreamer writing json value issue
URL: https://github.com/apache/hudi/issues/14443


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org