You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "nssalian (via GitHub)" <gi...@apache.org> on 2026/04/14 15:20:51 UTC

[I] AvroSchemaConverter silently drops typed_value when converting shredded variant schemas [parquet-java]

nssalian opened a new issue, #3477:
URL: https://github.com/apache/parquet-java/issues/3477

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   I noticed this on the parquet-cli while checking the schema for a shredded parquet file.
   
   Essentially, given this Parquet schema:
   
   ```
   message table {
     required group data (VARIANT(1)) {
       required binary metadata;
       optional binary value;
       optional group typed_value {
         required group name {
           optional binary value;
           optional binary typed_value (STRING);
         }
         required group age {
           optional binary value;
           optional int32 typed_value (INTEGER(8,true));
         }
       }
     }
   }
   ```
   
   AvroSchemaConverter.convert() produces:
   
   ```
   {
     "type": "record",
     "name": "data",
     "fields": [
       {"name": "metadata", "type": "bytes"},
       {"name": "value", "type": "bytes"}
     ]
   }
   ```
   
   `typed_value` is missing
   
   Expected: The converter should convert all children of the VARIANT group (including typed_value when present)
   
   Version: 1.18.0-SNAPSHOT
   
   I can add a fix to this myself.
   
   ### Component(s)
   
   Avro


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] AvroSchemaConverter silently drops typed_value when converting shredded variant schemas [parquet-java]

Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on issue #3477:
URL: https://github.com/apache/parquet-java/issues/3477#issuecomment-4248625010

   >Just to clarify, I'm not sure the Avro core functionality should change here (we probably don't want to return shredded for Avro). Assigned
   Thanks for clarifying and assigning. I'll patch the CLI functionality for now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] AvroSchemaConverter silently drops typed_value when converting shredded variant schemas [parquet-java]

Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on issue #3477:
URL: https://github.com/apache/parquet-java/issues/3477#issuecomment-4246557646

   I think the avro schema converter might be overloaded between CLI and actual avro schema conversion then?  I think other systems depend on avro not being shredded.  It does seem useful at the CLI level to be able to observed shredded columns though.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] AvroSchemaConverter silently drops typed_value when converting shredded variant schemas [parquet-java]

Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on issue #3477:
URL: https://github.com/apache/parquet-java/issues/3477#issuecomment-4245890500

   CC: @emkornfield @aihuaxu @steveloughran 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] AvroSchemaConverter silently drops typed_value when converting shredded variant schemas [parquet-java]

Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on issue #3477:
URL: https://github.com/apache/parquet-java/issues/3477#issuecomment-4248145631

   > I think the avro schema converter might be overloaded between CLI and actual avro schema conversion then? I think other systems depend on avro not being shredded. It does seem useful at the CLI level to be able to observed shredded columns though.
   
   Agreed. I do see that the converter serves both the CLI (parquet-cli/Schemas.java) for schema display and the Avro read/write paths (AvroReadSupport, AvroWriteSupport, AvroVariantConverter). I'll patch those as well. @emkornfield , do you mind assigning this to me so I can post a PR? Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] AvroSchemaConverter silently drops typed_value when converting shredded variant schemas [parquet-java]

Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on issue #3477:
URL: https://github.com/apache/parquet-java/issues/3477#issuecomment-4269225621

   should it convert or should it unshred the variant and attach that data?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] AvroSchemaConverter silently drops typed_value when converting shredded variant schemas [parquet-java]

Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on issue #3477:
URL: https://github.com/apache/parquet-java/issues/3477#issuecomment-4248161803

   > Agreed. I do see that the converter serves both the CLI (parquet-cli/Schemas.java) for schema display and the Avro read/write paths (AvroReadSupport, AvroWriteSupport, AvroVariantConverter). I'll patch those as well. @emkornfield , do you mind assigning this to me so I can post a PR? Thanks.
   
   Just to clarify, I'm not sure the Avro core functionality should change here (we probably don't want to return shredded for Avro).  Assigned


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org