You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Zheng Shao (Jira)" <ji...@apache.org> on 2020/11/11 19:00:00 UTC
[jira] [Updated] (PARQUET-1945) Add an option to allow auto
conversion from empty fields to NULL
[ https://issues.apache.org/jira/browse/PARQUET-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated PARQUET-1945:
--------------------------------
Description:
Right now, Parquet Writer throws out an exception:
{{Parquet record is malformed: empty fields are illegal, the field should be ommited completely instead}}
when an empty field (array or struct or map I guess?) is written.
The suggestion here is to add an option "auto_convert_empty_fields_to_null" that convert empty fields to null automatically on write.
The LOC to change is [here:|https://sourcegraph.com/github.com/apache/parquet-mr/-/blob/parquet-column/src/main/java/org/apache/parquet/io/MessageColumnIO.java#L328]
{quote}{{if (emptyField) {}}
{{ {{ throw new ParquetEncodingException("empty fields are illegal, the field should be ommited completely instead");}}}}
{{}}}{quote}
was:
Right now, Parquet Writer throws out an exception:
{{Parquet record is malformed: empty fields are illegal, the field should be ommited completely instead}}
when an empty field (array or struct or map I guess?) is written.
The suggestion here is to add an option "auto_convert_empty_fields_to_null" that convert empty fields to null automatically on write.
The LOC to change is [here:|https://sourcegraph.com/github.com/apache/parquet-mr/-/blob/parquet-column/src/main/java/org/apache/parquet/io/MessageColumnIO.java#L328]
{{ if (emptyField) {}}
{{ throw new ParquetEncodingException("empty fields are illegal, the field should be ommited completely instead");}}
{{ }}}
> Add an option to allow auto conversion from empty fields to NULL
> ----------------------------------------------------------------
>
> Key: PARQUET-1945
> URL: https://issues.apache.org/jira/browse/PARQUET-1945
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-format
> Reporter: Zheng Shao
> Priority: Minor
>
> Right now, Parquet Writer throws out an exception:
> {{Parquet record is malformed: empty fields are illegal, the field should be ommited completely instead}}
> when an empty field (array or struct or map I guess?) is written.
> The suggestion here is to add an option "auto_convert_empty_fields_to_null" that convert empty fields to null automatically on write.
> The LOC to change is [here:|https://sourcegraph.com/github.com/apache/parquet-mr/-/blob/parquet-column/src/main/java/org/apache/parquet/io/MessageColumnIO.java#L328]
> {quote}{{if (emptyField) {}}
> {{ {{ throw new ParquetEncodingException("empty fields are illegal, the field should be ommited completely instead");}}}}
> {{}}}{quote}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)