You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Neville Dipale (Jira)" <ji...@apache.org> on 2021/02/27 08:43:00 UTC
[jira] [Updated] (ARROW-11803) [Rust] Parquet] Support v2
LogicalType
[ https://issues.apache.org/jira/browse/ARROW-11803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neville Dipale updated ARROW-11803:
-----------------------------------
Description:
We currently do not read nor write the version 2 logical types. This is mainly because we do not have a mapping for it from parquet-format-rs.
To implement this, we can:
- convert "parquet::basic::LogicalType" to "parquet::basic::ConvertedType"
- implement "parquet::basic::LogicalType" which mirrors "parquet_format::LogicalType"
- create a mapping between ConvertedType and LogicalType
- write LogicalType to "parquet_format::SchemaElement" if v2 of the writer is used
This would be a good starting point for implementing 2.6 types (UUID, NANOS precision time & timestamp).
Follow-up work would be:
- parsing v2 of the schema [ARROW-11365]
- Using v2 in the Arrow writer (mostly schema conversion)
- Supporting nanosecond precision time & timestamp
was:
We currently do not read nor write the version 2 logical types. This is mainly because we do not have a mapping for it from parquet-format-rs.
To implement this, we can:
- convert "parquet::basic::LogicalType" to "parquet::basic::ConvertedType"
- implement "parquet::basic::LogicalType" which mirrors "parquet_format::LogicalType"
- create a mapping between ConvertedType and LogicalType
- write LogicalType to "parquet_format::SchemaElement" if v2 of the writer is used
This would be a good starting point for implementing 2.6 types (UUID, NANOS precision time & timestamp).
Follow-up work would be:
- parsing v2 of the schema
- Using v2 in the Arrow writer (mostly schema conversion)
- Supporting nanosecond precision time & timestamp
> [Rust] Parquet] Support v2 LogicalType
> --------------------------------------
>
> Key: ARROW-11803
> URL: https://issues.apache.org/jira/browse/ARROW-11803
> Project: Apache Arrow
> Issue Type: Sub-task
> Reporter: Neville Dipale
> Assignee: Neville Dipale
> Priority: Major
>
> We currently do not read nor write the version 2 logical types. This is mainly because we do not have a mapping for it from parquet-format-rs.
> To implement this, we can:
> - convert "parquet::basic::LogicalType" to "parquet::basic::ConvertedType"
> - implement "parquet::basic::LogicalType" which mirrors "parquet_format::LogicalType"
> - create a mapping between ConvertedType and LogicalType
> - write LogicalType to "parquet_format::SchemaElement" if v2 of the writer is used
> This would be a good starting point for implementing 2.6 types (UUID, NANOS precision time & timestamp).
> Follow-up work would be:
> - parsing v2 of the schema [ARROW-11365]
> - Using v2 in the Arrow writer (mostly schema conversion)
> - Supporting nanosecond precision time & timestamp
--
This message was sent by Atlassian Jira
(v8.3.4#803005)