You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@parquet.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/10/22 10:57:49 UTC
[PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
alamb opened a new pull request, #123:
URL: https://github.com/apache/parquet-site/pull/123
- Closes https://github.com/apache/parquet-site/issues/118
## Rationale
Three new logical types have been added to the Parquet specification. Let's ensure they are listed in the implementation status matrix
# Changes:
1. Add entries for Variant, Geometry, Geography types
2. Add links to relevant parts of the spec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #123:
URL: https://github.com/apache/parquet-site/pull/123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #123:
URL: https://github.com/apache/parquet-site/pull/123#issuecomment-3439122837
> I was peer-pressured by the parquet implementation status page π
The plan is working!
(@julienledem and I plan to do something similar for some of the other query engines that have a reputation for not being able to read newer files)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #123:
URL: https://github.com/apache/parquet-site/pull/123#discussion_r2465045940
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -43,30 +49,43 @@ Implementations:
### Logical types
-| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
-| ----------------------------------------- | ----- | ------------- | -------- | -------- | ----- | --------- | ------ |
-| STRING | β
| β
| β
| β
| β
| β
| β
|
-| ENUM | β | β
| β
| β
(1) | β | β
| β
|
-| UUID | β | β
| β
| β
(1) | β | β
| β
|
-| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
-| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
-| DATE | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
-| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
-| BSON | β | β
(1) | β
| β
(1) | β | β | β |
-| LIST | β
| β
| β
| β
| β
| (R) | β
|
-| MAP | β
| β
| β
| β
| β
| (R) | β
|
-| UNKNOWN (always null) | β
| β
| β
| β
| β
| β
| β
|
-| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+Logical types are defined by the [`union LogicalType` in parquet.thrift] and described in [LogicalTypes.md]
+
+[`union LogicalType` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L471
+[LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
+
+| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
+|-----------------------------------------|------| ------- | ------- | ------- | ---- | -------- |--------|
+| STRING | β
| β
| β
| β
| β
| β
| β
|
+| ENUM | β | β
| β
| β
(1) | β | β
| β
|
+| UUID | β | β
| β
| β
(1) | β | β
| β
|
+| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
+| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
+| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+| DATE | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
+| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
+| BSON | β | β
(1) | β
| β
(1) | β | β | β |
+| [VARIANT] | | | | | | | |
+| [GEOMETRY] | | | | | | | |
+| [GEOGRAPHY] | | | | | | | |
Review Comment:
@paleolimbot you [said](https://github.com/apache/parquet-site/pull/123#discussion_r2455434022):
> and geography is supported by everything except cudf and duckdb
Can you confirm what "supported" means? I I found an example file that has `GEOGRAPHY` column and duckdb seems to be able to read it fine (though I note the column says "geometry" rather than "geography"):
```sql
D select * from 'data/geospatial/crs-geography.parquet' limit 1;
ββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β wkt β geography β
β varchar β geometry β
ββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β POLYGON ((-111 45,β¦ β POLYGON ((-111 45, -111 44.9, -111 44.8, -111 44.7, -111 44.6, -111 44.5, -111 44.4, -111 44.3, β¦ β
ββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
https://github.com/apache/parquet-testing/tree/master/data/geospatial
>
> - `crs-geography.parquet`: Contains a GEOGRAPHY column with the crs
> omitted. This should be interpreted as OGC:CRS84 (i.e., longitude/latitude).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #123:
URL: https://github.com/apache/parquet-site/pull/123#issuecomment-3432426043
Thank you for the review @severo π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #123:
URL: https://github.com/apache/parquet-site/pull/123#discussion_r2469060564
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -43,30 +49,43 @@ Implementations:
### Logical types
-| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
-| ----------------------------------------- | ----- | ------------- | -------- | -------- | ----- | --------- | ------ |
-| STRING | β
| β
| β
| β
| β
| β
| β
|
-| ENUM | β | β
| β
| β
(1) | β | β
| β
|
-| UUID | β | β
| β
| β
(1) | β | β
| β
|
-| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
-| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
-| DATE | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
-| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
-| BSON | β | β
(1) | β
| β
(1) | β | β | β |
-| LIST | β
| β
| β
| β
| β
| (R) | β
|
-| MAP | β
| β
| β
| β
| β
| (R) | β
|
-| UNKNOWN (always null) | β
| β
| β
| β
| β
| β
| β
|
-| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+Logical types are defined by the [`union LogicalType` in parquet.thrift] and described in [LogicalTypes.md]
+
+[`union LogicalType` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L471
+[LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
+
+| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
+|-----------------------------------------|------| ------------- | ------- | --------- | ---- | -------- |--------|
+| STRING | β
| β
| β
| β
| β
| β
| β
|
+| ENUM | β | β
| β
| β
(1) | β | β
| β
|
+| UUID | β | β
| β
| β
(1) | β | β
| β
|
+| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
+| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
+| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+| DATE | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
+| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
+| BSON | β | β
(1) | β
| β
(1) | β | β | β |
Review Comment:
@neilechao and @aihuaxu I am working to characterize the support for Variant across readers. The arrow version says:
https://github.com/apache/arrow/blob/57f626189326251c8144d4fad89c28220c68ca75/cpp/src/parquet/arrow/variant_internal.h#L33
> EXPERIMENTAL: Variant is not yet fully supported.
However, looking around at the open issues in arrow https://github.com/apache/arrow/issues?q=is%3Aissue%20%20variant
It wasn't clear to me what the current status is (should arrow have a green β
or an β or something else?)
Some relevant PRs;
- https://github.com/apache/arrow/issues/47838
- https://github.com/apache/arrow/pull/45375
- https://github.com/apache/arrow/pull/47835
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #123:
URL: https://github.com/apache/parquet-site/pull/123#discussion_r2451683604
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -27,6 +28,11 @@ Implementations:
### Physical types
+Physical types are defined by the [`enum Type` parquet.thrift]
Review Comment:
I purposely chose not to make this a permalink so it always reflected the current version of the thrift file
This means over time the link may not point to the exact correct line, but I think people can find the relevant section using `union LogicalType` and we can update the links if needed
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -43,30 +49,43 @@ Implementations:
### Logical types
-| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
-| ----------------------------------------- | ----- | ------------- | -------- | -------- | ----- | --------- | ------ |
-| STRING | β
| β
| β
| β
| β
| β
| β
|
-| ENUM | β | β
| β
| β
(1) | β | β
| β
|
-| UUID | β | β
| β
| β
(1) | β | β
| β
|
-| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
-| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
-| DATE | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
-| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
-| BSON | β | β
(1) | β
| β
(1) | β | β | β |
-| LIST | β
| β
| β
| β
| β
| (R) | β
|
-| MAP | β
| β
| β
| β
| β
| (R) | β
|
-| UNKNOWN (always null) | β
| β
| β
| β
| β
| β
| β
|
-| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+Logical types are defined by the [`union LogicalType` in parquet.thrift] and described in [LogicalTypes.md]
+
+[`union LogicalType` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L471
+[LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
+
+| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
+|-----------------------------------------|------| ------- | ------- | ------- | ---- | -------- |--------|
+| STRING | β
| β
| β
| β
| β
| β
| β
|
+| ENUM | β | β
| β
| β
(1) | β | β
| β
|
+| UUID | β | β
| β
| β
(1) | β | β
| β
|
+| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
+| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
+| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+| DATE | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
+| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
+| BSON | β | β
(1) | β
| β
(1) | β | β | β |
+| [VARIANT] | | | | | | | |
Review Comment:
this table has three new rows for Variant, Geometry, Geography, and I moved FLOAT16 up with the rest of the numeric types, to be consistent with the list in https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #123:
URL: https://github.com/apache/parquet-site/pull/123#issuecomment-3477893970
I think this PR is an improvement over the current version on the website so will merge it in.
If there are additional checks that were missed or need to be added in the future, let's make those changes as a follow on PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "platypii (via GitHub)" <gi...@apache.org>.
platypii commented on code in PR #123:
URL: https://github.com/apache/parquet-site/pull/123#discussion_r2455620300
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -43,30 +49,43 @@ Implementations:
### Logical types
-| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
-| ----------------------------------------- | ----- | ------------- | -------- | -------- | ----- | --------- | ------ |
-| STRING | β
| β
| β
| β
| β
| β
| β
|
-| ENUM | β | β
| β
| β
(1) | β | β
| β
|
-| UUID | β | β
| β
| β
(1) | β | β
| β
|
-| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
-| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
-| DATE | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
-| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
-| BSON | β | β
(1) | β
| β
(1) | β | β | β |
-| LIST | β
| β
| β
| β
| β
| (R) | β
|
-| MAP | β
| β
| β
| β
| β
| (R) | β
|
-| UNKNOWN (always null) | β
| β
| β
| β
| β
| β
| β
|
-| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+Logical types are defined by the [`union LogicalType` in parquet.thrift] and described in [LogicalTypes.md]
+
+[`union LogicalType` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L471
+[LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
+
+| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
+|-----------------------------------------|------| ------- | ------- | ------- | ---- | -------- |--------|
+| STRING | β
| β
| β
| β
| β
| β
| β
|
+| ENUM | β | β
| β
| β
(1) | β | β
| β
|
+| UUID | β | β
| β
| β
(1) | β | β
| β
|
+| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
+| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
+| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+| DATE | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
+| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
+| BSON | β | β
(1) | β
| β
(1) | β | β | β |
+| [VARIANT] | | | | | | | |
+| [GEOMETRY] | | | | | | | |
+| [GEOGRAPHY] | | | | | | | |
Review Comment:
Does duckdb support geometery / geography? When I load geospatial files from the parquet-testing repo, duckdb does not decode them? (latest duckdb 1.4.1)
```
D from 'geospatial.parquet';
βββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β group β wkt β geometry β
β varchar β varchar β blob β
βββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β all β POINT (30 10) β \x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00>@\x00\x00\x00\x00\x00\x00$@ β
β all β LINESTRING (30 10,β¦ β \x01\x02\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00>@\x00\x00\x00\x00\x00\x00$@\x00\x00\x00\x00\x00\x00$@β¦ β
β all β POLYGON ((30 10, 4β¦ β \x01\x03\x00\x00\x00\x01\x00\x00\x00\x05\x00\x00\x00\x00\x00\x00\x00\x00\x00>@\x00\x00\x00\x00\x00\x00$@\x00\x00\xβ¦ β
β
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #123:
URL: https://github.com/apache/parquet-site/pull/123#issuecomment-3444838552
> hyparquet js:
>
> * does not (yet) support VARIANT β
> * **does** support GEOMETRY and GEOGRAPHY β
(edit from: ~(R)~)
Updated in a4ec40d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #123:
URL: https://github.com/apache/parquet-site/pull/123#issuecomment-3444855506
> In arrow-rs you currently have to try pretty hard to actually get the types to be written or read but it's tested and can be done.
I hope we can make this better in the upcoming releases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "severo (via GitHub)" <gi...@apache.org>.
severo commented on PR #123:
URL: https://github.com/apache/parquet-site/pull/123#issuecomment-3435456240
cc @platypii for hyparquet too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #123:
URL: https://github.com/apache/parquet-site/pull/123#issuecomment-3477894059
Thanks again @severo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #123:
URL: https://github.com/apache/parquet-site/pull/123#discussion_r2461918631
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -43,30 +49,43 @@ Implementations:
### Logical types
-| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
-| ----------------------------------------- | ----- | ------------- | -------- | -------- | ----- | --------- | ------ |
-| STRING | β
| β
| β
| β
| β
| β
| β
|
-| ENUM | β | β
| β
| β
(1) | β | β
| β
|
-| UUID | β | β
| β
| β
(1) | β | β
| β
|
-| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
-| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
-| DATE | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
-| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
-| BSON | β | β
(1) | β
| β
(1) | β | β | β |
-| LIST | β
| β
| β
| β
| β
| (R) | β
|
-| MAP | β
| β
| β
| β
| β
| (R) | β
|
-| UNKNOWN (always null) | β
| β
| β
| β
| β
| β
| β
|
-| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+Logical types are defined by the [`union LogicalType` in parquet.thrift] and described in [LogicalTypes.md]
+
+[`union LogicalType` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L471
+[LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
+
+| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
+|-----------------------------------------|------| ------- | ------- | ------- | ---- | -------- |--------|
+| STRING | β
| β
| β
| β
| β
| β
| β
|
+| ENUM | β | β
| β
| β
(1) | β | β
| β
|
+| UUID | β | β
| β
| β
(1) | β | β
| β
|
+| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
+| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
+| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+| DATE | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
+| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
+| BSON | β | β
(1) | β
| β
(1) | β | β | β |
+| [VARIANT] | | | | | | | |
Review Comment:
Done in 1d23cc2 (along with arrow-rs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #123:
URL: https://github.com/apache/parquet-site/pull/123#discussion_r2461934478
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -43,30 +49,43 @@ Implementations:
### Logical types
-| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
-| ----------------------------------------- | ----- | ------------- | -------- | -------- | ----- | --------- | ------ |
-| STRING | β
| β
| β
| β
| β
| β
| β
|
-| ENUM | β | β
| β
| β
(1) | β | β
| β
|
-| UUID | β | β
| β
| β
(1) | β | β
| β
|
-| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
-| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
-| DATE | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
-| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
-| BSON | β | β
(1) | β
| β
(1) | β | β | β |
-| LIST | β
| β
| β
| β
| β
| (R) | β
|
-| MAP | β
| β
| β
| β
| β
| (R) | β
|
-| UNKNOWN (always null) | β
| β
| β
| β
| β
| β
| β
|
-| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+Logical types are defined by the [`union LogicalType` in parquet.thrift] and described in [LogicalTypes.md]
+
+[`union LogicalType` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L471
+[LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
+
+| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
+|-----------------------------------------|------| ------- | ------- | ------- | ---- | -------- |--------|
+| STRING | β
| β
| β
| β
| β
| β
| β
|
+| ENUM | β | β
| β
| β
(1) | β | β
| β
|
+| UUID | β | β
| β
| β
(1) | β | β
| β
|
+| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
+| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
+| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+| DATE | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
+| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
+| BSON | β | β
(1) | β
| β
(1) | β | β | β |
+| [VARIANT] | | | | | | | |
+| [GEOMETRY] | | | | | | | |
+| [GEOGRAPHY] | | | | | | | |
Review Comment:
Updated in dbfbc4c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #123:
URL: https://github.com/apache/parquet-site/pull/123#issuecomment-3444855985
Here is what the current matrix looks like
<img width="795" height="818" alt="Screenshot 2025-10-24 at 4 44 22β―PM" src="https://github.com/user-attachments/assets/7b5b1212-7a4a-4989-af7f-9ada637fc498" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "platypii (via GitHub)" <gi...@apache.org>.
platypii commented on PR #123:
URL: https://github.com/apache/parquet-site/pull/123#issuecomment-3435523166
hyparquet js:
- does not (yet) support VARIANT β
- **does** support reading GEOMETRY and GEOGRAPHY (R)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "platypii (via GitHub)" <gi...@apache.org>.
platypii commented on PR #123:
URL: https://github.com/apache/parquet-site/pull/123#issuecomment-3438743920
Update: As of today, Hyparquet now fully supports reading AND writing GEOMETRY and GEOGRAPHY types. β
I was peer-pressured by the parquet implementation status page :laughing:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #123:
URL: https://github.com/apache/parquet-site/pull/123#discussion_r2457311328
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -43,30 +49,43 @@ Implementations:
### Logical types
-| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
-| ----------------------------------------- | ----- | ------------- | -------- | -------- | ----- | --------- | ------ |
-| STRING | β
| β
| β
| β
| β
| β
| β
|
-| ENUM | β | β
| β
| β
(1) | β | β
| β
|
-| UUID | β | β
| β
| β
(1) | β | β
| β
|
-| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
-| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
-| DATE | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
-| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
-| BSON | β | β
(1) | β
| β
(1) | β | β | β |
-| LIST | β
| β
| β
| β
| β
| (R) | β
|
-| MAP | β
| β
| β
| β
| β
| (R) | β
|
-| UNKNOWN (always null) | β
| β
| β
| β
| β
| β
| β
|
-| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+Logical types are defined by the [`union LogicalType` in parquet.thrift] and described in [LogicalTypes.md]
+
+[`union LogicalType` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L471
+[LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
+
+| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
+|-----------------------------------------|------| ------- | ------- | ------- | ---- | -------- |--------|
+| STRING | β
| β
| β
| β
| β
| β
| β
|
+| ENUM | β | β
| β
| β
(1) | β | β
| β
|
+| UUID | β | β
| β
| β
(1) | β | β
| β
|
+| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
+| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
+| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+| DATE | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
+| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
+| BSON | β | β
(1) | β
| β
(1) | β | β | β |
+| [VARIANT] | | | | | | | |
+| [GEOMETRY] | | | | | | | |
+| [GEOGRAPHY] | | | | | | | |
Review Comment:
Indeed it seems to be better after `load spatial`
```sql
D LOAD spatial;
D from 'geospatial.parquet';
IO Error:
No files found that match the pattern "geospatial.parquet"
D from 'geospatial/geospatial.parquet';
ββββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β group β wkt β geometry β
β varchar β varchar β geometry β
ββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β all β POINT (30 10) β POINT (30 10) β
β all β LINESTRING (30 10,β¦ β LINESTRING (30 10, 10 30, 40 40) β
β all β POLYGON ((30 10, 4β¦ β POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10)) β
β all β MULTIPOINT ((30 10)) β MULTIPOINT (30 10) β
β all β MULTILINESTRING ((β¦ β MULTILINESTRING ((30 10, 10 30, 40 40)) β
β all β MULTIPOLYGON (((30β¦ β MULTIPOLYGON (((30 10, 40 40, 20 40, 10 20, 30 10))) β
β all β GEOMETRYCOLLECTIONβ¦ β GEOMETRYCOLLECTION (POINT (30 10), LINESTRING (30 10, 10 30, 40 40), POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10β¦ β
β all β POINT Z (30 10 40) β POINT Z (30 10 40) β
β all β LINESTRING Z (30 1β¦ β LINESTRING Z (30 10 40, 10 30 40, 40 40 80) β
β all β POLYGON Z ((30 10 β¦ β POLYGON Z ((30 10 40, 40 40 80, 20 40 60, 10 20 30, 30 10 40)) β
β multilinestring-zm β MULTILINESTRING ZMβ¦ β MULTILINESTRING ZM EMPTY
...
ββββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β 196 rows (40 shown) 3 columns β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -43,30 +49,43 @@ Implementations:
### Logical types
-| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
-| ----------------------------------------- | ----- | ------------- | -------- | -------- | ----- | --------- | ------ |
-| STRING | β
| β
| β
| β
| β
| β
| β
|
-| ENUM | β | β
| β
| β
(1) | β | β
| β
|
-| UUID | β | β
| β
| β
(1) | β | β
| β
|
-| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
-| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
-| DATE | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
-| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
-| BSON | β | β
(1) | β
| β
(1) | β | β | β |
-| LIST | β
| β
| β
| β
| β
| (R) | β
|
-| MAP | β
| β
| β
| β
| β
| (R) | β
|
-| UNKNOWN (always null) | β
| β
| β
| β
| β
| β
| β
|
-| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+Logical types are defined by the [`union LogicalType` in parquet.thrift] and described in [LogicalTypes.md]
+
+[`union LogicalType` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L471
+[LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
+
+| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
+|-----------------------------------------|------| ------- | ------- | ------- | ---- | -------- |--------|
+| STRING | β
| β
| β
| β
| β
| β
| β
|
+| ENUM | β | β
| β
| β
(1) | β | β
| β
|
+| UUID | β | β
| β
| β
(1) | β | β
| β
|
+| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
+| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
+| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+| DATE | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
+| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
+| BSON | β | β
(1) | β
| β
(1) | β | β | β |
+| [VARIANT] | | | | | | | |
+| [GEOMETRY] | | | | | | | |
+| [GEOGRAPHY] | | | | | | | |
Review Comment:
Indeed it seems to be better after `load spatial`
```sql
D LOAD spatial;
No files found that match the pattern "geospatial.parquet"
D from 'geospatial/geospatial.parquet';
ββββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β group β wkt β geometry β
β varchar β varchar β geometry β
ββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β all β POINT (30 10) β POINT (30 10) β
β all β LINESTRING (30 10,β¦ β LINESTRING (30 10, 10 30, 40 40) β
β all β POLYGON ((30 10, 4β¦ β POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10)) β
β all β MULTIPOINT ((30 10)) β MULTIPOINT (30 10) β
β all β MULTILINESTRING ((β¦ β MULTILINESTRING ((30 10, 10 30, 40 40)) β
β all β MULTIPOLYGON (((30β¦ β MULTIPOLYGON (((30 10, 40 40, 20 40, 10 20, 30 10))) β
β all β GEOMETRYCOLLECTIONβ¦ β GEOMETRYCOLLECTION (POINT (30 10), LINESTRING (30 10, 10 30, 40 40), POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10β¦ β
β all β POINT Z (30 10 40) β POINT Z (30 10 40) β
β all β LINESTRING Z (30 1β¦ β LINESTRING Z (30 10 40, 10 30 40, 40 40 80) β
β all β POLYGON Z ((30 10 β¦ β POLYGON Z ((30 10 40, 40 40 80, 20 40 60, 10 20 30, 30 10 40)) β
β multilinestring-zm β MULTILINESTRING ZMβ¦ β MULTILINESTRING ZM EMPTY
...
ββββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β 196 rows (40 shown) 3 columns β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #123:
URL: https://github.com/apache/parquet-site/pull/123#discussion_r2461932481
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -43,30 +49,43 @@ Implementations:
### Logical types
-| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
-| ----------------------------------------- | ----- | ------------- | -------- | -------- | ----- | --------- | ------ |
-| STRING | β
| β
| β
| β
| β
| β
| β
|
-| ENUM | β | β
| β
| β
(1) | β | β
| β
|
-| UUID | β | β
| β
| β
(1) | β | β
| β
|
-| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
-| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
-| DATE | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
-| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
-| BSON | β | β
(1) | β
| β
(1) | β | β | β |
-| LIST | β
| β
| β
| β
| β
| (R) | β
|
-| MAP | β
| β
| β
| β
| β
| (R) | β
|
-| UNKNOWN (always null) | β
| β
| β
| β
| β
| β
| β
|
-| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+Logical types are defined by the [`union LogicalType` in parquet.thrift] and described in [LogicalTypes.md]
+
+[`union LogicalType` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L471
+[LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
+
+| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
+|-----------------------------------------|------| ------- | ------- | ------- | ---- | -------- |--------|
+| STRING | β
| β
| β
| β
| β
| β
| β
|
+| ENUM | β | β
| β
| β
(1) | β | β
| β
|
+| UUID | β | β
| β
| β
(1) | β | β
| β
|
+| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
+| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
+| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+| DATE | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
+| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
+| BSON | β | β
(1) | β
| β
(1) | β | β | β |
+| [VARIANT] | | | | | | | |
+| [GEOMETRY] | | | | | | | |
+| [GEOGRAPHY] | | | | | | | |
Review Comment:
I also confirmed duckdb 1.4.1 can read variant as well
```sql
D select * from 'case-001.parquet';
βββββββββ¬βββββββββββββββββββββ
β id β var β
β int32 β json β
βββββββββΌβββββββββββββββββββββ€
β 1 β ["comedy","drama"] β
βββββββββ΄βββββββββββββββββββββ
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "severo (via GitHub)" <gi...@apache.org>.
severo commented on code in PR #123:
URL: https://github.com/apache/parquet-site/pull/123#discussion_r2451875124
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -27,6 +28,11 @@ Implementations:
### Physical types
+Physical types are defined by the [`enum Type` parquet.thrift]
+
+[`enum Type` parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L32
Review Comment:
```suggestion
[`enum Type` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L32
```
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -27,6 +28,11 @@ Implementations:
### Physical types
+Physical types are defined by the [`enum Type` parquet.thrift]
Review Comment:
```suggestion
Physical types are defined by the [`enum Type` in parquet.thrift]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #123:
URL: https://github.com/apache/parquet-site/pull/123#discussion_r2455466385
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -43,30 +49,43 @@ Implementations:
### Logical types
-| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
-| ----------------------------------------- | ----- | ------------- | -------- | -------- | ----- | --------- | ------ |
-| STRING | β
| β
| β
| β
| β
| β
| β
|
-| ENUM | β | β
| β
| β
(1) | β | β
| β
|
-| UUID | β | β
| β
| β
(1) | β | β
| β
|
-| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
-| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
-| DATE | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
-| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
-| BSON | β | β
(1) | β
| β
(1) | β | β | β |
-| LIST | β
| β
| β
| β
| β
| (R) | β
|
-| MAP | β
| β
| β
| β
| β
| (R) | β
|
-| UNKNOWN (always null) | β
| β
| β
| β
| β
| β
| β
|
-| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+Logical types are defined by the [`union LogicalType` in parquet.thrift] and described in [LogicalTypes.md]
+
+[`union LogicalType` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L471
+[LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
+
+| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
+|-----------------------------------------|------| ------- | ------- | ------- | ---- | -------- |--------|
+| STRING | β
| β
| β
| β
| β
| β
| β
|
+| ENUM | β | β
| β
| β
(1) | β | β
| β
|
+| UUID | β | β
| β
| β
(1) | β | β
| β
|
+| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
+| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
+| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+| DATE | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
+| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
+| BSON | β | β
(1) | β
| β
(1) | β | β | β |
+| [VARIANT] | | | | | | | |
Review Comment:
arrow-go implemented variant so you can add the check mark there for it π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "platypii (via GitHub)" <gi...@apache.org>.
platypii commented on code in PR #123:
URL: https://github.com/apache/parquet-site/pull/123#discussion_r2455620300
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -43,30 +49,43 @@ Implementations:
### Logical types
-| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
-| ----------------------------------------- | ----- | ------------- | -------- | -------- | ----- | --------- | ------ |
-| STRING | β
| β
| β
| β
| β
| β
| β
|
-| ENUM | β | β
| β
| β
(1) | β | β
| β
|
-| UUID | β | β
| β
| β
(1) | β | β
| β
|
-| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
-| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
-| DATE | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
-| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
-| BSON | β | β
(1) | β
| β
(1) | β | β | β |
-| LIST | β
| β
| β
| β
| β
| (R) | β
|
-| MAP | β
| β
| β
| β
| β
| (R) | β
|
-| UNKNOWN (always null) | β
| β
| β
| β
| β
| β
| β
|
-| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+Logical types are defined by the [`union LogicalType` in parquet.thrift] and described in [LogicalTypes.md]
+
+[`union LogicalType` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L471
+[LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
+
+| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
+|-----------------------------------------|------| ------- | ------- | ------- | ---- | -------- |--------|
+| STRING | β
| β
| β
| β
| β
| β
| β
|
+| ENUM | β | β
| β
| β
(1) | β | β
| β
|
+| UUID | β | β
| β
| β
(1) | β | β
| β
|
+| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
+| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
+| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+| DATE | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
+| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
+| BSON | β | β
(1) | β
| β
(1) | β | β | β |
+| [VARIANT] | | | | | | | |
+| [GEOMETRY] | | | | | | | |
+| [GEOGRAPHY] | | | | | | | |
Review Comment:
Does duckdb support geometery / geography? When I load geospatial files from the parquet-testing repo, duckdb does not decode them?
```
D from 'geospatial.parquet';
βββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β group β wkt β geometry β
β varchar β varchar β blob β
βββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β all β POINT (30 10) β \x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00>@\x00\x00\x00\x00\x00\x00$@ β
β all β LINESTRING (30 10,β¦ β \x01\x02\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00>@\x00\x00\x00\x00\x00\x00$@\x00\x00\x00\x00\x00\x00$@β¦ β
β all β POLYGON ((30 10, 4β¦ β \x01\x03\x00\x00\x00\x01\x00\x00\x00\x05\x00\x00\x00\x00\x00\x00\x00\x00\x00>@\x00\x00\x00\x00\x00\x00$@\x00\x00\xβ¦ β
β
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #123:
URL: https://github.com/apache/parquet-site/pull/123#discussion_r2465061229
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -43,30 +49,43 @@ Implementations:
### Logical types
-| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
-| ----------------------------------------- | ----- | ------------- | -------- | -------- | ----- | --------- | ------ |
-| STRING | β
| β
| β
| β
| β
| β
| β
|
-| ENUM | β | β
| β
| β
(1) | β | β
| β
|
-| UUID | β | β
| β
| β
(1) | β | β
| β
|
-| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
-| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
-| DATE | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
-| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
-| BSON | β | β
(1) | β
| β
(1) | β | β | β |
-| LIST | β
| β
| β
| β
| β
| (R) | β
|
-| MAP | β
| β
| β
| β
| β
| (R) | β
|
-| UNKNOWN (always null) | β
| β
| β
| β
| β
| β
| β
|
-| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+Logical types are defined by the [`union LogicalType` in parquet.thrift] and described in [LogicalTypes.md]
+
+[`union LogicalType` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L471
+[LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
+
+| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
+|-----------------------------------------|------| ------- | ------- | ------- | ---- | -------- |--------|
+| STRING | β
| β
| β
| β
| β
| β
| β
|
+| ENUM | β | β
| β
| β
(1) | β | β
| β
|
+| UUID | β | β
| β
| β
(1) | β | β
| β
|
+| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
+| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
+| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+| DATE | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
+| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
+| BSON | β | β
(1) | β
| β
(1) | β | β | β |
+| [VARIANT] | | | | | | | |
+| [GEOMETRY] | | | | | | | |
+| [GEOGRAPHY] | | | | | | | |
Review Comment:
> @alamb I think @paleolimbot is right. Parquet Geometry / Geography are widely supported now including Arrow C++ and Parquet Java official releases which were done by @paleolimbot and @zhangfengcdt a while ago
Thank you @jiayuasu -- updated in 86692aa
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #123:
URL: https://github.com/apache/parquet-site/pull/123#discussion_r2462586817
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -43,30 +49,43 @@ Implementations:
### Logical types
-| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
-| ----------------------------------------- | ----- | ------------- | -------- | -------- | ----- | --------- | ------ |
-| STRING | β
| β
| β
| β
| β
| β
| β
|
-| ENUM | β | β
| β
| β
(1) | β | β
| β
|
-| UUID | β | β
| β
| β
(1) | β | β
| β
|
-| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
-| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
-| DATE | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
-| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
-| BSON | β | β
(1) | β
| β
(1) | β | β | β |
-| LIST | β
| β
| β
| β
| β
| (R) | β
|
-| MAP | β
| β
| β
| β
| β
| (R) | β
|
-| UNKNOWN (always null) | β
| β
| β
| β
| β
| β
| β
|
-| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+Logical types are defined by the [`union LogicalType` in parquet.thrift] and described in [LogicalTypes.md]
+
+[`union LogicalType` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L471
+[LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
+
+| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
+|-----------------------------------------|------| ------- | ------- | ------- | ---- | -------- |--------|
+| STRING | β
| β
| β
| β
| β
| β
| β
|
+| ENUM | β | β
| β
| β
(1) | β | β
| β
|
+| UUID | β | β
| β
| β
(1) | β | β
| β
|
+| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
+| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
+| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+| DATE | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
+| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
+| BSON | β | β
(1) | β
| β
(1) | β | β | β |
+| [VARIANT] | | | | | | | |
+| [GEOMETRY] | | | | | | | |
+| [GEOGRAPHY] | | | | | | | |
Review Comment:
@alamb I think @paleolimbot is right. Parquet Geometry / Geography are widely supported now including `Arrow C++` and `Parquet Java` official releases which were done by @paleolimbot and @zhangfengcdt a while ago
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #123:
URL: https://github.com/apache/parquet-site/pull/123#discussion_r2455434022
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -43,30 +49,43 @@ Implementations:
### Logical types
-| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
-| ----------------------------------------- | ----- | ------------- | -------- | -------- | ----- | --------- | ------ |
-| STRING | β
| β
| β
| β
| β
| β
| β
|
-| ENUM | β | β
| β
| β
(1) | β | β
| β
|
-| UUID | β | β
| β
| β
(1) | β | β
| β
|
-| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
-| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
-| DATE | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
-| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
-| BSON | β | β
(1) | β
| β
(1) | β | β | β |
-| LIST | β
| β
| β
| β
| β
| (R) | β
|
-| MAP | β
| β
| β
| β
| β
| (R) | β
|
-| UNKNOWN (always null) | β
| β
| β
| β
| β
| β
| β
|
-| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+Logical types are defined by the [`union LogicalType` in parquet.thrift] and described in [LogicalTypes.md]
+
+[`union LogicalType` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L471
+[LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
+
+| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
+|-----------------------------------------|------| ------- | ------- | ------- | ---- | -------- |--------|
+| STRING | β
| β
| β
| β
| β
| β
| β
|
+| ENUM | β | β
| β
| β
(1) | β | β
| β
|
+| UUID | β | β
| β
| β
(1) | β | β
| β
|
+| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
+| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
+| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+| DATE | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
+| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
+| BSON | β | β
(1) | β
| β
(1) | β | β | β |
+| [VARIANT] | | | | | | | |
+| [GEOMETRY] | | | | | | | |
+| [GEOGRAPHY] | | | | | | | |
Review Comment:
My mental ASCII interpolation might be off but I believe that geometry is supported by everything except arrow-go and cudf (read support only for hyparquet); and geography is supported by everything except cudf and duckdb (read support only for hyparquet). In arrow-rs you currently have to try pretty hard to actually get the types to be written or read but it's tested and can be done.
```suggestion
| [GEOMETRY] | β
| β
| | β
| | (R) | β
|
| [GEOGRAPHY] | β
| β
| | β
| | (R) | |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Variant, Geometry, Geography types to implementation status page [parquet-site]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #123:
URL: https://github.com/apache/parquet-site/pull/123#discussion_r2455707881
##########
content/en/docs/File Format/implementationstatus.md:
##########
@@ -43,30 +49,43 @@ Implementations:
### Logical types
-| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
-| ----------------------------------------- | ----- | ------------- | -------- | -------- | ----- | --------- | ------ |
-| STRING | β
| β
| β
| β
| β
| β
| β
|
-| ENUM | β | β
| β
| β
(1) | β | β
| β
|
-| UUID | β | β
| β
| β
(1) | β | β
| β
|
-| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
-| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
-| DATE | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
-| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
-| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
-| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
-| BSON | β | β
(1) | β
| β
(1) | β | β | β |
-| LIST | β
| β
| β
| β
| β
| (R) | β
|
-| MAP | β
| β
| β
| β
| β
| (R) | β
|
-| UNKNOWN (always null) | β
| β
| β
| β
| β
| β
| β
|
-| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+Logical types are defined by the [`union LogicalType` in parquet.thrift] and described in [LogicalTypes.md]
+
+[`union LogicalType` in parquet.thrift]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L471
+[LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
+
+| Data type | arrow | parquet-java | arrow-go | arrow-rs | cudf | hyparquet | duckdb |
+|-----------------------------------------|------| ------- | ------- | ------- | ---- | -------- |--------|
+| STRING | β
| β
| β
| β
| β
| β
| β
|
+| ENUM | β | β
| β
| β
(1) | β | β
| β
|
+| UUID | β | β
| β
| β
(1) | β | β
| β
|
+| 8, 16, 32, 64 bit signed and unsigned INT | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| DECIMAL (BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| (R) |
+| DECIMAL (FIXED_LEN_BYTE_ARRAY) | β
| β
| β
| β
| β
| β
| β
|
+| FLOAT16 | β
| β
(1) | β
| β
| β
| β
| β
|
+| DATE | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT32) | β
| β
| β
| β
| β
| β
| β
|
+| TIME (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| TIMESTAMP (INT64) | β
| β
| β
| β
| β
| β
| β
|
+| INTERVAL | β
| β
(1) | β
| β
| β | β
| β
|
+| JSON | β
| β
(1) | β
| β
(1) | β | β
| β
|
+| BSON | β | β
(1) | β
| β
(1) | β | β | β |
+| [VARIANT] | | | | | | | |
+| [GEOMETRY] | | | | | | | |
+| [GEOGRAPHY] | | | | | | | |
Review Comment:
I wrote that based on https://github.com/duckdb/duckdb/pull/19244 , where between 1.4.0 and 1.4.1 the writer was updated to not write GEOMETRY by default (I assumed that if it was writing it by default it could also read it but I haven't checked). Probably `LOAD spatial;` is needed first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org