You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@parquet.apache.org by ga...@apache.org on 2025/03/26 12:36:46 UTC
(parquet-testing) branch master updated: Add test file with unknown logical type (#72)
This is an automated email from the ASF dual-hosted git repository.
gangwu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/parquet-testing.git
The following commit(s) were added to refs/heads/master by this push:
new 18d1754 Add test file with unknown logical type (#72)
18d1754 is described below
commit 18d17540097fca7c40be3d42c167e6bfad90763c
Author: Dewey Dunnington <de...@dunnington.ca>
AuthorDate: Wed Mar 26 07:36:41 2025 -0500
Add test file with unknown logical type (#72)
---
bad_data/README.md | 2 +-
data/README.md | 5 +++--
data/unknown-logical-type.parquet | Bin 0 -> 1051 bytes
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/bad_data/README.md b/bad_data/README.md
index 0a030a0..bb12b6f 100644
--- a/bad_data/README.md
+++ b/bad_data/README.md
@@ -24,7 +24,7 @@ These are files used for reproducing various bugs that have been reported.
corrupted.
* ARROW-RS-GH-6229-DICTHEADER.parquet: tests a case where the number of values
stored in dictionary page header is negative.
-* ARROW-RS-GH-6229-LEVELS.parquet: tests a case where a page has insufficient
+* ARROW-RS-GH-6229-LEVELS.parquet: tests a case where a page has insufficient
repetition levels.
* ARROW-GH-41321.parquet: test case of https://github.com/apache/arrow/issues/41321
where decoded rep / def levels is less than num_values in page_header.
diff --git a/data/README.md b/data/README.md
index df8690e..cc7909b 100644
--- a/data/README.md
+++ b/data/README.md
@@ -57,6 +57,7 @@
| repeated_primitive_no_list.parquet | REPEATED INT32 and BYTE_ARRAY fields without LIST annotation. See [note](#REPEATED-primitive-fields-with-no-LIST-annotation) |
| map_no_value.parquet | MAP with null values, MAP with INT32 keys and no values, and LIST<INT32> column with same values as the MAP keys. See [map_no_value.md](map_no_value.md) |
| page_v2_empty_compressed.parquet | An INT32 column with DataPageV2, all values are null, the zero-sized data is compressed using ZSTD |
+| unknown-logical-type.parquet | A file containing a column annotated with a LogicalType whose identifier has been set to an abitrary high value to check the behaviour of an old reader reading a file written by a new writer containing an unsupported type (see [related issue](https://github.com/apache/arrow/issues/41764)). |
TODO: Document what each file is in the table above.
@@ -403,8 +404,8 @@ where the Map key fields are marked as optional rather than required.
This is not spec-compliant, yet appears in a number of existing data files in the wild.
This issue has been fixed in:
-- [Trino v386+](https://github.com/trinodb/trino/commit/3247bd2e64d7422bd13e805cd67cfca3fa8ba520)
-- [Presto v0.274+](https://github.com/prestodb/presto/commit/842b46972c11534a7729d0a18e3abc5347922d1a)
+- [Trino v386+](https://github.com/trinodb/trino/commit/3247bd2e64d7422bd13e805cd67cfca3fa8ba520)
+- [Presto v0.274+](https://github.com/prestodb/presto/commit/842b46972c11534a7729d0a18e3abc5347922d1a)
We can recreate these problematic files for testing [arrow-rs #5630](https://github.com/apache/arrow-rs/pull/5630)
with relevant Presto/Trino CLI, or with AWS Athena Console:
diff --git a/data/unknown-logical-type.parquet b/data/unknown-logical-type.parquet
new file mode 100644
index 0000000..0548911
Binary files /dev/null and b/data/unknown-logical-type.parquet differ