You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "alexey-milovidov (via GitHub)" <gi...@apache.org> on 2025/09/29 11:36:07 UTC
[I] parquet-reader outputting invalid JSON [arrow]
alexey-milovidov opened a new issue, #47668:
URL: https://github.com/apache/arrow/issues/47668
### Describe the bug, including details regarding any error messages, version, and platform.
Example output without this:
```json
{
"FileName": "/home/ubuntu/ClickHouse/tests/queries/0_stateless/data_parquet/68131.parquet",
"Version": "1.0",
"CreatedBy": "parquet-mr version 1.12.2 (build 77e30c8093386ec52c3cfa6c34b7ef3321322c94)",
"TotalRows": "1",
"NumberOfRowGroups": "1",
"NumberOfRealColumns": "1",
"NumberOfColumns": "1",
"Columns": [
{ "Id": "0", "Name": "f", "PhysicalType": "INT32", "ConvertedType": "NONE", "LogicalType": {"Type": "None"} }
],
"RowGroups": [
{
"Id": "0", "TotalBytes": "43", "TotalCompressedBytes": "43", "Rows": "1",
"ColumnChunks": [
{"Id": "0", "Values": "2", "StatsSet": "True", "Stats": {"NumNulls": "0", "Max": "2", "Min": "1" },
"Compression": "UNCOMPRESSED", "Encodings": "PLAIN", "UncompressedSize": "43", "CompressedSize": "43", ColumnIndex {"offset": "47", "length": "23"}", OffsetIndex {"offset": "70", "length": "10"}" }
]
}
]
}
```
It goes off the rails starting at `ColumnIndex`.
ClickHouse uses this tool in test `00900_long_parquet_load` (ran manually to get table schemas).
The fix is here: https://github.com/ClickHouse/arrow/pull/72
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] parquet-reader outputting invalid JSON [arrow]
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve closed issue #47668: parquet-reader outputting invalid JSON
URL: https://github.com/apache/arrow/issues/47668
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] parquet-reader outputting invalid JSON [arrow]
Posted by "stackedsax (via GitHub)" <gi...@apache.org>.
stackedsax commented on issue #47668:
URL: https://github.com/apache/arrow/issues/47668#issuecomment-5981235919
This was fixed upstream in #44532 (GH-44101), released in Arrow 19.0.0. The malformed `ColumnIndex {...}"` output came from a stray string literal in `printer.cc` introduced by #38860 (18.0.0); #44532 corrected it to emit `"ColumnIndex": {"offset": ..., "length": ...}`.
Since then the printer has been migrated to a proper `JsonWriter` (#50877, #50990), so malformed keys of this kind are structurally impossible now, and #47190 added a `ValidJsonOutput` test in `reader_test.cc` that parses the printer's output to make sure it stays valid JSON.
ClickHouse's fork (ClickHouse/arrow#72, linked above) is based on `release/16.1.0`, which predates the fix, so the fork would need to rebase or cherry-pick f5691d467c rather than anything changing here.
@thisisnic I think this one can be closed as fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] parquet-reader outputting invalid JSON [arrow]
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on issue #47668:
URL: https://github.com/apache/arrow/issues/47668#issuecomment-6157486975
Thanks for the investigation @stackedsax!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org