You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "nssalian (via GitHub)" <gi...@apache.org> on 2026/07/29 15:49:14 UTC
[PR] GH-101: Add JSON and BSON logical type test files [parquet-testing]
nssalian opened a new pull request, #118:
URL: https://github.com/apache/parquet-testing/pull/118
Adds two fixtures for the JSON and BSON logical types along with the README update.
- data/json.parquet (4 rows): {"a":1}, {"a":1,"b":null}, [1,null,3], and a NULL row.
- data/bson.parquet (3 rows): BSON {"a":1}, {"a":1,"b":null}, and a NULL row.
- data/README.md has a "JSON and BSON logical types"
Both are a single optional BYTE_ARRAY column and were generated by parquet-mr 1.18.0-SNAPSHOT
Verified readable by both parquet-java's reader and pyarrow, and the BSON payloads decode with a BSON library. data/README.md has a "JSON and BSON logical types" section with the generator programs and the read-back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-101: Add JSON and BSON logical type test files [parquet-testing]
Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on PR #118:
URL: https://github.com/apache/parquet-testing/pull/118#issuecomment-5136345408
@emkornfield @alamb PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-101: Add JSON and BSON logical type test files [parquet-testing]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #118:
URL: https://github.com/apache/parquet-testing/pull/118#discussion_r3707244134
##########
data/bson.parquet:
##########
Review Comment:
I verified the contents of this with datafusion-cli:
```sql
$ datafusion-cli -c "SELECT * from 'data/json.parquet'"
DataFusion CLI v54.1.0
+------------------+
| json_field |
+------------------+
| {"a":1} |
| {"a":1,"b":null} |
| [1,null,3] |
| NULL |
+------------------+
4 row(s) fetched.
Elapsed 0.039 seconds.
```
##########
data/json.parquet:
##########
Review Comment:
Contents look good:
```sql
datafusion-cli -c "SELECT * from 'data/bson.parquet'"
DataFusion CLI v54.1.0
+--------------------------------+
| bson_field |
+--------------------------------+
| 0c0000001061000100000000 |
| 0f000000106100010000000a620000 |
| NULL |
+--------------------------------+
3 row(s) fetched.
```
I decoded into bson and it matches the contents:
```python
andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ uv run --with pymongo --with pyarrow python3 -c "
import bson, pyarrow.parquet as pq
t = pq.read_table('data/bson.parquet')
for v in t.column('bson_field'):
print('NULL' if not v.is_valid else bson.decode(v.as_py()))
"
{'a': 1}
{'a': 1, 'b': None}
NULL
```
As does metadata
```shell
andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ ~/Software/arrow-rs/target/release/parquet-schema data/bson.parquet
Metadata for file: data/bson.parquet
version: 1
num of rows: 3
created by: parquet-mr version 1.18.0-SNAPSHOT (build 1a9e455655604acf09cdd45b4e2958661d38281c)
metadata:
writer.model.name: example
message bson_fixture {
OPTIONAL BYTE_ARRAY bson_field (BSON);
}
```
##########
data/bson.parquet:
##########
Review Comment:
Metadata is good too
```shell
andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ ~/Software/arrow-rs/target/release/parquet-schema data/json.parquet
Metadata for file: data/json.parquet
version: 1
num of rows: 4
created by: parquet-mr version 1.18.0-SNAPSHOT (build 1a9e455655604acf09cdd45b4e2958661d38281c)
metadata:
writer.model.name: example
message json_fixture {
OPTIONAL BYTE_ARRAY json_field (JSON);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Add JSON and BSON logical type test files [parquet-testing]
Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on PR #118:
URL: https://github.com/apache/parquet-testing/pull/118#issuecomment-5172119485
Thank you for the review and testing it out @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Add JSON and BSON logical type test files [parquet-testing]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #118:
URL: https://github.com/apache/parquet-testing/pull/118#issuecomment-5209193793
Thanks again @nssalian
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Add JSON and BSON logical type test files [parquet-testing]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #118:
URL: https://github.com/apache/parquet-testing/pull/118
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org