Re: [PR] GH-101: Add JSON and BSON logical type test files [parquet-testing]

["alamb (via GitHub)" <gi...@apache.org>]

alamb commented on code in PR #118:
URL: https://github.com/apache/parquet-testing/pull/118#discussion_r3707244134


##########
data/bson.parquet:
##########


Review Comment:
   I verified the contents of this with datafusion-cli:
   
   ```sql
   $ datafusion-cli  -c "SELECT * from 'data/json.parquet'"
   DataFusion CLI v54.1.0
   +------------------+
   | json_field       |
   +------------------+
   | {"a":1}          |
   | {"a":1,"b":null} |
   | [1,null,3]       |
   | NULL             |
   +------------------+
   4 row(s) fetched.
   Elapsed 0.039 seconds.
   ```



##########
data/json.parquet:
##########


Review Comment:
   Contents look good:
   ```sql
   datafusion-cli  -c "SELECT * from 'data/bson.parquet'"
   DataFusion CLI v54.1.0
   +--------------------------------+
   | bson_field                     |
   +--------------------------------+
   | 0c0000001061000100000000       |
   | 0f000000106100010000000a620000 |
   | NULL                           |
   +--------------------------------+
   3 row(s) fetched.
   ```
   
   I decoded into bson and it matches the contents:
   
   ```python
   andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ uv run --with pymongo --with pyarrow python3 -c "
     import bson, pyarrow.parquet as pq
     t = pq.read_table('data/bson.parquet')
     for v in t.column('bson_field'):
         print('NULL' if not v.is_valid else bson.decode(v.as_py()))
     "
   {'a': 1}
   {'a': 1, 'b': None}
   NULL
   ```
   
   As does metadata
   
   ```shell
   andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ ~/Software/arrow-rs/target/release/parquet-schema data/bson.parquet
   Metadata for file: data/bson.parquet
   
   version: 1
   num of rows: 3
   created by: parquet-mr version 1.18.0-SNAPSHOT (build 1a9e455655604acf09cdd45b4e2958661d38281c)
   metadata:
     writer.model.name: example
   message bson_fixture {
     OPTIONAL BYTE_ARRAY bson_field (BSON);
   }
   ```



##########
data/bson.parquet:
##########


Review Comment:
   Metadata is good too
   ```shell
   andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ ~/Software/arrow-rs/target/release/parquet-schema data/json.parquet
   Metadata for file: data/json.parquet
   
   version: 1
   num of rows: 4
   created by: parquet-mr version 1.18.0-SNAPSHOT (build 1a9e455655604acf09cdd45b4e2958661d38281c)
   metadata:
     writer.model.name: example
   message json_fixture {
     OPTIONAL BYTE_ARRAY json_field (JSON);
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org