You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Jefffrey (via GitHub)" <gi...@apache.org> on 2023/11/30 20:35:30 UTC
[I] Parquet: ColumnOrder not being written when writing parquet files [arrow-rs]
Jefffrey opened a new issue, #5152:
URL: https://github.com/apache/arrow-rs/issues/5152
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
Column order needs to be written in FileMetadata, see thrift spec:
https://github.com/apache/parquet-format/blob/066f9817332da32bdc6dc6dea833b6ee9c269934/src/main/thrift/parquet.thrift#L1133-L1149
**To Reproduce**
<!--
Steps to reproduce the behavior:
-->
In `arrow_writer/mod.rs`:
```rust
#[test]
fn test_123() {
let a = Int32Array::from(vec![1, 2, 3, 4, 5]);
let b = IntervalDayTimeArray::from(vec![0; 5]);
let batch = RecordBatch::try_from_iter(vec![
("a", Arc::new(a) as ArrayRef),
("b", Arc::new(b) as ArrayRef),
])
.unwrap();
let mut buf = Vec::with_capacity(1024);
let mut writer = ArrowWriter::try_new(&mut buf, batch.schema(), None).unwrap();
writer.write(&batch).unwrap();
writer.close().unwrap();
let bytes = Bytes::from(buf);
let options = ReadOptionsBuilder::new().with_page_index().build();
let reader = SerializedFileReader::new_with_options(bytes, options).unwrap();
dbg!(reader.metadata().file_metadata().column_orders());
}
```
Currently outputting `None`
**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
Should write ColumnOrder for each column
**Additional context**
<!--
Add any other context about the problem here.
-->
See here:
https://github.com/apache/arrow-rs/blob/6d4b8bbad95c7e4fec0c4f1fb755ad7a1c542983/parquet/src/file/writer.rs#L326-L336
Currently being set to `None` always when writing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet: ColumnOrder not being written when writing parquet files [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #5152: Parquet: ColumnOrder not being written when writing parquet files
URL: https://github.com/apache/arrow-rs/issues/5152
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet: ColumnOrder not being written when writing parquet files [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5152:
URL: https://github.com/apache/arrow-rs/issues/5152#issuecomment-1878523459
`label_issue.py` automatically added labels {'parquet'} from #5147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet: ColumnOrder not being written when writing parquet files [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #5152:
URL: https://github.com/apache/arrow-rs/issues/5152#issuecomment-1836918970
Planning to take a shot at this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org