You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "setop (via GitHub)" <gi...@apache.org> on 2024/05/27 11:15:11 UTC
[I] "schema" and "arrow_schema" incompatibility when concatenate [arrow-rs]
setop opened a new issue, #5804:
URL: https://github.com/apache/arrow-rs/issues/5804
## Describe the bug
Tabular data split into two parquet files, with the same schema, one produced with parquet-ccp-arrow, the other with parquet-rs.
When concatenating I get `Error: General("inputs must have the same schema, [...]")`
## To Reproduce
* `pip install pyarrow`
* create csv
```bash
cat << EOF > in.csv
x,y
1,2
EOF
```
* convert csv to parquet using python wrapper over [parquet-cpp](https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html)
```python
# csv_to_parquet.py
import sys
from pyarrow import csv, parquet as pq
table = csv.read_csv(sys.argv[1])
pq.write_table(table, sys.argv[2])
```
`python csv_to_parquet.py in.csv a.parquet`
* create schema file
```bash
cat << EOF > schema.txt
message schema {
OPTIONAL INT64 x;
OPTIONAL INT64 y;
}
EEE
```
* convert csv to parquet using parquet-rs
```bash
parquet-fromcsv --schema schema.txt --input-file a.csv -h --output-file b.parquet -w 2
```
* concatenate these two files `parquet-concat c.parquet {a,b}.parquet`
## Expected behavior
Files get concatenated, two rows.
## Additional context
The only difference is the message name.
```diff
GroupType {
basic_info: BasicTypeInfo {
- name: \"schema\",
+ name: \"arrow_schema\",
repetition: None,
converted_type: NONE,
logical_type: None,
```
### schemas
```
Metadata for file: a.parquet
version: 2
num of rows: 1
created by: parquet-cpp-arrow version 16.1.0
metadata:
ARROW:schema: /////6gAAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAMAAAACAAIAAAABAAIAAAABAAAAAIAAABEAAAABAAAANT///8AAAECEAAAABQAAAAEAAAAAAAAAAEAAAB5AAAAxP///wAAAAFAAAAAEAAUAAgABgAHAAwAAAAQABAAAAAAAAECEAAAABwAAAAEAAAAAAAAAAEAAAB4AAAACAAMAAgABwAIAAAAAAAAAUAAAAA=
message schema {
OPTIONAL INT64 x;
OPTIONAL INT64 y;
}
```
```
Metadata for file: b.parquet
version: 2
num of rows: 1
created by: parquet-rs version 51.0.0
metadata:
ARROW:schema: /////6gAAAAQAAAAAAAKAAwACgAJAAQACgAAABAAAAAAAQQACAAIAAAABAAIAAAABAAAAAIAAABEAAAABAAAANT///8QAAAAGAAAAAAAAQIUAAAAxP///0AAAAAAAAABAAAAAAEAAAB5AAAAEAAUABAADgAPAAQAAAAIABAAAAAYAAAAIAAAAAAAAQIcAAAACAAMAAQACwAIAAAAQAAAAAAAAAEAAAAAAQAAAHgAAAA=
message arrow_schema {
OPTIONAL INT64 x;
OPTIONAL INT64 y;
}
```
I also tested with both files in writer version 1, with a mix of version 1 and 2. Same error.
---
I started commenting on #4799, but since it is closed, not sure it'll get noticed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support Different Schemas in parquet-concat [arrow-rs]
Posted by "setop (via GitHub)" <gi...@apache.org>.
setop commented on issue #5804:
URL: https://github.com/apache/arrow-rs/issues/5804#issuecomment-2133913561
I ask `parquet-fromcsv` to create a second parquet file with the same schema as the first one.
So why "message schema" is silently changed to "message arrow_schema" ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Schema Root Message Name Ignored by parquet-concat [arrow-rs]
Posted by "hesampakdaman (via GitHub)" <gi...@apache.org>.
hesampakdaman commented on issue #5804:
URL: https://github.com/apache/arrow-rs/issues/5804#issuecomment-2136176772
Is this due to `arrow_to_parquet_schema` which always sets the group name to "arrow_schema"? It's used by the writer in `parquet-fromcsv`.
```rust
/// Convert arrow schema to parquet schema
pub fn arrow_to_parquet_schema(schema: &Schema) -> Result<SchemaDescriptor> {
let fields = schema
.fields()
.iter()
.map(|field| arrow_to_parquet_type(field).map(Arc::new))
.collect::<Result<_>>()?;
let group = Type::group_type_builder("arrow_schema")
.with_fields(fields)
.build()?;
Ok(SchemaDescriptor::new(Arc::new(group)))
}
```
If this is the case, is it a viable solution to just add metadata from `arrow_schema_from_string` in `parquet-fromcsv`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Schema Root Message Name Ignored by parquet-fromcsv [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5804:
URL: https://github.com/apache/arrow-rs/issues/5804#issuecomment-2144831146
`label_issue.py` automatically added labels {'parquet'} from #5814
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Schema Root Message Name Ignored by parquet-fromcsv [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #5804: Schema Root Message Name Ignored by parquet-fromcsv
URL: https://github.com/apache/arrow-rs/issues/5804
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] "schema" and "arrow_schema" incompatibility when concatenate [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5804:
URL: https://github.com/apache/arrow-rs/issues/5804#issuecomment-2133633514
This is expected and not a bug, you're attempting to concatenate two parquet files with differing schema which is not supported.
As the schema are not the same, not only is it not clear what the output schema should be, but more fundamentally as the tool concatenates encoded data, we can't permit schema changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org