You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "valkum (via GitHub)" <gi...@apache.org> on 2025/09/04 17:37:58 UTC
[I] Fail to merge schema field for patitioned table with dict. [datafusion]
valkum opened a new issue, #17421:
URL: https://github.com/apache/datafusion/issues/17421
### Describe the bug
I am trying out datafusion for some refactoring. I am testing with the following setup:
```
+-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-------------+
| column_name | data_type | is_nullable |
+-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-------------+
| name | Utf8View | NO |
| group | Dictionary(UInt16, Utf8) | NO |
| market | Struct([Field { name: "market", data_type: Dictionary(UInt16, Utf8), nullable: true, dict_id: 1, dict_is_ordered: false, metadata: {} }, ...])| YES |
|... | ... | ... |
+-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-------------+
```
I am trying to use `group` as a partition key and storing the table as a parquet hive.
When reading the hive either with datafusion-cli or using Rust with
```
ctx.register_parquet(
"test",
src,
ParquetReadOptions::default()
.file_sort_order(vec![vec![
SortExpr::new(col("name"), true, false),
SortExpr::new(col("group"), true, false),
]])
.table_partition_cols(vec![
(
"group".to_string(),
DataType::Dictionary(Box::new(DataType::UInt16), Box::new(DataType::Utf8)),
),
]),
)
.await
```
I am getting the following error:
`Arrow error: Schema error: Fail to merge schema field 'market' because from dict_id = 1 does not match 0`.
I assume this is caused because the order of dict creation is different. Do note that this does not occur with `datafusion.execution.keep_partition_by_columns = True`. But then you run into https://github.com/apache/datafusion/issues/17420
### To Reproduce
Create table with schema from above.
```
COPY test TO 'test_out' STORED AS PARQUET PARTITIONED BY group;
CREATE EXTERNAL TABLE test2
STORED AS PARQUET
PARTITIONED BY (group)
LOCATION 'test_out';
```
### Expected behavior
The hive/partitioned table should be loaded.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Fail to merge schema field for partitioned table with dict [datafusion]
Posted by "VaibhaveS (via GitHub)" <gi...@apache.org>.
VaibhaveS commented on issue #17421:
URL: https://github.com/apache/datafusion/issues/17421#issuecomment-5230726281
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Fail to merge schema field for partitioned table with dict [datafusion]
Posted by "VaibhaveS (via GitHub)" <gi...@apache.org>.
VaibhaveS commented on issue #17421:
URL: https://github.com/apache/datafusion/issues/17421#issuecomment-5230888218
Verified on current main with nested `[Dictionary(UInt16, Utf8)]` field using `dict_id = 1`, partitioned Parquet COPY/read succeeds; unable to reproduce.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org