You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "crepererum (via GitHub)" <gi...@apache.org> on 2023/03/10 11:27:42 UTC
[GitHub] [arrow-datafusion] crepererum opened a new pull request, #5545: refactor: user may choose to dict-encode partition values
crepererum opened a new pull request, #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545
# Which issue does this PR close?
\-
# Rationale for this change
Let the user decide if they may want to encode partition values for file-based data sources. Dictionary encoding makes sense for string values but is probably pointless or even counterproductive for integer types.
# What changes are included in this PR?
- the data type specified `partition_columns` is now the actual output type
- SQL mapping has been changed to follow that EXCEPT the default for unspecified types which I kept at `Dict(u16, utf8)`
- I kept the "buffer reuse" logic that was in there but extended it more dictionary types. I use a type-safe approach here because Arrow buffers may become typed at some point w/ the arrow2 merger
# Are these changes tested?
Adjusted existing tests.
# Are there any user-facing changes?
**BREAKING:** Types for partition columns in file-based sources are no longer dictionary encoded by default. The user MUST choose a dictionary type if they want to achieve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] comphead commented on a diff in pull request #5545: refactor: user may choose to dict-encode partition values
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#discussion_r1132641617
##########
datafusion/core/src/physical_plan/file_format/parquet.rs:
##########
@@ -1656,26 +1655,50 @@ mod tests {
object_meta: meta,
partition_values: vec![
ScalarValue::Utf8(Some("2021".to_owned())),
- ScalarValue::Utf8(Some("10".to_owned())),
- ScalarValue::Utf8(Some("26".to_owned())),
+ ScalarValue::UInt8(Some(10)),
+ ScalarValue::Dictionary(
+ Box::new(DataType::UInt16),
+ Box::new(ScalarValue::Utf8(Some("26".to_owned()))),
+ ),
],
range: None,
extensions: None,
};
+ let expected_schema = Schema::new(vec![
+ Field::new("id", DataType::Int32, true),
+ Field::new("bool_col", DataType::Boolean, true),
+ Field::new("tinyint_col", DataType::Int32, true),
Review Comment:
thats looks like typo -> `tinyint_col` should be `DataType::UInt8` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1466485874
Added docs here https://github.com/apache/arrow-datafusion/pull/5576
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] crepererum commented on pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "crepererum (via GitHub)" <gi...@apache.org>.
crepererum commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1467741662
I've made the following changes:
- `partition_type_wrap` and `partition_value_wrap` helper methods are now public
- docs mention these helpers
- there's a transition auto-fix that adds the dictionary wrapping to the value if the type demands it but spits out a warning. this is also tested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] wjones127 commented on pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1527920960
Noted 👍 I'm not objecting to this change. Just wanted to provide information to any other developers who end up reading this PR and are thinking about how they will adapt their code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] crepererum commented on a diff in pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "crepererum (via GitHub)" <gi...@apache.org>.
crepererum commented on code in PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#discussion_r1135372688
##########
datafusion/core/src/physical_plan/file_format/parquet.rs:
##########
@@ -1646,10 +1646,10 @@ mod tests {
let meta = local_unpartitioned_file(filename);
- let schema = ParquetFormat::default()
+ let schema = dbg!(ParquetFormat::default()
Review Comment:
no, fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1466517323
> Field contains other information that is irrelevant here like nullability. We could assert on the correct value though (i.e. not nullable). Also this only concerns on of the two API changes, the other one (PartitionedFile) is still silently breaking. To avoid that we would need to rename partition_values to something else. However I'm not sure if a rename would trigger people to not only follow the rename but also change the types correctly.
Good point re `Field` and non relevant columns
What about a type that made the dictionary encoding explict, something like
```rust
struct PartitionedColumn {
name: String,
data_type: DataType,
dictionary_index_type: Option<DataType>
}
```
And then have a `new` function that defaults to Dictionary 🤔 But that is kind of ugly as well and now what happens if the data_type was a dictionary in the first place ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] crepererum commented on a diff in pull request #5545: refactor: user may choose to dict-encode partition values
Posted by "crepererum (via GitHub)" <gi...@apache.org>.
crepererum commented on code in PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#discussion_r1133692368
##########
datafusion/core/src/physical_plan/file_format/parquet.rs:
##########
@@ -1656,26 +1655,50 @@ mod tests {
object_meta: meta,
partition_values: vec![
ScalarValue::Utf8(Some("2021".to_owned())),
- ScalarValue::Utf8(Some("10".to_owned())),
- ScalarValue::Utf8(Some("26".to_owned())),
+ ScalarValue::UInt8(Some(10)),
+ ScalarValue::Dictionary(
+ Box::new(DataType::UInt16),
+ Box::new(ScalarValue::Utf8(Some("26".to_owned()))),
+ ),
],
range: None,
extensions: None,
};
+ let expected_schema = Schema::new(vec![
+ Field::new("id", DataType::Int32, true),
+ Field::new("bool_col", DataType::Boolean, true),
+ Field::new("tinyint_col", DataType::Int32, true),
Review Comment:
The schema that I get from `infer_schema` (see two statements above) contains an `Int32`. I haven't touched this code and this is the current state on `main`. Not saying this is correct though, it might be a pre-existing bug. Note that the schema wasn't tested before, that's why I added this assertion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1469826327
This PR had a merge conflict so I took the liberty of fixing that and pushing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] crepererum commented on pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "crepererum (via GitHub)" <gi...@apache.org>.
crepererum commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1527261880
@wjones127 you can still dictionary-encode integers by specifying the column type in `FileScanConfig` as `DataType::Dictionary(Box::new(DataType::Int64), Box::new(DataType::UInt8))` (for example), see https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1465873451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1466623543
> You get a double-encoded dictionary? I don't know, but IMHO data_type+dictionary_index_type screams "just use DataType instead of inventing yet another type"
Yes I agree -- that is what probably should have been done in the first place, but it was not. So now we need to figure out a reasonable way to help avoid subtle regressions.
Maybe we can handle it by updating the docs / making it easy to do the right thing (use dictionary) 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] crepererum commented on a diff in pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "crepererum (via GitHub)" <gi...@apache.org>.
crepererum commented on code in PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#discussion_r1135375222
##########
datafusion/core/src/physical_plan/file_format/mod.rs:
##########
@@ -64,6 +65,26 @@ use std::{
use super::{ColumnStatistics, Statistics};
+/// Convert logical type of partition column to physical type: `Dictionary(UInt16, val_type)`.
+///
+/// You CAN use this to specify types for partition columns. However you MAY also choose not to dictionary-encode the
Review Comment:
renamed as suggested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1469925891
BTW another reason this change is an improvement, in my mind, is that it allows users to tradeoff between more efficient encoding (e.g. Dict(Int8, Utf8)) and supporting more distinct values (Dict(Int16, Utf8)).
I think originally the code always used Dictionary(Int8, ..) and someone had the usecase with more than 256 distinct values (files) so we increased the size to Dict(Int16, ...).
So this change now allows people to make that tradeoff explcitly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1466461923
(BTW I can find time to work on the API in the next day or two if needed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] wjones127 commented on pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1526917997
Got here as a downstream user who is affected by this change. Thinking through this, I wouldn't totally write off dictionary encoding integers as useless, since there still are benefits to dictionary arrays besides space savings. They essentially mark columns as having low cardinality and provide the set of unique values. Any scalar compute functions run on these columns can be applied to the dictionary while leaving the indices buffer untouched. That is an easy to way to achieve what I would expect out of a "smart" compute engine: when projecting partition columns, project the distinct values rather than the expanded/materialized array. It's possible DataFusion already handles this in a smart way I'm unaware of though.
I'd also note that the ideal partition column types are probably [run-end encoded arrays](https://github.com/apache/arrow-rs/issues/3520) (`RunArray`), once they are implemented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] comphead commented on pull request #5545: refactor: user may choose to dict-encode partition values
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1464129187
BREAKING: Types for partition columns in file-based sources are no longer dictionary encoded by default. The user MUST choose a dictionary type if they want to achieve this. -> @crepererum could you please provide a code example of how user code should be changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold commented on pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1587656337
> provide the set of unique values
FWIW this isn't how dictionaries are implemented today, there are various situations where the dictionary will contain values not referenced by an index, and/or the same value repeated multiple times. As it currently stands primitive dictionaries will almost always be less efficient both from a memory usage and performance standpoint
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] crepererum commented on pull request #5545: refactor: user may choose to dict-encode partition values
Posted by "crepererum (via GitHub)" <gi...@apache.org>.
crepererum commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1465873451
# Breaking Change
## Before
```rust
let file_scan_config = FileScanConfig {
table_partition_cols: vec![
(
"group".to_owned(),
DataType::Utf8,
),
...
],
...
};
let partitioned_file = PartitionedFile {
partition_values: vec![
ScalarValue::Utf8(Some("foo".to_owned())),
...
],
...
};
```
## After (exact)
If you want an exact conversion:
```rust
let file_scan_config = FileScanConfig {
table_partition_cols: vec![
(
"group".to_owned(),
DataType::Dictionary(
Box::new(DataType::UInt16),
Box::new(DataType::Utf8),
),
),
...
],
...
};
let partitioned_file = PartitionedFile {
partition_values: vec![
ScalarValue::Dictionary(
Box::new(DataType::UInt16),
Box::new(ScalarValue::Utf8(Some("foo".to_owned()))),
),
...
],
...
};
```
## After (alternative)
You may just decide that you don't to dictionary-encode at all:
```rust
let file_scan_config = FileScanConfig {
table_partition_cols: vec![
(
"group".to_owned(),
DataType::Utf8,
),
...
],
...
};
let partitioned_file = PartitionedFile {
partition_values: vec![
ScalarValue::Utf8(Some("foo".to_owned())),
...
],
...
};
```
or that you want a different dictionary key type:
```rust
let file_scan_config = FileScanConfig {
table_partition_cols: vec![
(
"group".to_owned(),
DataType::Dictionary(
Box::new(DataType::Int8),
Box::new(DataType::Utf8),
),
),
...
],
...
};
let partitioned_file = PartitionedFile {
partition_values: vec![
ScalarValue::Dictionary(
Box::new(DataType::Int8),
Box::new(ScalarValue::Utf8(Some("foo".to_owned()))),
),
...
],
...
};
```
Note that in all cases, the types in `FileScanConfig::table_partition_cols` and `PartitionedFile::partition_values` MUST be in-sync (strictly speaking that hasn't changed, before this PR both had been converted to UInt16 dictionary types).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#discussion_r1135343654
##########
datafusion/core/src/physical_plan/file_format/mod.rs:
##########
@@ -64,6 +65,26 @@ use std::{
use super::{ColumnStatistics, Statistics};
+/// Convert logical type of partition column to physical type: `Dictionary(UInt16, val_type)`.
+///
+/// You CAN use this to specify types for partition columns. However you MAY also choose not to dictionary-encode the
Review Comment:
This looks good -- I think is important to try and help people choose when to use these functions, but I can add that to https://github.com/apache/arrow-datafusion/pull/5576 as a follow on
I also think these functions might be easier to find if they are named something more connected to what they do (dictionary encode). Perhaps `wrap_partition_type_in_dict` but that is just a preference
##########
datafusion/core/src/physical_plan/file_format/parquet.rs:
##########
@@ -1646,10 +1646,10 @@ mod tests {
let meta = local_unpartitioned_file(filename);
- let schema = ParquetFormat::default()
+ let schema = dbg!(ParquetFormat::default()
Review Comment:
Did you intend to leave this `dbg` in?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] crepererum commented on pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "crepererum (via GitHub)" <gi...@apache.org>.
crepererum commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1466503229
> to something like (to force the API to change) ... `Field`
`Field` contains other information that is irrelevant here like nullability. We could assert on the correct value though (i.e. not nullable). Also this only concerns on of the two API changes, the other one (`PartitionedFile`) is still silently breaking. To avoid that we would need to rename `partition_values` to something else. However I'm not sure if a rename would trigger people to not only follow the rename but also change the types correctly.
> And then add some a method and docs to help the upgrade
Since the user needs to adjust their code anyways, I'm not sure if such a random method helps. It's hard to find and because the uint16 dict type is so arbitrary, I'm not sure it should exist in the long run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] crepererum commented on pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "crepererum (via GitHub)" <gi...@apache.org>.
crepererum commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1466528055
> now what happens if the data_type was a dictionary in the first place ...
You get a double-encoded dictionary? I don't know, but IMHO `data_type`+`dictionary_index_type` screams "just use `DataType` instead of inventing yet another type".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] crepererum commented on pull request #5545: Support arbitrary user defined partition column in `ListingTable` (rather than assuming they are always Dictionary encoded)
Posted by "crepererum (via GitHub)" <gi...@apache.org>.
crepererum commented on PR #5545:
URL: https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1527262595
@wjones127 you can still dictionary-encode integers by specifying the column type in `FileScanConfig` as `DataType::Dictionary(Box::new(DataType::Int64), Box::new(DataType::UInt8))` (for example), see https://github.com/apache/arrow-datafusion/pull/5545#issuecomment-1465873451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org