You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/06/14 19:54:33 UTC
[I] use StringViewArray when reading String columns from Parquet [datafusion]
alamb opened a new issue, #10921:
URL: https://github.com/apache/datafusion/issues/10921
### Is your feature request related to a problem or challenge?
Part of https://github.com/apache/datafusion/issues/10918
In order to take advantage of the parquet writer generating StringViewArrays ( https://github.com/apache/arrow-rs/issues/5530 from @ariesdevil (❤️ ) ) we need to make sure datafusion doesn't immediately cast the array back to `StringView` which would undo the benefits
```
▲
┌ ─ ─ ─ ─ ─ ─ ┐ │ After filtering,
StringArray │ any unfiltered rows
└ ─ ─ ─ ─ ─ ─ ┘ │ are gathered via
... │ the `take` kernel
│
┌────────────────────────────┐
│ │
│ FilterExec │
│ │
└────────────────────────────┘
▲
┌ ─ ─ ─ ─ ─ ─ ┐ │
StringArray │
└ ─ ─ ─ ─ ─ ─ ┘ │ Reading String data
│ from a Parquet file
... │ results in
│ StringArrays passed
┌ ─ ─ ─ ─ ─ ─ ┐ │
StringArray │
└ ─ ─ ─ ─ ─ ─ ┘ │
│
┌────────────────────────────┐
│ │
│ ParquetExec │
│ │
└────────────────────────────┘
Current situation
```
### Describe the solution you'd like
To support a phased rollout of this feature, I recommend we focus at first on only the first filtering operation
Specifically get to the point where the parquet reader will read data out as StringView like this:
```
▲
┌ ─ ─ ─ ─ ─ ─ ┐ │
StringArray │
└ ─ ─ ─ ─ ─ ─ ┘ │
... │
│
┌────────────────────────────┐
│ │
│ FilterExec │
│ │
└────────────────────────────┘
┌ ─ ─ ─ ─ ─ ─ ┐ ▲
StringViewArr │
│ ay │ │
─ ─ ─ ─ ─ ─ ─ │
... │
│
┌ ─ ─ ─ ─ ─ ─ ┐ │
StringViewArr │
│ ay │ │
─ ─ ─ ─ ─ ─ ─ │
│
┌────────────────────────────┐
│ │
│ ParquetExec │
│ │
└────────────────────────────┘
Intermediate
Situation 1: pass
StringViewArray
between ParquetExec
```
### Describe alternatives you've considered
I suggest we:
1. Make a [configuration setting](https://datafusion.apache.org/user-guide/configs.html) like "force StringViewArray" when reading parquet so we can test this. When this setting is enabled, DataFusion should configure the ParquetExec to produce `StringViewArray` regardless of the type stored in the parquet file
2. Then work on incrementally rolling out support / testing for various filter expressions (especially string functions like substring and https://github.com/apache/datafusion/issues/10919)
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] use StringViewArray when reading String columns from Parquet [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #10921: use StringViewArray when reading String columns from Parquet
URL: https://github.com/apache/datafusion/issues/10921
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] use StringViewArray when reading String columns from Parquet [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10921:
URL: https://github.com/apache/datafusion/issues/10921#issuecomment-2256050196
https://github.com/apache/datafusion/pull/11667 is ready for review 🥳
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] use StringViewArray when reading String columns from Parquet [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10921:
URL: https://github.com/apache/datafusion/issues/10921#issuecomment-2254115879
This is done on the string-view2 branch. Once we mereg https://github.com/apache/datafusion/pull/11667 we can close this ticket I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] use StringViewArray when reading String columns from Parquet [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10921:
URL: https://github.com/apache/datafusion/issues/10921#issuecomment-2218767968
Update -- @XiangpengHao found the root cause of the "small string is slower" -- read about it in this great writeup: https://github.com/apache/arrow-rs/pull/6031
TLDR is that we can make arrow/parquet reading faster than StringView always with (even) more! work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] use StringViewArray when reading String columns from Parquet [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10921:
URL: https://github.com/apache/datafusion/issues/10921#issuecomment-2170282429
> I'll take this one, can you assign me @alamb ?
BTW you can assign yourself (single word comment `take`): https://datafusion.apache.org/contributor-guide/index.html#finding-and-creating-issues-to-work-on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] use StringViewArray when reading String columns from Parquet [datafusion]
Posted by "XiangpengHao (via GitHub)" <gi...@apache.org>.
XiangpengHao commented on issue #10921:
URL: https://github.com/apache/datafusion/issues/10921#issuecomment-2170079350
I'll take this one, can you assign me @alamb ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] use StringViewArray when reading String columns from Parquet [datafusion]
Posted by "XiangpengHao (via GitHub)" <gi...@apache.org>.
XiangpengHao commented on issue #10921:
URL: https://github.com/apache/datafusion/issues/10921#issuecomment-2217882361
Want to share some thoughts here on when to use `StringViewArray` and when not.
We only consider the cost of loading data from parquet to narrow the scope.
To load a `StringArray`, we need to copy the data to a new buffer and build offset array. The extra memory we need to setup is `array len * (string len + offset size)`, for `StringArray` it is `array len * (string len + 4)`, for `BigStringArray` it is `array len * (string len + 8)`
To load a `StringViewArray`, we only need to build view array and can reuse the buffer from parquet decoder. The extra memory to setup is `array len * view size `, i.e., `array len * 16`. Note that the memory consumption of `StringViewArray` is constant to string length, i.e., it takes 16 bytes of memory no matter how long the underlying string is.
For a sufficiently large array, the time to build the array should be proportional to the extra memory we set up.
This means that if each of the individual string is small, i.e., smaller than 12 bytes, `StringArray` is actually faster than `StringViewArray`. In other words, we should use `StringViewArray` only when strings are larger than 12 bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] use StringViewArray when reading String columns from Parquet [datafusion]
Posted by "XiangpengHao (via GitHub)" <gi...@apache.org>.
XiangpengHao commented on issue #10921:
URL: https://github.com/apache/datafusion/issues/10921#issuecomment-2177420089
It seems that the current filter and parquet exec play nicely, and the following code will directly filter on the string view array (instead of converting to `StringArrary`), which is quite nice.
(the test case requires the latest arrow-rs to run)
```rust
#[tokio::test]
async fn parquet_read_filter_string_view() {
let tmp_dir = TempDir::new().unwrap();
let values = vec![Some("small"), None, Some("Larger than 12 bytes array")];
let c1: ArrayRef = Arc::new(StringViewArray::from_iter(values.iter()));
let c2: ArrayRef = Arc::new(StringArray::from_iter(values.iter()));
let batch =
RecordBatch::try_from_iter(vec![("c1", c1.clone()), ("c2", c2.clone())]).unwrap();
let file_name = {
let table_dir = tmp_dir.path().join("parquet_test");
std::fs::create_dir(&table_dir).unwrap();
let file_name = table_dir.join("part-0.parquet");
let mut writer = ArrowWriter::try_new(
fs::File::create(&file_name).unwrap(),
batch.schema(),
None,
)
.unwrap();
writer.write(&batch).unwrap();
writer.close().unwrap();
file_name
};
let ctx = SessionContext::new();
ctx.register_parquet("t", file_name.to_str().unwrap(), Default::default())
.await
.unwrap();
async fn display_result(sql: &str, ctx: &SessionContext) {
let result = ctx.sql(sql).await.unwrap().collect().await.unwrap();
arrow::util::pretty::print_batches(&result).unwrap();
for b in result {
println!("schema: {:?}", b.schema());
}
}
display_result("SELECT * from t", &ctx).await;
display_result("SELECT * from t where c1 <> 'small'", &ctx).await;
display_result("SELECT * from t where c2 <> 'small'", &ctx).await;
}
```
I'll take a closer look at the generated logical/physical plans to verify that the string view array is never being converted to string array. And if that is the case, the remaining work of this issue is probably (1) add an option to force reading StringArray as StringView array, and (2) add more tests and potentially test the generated plan uses StringViewArray consistently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org