You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/11/10 11:09:27 UTC
[GitHub] [arrow-rs] crepererum opened a new pull request, #3080: early type checks in `RowConverter`
crepererum opened a new pull request, #3080:
URL: https://github.com/apache/arrow-rs/pull/3080
# Which issue does this PR close?
Closes #3077.
# Rationale for this change
1. There should be an easy way to check if a row format supports a certain type before conversion. This can be useful for DataFusion, where certain decisions are made during planning time.
2. Improve `RowConverter` API by implementing "if you can create it, you can also feed data into it (modulo overflows)".
# What changes are included in this PR?
- `RowConverter::supports_fields` to check if fields are supported by row format
- `RowConverter::new` can now fail
- Small clean (remove unnecessary match arm)
# Are there any user-facing changes?
**BREAKING CHANGE:** `RowConverter::new` can now fail
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] ursabot commented on pull request #3080: early type checks in `RowConverter`
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #3080:
URL: https://github.com/apache/arrow-rs/pull/3080#issuecomment-1310922969
Benchmark runs are scheduled for baseline = ed20bf1431de784c6193cf2e21bcc6d178aa5de1 and contender = 8d364fe430c39d99ed68665c8c4223e02f54ab56. 8d364fe430c39d99ed68665c8c4223e02f54ab56 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/782abc35cf4946b7ba206ea45ec3a6a4...02d6b721418a4577a637e1b372ba73ff/)
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/aa5ae54223c54017a628e2486b6ef56f...927010c2b39e47f8bca50280f075239a/)
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/7f0236152d6148da873980a14cef5f7f...7afb9f1d77ca4f36bab231e56acdfbe3/)
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/aa104e493f734902b2bd4b25634fa991...436707c3740e4d37b7cfb2c2f332f0a5/)
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #3080: early type checks in `RowConverter`
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #3080:
URL: https://github.com/apache/arrow-rs/pull/3080#discussion_r1019551725
##########
arrow/src/row/mod.rs:
##########
@@ -229,14 +229,70 @@ impl SortField {
}
}
+/// Helper that is used for [`downcast_primitive`] and just returns `true`.
+macro_rules! always_true {
+ ($t:ty) => {
+ true
+ };
+}
+
impl RowConverter {
/// Create a new [`RowConverter`] with the provided schema
- pub fn new(fields: Vec<SortField>) -> Self {
+ pub fn new(fields: Vec<SortField>) -> Result<Self> {
+ if !Self::supports_fields(&fields) {
+ return Err(ArrowError::NotYetImplemented(format!(
+ "not yet implemented: {:?}",
+ fields
+ )));
+ }
+
let interners = (0..fields.len()).map(|_| None).collect();
- Self {
+ Ok(Self {
fields: fields.into(),
interners,
+ })
+ }
+
+ /// Check if the given fields are supported by the row format.
+ pub fn supports_fields(fields: &[SortField]) -> bool {
+ for f in fields {
+ let data_type = &f.data_type;
Review Comment:
I think this can be simplified to something like
```
match data_type {
DataType::Dictionary(_, v) => !DataType::is_nested(v.as_ref()),
_ => !DataType::is_nested(data_type),
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on pull request #3080: early type checks in `RowConverter`
Posted by GitBox <gi...@apache.org>.
tustvold commented on PR #3080:
URL: https://github.com/apache/arrow-rs/pull/3080#issuecomment-1310892108
I took the liberty of simplifying this so it can make the arrow release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #3080: early type checks in `RowConverter`
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #3080:
URL: https://github.com/apache/arrow-rs/pull/3080#discussion_r1019559696
##########
arrow/src/row/mod.rs:
##########
@@ -229,14 +229,70 @@ impl SortField {
}
}
+/// Helper that is used for [`downcast_primitive`] and just returns `true`.
+macro_rules! always_true {
+ ($t:ty) => {
+ true
+ };
+}
+
impl RowConverter {
/// Create a new [`RowConverter`] with the provided schema
- pub fn new(fields: Vec<SortField>) -> Self {
+ pub fn new(fields: Vec<SortField>) -> Result<Self> {
+ if !Self::supports_fields(&fields) {
+ return Err(ArrowError::NotYetImplemented(format!(
+ "not yet implemented: {:?}",
+ fields
+ )));
+ }
+
let interners = (0..fields.len()).map(|_| None).collect();
- Self {
+ Ok(Self {
fields: fields.into(),
interners,
+ })
+ }
+
+ /// Check if the given fields are supported by the row format.
+ pub fn supports_fields(fields: &[SortField]) -> bool {
+ for f in fields {
+ let data_type = &f.data_type;
Review Comment:
https://github.com/apache/arrow-rs/pull/3083 will make this even simpler
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold merged pull request #3080: early type checks in `RowConverter`
Posted by GitBox <gi...@apache.org>.
tustvold merged PR #3080:
URL: https://github.com/apache/arrow-rs/pull/3080
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org