You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "notfilippo (via GitHub)" <gi...@apache.org> on 2024/08/14 10:00:04 UTC
[PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
notfilippo opened a new pull request, #11978:
URL: https://github.com/apache/datafusion/pull/11978
This PR removes LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue, following the discussion on #11513
---
## Open questions
### Top level ScalarValue cast
This change currently fails this test (and potentially others) and before proceeding I would like to get some feedback on how to solve this issue. The test fails because of an interaction of `expression_simplifier` + `optimize_projections`.
The query is: `SELECT arrow_cast(1234, 'Float64') as f64, arrow_cast('foo', 'LargeUtf8') as large`
1. expression_simplifier correctly tries to evaluate_to_scalar the `arrow_cast('foo', 'LargeUtf8')` cast ([here](https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs#L626-L626))
2. this leads to the cast physical expression ([here](https://github.com/apache/datafusion/blob/main/datafusion/expr-common/src/columnar_value.rs#L187-L187)), but this is a no-op since it transforms `Utf8('foo')` (scalar) → `LargeUtf8('foo')` (array) → `Utf8('foo')` (scalar)
- TODO: if this is what we want this should probably not execute at all
3. This edit doesn't change the underlying schema of the LogicalPlan
4. `optimize_projections` rewrites the Projection and updates the schema, seeing the `Utf8('foo')` it correctly assumes that the LogicalPlan's schema field for this value should have DataType == Utf8
[This check](https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/optimizer.rs#L471-L471) is the one raising this error but I guess it should instead check if schema fields are _logically_ equivalent to eachother. I'm not totally convinced this is the correct solution because it removes some guarantees that might be expected by users downstream. Happy to hear everyone's opinion on this.
```
---- optimizer::select_arrow_cast stdout ----
thread 'optimizer::select_arrow_cast' panicked at datafusion/core/tests/optimizer/mod.rs:118:30:
called `Result::unwrap()` on an `Err` value: Context("Optimizer rule 'optimize_projections' failed", Context("optimize_projections", Internal("Failed due to a difference in schemas, original schema: DFSchema { inner: Schema { fields: [Field { name: \"f64\", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"large\", data_type: LargeUtf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }, field_qualifiers: [None, None], functional_dependencies: FunctionalDependencies { deps: [] } }, new schema: DFSchema { inner: Schema { fields: [Field { name: \"f64\", data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"large\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }, field_qualifiers: [None, None], functional_dependencies: FunctionalDependencies { deps: [] } }")))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1720046566
##########
datafusion/sqllogictest/test_files/string_view.slt:
##########
@@ -270,37 +270,37 @@ explain SELECT column1_utf8 from test where 'Andrew' = column1_utf8view;
----
logical_plan
01)Projection: test.column1_utf8
-02)--Filter: test.column1_utf8view = Utf8View("Andrew")
+02)--Filter: CAST(Utf8("Andrew") AS Utf8View) = test.column1_utf8view
03)----TableScan: test projection=[column1_utf8, column1_utf8view]
query TT
explain SELECT column1_utf8 from test where column1_utf8 = arrow_cast('Andrew', 'Utf8View');
----
logical_plan
-01)Filter: test.column1_utf8 = Utf8("Andrew")
+01)Filter: CAST(test.column1_utf8 AS Utf8View) = CAST(Utf8("Andrew") AS Utf8View)
Review Comment:
Totally agree! I'm aiming at restoring most of the sqllogictests (and this behaviour should be easy to catch) to their original expected results so consider the edits just as a way to make the test progress so it can surface more weird behaviours.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1720763374
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
I think there was a discussion about replacing `ColumnarValue::Scalar` or `ScalarValue` with `Datum`s `Scalar` when it is introduced. But the cost of overhead might not be negligible so we stick with `ScalarValue`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2297514514
I've pushed a fairly big experiment. I've tried to change ColumnarValue to
```rust
#[derive(Clone, Debug)]
pub enum ColumnarValue {
/// Array of values
Array(ArrayRef),
/// A single value
Scalar(ScalarValue),
Scalar(Scalar),
}
#[derive(Clone, Debug)]
pub struct Scalar {
value: ScalarValue,
data_type: DataType,
}
```
which follows the approache that was discussed with @jayzhan211 in the comments above. I've opted for this hybrid solution to retain most of the flexibility of the original ColumnarValue and I'm mostly satisfied with how it turned out. Curious to hear your thoughts @jayzhan211 and @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2297870220
It seems the trick you did is to get the first index of ArrayRef as ScalarValue but we still ends up require DataType to keep the type information.
However, I think we could move on with this approach, we could figure out if there is better approach later on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2306639743
> UIpd: It seems there is only one place that calls Accumulator::evaluate and transform it to ArrayRef, I think we can just change the return value to ArrayRef for Accumulator::evaluate
It seems like this solution is not that easy as the state needs to also be accounted for, but it's definitely a good start!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1764457333
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -89,7 +89,94 @@ pub enum ColumnarValue {
/// Array of values
Array(ArrayRef),
/// A single value
- Scalar(ScalarValue),
+ Scalar(Scalar),
+}
+
+#[derive(Clone, Debug)]
+pub struct Scalar {
+ value: ScalarValue,
+ data_type: DataType,
+}
Review Comment:
This is the main change of this PR
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -89,7 +89,94 @@ pub enum ColumnarValue {
/// Array of values
Array(ArrayRef),
/// A single value
- Scalar(ScalarValue),
+ Scalar(Scalar),
+}
+
+#[derive(Clone, Debug)]
+pub struct Scalar {
+ value: ScalarValue,
+ data_type: DataType,
+}
Review Comment:
I believe this is the main change of this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1716741382
##########
datafusion/core/src/datasource/physical_plan/file_scan_config.rs:
##########
@@ -480,155 +422,17 @@ impl PartitionColumnProjector {
}
}
-#[derive(Debug, Default)]
-struct ZeroBufferGenerators {
- gen_i8: ZeroBufferGenerator<i8>,
- gen_i16: ZeroBufferGenerator<i16>,
- gen_i32: ZeroBufferGenerator<i32>,
- gen_i64: ZeroBufferGenerator<i64>,
- gen_u8: ZeroBufferGenerator<u8>,
- gen_u16: ZeroBufferGenerator<u16>,
- gen_u32: ZeroBufferGenerator<u32>,
- gen_u64: ZeroBufferGenerator<u64>,
-}
-
-/// Generate a arrow [`Buffer`] that contains zero values.
-#[derive(Debug, Default)]
-struct ZeroBufferGenerator<T>
-where
- T: ArrowNativeType,
-{
- cache: Option<Buffer>,
- _t: PhantomData<T>,
-}
-
-impl<T> ZeroBufferGenerator<T>
-where
- T: ArrowNativeType,
-{
- const SIZE: usize = std::mem::size_of::<T>();
-
- fn get_buffer(&mut self, n_vals: usize) -> Buffer {
- match &mut self.cache {
- Some(buf) if buf.len() >= n_vals * Self::SIZE => {
- buf.slice_with_length(0, n_vals * Self::SIZE)
- }
- _ => {
- let mut key_buffer_builder = BufferBuilder::<T>::new(n_vals);
- key_buffer_builder.advance(n_vals); // keys are all 0
- self.cache.insert(key_buffer_builder.finish()).clone()
- }
- }
- }
-}
-
-fn create_dict_array<T>(
- buffer_gen: &mut ZeroBufferGenerator<T>,
- dict_val: &ScalarValue,
- len: usize,
- data_type: DataType,
-) -> Result<ArrayRef>
-where
- T: ArrowNativeType,
-{
- let dict_vals = dict_val.to_array()?;
-
- let sliced_key_buffer = buffer_gen.get_buffer(len);
-
- // assemble pieces together
- let mut builder = ArrayData::builder(data_type)
- .len(len)
- .add_buffer(sliced_key_buffer);
- builder = builder.add_child_data(dict_vals.to_data());
- Ok(Arc::new(DictionaryArray::<UInt16Type>::from(
- builder.build().unwrap(),
- )))
-}
-
-fn create_output_array(
- key_buffer_cache: &mut ZeroBufferGenerators,
- val: &ScalarValue,
- len: usize,
-) -> Result<ArrayRef> {
- if let ScalarValue::Dictionary(key_type, dict_val) = &val {
- match key_type.as_ref() {
- DataType::Int8 => {
- return create_dict_array(
- &mut key_buffer_cache.gen_i8,
- dict_val,
- len,
- val.data_type(),
- );
- }
- DataType::Int16 => {
- return create_dict_array(
- &mut key_buffer_cache.gen_i16,
- dict_val,
- len,
- val.data_type(),
- );
- }
- DataType::Int32 => {
- return create_dict_array(
- &mut key_buffer_cache.gen_i32,
- dict_val,
- len,
- val.data_type(),
- );
- }
- DataType::Int64 => {
- return create_dict_array(
- &mut key_buffer_cache.gen_i64,
- dict_val,
- len,
- val.data_type(),
- );
- }
- DataType::UInt8 => {
- return create_dict_array(
- &mut key_buffer_cache.gen_u8,
- dict_val,
- len,
- val.data_type(),
- );
- }
- DataType::UInt16 => {
- return create_dict_array(
- &mut key_buffer_cache.gen_u16,
- dict_val,
- len,
- val.data_type(),
- );
- }
- DataType::UInt32 => {
- return create_dict_array(
- &mut key_buffer_cache.gen_u32,
- dict_val,
- len,
- val.data_type(),
- );
- }
- DataType::UInt64 => {
- return create_dict_array(
- &mut key_buffer_cache.gen_u64,
- dict_val,
- len,
- val.data_type(),
- );
- }
- _ => {}
- }
- }
-
+fn create_output_array(val: &ScalarValue, len: usize) -> Result<ArrayRef> {
+ // TODO(@notfilippo): should we reintroduce a way to encode as dictionaries?
Review Comment:
I'm missing some context for partitions so I'm not really sure if it's the right call to remove the Dictionary output array. IIUC it seems like it's very useful to save space and optimise columns added this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1717686665
##########
datafusion/physical-plan/src/projection.rs:
##########
@@ -306,9 +306,10 @@ impl ProjectionStream {
let arrays = self
.expr
.iter()
- .map(|expr| {
+ .zip(&self.schema.fields)
+ .map(|(expr, field)| {
expr.evaluate(batch)
- .and_then(|v| v.into_array(batch.num_rows()))
+ .and_then(|v| v.into_array_of_type(batch.num_rows(), field.data_type()))
Review Comment:
It looks incorrect to me 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1717686665
##########
datafusion/physical-plan/src/projection.rs:
##########
@@ -306,9 +306,10 @@ impl ProjectionStream {
let arrays = self
.expr
.iter()
- .map(|expr| {
+ .zip(&self.schema.fields)
+ .map(|(expr, field)| {
expr.evaluate(batch)
- .and_then(|v| v.into_array(batch.num_rows()))
+ .and_then(|v| v.into_array_of_type(batch.num_rows(), field.data_type()))
Review Comment:
It looks incorrect to me 🤔 I think we should get the corresponding physical type (DataType) either building physical expression or the rule in physical optimizer. We could do it as follow up PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1719925643
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
This function is also used in physical expr `CastExpr`. But we should not do logical equality check for physical casting, we do actual arrow cast instead. I think it makes sense to have 2 casting function, one do logical eq check, one do actual casting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1722625770
##########
datafusion/common/src/logical/mod.rs:
##########
@@ -0,0 +1,18 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+pub mod eq;
Review Comment:
Maybe a full name for module name? like equality?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1719641899
##########
datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs:
##########
@@ -627,15 +629,34 @@ impl<'a> ConstEvaluator<'a> {
return ConstSimplifyResult::NotSimplified(s);
}
+ let start_type = match expr.get_type(&self.input_schema) {
+ Ok(t) => t,
+ Err(err) => return ConstSimplifyResult::SimplifyRuntimeError(err, expr),
+ };
+
let phys_expr =
match create_physical_expr(&expr, &self.input_schema, self.execution_props) {
Ok(e) => e,
Err(err) => return ConstSimplifyResult::SimplifyRuntimeError(err, expr),
};
+
let col_val = match phys_expr.evaluate(&self.input_batch) {
Ok(v) => v,
Err(err) => return ConstSimplifyResult::SimplifyRuntimeError(err, expr),
};
+
+ // TODO(@notfilippo): a fix for the select_arrow_cast error
+ let end_type = col_val.data_type();
+ if end_type.logically_eq(&start_type) && start_type != end_type {
+ return ConstSimplifyResult::SimplifyRuntimeError(
+ DataFusionError::Execution(format!(
Review Comment:
`exec_err` returns a result but I could use `exec_datafusion_err`. thx!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1720038132
##########
datafusion/functions/src/string/octet_length.rs:
##########
@@ -81,12 +81,6 @@ impl ScalarUDFImpl for OctetLengthFunc {
ScalarValue::Utf8(v) => Ok(ColumnarValue::Scalar(ScalarValue::Int32(
v.as_ref().map(|x| x.len() as i32),
))),
- ScalarValue::LargeUtf8(v) => Ok(ColumnarValue::Scalar(
Review Comment:
I like how removing the detailed physical types on the ScalarValue makes this kind of code much simpler (it doesn't heed to also handle two different "types" of strings
##########
datafusion/sqllogictest/test_files/string_view.slt:
##########
@@ -270,37 +270,37 @@ explain SELECT column1_utf8 from test where 'Andrew' = column1_utf8view;
----
logical_plan
01)Projection: test.column1_utf8
-02)--Filter: test.column1_utf8view = Utf8View("Andrew")
+02)--Filter: CAST(Utf8("Andrew") AS Utf8View) = test.column1_utf8view
03)----TableScan: test projection=[column1_utf8, column1_utf8view]
query TT
explain SELECT column1_utf8 from test where column1_utf8 = arrow_cast('Andrew', 'Utf8View');
----
logical_plan
-01)Filter: test.column1_utf8 = Utf8("Andrew")
+01)Filter: CAST(test.column1_utf8 AS Utf8View) = CAST(Utf8("Andrew") AS Utf8View)
Review Comment:
I think this is not a good change -- it shows the column itself being cast (which is a substantial overhead at runtime)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2294831000
> Otherwise, IMO this is just a code simplification (Still a nice step forward) but not the goal of decoupling types https://github.com/apache/datafusion/issues/11513 ,
Yes I agree with this assesment
🤔 so that leaves us with the question of how to incrementally try and introduce logical types into logical plans 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2294848052
Yeah, I think Datum is actually what we need. After casting the array, we don't need to convert it back to ScalarValue, it has additional conversion cost and the physical type information is lost too. Ideally we could replace ColumnarValue with Datum, but it seems the change is quite large.
We could try replacing ScalarValue with Scalar.
```rust
enum ColumnarValue {
Array(Arc<dyn Array>),
Scalar(arrow_array::Scalar)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1765282450
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -89,7 +89,94 @@ pub enum ColumnarValue {
/// Array of values
Array(ArrayRef),
/// A single value
- Scalar(ScalarValue),
+ Scalar(Scalar),
+}
+
+#[derive(Clone, Debug)]
+pub struct Scalar {
+ value: ScalarValue,
+ data_type: DataType,
+}
Review Comment:
I agree with @findepi 's suggestion to make the smallest PR as possible to begin. I think splitting this PR out into a small part to begin, along with comments explaining the planned end state (or linking back to the ticket that describes this) would be very helpful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2308143053
Will `evaluate_as_scalar` replaces `evaluate` and `state_as_scalars` replaces `state`? If it is, then it looks good to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1719935878
##########
datafusion/core/src/datasource/physical_plan/file_scan_config.rs:
##########
@@ -445,30 +410,7 @@ impl PartitionColumnProjector {
"Invalid partitioning found on disk".to_string(),
))?;
- let mut partition_value = Cow::Borrowed(p_value);
-
- // check if user forgot to dict-encode the partition value
- let field = self.projected_schema.field(sidx);
Review Comment:
This should be fixed now as I use the schema information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1717989216
##########
datafusion/core/src/datasource/physical_plan/file_scan_config.rs:
##########
@@ -445,30 +410,7 @@ impl PartitionColumnProjector {
"Invalid partitioning found on disk".to_string(),
))?;
- let mut partition_value = Cow::Borrowed(p_value);
-
- // check if user forgot to dict-encode the partition value
- let field = self.projected_schema.field(sidx);
Review Comment:
I check the `field.data_type()` and got `Utf8` but not `Dictionary(UInt16, Utf8)`. I'm not sure why, but I expect `Dictionary(UInt16, Utf8)` is preserved at this point, since we only remove some types in ScalarValue. The DataType in Schema should be the same 🤔
##########
datafusion/core/src/datasource/physical_plan/file_scan_config.rs:
##########
@@ -445,30 +410,7 @@ impl PartitionColumnProjector {
"Invalid partitioning found on disk".to_string(),
))?;
- let mut partition_value = Cow::Borrowed(p_value);
-
- // check if user forgot to dict-encode the partition value
- let field = self.projected_schema.field(sidx);
Review Comment:
I check the `field.data_type()` and got `Utf8` but not `Dictionary(UInt16, Utf8)`. I'm not sure why, but I expect `Dictionary(UInt16, Utf8)` is preserved at this point, since we only remove some types in ScalarValue. The DataType in Schema should be unchanged 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1722625285
##########
datafusion/common/src/logical/eq.rs:
##########
@@ -0,0 +1,26 @@
+use arrow_schema::DataType;
+
+pub trait LogicallyEq<Rhs: ?Sized = Self> {
+ #[must_use]
+ fn logically_eq(&self, other: &Rhs) -> bool;
+}
+
+impl LogicallyEq for DataType {
+ fn logically_eq(&self, other: &Self) -> bool {
+ use DataType::*;
+
+ match (self, other) {
+ (Utf8 | LargeUtf8 | Utf8View, Utf8 | LargeUtf8 | Utf8View)
+ | (Binary | LargeBinary | BinaryView, Binary | LargeBinary | BinaryView) => {
+ true
+ }
+ (Dictionary(_, inner), other) | (other, Dictionary(_, inner)) => {
Review Comment:
We could avoid unnecessary call if we handle both Dict case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1717686665
##########
datafusion/physical-plan/src/projection.rs:
##########
@@ -306,9 +306,10 @@ impl ProjectionStream {
let arrays = self
.expr
.iter()
- .map(|expr| {
+ .zip(&self.schema.fields)
+ .map(|(expr, field)| {
expr.evaluate(batch)
- .and_then(|v| v.into_array(batch.num_rows()))
+ .and_then(|v| v.into_array_of_type(batch.num_rows(), field.data_type()))
Review Comment:
It looks incorrect to me 🤔
Is it possible to coerce types in `type_coercion`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1717933633
##########
datafusion/common/src/scalar/mod.rs:
##########
@@ -3173,6 +2986,11 @@ impl ScalarValue {
.map(|sv| sv.size() - std::mem::size_of_val(sv))
.sum::<usize>()
}
+
+ pub fn supported_datatype(data_type: &DataType) -> Result<DataType, DataFusionError> {
Review Comment:
It seems not used
##########
datafusion/common/src/scalar/mod.rs:
##########
@@ -3173,6 +2986,11 @@ impl ScalarValue {
.map(|sv| sv.size() - std::mem::size_of_val(sv))
.sum::<usize>()
}
+
+ pub fn supported_datatype(data_type: &DataType) -> Result<DataType, DataFusionError> {
Review Comment:
It seems not used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1719925643
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
This function is also used in physical expr `CastExpr`. But we should not do logical equality check for physical casting, we do actual arrow cast instead. I think it makes sense to have 2 casting function, one do logical eq check, one do actual casting.
With this, I think `into_array_of_type ` is no longer required
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
This function is also used in physical expr `CastExpr`. But we should not do logical equality check for physical casting, we do actual arrow cast instead. I think it makes sense to have 2 casting function, one do logical eq check, one do actual casting.
With this, I think `into_array_of_type` is no longer required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1719931207
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
I think the general idea is we do logical type eq check in logical layer. Do actual casting in physical layer.
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
I think the general idea is that we do logical type eq check in logical layer. Do actual casting in physical layer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1719925643
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
`cast_to` is also used in physical expr `CastExpr`. But we should not do logical equality check for physical casting, we do actual arrow cast instead. I think it makes sense to have 2 casting function, one do logical eq check, one do actual casting.
With this, I think `into_array_of_type` is no longer required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1720513958
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
`actual casting in physical layer` what in my mind is array with arrow::cast
```rust
let cast_array = kernels::cast::cast_with_options(
&scalar_array,
cast_type,
&cast_options,
)?;
```
Btw, I'm confused about what is the difference between `ColumnarValue` and `Datum`. It seems like we could replace `ColumnarValue` with `Datum` 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2293622109
Since you have no commit yet, so CI is not running.
You could run these command to pass CI
`cargo test --test sqllogictests` (slt files)
`./dev/rust_lint.sh`
`cargo test --lib --tests --bins --features avro,json` (rust test)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1719702630
##########
datafusion/common/src/logical/eq.rs:
##########
@@ -0,0 +1,26 @@
+use arrow_schema::DataType;
+
+pub trait LogicallyEq<Rhs: ?Sized = Self> {
+ #[must_use]
+ fn logically_eq(&self, other: &Rhs) -> bool;
+}
+
+impl LogicallyEq for DataType {
+ fn logically_eq(&self, other: &Self) -> bool {
+ use DataType::*;
+
+ match (self, other) {
+ (Utf8 | LargeUtf8 | Utf8View, Utf8 | LargeUtf8 | Utf8View)
+ | (Binary | LargeBinary | BinaryView, Binary | LargeBinary | BinaryView) => {
+ true
+ }
+ (Dictionary(_, inner), other) | (other, Dictionary(_, inner)) => {
Review Comment:
Technically it would work with the recursion but I could add a special case to prevent it from doing multiple recursive calls.
##########
datafusion/common/src/logical/eq.rs:
##########
@@ -0,0 +1,26 @@
+use arrow_schema::DataType;
+
+pub trait LogicallyEq<Rhs: ?Sized = Self> {
+ #[must_use]
+ fn logically_eq(&self, other: &Rhs) -> bool;
+}
+
+impl LogicallyEq for DataType {
+ fn logically_eq(&self, other: &Self) -> bool {
+ use DataType::*;
+
+ match (self, other) {
+ (Utf8 | LargeUtf8 | Utf8View, Utf8 | LargeUtf8 | Utf8View)
+ | (Binary | LargeBinary | BinaryView, Binary | LargeBinary | BinaryView) => {
+ true
+ }
+ (Dictionary(_, inner), other) | (other, Dictionary(_, inner)) => {
Review Comment:
Technically it _should_ work with the recursion but I could add a special case to prevent it from doing multiple recursive calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2293500975
> Given that we have ScalarValue::Utf8, and we have StringView in schema. We can then get the corresponding StrginViewArray.
I feel like this is a little bit more complex because casting a scalar value to a logically equal type (`arrow_cast('test', 'Dictionary(Int32, Utf8)')`) is a no-op for all cases but one the cast is at the top level, which indicates that you actually want an __array__ of that type. So we need a way to distinguish those two scenarios, which is why I've added this:
https://github.com/apache/datafusion/blob/78dc034d4b7d644e6c34745286e82a6332dd7606/datafusion/physical-plan/src/projection.rs#L312
But I'm not really sure about it...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1719955354
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
Seems like ColumnarValue is used in both logical and physical layer so I'm not really sure how to approach this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1717989216
##########
datafusion/core/src/datasource/physical_plan/file_scan_config.rs:
##########
@@ -445,30 +410,7 @@ impl PartitionColumnProjector {
"Invalid partitioning found on disk".to_string(),
))?;
- let mut partition_value = Cow::Borrowed(p_value);
-
- // check if user forgot to dict-encode the partition value
- let field = self.projected_schema.field(sidx);
Review Comment:
I check the `field.data_type()` and got the value type (Utf8) but not Dictionary. I'm not sure why, but I expect Dict is preserved at this point 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1717686665
##########
datafusion/physical-plan/src/projection.rs:
##########
@@ -306,9 +306,10 @@ impl ProjectionStream {
let arrays = self
.expr
.iter()
- .map(|expr| {
+ .zip(&self.schema.fields)
+ .map(|(expr, field)| {
expr.evaluate(batch)
- .and_then(|v| v.into_array(batch.num_rows()))
+ .and_then(|v| v.into_array_of_type(batch.num_rows(), field.data_type()))
Review Comment:
It looks incorrect to me 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2294834051
> 🤔 so that leaves us with the question of how to incrementally try and introduce logical types into logical plans 🤔
As @jayzhan211 says, the type information is still retained in the Schema, but it would be impractical to modify all expression orders to support the fact that the type is not stored in the ColumnarValue.
My discussion around Datum revolves around the fact that to retain physical type information maybe `ColumnarValue` enum should instead be:
```rust
enum ColumnarValue {
Array(Arc<dyn Array>),
Scalar(Arc<dyn Array>) // this is an array of len = 1
}
```
or that ScalarValue is kept as is and a LogicalScalarValue is introduced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1720763663
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
https://github.com/apache/datafusion/issues/7353
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2312544742
> Will evaluate_as_scalar replaces evaluate and state_as_scalars replaces state? If it is, then it looks good to me
Yes that's the plan!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1719955354
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
Seems like ColumnarValue is used in both logical and physical layer so I'm not really sure how to approach this. I feel like maybe it would be better to have ColumnarValue as a logical representation and have Datum as the physical one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2293766002
Starting to check this out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1719931672
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
Yes, I agree! The issue is that that function still returns a ColumnarValue for physical execution, which would mean that the Cast, even when physical, would lose the type information (and be a no-op) since it would result in a ColumnarValue::ScalarValue(...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1720513958
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
`actual casting in physical layer` what in my mind is array with arrow::cast
```rust
let cast_array = kernels::cast::cast_with_options(
&scalar_array,
cast_type,
&cast_options,
)?;
```
`Datum` is another concept other than `ColumnarValue`. It is `ArrayRef` or `Scalar`
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
`actual casting in physical layer` what in my mind is array with arrow::cast
```rust
let cast_array = kernels::cast::cast_with_options(
&scalar_array,
cast_type,
&cast_options,
)?;
```
`Datum` is another concept other than `ColumnarValue`. It could be `Array` or `Scalar`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2294518747
> I wonder if we could do some sort of hybrid approach where `ScalarValue` still retains a DataType, type, but it isn't encoded in the variant
>
> like
>
> ```rust
> enum ScalarValue {
> ...
> /// A logical string value, with the specified data type
> ScalarValue::Utf8(String, DataType) ,
> }
> ```
>
> That way you would still have the `DataType` for each `ScalarValue` 🤔
This should be the last resort if we don't have any other way to figure the actual target type to cast to. Otherwise, IMO this is just a code simplification (Still a nice step forward) but not the goal of decoupling types #11513 , since we still hide the physical type inside `ScalarValue`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2308124082
> > UIpd: It seems there is only one place that calls Accumulator::evaluate and transform it to ArrayRef, I think we can just change the return value to ArrayRef for Accumulator::evaluate
>
> It seems like this solution is not that easy as the state needs to also be accounted for, but it's definitely a good start!
Hmm, I don't think there is any issue with `state`. You don't need to change `state`, only `evaluate` is enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1717989216
##########
datafusion/core/src/datasource/physical_plan/file_scan_config.rs:
##########
@@ -445,30 +410,7 @@ impl PartitionColumnProjector {
"Invalid partitioning found on disk".to_string(),
))?;
- let mut partition_value = Cow::Borrowed(p_value);
-
- // check if user forgot to dict-encode the partition value
- let field = self.projected_schema.field(sidx);
Review Comment:
I check the `field.data_type()`, and only get the value type but not Dictionary. I'm not sure why, but I expect Dict is preserved at this point 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2290739554
```
statement ok
create table test_source as values
('Andrew', 'X'),
('Xiangpeng', 'Xiangpeng'),
('Raphael', 'R'),
(NULL, 'R')
;
statement ok
create table t as
SELECT
arrow_cast(column1, 'Utf8') as column1_utf8,
arrow_cast(column2, 'Utf8') as column2_utf8,
arrow_cast(column1, 'LargeUtf8') as column1_large_utf8,
arrow_cast(column2, 'LargeUtf8') as column2_large_utf8,
arrow_cast(column1, 'Utf8View') as column1_utf8view,
arrow_cast(column2, 'Utf8View') as column2_utf8view,
arrow_cast(column1, 'Dictionary(Int32, Utf8)') as column1_dict,
arrow_cast(column2, 'Dictionary(Int32, Utf8)') as column2_dict
FROM test_source;
query error DataFusion error: Arrow error: Invalid argument error: column types must match schema types, expected Utf8View but found Utf8 at column index 0
select min(column1_utf8view) from t;
```
I run this test and failed. It success on main branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1717989216
##########
datafusion/core/src/datasource/physical_plan/file_scan_config.rs:
##########
@@ -445,30 +410,7 @@ impl PartitionColumnProjector {
"Invalid partitioning found on disk".to_string(),
))?;
- let mut partition_value = Cow::Borrowed(p_value);
-
- // check if user forgot to dict-encode the partition value
- let field = self.projected_schema.field(sidx);
Review Comment:
I check the `field.data_type()` and got `Utf8` but not `Dictionary(UInt16, Utf8)`. I'm not sure why, but I expect `Dictionary(UInt16, Utf8)` is preserved at this point, since we only remove some types in ScalarValue. DataType in Schema should remained the same 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1720513958
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
`actual casting in physical layer` what in my mind is array with arrow::cast
```rust
let cast_array = kernels::cast::cast_with_options(
&scalar_array,
cast_type,
&cast_options,
)?;
```
Btw, I'm confused about what is the difference between `ColumnarValue` and `Datum`. `ColumnarValue` is in an early day where `Datum` is not exist yet. Now, It seems like we could replace `ColumnarValue` with `Datum` 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1720513958
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
`actual casting in physical layer` what in my mind is array with arrow::cast
```rust
let cast_array = kernels::cast::cast_with_options(
&scalar_array,
cast_type,
&cast_options,
)?;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2298832975
The issue is that we have only `ScalarValue` but no `DataType` to get the actual ArrayRef back.
One of the solution is to store `DataType` in `MaxAccumulator` and call `to_array_of_type` to return `ArrayRef` as the output of `evaluate`, maybe `evaluate_as_array` to avoid breaking existing function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1765286489
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -89,7 +89,94 @@ pub enum ColumnarValue {
/// Array of values
Array(ArrayRef),
/// A single value
- Scalar(ScalarValue),
+ Scalar(Scalar),
+}
+
+#[derive(Clone, Debug)]
+pub struct Scalar {
+ value: ScalarValue,
+ data_type: DataType,
+}
Review Comment:
I agree with @findepi 's suggestion to make the smallest PR as possible to begin. I think splitting this PR out into a small part to begin, along with comments explaining the planned end state (or linking back to the ticket that describes this) would be very helpful
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -89,7 +89,94 @@ pub enum ColumnarValue {
/// Array of values
Array(ArrayRef),
/// A single value
- Scalar(ScalarValue),
+ Scalar(Scalar),
+}
+
+#[derive(Clone, Debug)]
+pub struct Scalar {
+ value: ScalarValue,
+ data_type: DataType,
+}
Review Comment:
I agree with @findepi 's suggestion to make the smallest PR as possible to begin. I think splitting this PR out into a small part to begin, along with comments explaining the planned end state (or linking back to the ticket that describes this) would be very helpful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2353519265
I'm back from vacation and I've rebased my PR to the latest upstream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2306701596
Marking as "ready for review" because all tests pass on my end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1717282407
##########
datafusion/physical-plan/src/projection.rs:
##########
@@ -306,9 +306,10 @@ impl ProjectionStream {
let arrays = self
.expr
.iter()
- .map(|expr| {
+ .zip(&self.schema.fields)
+ .map(|(expr, field)| {
expr.evaluate(batch)
- .and_then(|v| v.into_array(batch.num_rows()))
+ .and_then(|v| v.into_array_of_type(batch.num_rows(), field.data_type()))
Review Comment:
Not sure if casting here is a correct approach.
##########
datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs:
##########
@@ -627,15 +629,34 @@ impl<'a> ConstEvaluator<'a> {
return ConstSimplifyResult::NotSimplified(s);
}
+ let start_type = match expr.get_type(&self.input_schema) {
+ Ok(t) => t,
+ Err(err) => return ConstSimplifyResult::SimplifyRuntimeError(err, expr),
+ };
+
let phys_expr =
match create_physical_expr(&expr, &self.input_schema, self.execution_props) {
Ok(e) => e,
Err(err) => return ConstSimplifyResult::SimplifyRuntimeError(err, expr),
};
+
let col_val = match phys_expr.evaluate(&self.input_batch) {
Ok(v) => v,
Err(err) => return ConstSimplifyResult::SimplifyRuntimeError(err, expr),
};
+
+ // TODO(@notfilippo): a fix for the select_arrow_cast error
Review Comment:
This check partially fixes the issue with `select_arrow_cast`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1719993369
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
> I think the general idea is that we do logical type eq check in logical layer. Do actual casting in physical layer.
So my actual question could be summarised as: what would the "actual casting in physical layer" return? A Datum?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2293837444
I wonder if we could do some sort of hybrid approach where `ScalarValue` still retains a DataType, type, but it isn't encoded in the variant
like
```rust
enum ScalarValue {
...
/// A logical string value, with the specified data type
ScalarValue::Utf8(String, DataType) ,
}
```
That way you would still have the `DataType` for each `ScalarValue` 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1720763044
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
I've clarified what I mean when discussing `Datum` in this comment:
https://github.com/apache/datafusion/pull/11978#issuecomment-2294834051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1717253809
##########
datafusion/common/src/scalar/mod.rs:
##########
@@ -1714,6 +1631,15 @@ impl ScalarValue {
Some(sv) => sv.data_type(),
};
+ Self::iter_to_array_of_type(scalars.collect(), &data_type)
+ }
+
+ fn iter_to_array_of_type(
Review Comment:
This was introduced because of aggregations that use ScalarValue as state and so they could lose type information when emitting state.
##########
datafusion/sqllogictest/test_files/string_view.slt:
##########
@@ -261,7 +261,7 @@ explain SELECT column1_utf8 from test where column1_utf8view = 'Andrew';
----
logical_plan
01)Projection: test.column1_utf8
-02)--Filter: test.column1_utf8view = Utf8View("Andrew")
+02)--Filter: test.column1_utf8view = CAST(Utf8("Andrew") AS Utf8View)
Review Comment:
Casts are not simplified to retain the type information to fix the `optimize_projection` issue above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1717989216
##########
datafusion/core/src/datasource/physical_plan/file_scan_config.rs:
##########
@@ -445,30 +410,7 @@ impl PartitionColumnProjector {
"Invalid partitioning found on disk".to_string(),
))?;
- let mut partition_value = Cow::Borrowed(p_value);
-
- // check if user forgot to dict-encode the partition value
- let field = self.projected_schema.field(sidx);
Review Comment:
I check the `field.data_type()` and got `Utf8` but not `Dictionary(UInt16, Utf8)`. I'm not sure why, but I expect `Dictionary(UInt16, Utf8)` is preserved at this point 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1717949118
##########
datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs:
##########
@@ -627,15 +629,34 @@ impl<'a> ConstEvaluator<'a> {
return ConstSimplifyResult::NotSimplified(s);
}
+ let start_type = match expr.get_type(&self.input_schema) {
+ Ok(t) => t,
+ Err(err) => return ConstSimplifyResult::SimplifyRuntimeError(err, expr),
+ };
+
let phys_expr =
match create_physical_expr(&expr, &self.input_schema, self.execution_props) {
Ok(e) => e,
Err(err) => return ConstSimplifyResult::SimplifyRuntimeError(err, expr),
};
+
let col_val = match phys_expr.evaluate(&self.input_batch) {
Ok(v) => v,
Err(err) => return ConstSimplifyResult::SimplifyRuntimeError(err, expr),
};
+
+ // TODO(@notfilippo): a fix for the select_arrow_cast error
+ let end_type = col_val.data_type();
+ if end_type.logically_eq(&start_type) && start_type != end_type {
+ return ConstSimplifyResult::SimplifyRuntimeError(
+ DataFusionError::Execution(format!(
Review Comment:
we could use `exec_err`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1765157634
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -89,7 +89,94 @@ pub enum ColumnarValue {
/// Array of values
Array(ArrayRef),
/// A single value
- Scalar(ScalarValue),
+ Scalar(Scalar),
+}
+
+#[derive(Clone, Debug)]
+pub struct Scalar {
+ value: ScalarValue,
+ data_type: DataType,
+}
Review Comment:
I was thinking that I could open a new PR introducing just introducing this columnar value type, and then little by little removing the scalar value variants (Dictionary, Large* and *View...)
I'm mentioning this because of the discussion in https://github.com/apache/datafusion/pull/12488#issuecomment-2358577025, as I think it will be very beneficial to split this PR in multiple changes, both for reviewers and datafusion users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1717987278
##########
datafusion/common/src/logical/eq.rs:
##########
@@ -0,0 +1,26 @@
+use arrow_schema::DataType;
+
+pub trait LogicallyEq<Rhs: ?Sized = Self> {
+ #[must_use]
+ fn logically_eq(&self, other: &Rhs) -> bool;
+}
+
+impl LogicallyEq for DataType {
+ fn logically_eq(&self, other: &Self) -> bool {
+ use DataType::*;
+
+ match (self, other) {
+ (Utf8 | LargeUtf8 | Utf8View, Utf8 | LargeUtf8 | Utf8View)
+ | (Binary | LargeBinary | BinaryView, Binary | LargeBinary | BinaryView) => {
+ true
+ }
+ (Dictionary(_, inner), other) | (other, Dictionary(_, inner)) => {
Review Comment:
How about both Dictionary cases?
##########
datafusion/common/src/logical/eq.rs:
##########
@@ -0,0 +1,26 @@
+use arrow_schema::DataType;
+
+pub trait LogicallyEq<Rhs: ?Sized = Self> {
+ #[must_use]
+ fn logically_eq(&self, other: &Rhs) -> bool;
+}
+
+impl LogicallyEq for DataType {
+ fn logically_eq(&self, other: &Self) -> bool {
+ use DataType::*;
+
+ match (self, other) {
+ (Utf8 | LargeUtf8 | Utf8View, Utf8 | LargeUtf8 | Utf8View)
+ | (Binary | LargeBinary | BinaryView, Binary | LargeBinary | BinaryView) => {
+ true
+ }
+ (Dictionary(_, inner), other) | (other, Dictionary(_, inner)) => {
Review Comment:
How about both Dictionary case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1717686665
##########
datafusion/physical-plan/src/projection.rs:
##########
@@ -306,9 +306,10 @@ impl ProjectionStream {
let arrays = self
.expr
.iter()
- .map(|expr| {
+ .zip(&self.schema.fields)
+ .map(|(expr, field)| {
expr.evaluate(batch)
- .and_then(|v| v.into_array(batch.num_rows()))
+ .and_then(|v| v.into_array_of_type(batch.num_rows(), field.data_type()))
Review Comment:
It looks incorrect to me 🤔 I think we should get the corresponding physical type (DataType) either building physical expression or the rule in physical optimizer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1765157634
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -89,7 +89,94 @@ pub enum ColumnarValue {
/// Array of values
Array(ArrayRef),
/// A single value
- Scalar(ScalarValue),
+ Scalar(Scalar),
+}
+
+#[derive(Clone, Debug)]
+pub struct Scalar {
+ value: ScalarValue,
+ data_type: DataType,
+}
Review Comment:
I was thinking that I could open a new PR introducing just introducing this columnar value type, and then little by little removing the scalar value variants (Dictionary, Large* and *View...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2365151653
To make it clear this PR is not waiting on more review (we are discussing on https://github.com/apache/datafusion/pull/12536) marking this as draft
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1719702630
##########
datafusion/common/src/logical/eq.rs:
##########
@@ -0,0 +1,26 @@
+use arrow_schema::DataType;
+
+pub trait LogicallyEq<Rhs: ?Sized = Self> {
+ #[must_use]
+ fn logically_eq(&self, other: &Rhs) -> bool;
+}
+
+impl LogicallyEq for DataType {
+ fn logically_eq(&self, other: &Self) -> bool {
+ use DataType::*;
+
+ match (self, other) {
+ (Utf8 | LargeUtf8 | Utf8View, Utf8 | LargeUtf8 | Utf8View)
+ | (Binary | LargeBinary | BinaryView, Binary | LargeBinary | BinaryView) => {
+ true
+ }
+ (Dictionary(_, inner), other) | (other, Dictionary(_, inner)) => {
Review Comment:
Technically it would work with the recursion but I could add a special case just to be sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2298396570
I'm happy to report that I've got most sqllogictests to run successfully (albeit there is still the issue pointed out by @alamb, which i plan to address after I've got all tests passing). The only issues I'm having are the following:
### Aggregates using ScalarValues as state
Aggregates use ScalarValue to represent state and evaluate to a result. Should I look into restricting their return type to the subset of types which can be represented by ScalarValues?
```
External error: query failed: DataFusion error: Arrow error: Invalid argument error: column types must match schema types, expected LargeUtf8 but found Utf8 at column index 3
[SQL] SELECT
min(utf8),
max(utf8),
count(utf8),
min(largeutf8),
max(largeutf8),
count(largeutf8)
FROM t
at test_files/aggregate.slt:4104
```
### Weird error I don't quite understand
Removing this query doesn't yield any other error in the slt file. I don't have any other clue and I'm not sure where to start 🤷 .
```
External error: query result mismatch:
[SQL] SELECT
count,
LAG(timestamp, 1) OVER (ORDER BY timestamp),
arrow_typeof(LAG(timestamp, 1) OVER (ORDER BY timestamp))
FROM timestamp_with_tz
LIMIT 10;
[Diff] (-expected|+actual)
0 NULL Timestamp(Millisecond, Some("UTC"))
0 2014-08-27T14:00:00Z Timestamp(Millisecond, Some("UTC"))
0 2014-08-27T14:00:00Z Timestamp(Millisecond, Some("UTC"))
- 4 2014-08-27T14:00:00Z Timestamp(Millisecond, Some("UTC"))
0 2014-08-27T14:00:00Z Timestamp(Millisecond, Some("UTC"))
- 0 2014-08-27T14:00:00Z Timestamp(Millisecond, Some("UTC"))
+ 12 2014-08-27T14:00:00Z Timestamp(Millisecond, Some("UTC"))
+ 2 2014-08-27T14:00:00Z Timestamp(Millisecond, Some("UTC"))
0 2014-08-27T14:00:00Z Timestamp(Millisecond, Some("UTC"))
14 2014-08-27T14:00:00Z Timestamp(Millisecond, Some("UTC"))
- 0 2014-08-27T14:00:00Z Timestamp(Millisecond, Some("UTC"))
- 0 2014-08-27T14:00:00Z Timestamp(Millisecond, Some("UTC"))
+ 5 2014-08-27T14:00:00Z Timestamp(Millisecond, Some("UTC"))
+ 1 2014-08-27T14:00:00Z Timestamp(Millisecond, Some("UTC"))
at test_files/parquet.slt:255
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#issuecomment-2293628576
> Since you have no commit yet, so CI is not running.
>
> You could run these command to pass CI `cargo test --test sqllogictests` (slt files) `./dev/rust_lint.sh` `cargo test --lib --tests --bins --features avro,json` (rust test)
Thanks! Don't worry about turning it on as I can still get results from my fork.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Remove LargeUtf8|Binary, Utf8|BinaryView, and Dictionary from ScalarValue [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11978:
URL: https://github.com/apache/datafusion/pull/11978#discussion_r1720760725
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -195,6 +214,10 @@ impl ColumnarValue {
kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
)),
ColumnarValue::Scalar(scalar) => {
+ if scalar.data_type().logically_eq(cast_type) {
Review Comment:
> Datum is another concept other than ColumnarValue. It could be Array or Scalar
`Datum` is the arrow-rs concept equivalent of `ColumnarValue::Scalar` (though its implementation is different). Also, it is not yet widely supported in arrow-rs kernels (it is mostly now supported in the core comparison /equality kernels)
`ColumnarValue` and `ScalarValue` both significantly pre-date the introduction of `Datum` to arrow-rs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org