You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "yukkit (via GitHub)" <gi...@apache.org> on 2023/11/12 08:53:25 UTC
[PR] [draft] try to support user-defined types [arrow-datafusion]
yukkit opened a new pull request, #8143:
URL: https://github.com/apache/arrow-datafusion/pull/8143
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #7923 .
Current Pull Request is an Experimental Demo for Validating the Feasibility of Logical Types
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Features
- Create User-Defined Types (UDTs) through SQL, specifying the field types as UDTs during table creation.
- Support the use of `UDT` as a function signature in `udf/udaf`.
- Register extension types through the `register_data_type` function in the `SessionContext`.
## New Additions
- `LogicalType` struct.
- `ExtensionType` trait. Abstraction for extension types.
- `TypeSignature` struct. Uniquely identifies a data type.
## Major Changes
- Added `get_data_type(&self, _name: &TypeSignature) -> Option<LogicalType>` function to the `ContextProvider` trait.
- In `DFSchema`, `DFField` now uses `LogicalType`, removing arrow `Field` and retaining only `data_type`, `nullable`, `metadata` since `dict_id`, `dict_is_ordered` are not necessary at the logical stage.
- `ExprSchemable` and `ExprSchema` now use `LogicalType`.
- `ast` to logical plan conversion now uses `LogicalType`.
## To Be Implemented
- `TypeCoercionRewriter` in the analyze stage uses logical types. For example, functions like `comparison_coercion`, `get_input_types`, `get_valid_types`, etc.
- Functions signatures for `udf/udaf` use `TypeSignature` instead of the existing `DataType` for ease of use in `udf/udaf`.
## To Be Determined
- Should `ScalarValue` use `LogicalType` or arrow `DataType`?
- Should `TableSource` return `DFSchema` or arrow `Schema`?
* Still using `Schema`.
- Conversion between physical types and logical types (in Datafusion, type conversion is achieved through the conversion of `DFSchema` to `Schema`; logical plans use `DFSchema`, physical plans use `Schema`).
- Conversion between `Schema` and `DFSchema`
* When to convert `Schema` to `DFSchema`? During the construction of the logical `TableScan` node, obtain arrow `Schema` through `TableSource` and then convert it to `DFSchema`.
* When to convert `DFSchema` to `Schema`? Directly obtain arrow `Schema` from `TableSource` in physical planner, no need for conversion.
## Some Thoughts
- In [this comment](https://github.com/apache/arrow-rs/issues/4472#issuecomment-1749805039), the use case of converting from `dyn Array` to `LineStringArray` or `MultiPointArray` was raised. In my perspective, assuming there is a function specifically designed for handling `LineString` data, the function signature can be defined as `LineString`, ensuring that the input data must be of a type acceptable by `LineStringArray`.
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [draft] Add `LogicalType`, try to support user-defined types [arrow-datafusion]
Posted by "yukkit (via GitHub)" <gi...@apache.org>.
yukkit commented on code in PR #8143:
URL: https://github.com/apache/arrow-datafusion/pull/8143#discussion_r1393883980
##########
datafusion/common/src/logical_type.rs:
##########
@@ -0,0 +1,411 @@
+use std::{borrow::Cow, fmt::Display, sync::Arc};
+
+use crate::error::Result;
+use arrow_schema::{DataType, Field, IntervalUnit, TimeUnit};
+
+#[derive(Clone, Debug)]
+pub enum LogicalType {
+ Null,
+ Boolean,
+ Int8,
+ Int16,
+ Int32,
+ Int64,
+ UInt8,
+ UInt16,
+ UInt32,
+ UInt64,
+ Float16,
+ Float32,
+ Float64,
+ String,
+ LargeString,
+ Date32,
+ Date64,
+ Time32(TimeUnit),
+ Time64(TimeUnit),
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ Duration(TimeUnit),
+ Interval(IntervalUnit),
+ Binary,
+ FixedSizeBinary(i32),
+ LargeBinary,
+ Utf8,
+ LargeUtf8,
+ List(Box<LogicalType>),
+ FixedSizeList(Box<LogicalType>, i32),
+ LargeList(Box<LogicalType>),
+ Struct(Fields),
+ Map(NamedLogicalTypeRef, bool),
+ // union
+ Decimal128(u8, i8),
+ Decimal256(u8, i8),
+ Extension(ExtensionTypeRef),
+}
+
+impl PartialEq for LogicalType {
+ fn eq(&self, other: &Self) -> bool {
+ self.type_signature() == other.type_signature()
+ }
+}
+
+impl Eq for LogicalType {}
+
+impl std::hash::Hash for LogicalType {
+ fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+ self.type_signature().hash(state)
+ }
+}
+
+impl Display for LogicalType {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ write!(f, "{}", self.display_name())
+ }
+}
+
+pub type Fields = Arc<[NamedLogicalTypeRef]>;
+pub type NamedLogicalTypeRef = Arc<NamedLogicalType>;
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub struct NamedLogicalType {
+ name: String,
+ data_type: LogicalType,
+}
+
+impl NamedLogicalType {
+ pub fn new(name: impl Into<String>, data_type: LogicalType) -> Self {
+ Self {
+ name: name.into(),
+ data_type,
+ }
+ }
+
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ pub fn data_type(&self) -> &LogicalType {
+ &self.data_type
+ }
+}
+
+pub type OwnedTypeSignature = TypeSignature<'static>;
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct TypeSignature<'a> {
+ // **func_name**(p1, p2)
+ name: Cow<'a, str>,
+ // func_name(**p1**, **p2**)
+ params: Vec<Cow<'a, str>>,
+}
+
+impl<'a> TypeSignature<'a> {
+ pub fn new(name: impl Into<Cow<'a, str>>) -> Self {
+ Self::new_with_params(name, vec![])
+ }
+
+ pub fn new_with_params(
+ name: impl Into<Cow<'a, str>>,
+ params: Vec<Cow<'a, str>>,
+ ) -> Self {
+ Self {
+ name: name.into(),
+ params,
+ }
+ }
+
+ pub fn to_owned_type_signature(&self) -> OwnedTypeSignature {
+ OwnedTypeSignature {
+ name: self.name.to_string().into(),
+ params: self.params.iter().map(|p| p.to_string().into()).collect(),
+ }
+ }
+}
+
+pub type ExtensionTypeRef = Arc<dyn ExtensionType + Send + Sync>;
+
+pub trait ExtensionType: std::fmt::Debug {
+ fn display_name(&self) -> &str;
+ fn type_signature(&self) -> TypeSignature;
+ fn physical_type(&self) -> DataType;
+
+ fn is_comparable(&self) -> bool;
+ fn is_orderable(&self) -> bool;
+ fn is_numeric(&self) -> bool;
+}
+
+pub trait TypeManager {
+ fn register_data_type(
+ &mut self,
+ signature: impl Into<TypeSignature<'static>>,
+ extension_type: ExtensionTypeRef,
+ ) -> Result<()>;
+
+ fn data_type(&self, signature: &TypeSignature) -> Result<Option<ExtensionTypeRef>>;
+}
+
+impl ExtensionType for LogicalType {
+ fn display_name(&self) -> &str {
+ match self {
+ Self::Null => "NULL",
+ Self::Boolean => "BOOLEAN",
+ Self::Int8 => "INT8",
+ Self::Int16 => "INT16",
+ Self::Int32 => "INT32",
+ Self::Int64 => "INT64",
+ Self::UInt8 => "UINT8",
+ Self::UInt16 => "UINT16",
+ Self::UInt32 => "UINT32",
+ Self::UInt64 => "UINT64",
+ Self::Float16 => "FLOAT16",
+ Self::Float32 => "Float16",
+ Self::Float64 => "Float64",
+ Self::String => "String",
+ Self::LargeString => "LargeString",
+ Self::Date32 => "Date32",
+ Self::Date64 => "Date64",
+ Self::Time32(_) => "Time32",
+ Self::Time64(_) => "Time64",
+ Self::Timestamp(_, _) => "Timestamp",
+ Self::Duration(_) => "Duration",
+ Self::Interval(_) => "Interval",
+ Self::Binary => "Binary",
+ Self::FixedSizeBinary(_) => "FixedSizeBinary",
+ Self::LargeBinary => "LargeBinary",
+ Self::Utf8 => "Utf8",
+ Self::LargeUtf8 => "LargeUtf8",
+ Self::List(_) => "List",
+ Self::FixedSizeList(_, _) => "FixedSizeList",
+ Self::LargeList(_) => "LargeList",
+ Self::Struct(_) => "Struct",
+ Self::Map(_, _) => "Map",
+ Self::Decimal128(_, _) => "Decimal128",
+ Self::Decimal256(_, _) => "Decimal256",
+ Self::Extension(ext) => ext.display_name(),
+ }
+ }
+
+ fn type_signature(&self) -> TypeSignature {
+ match self {
+ Self::Boolean => TypeSignature::new("boolean"),
+ Self::Int32 => TypeSignature::new("int32"),
+ Self::Int64 => TypeSignature::new("int64"),
+ Self::UInt64 => TypeSignature::new("uint64"),
+ Self::Float32 => TypeSignature::new("float32"),
+ Self::Float64 => TypeSignature::new("float64"),
+ Self::String => TypeSignature::new("string"),
+ Self::Timestamp(tu, zone) => {
+ let tu = match tu {
+ TimeUnit::Second => "second",
+ TimeUnit::Millisecond => "millisecond",
+ TimeUnit::Microsecond => "microsecond",
+ TimeUnit::Nanosecond => "nanosecond",
+ };
+
+ let params = if let Some(zone) = zone {
+ vec![tu.into(), zone.as_ref().into()]
+ } else {
+ vec![tu.into()]
+ };
+
+ TypeSignature::new_with_params("timestamp", params)
+ }
+ Self::Binary => TypeSignature::new("binary"),
+ Self::Utf8 => TypeSignature::new("string"),
+ Self::Extension(ext) => ext.type_signature(),
+ Self::Struct(fields) => {
+ let params = fields.iter().map(|f| f.name().into()).collect();
+ TypeSignature::new_with_params("struct", params)
+ }
+ other => panic!("not implemented: {other:?}"),
+ }
+ }
+
+ fn physical_type(&self) -> DataType {
+ match self {
+ Self::Boolean => DataType::Boolean,
+ Self::Int32 => DataType::Int32,
+ Self::Int64 => DataType::Int64,
+ Self::UInt64 => DataType::UInt64,
+ Self::Float32 => DataType::Float32,
+ Self::Float64 => DataType::Float64,
+ Self::String => DataType::Utf8,
+ Self::Timestamp(tu, zone) => DataType::Timestamp(tu.clone(), zone.clone()),
+ Self::Binary => DataType::Binary,
+ Self::Utf8 => DataType::Utf8,
+ Self::Extension(ext) => ext.physical_type(),
+ Self::Struct(fields) => {
+ let fields = fields
+ .iter()
+ .map(|f| {
+ let name = f.name();
+ let data_type = f.physical_type();
+ Arc::new(Field::new(name, data_type, true))
+ })
+ .collect::<Vec<_>>();
+ DataType::Struct(fields.into())
+ }
+ other => panic!("not implemented {other:?}"),
Review Comment:
Thanks for the heads up. This should include `DataType::Dictionary` and `DataType::RunEndEncoded` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [draft] try to support user-defined types [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8143:
URL: https://github.com/apache/arrow-datafusion/pull/8143#issuecomment-1808781598
Thanks @yukkit -- I plan to give this a look, but probably will not have time until tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [draft] Add `LogicalType`, try to support user-defined types [datafusion]
Posted by "therealsharath (via GitHub)" <gi...@apache.org>.
therealsharath commented on PR #8143:
URL: https://github.com/apache/datafusion/pull/8143#issuecomment-2108970890
Hello, sorry if this is a redundant question. What is the status of this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [draft] Add `LogicalType`, try to support user-defined types [arrow-datafusion]
Posted by "yukkit (via GitHub)" <gi...@apache.org>.
yukkit commented on PR #8143:
URL: https://github.com/apache/arrow-datafusion/pull/8143#issuecomment-1986785323
Please accept my apologies for the delay. Due to personal circumstances, I have been unable to attend to any work. I will now proceed to resume work on this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [draft] try to support user-defined types [arrow-datafusion]
Posted by "yukkit (via GitHub)" <gi...@apache.org>.
yukkit commented on PR #8143:
URL: https://github.com/apache/arrow-datafusion/pull/8143#issuecomment-1808087054
I've organized the logic for the mutual conversion between `DFSchema` and `Schema` in datafusion. In theory, there should be no conversion logic from `Schema` to `DFSchema`. I've outlined all the modifications below.
## DFSchema to Schema
### No need to change
DefaultPhysicalPlanner
- DescribeTable
- Values -> ValuesExec
- EmptyRelation -> EmptyExec
- Unnest -> UnnestExec
- CopyTo
- Explain
- Analyze
### To be changed
- [ ] TableProvider::schema
- [ ] ViewTable
- [ ] ListingTable
- [ ] EmptyTable
- [ ] MemTable
- [ ] StreamingTable
- DataFrame
- [x] write_table: replace with DFSchema
- [ ] cache: build MemTable
## Schema to DFSchema (To be changed)
- [x] LogicalPlanBuilder::insert_into: can directly use DFSchema
- [x] LogicalPlanBuilder::explain: can directly use DFSchema
- [x] ConstEvaluator: construct DFSchema then to Schema
- [x] SqlToRel::explain_to_plan: output schema can directly use DFSchema
- [x] SqlToRel::describe_table_to_plan: output schema can directly use DFSchema
- [ ] SqlToRel::insert_to_plan: depends on `table_source.schema()`
- [ ] SqlToRel::delete_to_plan: depends on `table_source.schema()`
- [ ] ListingTable::scan: used to create_physical_expr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [draft] Add `LogicalType`, try to support user-defined types [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #8143: [draft] Add `LogicalType`, try to support user-defined types
URL: https://github.com/apache/datafusion/pull/8143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [draft] Add `LogicalType`, try to support user-defined types [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8143:
URL: https://github.com/apache/arrow-datafusion/pull/8143#discussion_r1393350639
##########
datafusion/common/src/dfschema.rs:
##########
@@ -427,48 +443,16 @@ impl DFSchema {
self_fields.zip(other_fields).all(|(f1, f2)| {
f1.qualifier() == f2.qualifier()
&& f1.name() == f2.name()
- && Self::datatype_is_semantically_equal(f1.data_type(), f2.data_type())
+ && Self::datatype_is_logically_equal(f1.data_type(), f2.data_type())
})
}
/// Checks if two [`DataType`]s are logically equal. This is a notably weaker constraint
/// than datatype_is_semantically_equal in that a Dictionary<K,V> type is logically
/// equal to a plain V type, but not semantically equal. Dictionary<K1, V1> is also
/// logically equal to Dictionary<K2, V1>.
- fn datatype_is_logically_equal(dt1: &DataType, dt2: &DataType) -> bool {
- // check nested fields
- match (dt1, dt2) {
- (DataType::Dictionary(_, v1), DataType::Dictionary(_, v2)) => {
- v1.as_ref() == v2.as_ref()
- }
- (DataType::Dictionary(_, v1), othertype) => v1.as_ref() == othertype,
- (othertype, DataType::Dictionary(_, v1)) => v1.as_ref() == othertype,
- (DataType::List(f1), DataType::List(f2))
- | (DataType::LargeList(f1), DataType::LargeList(f2))
- | (DataType::FixedSizeList(f1, _), DataType::FixedSizeList(f2, _))
- | (DataType::Map(f1, _), DataType::Map(f2, _)) => {
- Self::field_is_logically_equal(f1, f2)
- }
- (DataType::Struct(fields1), DataType::Struct(fields2)) => {
- let iter1 = fields1.iter();
- let iter2 = fields2.iter();
- fields1.len() == fields2.len() &&
- // all fields have to be the same
- iter1
- .zip(iter2)
- .all(|(f1, f2)| Self::field_is_logically_equal(f1, f2))
- }
- (DataType::Union(fields1, _), DataType::Union(fields2, _)) => {
- let iter1 = fields1.iter();
- let iter2 = fields2.iter();
- fields1.len() == fields2.len() &&
- // all fields have to be the same
- iter1
- .zip(iter2)
- .all(|((t1, f1), (t2, f2))| t1 == t2 && Self::field_is_logically_equal(f1, f2))
- }
- _ => dt1 == dt2,
- }
+ fn datatype_is_logically_equal(dt1: &LogicalType, dt2: &LogicalType) -> bool {
+ dt1 == dt2
Review Comment:
that is certainly nicer
##########
datafusion/common/src/logical_type.rs:
##########
@@ -0,0 +1,411 @@
+use std::{borrow::Cow, fmt::Display, sync::Arc};
+
+use crate::error::Result;
+use arrow_schema::{DataType, Field, IntervalUnit, TimeUnit};
+
+#[derive(Clone, Debug)]
+pub enum LogicalType {
Review Comment:
Eventually we should add doc comments here, but it also makes sense to avoid over doing it on RFC / draft.
##########
datafusion/common/src/logical_type.rs:
##########
@@ -0,0 +1,411 @@
+use std::{borrow::Cow, fmt::Display, sync::Arc};
+
+use crate::error::Result;
+use arrow_schema::{DataType, Field, IntervalUnit, TimeUnit};
+
+#[derive(Clone, Debug)]
+pub enum LogicalType {
+ Null,
+ Boolean,
+ Int8,
+ Int16,
+ Int32,
+ Int64,
+ UInt8,
+ UInt16,
+ UInt32,
+ UInt64,
+ Float16,
+ Float32,
+ Float64,
+ String,
+ LargeString,
+ Date32,
+ Date64,
+ Time32(TimeUnit),
+ Time64(TimeUnit),
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ Duration(TimeUnit),
+ Interval(IntervalUnit),
+ Binary,
+ FixedSizeBinary(i32),
+ LargeBinary,
+ Utf8,
+ LargeUtf8,
+ List(Box<LogicalType>),
+ FixedSizeList(Box<LogicalType>, i32),
+ LargeList(Box<LogicalType>),
+ Struct(Fields),
+ Map(NamedLogicalTypeRef, bool),
+ // union
+ Decimal128(u8, i8),
+ Decimal256(u8, i8),
+ Extension(ExtensionTypeRef),
+}
+
+impl PartialEq for LogicalType {
+ fn eq(&self, other: &Self) -> bool {
+ self.type_signature() == other.type_signature()
+ }
+}
+
+impl Eq for LogicalType {}
+
+impl std::hash::Hash for LogicalType {
+ fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+ self.type_signature().hash(state)
+ }
+}
+
+impl Display for LogicalType {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ write!(f, "{}", self.display_name())
+ }
+}
+
+pub type Fields = Arc<[NamedLogicalTypeRef]>;
+pub type NamedLogicalTypeRef = Arc<NamedLogicalType>;
+
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub struct NamedLogicalType {
+ name: String,
+ data_type: LogicalType,
+}
+
+impl NamedLogicalType {
+ pub fn new(name: impl Into<String>, data_type: LogicalType) -> Self {
+ Self {
+ name: name.into(),
+ data_type,
+ }
+ }
+
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ pub fn data_type(&self) -> &LogicalType {
+ &self.data_type
+ }
+}
+
+pub type OwnedTypeSignature = TypeSignature<'static>;
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct TypeSignature<'a> {
+ // **func_name**(p1, p2)
+ name: Cow<'a, str>,
+ // func_name(**p1**, **p2**)
+ params: Vec<Cow<'a, str>>,
+}
+
+impl<'a> TypeSignature<'a> {
+ pub fn new(name: impl Into<Cow<'a, str>>) -> Self {
+ Self::new_with_params(name, vec![])
+ }
+
+ pub fn new_with_params(
+ name: impl Into<Cow<'a, str>>,
+ params: Vec<Cow<'a, str>>,
+ ) -> Self {
+ Self {
+ name: name.into(),
+ params,
+ }
+ }
+
+ pub fn to_owned_type_signature(&self) -> OwnedTypeSignature {
+ OwnedTypeSignature {
+ name: self.name.to_string().into(),
+ params: self.params.iter().map(|p| p.to_string().into()).collect(),
+ }
+ }
+}
+
+pub type ExtensionTypeRef = Arc<dyn ExtensionType + Send + Sync>;
+
+pub trait ExtensionType: std::fmt::Debug {
+ fn display_name(&self) -> &str;
+ fn type_signature(&self) -> TypeSignature;
+ fn physical_type(&self) -> DataType;
+
+ fn is_comparable(&self) -> bool;
+ fn is_orderable(&self) -> bool;
+ fn is_numeric(&self) -> bool;
+}
+
+pub trait TypeManager {
+ fn register_data_type(
+ &mut self,
+ signature: impl Into<TypeSignature<'static>>,
+ extension_type: ExtensionTypeRef,
+ ) -> Result<()>;
+
+ fn data_type(&self, signature: &TypeSignature) -> Result<Option<ExtensionTypeRef>>;
+}
+
+impl ExtensionType for LogicalType {
+ fn display_name(&self) -> &str {
+ match self {
+ Self::Null => "NULL",
+ Self::Boolean => "BOOLEAN",
+ Self::Int8 => "INT8",
+ Self::Int16 => "INT16",
+ Self::Int32 => "INT32",
+ Self::Int64 => "INT64",
+ Self::UInt8 => "UINT8",
+ Self::UInt16 => "UINT16",
+ Self::UInt32 => "UINT32",
+ Self::UInt64 => "UINT64",
+ Self::Float16 => "FLOAT16",
+ Self::Float32 => "Float16",
+ Self::Float64 => "Float64",
+ Self::String => "String",
+ Self::LargeString => "LargeString",
+ Self::Date32 => "Date32",
+ Self::Date64 => "Date64",
+ Self::Time32(_) => "Time32",
+ Self::Time64(_) => "Time64",
+ Self::Timestamp(_, _) => "Timestamp",
+ Self::Duration(_) => "Duration",
+ Self::Interval(_) => "Interval",
+ Self::Binary => "Binary",
+ Self::FixedSizeBinary(_) => "FixedSizeBinary",
+ Self::LargeBinary => "LargeBinary",
+ Self::Utf8 => "Utf8",
+ Self::LargeUtf8 => "LargeUtf8",
+ Self::List(_) => "List",
+ Self::FixedSizeList(_, _) => "FixedSizeList",
+ Self::LargeList(_) => "LargeList",
+ Self::Struct(_) => "Struct",
+ Self::Map(_, _) => "Map",
+ Self::Decimal128(_, _) => "Decimal128",
+ Self::Decimal256(_, _) => "Decimal256",
+ Self::Extension(ext) => ext.display_name(),
+ }
+ }
+
+ fn type_signature(&self) -> TypeSignature {
+ match self {
+ Self::Boolean => TypeSignature::new("boolean"),
+ Self::Int32 => TypeSignature::new("int32"),
+ Self::Int64 => TypeSignature::new("int64"),
+ Self::UInt64 => TypeSignature::new("uint64"),
+ Self::Float32 => TypeSignature::new("float32"),
+ Self::Float64 => TypeSignature::new("float64"),
+ Self::String => TypeSignature::new("string"),
+ Self::Timestamp(tu, zone) => {
+ let tu = match tu {
+ TimeUnit::Second => "second",
+ TimeUnit::Millisecond => "millisecond",
+ TimeUnit::Microsecond => "microsecond",
+ TimeUnit::Nanosecond => "nanosecond",
+ };
+
+ let params = if let Some(zone) = zone {
+ vec![tu.into(), zone.as_ref().into()]
+ } else {
+ vec![tu.into()]
+ };
+
+ TypeSignature::new_with_params("timestamp", params)
+ }
+ Self::Binary => TypeSignature::new("binary"),
+ Self::Utf8 => TypeSignature::new("string"),
+ Self::Extension(ext) => ext.type_signature(),
+ Self::Struct(fields) => {
+ let params = fields.iter().map(|f| f.name().into()).collect();
+ TypeSignature::new_with_params("struct", params)
+ }
+ other => panic!("not implemented: {other:?}"),
+ }
+ }
+
+ fn physical_type(&self) -> DataType {
+ match self {
+ Self::Boolean => DataType::Boolean,
+ Self::Int32 => DataType::Int32,
+ Self::Int64 => DataType::Int64,
+ Self::UInt64 => DataType::UInt64,
+ Self::Float32 => DataType::Float32,
+ Self::Float64 => DataType::Float64,
+ Self::String => DataType::Utf8,
+ Self::Timestamp(tu, zone) => DataType::Timestamp(tu.clone(), zone.clone()),
+ Self::Binary => DataType::Binary,
+ Self::Utf8 => DataType::Utf8,
+ Self::Extension(ext) => ext.physical_type(),
+ Self::Struct(fields) => {
+ let fields = fields
+ .iter()
+ .map(|f| {
+ let name = f.name();
+ let data_type = f.physical_type();
+ Arc::new(Field::new(name, data_type, true))
+ })
+ .collect::<Vec<_>>();
+ DataType::Struct(fields.into())
+ }
+ other => panic!("not implemented {other:?}"),
Review Comment:
is the idea that [`DataType::Dictionary`](https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html#variant.Dictionary) and [DataType::RunEndEncoded](https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html#variant.RunEndEncoded) would also be included here? If so it makes sense
##########
datafusion/common/src/dfschema.rs:
##########
@@ -715,33 +715,61 @@ impl ExprSchema for DFSchema {
}
/// DFField wraps an Arrow field and adds an optional qualifier
-#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DFField {
/// Optional qualifier (usually a table or relation name)
qualifier: Option<OwnedTableReference>,
/// Arrow field definition
- field: FieldRef,
+ // field: FieldRef,
+ name: String,
+ data_type: LogicalType,
+ nullable: bool,
+ /// A map of key-value pairs containing additional custom meta data.
+ metadata: HashMap<String, String>,
Review Comment:
Maybe we could store in a BTreeMap to avoid sorting them each time 🤔
##########
datafusion/core/src/datasource/listing/table.rs:
##########
@@ -253,6 +253,7 @@ pub struct ListingOptions {
pub format: Arc<dyn FileFormat>,
/// The expected partition column names in the folder structure.
/// See [Self::with_table_partition_cols] for details
+ /// TODO this maybe LogicalType
Review Comment:
I think one usecase is to use dictionary encoding for the partition columns to minimize the overhead of creating such columns. As long as they can be represented / specified with `LogicalType` I think it is a good idea to try changing this too.
##########
datafusion/common/src/dfschema.rs:
##########
@@ -715,33 +715,61 @@ impl ExprSchema for DFSchema {
}
/// DFField wraps an Arrow field and adds an optional qualifier
-#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DFField {
/// Optional qualifier (usually a table or relation name)
qualifier: Option<OwnedTableReference>,
/// Arrow field definition
- field: FieldRef,
+ // field: FieldRef,
+ name: String,
+ data_type: LogicalType,
+ nullable: bool,
+ /// A map of key-value pairs containing additional custom meta data.
+ metadata: HashMap<String, String>,
Review Comment:
Maybe we could store in a BTreeMap to avoid sorting them each time 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [draft] Add `LogicalType`, try to support user-defined types [arrow-datafusion]
Posted by "lewiszlw (via GitHub)" <gi...@apache.org>.
lewiszlw commented on PR #8143:
URL: https://github.com/apache/arrow-datafusion/pull/8143#issuecomment-1884406664
What's the status of this pr? This should be a very useful feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [draft] try to support user-defined types [arrow-datafusion]
Posted by "yukkit (via GitHub)" <gi...@apache.org>.
yukkit commented on code in PR #8143:
URL: https://github.com/apache/arrow-datafusion/pull/8143#discussion_r1391953675
##########
datafusion/core/src/execution/context/mod.rs:
##########
@@ -661,6 +667,28 @@ impl SessionContext {
}
}
+ async fn create_type(&self, cmd: CreateType) -> Result<DataFrame> {
Review Comment:
This is the core processing logic for creating SQL UDT.
##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -37,7 +37,7 @@ use std::sync::Arc;
/// trait to allow expr to typable with respect to a schema
pub trait ExprSchemable {
/// given a schema, return the type of the expr
- fn get_type<S: ExprSchema>(&self, schema: &S) -> Result<DataType>;
+ fn get_type<S: ExprSchema>(&self, schema: &S) -> Result<LogicalType>;
Review Comment:
Replace all arrow `DataType` used in the logical planning stage with `LogicalType`(literal ScalarValue TBD).
##########
datafusion/expr/src/field_util.rs:
##########
@@ -76,22 +74,72 @@ impl GetFieldAccessSchema {
(other, _) => plan_err!("The expression to get an indexed field is only valid for `List`, `Struct`, or `Map` types, got {other}"),
}
}
- Self::ListIndex{ key_dt } => {
- match (data_type, key_dt) {
- (DataType::List(lt), DataType::Int64) => Ok(Field::new("list", lt.data_type().clone(), true)),
- (DataType::List(_), _) => plan_err!(
- "Only ints are valid as an indexed field in a list"
- ),
- (other, _) => plan_err!("The expression to get an indexed field is only valid for `List` or `Struct` types, got {other}"),
+ Self::ListIndex => {
+ match data_type {
+ DataType::List(lt) => Ok(Field::new("list", lt.data_type().clone(), true)),
+ other => plan_err!("The expression to get an indexed field is only valid for `List` or `Struct` types, got {other}"),
+ }
+ }
+ Self::ListRange => {
+ match data_type {
+ DataType::List(_) => Ok(Field::new("list", data_type.clone(), true)),
+ other => plan_err!("The expression to get an indexed field is only valid for `List` or `Struct` types, got {other}"),
}
}
- Self::ListRange{ start_dt, stop_dt } => {
- match (data_type, start_dt, stop_dt) {
- (DataType::List(_), DataType::Int64, DataType::Int64) => Ok(Field::new("list", data_type.clone(), true)),
- (DataType::List(_), _, _) => plan_err!(
- "Only ints are valid as an indexed field in a list"
+ }
+ }
+ /// Returns the schema [`DFField`] from a [`LogicalType::List`] or
+ /// [`LogicalType::Struct`] indexed by this structure
+ ///
+ /// # Error
+ /// Errors if
+ /// * the `data_type` is not a Struct or a List,
+ /// * the `data_type` of the name/index/start-stop do not match a supported index type
+ pub fn get_accessed_df_field(&self, data_type: &LogicalType) -> Result<DFField> {
Review Comment:
TODO: More elegant implementation to avoid duplication of code
##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -142,31 +180,39 @@ impl ExprSchemable for Expr {
| Expr::IsUnknown(_)
| Expr::IsNotTrue(_)
| Expr::IsNotFalse(_)
- | Expr::IsNotUnknown(_) => Ok(DataType::Boolean),
+ | Expr::IsNotUnknown(_) => Ok(LogicalType::Boolean),
Expr::ScalarSubquery(subquery) => {
Ok(subquery.subquery.schema().field(0).data_type().clone())
}
Expr::BinaryExpr(BinaryExpr {
ref left,
ref right,
ref op,
- }) => get_result_type(&left.get_type(schema)?, op, &right.get_type(schema)?),
- Expr::Like { .. } | Expr::SimilarTo { .. } => Ok(DataType::Boolean),
+ }) => {
+ // TODO not convert to physical DataType
+ let physical_type = get_result_type(
Review Comment:
TODO: Modify the function `get_result_type` to accept LogicalType as parameters.
##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -332,8 +382,8 @@ impl ExprSchemable for Expr {
// TODO(kszucs): most of the operations do not validate the type correctness
// like all of the binary expressions below. Perhaps Expr should track the
// type of the expression?
-
- if can_cast_types(&this_type, cast_to_type) {
+ // TODO The basis for whether cast can be executed should be the logical type
+ if can_cast_types(&this_type.physical_type(), &cast_to_type.physical_type()) {
Review Comment:
TODO: Modify the function `can_cast_types` to accept LogicalType as parameters.
##########
datafusion/expr/src/logical_plan/builder.rs:
##########
@@ -1223,17 +1222,21 @@ pub fn union(left_plan: LogicalPlan, right_plan: LogicalPlan) -> Result<LogicalP
)
.map(|(left_field, right_field)| {
let nullable = left_field.is_nullable() || right_field.is_nullable();
- let data_type =
- comparison_coercion(left_field.data_type(), right_field.data_type())
- .ok_or_else(|| {
- plan_datafusion_err!(
+ // FIXME support logical data types, not convert to DataType
+ let data_type = comparison_coercion(
Review Comment:
Modify the function `comparison_coercion ` to accept LogicalType as parameters.
##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -87,14 +91,27 @@ impl ExprSchemable for Expr {
.iter()
.map(|e| e.get_type(schema))
.collect::<Result<Vec<_>>>()?;
- Ok((fun.return_type)(&data_types)?.as_ref().clone())
+
+ // TODO not convert to DataType
+ let data_types = data_types
+ .into_iter()
+ .map(|e| e.physical_type())
+ .collect::<Vec<_>>();
+
+ Ok((fun.return_type)(&data_types)?.as_ref().clone().into())
Review Comment:
TODO: Functions signatures for `udf/udaf` use `TypeSignature` instead of the arrow `DataType`
##########
datafusion/expr/src/utils.rs:
##########
@@ -886,39 +887,34 @@ pub(crate) fn find_column_indexes_referenced_by_expr(
/// can this data type be used in hash join equal conditions??
/// data types here come from function 'equal_rows', if more data types are supported
/// in equal_rows(hash join), add those data types here to generate join logical plan.
-pub fn can_hash(data_type: &DataType) -> bool {
+pub fn can_hash(data_type: &LogicalType) -> bool {
Review Comment:
This function `can_hash ` might be abstracted as a property of logical data types.
##########
datafusion/core/src/execution/context/mod.rs:
##########
@@ -1846,6 +1895,22 @@ impl SessionState {
}
}
+impl TypeManager for SessionState {
Review Comment:
SessionState implements TymeManager for registering and retrieving extended data types.
##########
datafusion/sql/src/planner.rs:
##########
@@ -295,15 +303,16 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
})
}
- pub(crate) fn convert_data_type(&self, sql_type: &SQLDataType) -> Result<DataType> {
+ pub(crate) fn convert_data_type(
Review Comment:
Convert data types in the `AST` to logical data types.
##########
datafusion/core/src/execution/context/mod.rs:
##########
@@ -661,6 +667,28 @@ impl SessionContext {
}
}
+ async fn create_type(&self, cmd: CreateType) -> Result<DataFrame> {
+ let CreateType {
+ name, data_type, ..
+ } = cmd;
+
+ let extension_type = Arc::new(NamedLogicalType::new(name, data_type));
+ let name = extension_type.display_name();
+ let type_signature = extension_type.type_signature();
+
+ let exists_type = self.data_type(&type_signature)?;
+ if exists_type.is_some() {
+ return exec_err!("DataType '{name}' already exists");
+ }
+
+ self.register_data_type(
+ type_signature.to_owned_type_signature(),
+ extension_type,
+ )?;
Review Comment:
Register the extension types created through SQL UDT.
##########
datafusion/optimizer/src/analyzer/type_coercion.rs:
##########
@@ -209,8 +211,9 @@ impl TreeNodeRewriter for TypeCoercionRewriter {
escape_char,
case_insensitive,
}) => {
- let left_type = expr.get_type(&self.schema)?;
- let right_type = pattern.get_type(&self.schema)?;
+ // FIXME like_coercion use LogicalType
+ let left_type = expr.get_type(&self.schema)?.physical_type();
+ let right_type = pattern.get_type(&self.schema)?.physical_type();
let coerced_type = like_coercion(&left_type, &right_type).ok_or_else(|| {
Review Comment:
TODO: Modify the function `like_coercion ` to accept LogicalType as parameters.
##########
datafusion/sql/src/planner.rs:
##########
@@ -370,11 +379,21 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
};
make_decimal_type(precision, scale)
}
- SQLDataType::Bytea => Ok(DataType::Binary),
- SQLDataType::Interval => Ok(DataType::Interval(IntervalUnit::MonthDayNano)),
+ SQLDataType::Bytea => Ok(LogicalType::Binary),
+ SQLDataType::Interval => Ok(LogicalType::Interval(IntervalUnit::MonthDayNano)),
// Explicitly list all other types so that if sqlparser
// adds/changes the `SQLDataType` the compiler will tell us on upgrade
// and avoid bugs like https://github.com/apache/arrow-datafusion/issues/3059
+ SQLDataType::Custom(name, params) => {
+ let name = object_name_to_string(name);
+ let params = params.iter().map(Into::into).collect();
+ let type_signature = TypeSignature::new_with_params(name, params);
+ if let Some(data_type) = self.context_provider.get_data_type(&type_signature) {
+ return Ok(data_type);
+ }
+
+ plan_err!("User-Defined SQL type {sql_type:?} not found")
+ }
Review Comment:
Convert custom data types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [draft] Add `LogicalType`, try to support user-defined types [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8143:
URL: https://github.com/apache/datafusion/pull/8143#issuecomment-2101780981
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [draft] Add `LogicalType`, try to support user-defined types [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8143:
URL: https://github.com/apache/datafusion/pull/8143#issuecomment-2242780414
FYI https://github.com/apache/datafusion/pull/11160 tracks a new proposal for this feature. It seems to be gaining traction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [draft] Add `LogicalType`, try to support user-defined types [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8143:
URL: https://github.com/apache/arrow-datafusion/pull/8143#issuecomment-1887797122
I think this PR is stalled and I don't have any update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [draft] try to support user-defined types [arrow-datafusion]
Posted by "yukkit (via GitHub)" <gi...@apache.org>.
yukkit commented on PR #8143:
URL: https://github.com/apache/arrow-datafusion/pull/8143#issuecomment-1807061643
Current PR has Some Unresolved Issues Requiring Collaboration for Discussion
Once there is a consensus on all the issues among the team, I will reorganize the PR accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [draft] Add `LogicalType`, try to support user-defined types [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8143:
URL: https://github.com/apache/arrow-datafusion/pull/8143#issuecomment-1986801969
No worries at all -- I hope all is well and we look forward to this work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [draft] Add `LogicalType`, try to support user-defined types [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8143:
URL: https://github.com/apache/datafusion/pull/8143#issuecomment-2227164276
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [draft] Add `LogicalType`, try to support user-defined types [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8143:
URL: https://github.com/apache/datafusion/pull/8143#issuecomment-2110981316
> Hello, sorry if this is a redundant question. What is the status of this PR?
I think it is stale and on track to be closed from what I can see
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org