You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "notfilippo (via GitHub)" <gi...@apache.org> on 2024/10/10 14:50:56 UTC
[PR] [logical-types] add NativeType and LogicalType [datafusion]
notfilippo opened a new pull request, #12853:
URL: https://github.com/apache/datafusion/pull/12853
This PR is part of #12622. It introduces the notion of DataFusion `NativeType` and the `LogicalType` trait.
The changes stem from various discussions of #11513 #12644 #12635 and other. The goal of this PR is to unify those discussions into a concrete code proposal to be merged to `main` [1] in order to unblock work on #12644 #12635.
[1] as opposed to the `logical-types` branch used currently for experimenting #12622
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2437180263
> Otherwise, I think we have a question that which DataType::Utf8 or DataType::Utf8View we need to cast to.
Ah! Now I understand the question. I think it makes sense to me that if there are no hints in the environment that suggest what specific type we should cast to then there is no value in choosing a specific cast type and we could fallback to some generic mapping e.g. casting to `Native::String` -> casting to `DataType::Utf8`. Not sure if there is any added benefit of letting the logical type choose it's sequence of casts instead of providing just one fallback type but maybe I'm missing something.
Now if a signature accepts DataType::Utf8, LargeUtf8 and Utf8View and we receive DataType::Int64 what do we do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1817702282
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,391 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{
+ LogicalField, LogicalFieldRef, LogicalFields, LogicalType, LogicalUnionFields,
+ TypeSignature,
+};
+use crate::error::{Result, _internal_err};
+use arrow::compute::can_cast_types;
+use arrow_schema::{
+ DataType, Field, FieldRef, Fields, IntervalUnit, TimeUnit, UnionFields,
+};
+use std::sync::Arc;
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean type representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ Time(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ Utf8,
+ /// A list of some logical data type with variable length.
+ List(LogicalFieldRef),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(LogicalFieldRef, i32),
+ /// A nested type that contains a number of sub-fields.
+ Struct(LogicalFields),
+ /// A nested type that can represent slots of differing types.
+ Union(LogicalUnionFields),
+ /// Decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal(u8, i8),
+ /// A Map is a type that an association between a key and a value.
+ ///
+ /// The key and value types are not constrained, but keys should be
+ /// hashable and unique.
+ ///
+ /// In a field with Map type, key type and the second the value type. The names of the
+ /// child fields may be respectively "entries", "key", and "value", but this is
+ /// not enforced.
+ Map(LogicalFieldRef),
+}
+
+impl LogicalType for NativeType {
+ fn native(&self) -> &NativeType {
+ self
+ }
+
+ fn signature(&self) -> TypeSignature<'_> {
+ TypeSignature::Native(self)
+ }
+
+ fn default_cast_for(&self, origin: &DataType) -> Result<DataType> {
+ use DataType::*;
+
+ fn default_field_cast(to: &LogicalField, from: &Field) -> Result<FieldRef> {
+ Ok(Arc::new(Field::new(
+ to.name.clone(),
+ to.logical_type.default_cast_for(from.data_type())?,
+ to.nullable,
+ )))
+ }
+
+ Ok(match (self, origin) {
+ (Self::Null, _) => Null,
+ (Self::Boolean, _) => Boolean,
+ (Self::Int8, _) => Int8,
+ (Self::Int16, _) => Int16,
+ (Self::Int32, _) => Int32,
+ (Self::Int64, _) => Int64,
+ (Self::UInt8, _) => UInt8,
+ (Self::UInt16, _) => UInt16,
+ (Self::UInt32, _) => UInt32,
+ (Self::UInt64, _) => UInt64,
+ (Self::Float16, _) => Float16,
+ (Self::Float32, _) => Float32,
+ (Self::Float64, _) => Float64,
+ (Self::Decimal(p, s), _) if p <= &38 => Decimal128(*p, *s),
+ (Self::Decimal(p, s), _) => Decimal256(*p, *s),
+ (Self::Timestamp(tu, tz), _) => Timestamp(*tu, tz.clone()),
+ (Self::Date, _) => Date32,
+ (Self::Time(tu), _) => match tu {
+ TimeUnit::Second | TimeUnit::Millisecond => Time32(*tu),
+ TimeUnit::Microsecond | TimeUnit::Nanosecond => Time64(*tu),
+ },
+ (Self::Duration(tu), _) => Duration(*tu),
+ (Self::Interval(iu), _) => Interval(*iu),
+ (Self::Binary, LargeUtf8) => LargeBinary,
+ (Self::Binary, Utf8View) => BinaryView,
+ (Self::Binary, data_type) if can_cast_types(data_type, &BinaryView) => {
+ BinaryView
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &LargeBinary) => {
+ LargeBinary
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &Binary) => Binary,
+ (Self::FixedSizeBinary(size), _) => FixedSizeBinary(*size),
+ (Self::Utf8, LargeBinary) => LargeUtf8,
+ (Self::Utf8, BinaryView) => Utf8View,
+ (Self::Utf8, data_type) if can_cast_types(data_type, &Utf8View) => Utf8View,
+ (Self::Utf8, data_type) if can_cast_types(data_type, &LargeUtf8) => LargeUtf8,
+ (Self::Utf8, data_type) if can_cast_types(data_type, &Utf8) => Utf8,
+ (Self::List(to_field), List(from_field) | FixedSizeList(from_field, _)) => {
+ List(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeList(from_field)) => {
+ LargeList(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), ListView(from_field)) => {
+ ListView(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeListView(from_field)) => {
+ LargeListView(default_field_cast(to_field, from_field)?)
+ }
+ // List array where each element is a len 1 list of the origin type
+ (Self::List(field), _) => List(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ ))),
+ (
+ Self::FixedSizeList(to_field, to_size),
+ FixedSizeList(from_field, from_size),
+ ) if from_size == to_size => {
+ FixedSizeList(default_field_cast(to_field, from_field)?, *to_size)
+ }
+ (
+ Self::FixedSizeList(to_field, size),
+ List(from_field)
+ | LargeList(from_field)
+ | ListView(from_field)
+ | LargeListView(from_field),
+ ) => FixedSizeList(default_field_cast(to_field, from_field)?, *size),
+ // FixedSizeList array where each element is a len 1 list of the origin type
+ (Self::FixedSizeList(field, size), _) => FixedSizeList(
+ Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ )),
+ *size,
+ ),
+ // From https://github.com/apache/arrow-rs/blob/56525efbd5f37b89d1b56aa51709cab9f81bc89e/arrow-cast/src/cast/mod.rs#L189-L196
+ (Self::Struct(to_fields), Struct(from_fields))
+ if from_fields.len() == to_fields.len() =>
+ {
+ Struct(
+ from_fields
+ .iter()
+ .zip(to_fields.iter())
+ .map(|(from, to)| default_field_cast(to, from))
+ .collect::<Result<Fields>>()?,
+ )
+ }
+ (Self::Struct(to_fields), Null) => Struct(
+ to_fields
+ .iter()
+ .map(|field| {
+ Ok(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(&Null)?,
+ field.nullable,
+ )))
+ })
+ .collect::<Result<Fields>>()?,
+ ),
+ (Self::Map(to_field), Map(from_field, sorted)) => {
+ Map(default_field_cast(to_field, from_field)?, *sorted)
+ }
+ (Self::Map(field), Null) => Map(
+ Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(&Null)?,
+ field.nullable,
+ )),
+ false,
+ ),
+ (Self::Union(to_fields), Union(from_fields, mode))
+ if from_fields.len() == to_fields.len() =>
+ {
+ Union(
+ from_fields
+ .iter()
+ .zip(to_fields.iter())
+ .map(|((_, from), (i, to))| {
+ Ok((*i, default_field_cast(to, from)?))
+ })
+ .collect::<Result<UnionFields>>()?,
+ *mode,
+ )
+ }
+ _ => {
+ return _internal_err!(
+ "Unavailable default cast for native type {:?} from physical type {:?}",
+ self,
+ origin
+ )
+ }
+ })
+ }
+}
+
+// The following From<DataType>, From<Field>, ... implementations are temporary
+// mapping solutions to provide backwards compatibility while transitioning from
+// the purely physical system to a logical / physical system.
+
+impl From<DataType> for NativeType {
Review Comment:
We can add this for convenience
```rust
impl From<&DataType> for NativeType {
fn from(value: &DataType) -> Self {
value.clone().into()
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1811365717
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,328 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{ops::Deref, sync::Arc};
+
+use arrow_schema::{DataType, Field, Fields, IntervalUnit, TimeUnit, UnionFields};
+
+use super::{LogicalType, TypeSignature};
+
+/// A record of a native type, its name and its nullability.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct NativeField {
+ name: String,
+ native_type: NativeType,
+ nullable: bool,
+}
+
+impl NativeField {
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ pub fn native_type(&self) -> &NativeType {
+ &self.native_type
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.nullable
+ }
+}
+
+/// A reference counted [`NativeField`].
+pub type NativeFieldRef = Arc<NativeField>;
+
+/// A cheaply cloneable, owned collection of [`NativeFieldRef`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct NativeFields(Arc<[NativeFieldRef]>);
+
+impl Deref for NativeFields {
+ type Target = [NativeFieldRef];
+
+ fn deref(&self) -> &Self::Target {
+ self.0.as_ref()
+ }
+}
+
+/// A cheaply cloneable, owned collection of [`NativeFieldRef`] and their
+/// corresponding type ids.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct NativeUnionFields(Arc<[(i8, NativeFieldRef)]>);
+
+impl Deref for NativeUnionFields {
+ type Target = [(i8, NativeFieldRef)];
+
+ fn deref(&self) -> &Self::Target {
+ self.0.as_ref()
+ }
+}
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean type representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ Time(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ Utf8,
+ /// A list of some logical data type with variable length.
+ List(NativeFieldRef),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(NativeFieldRef, i32),
+ /// A nested type that contains a number of sub-fields.
+ Struct(NativeFields),
+ /// A nested type that can represent slots of differing types.
+ Union(NativeUnionFields),
+ /// Decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal(u8, i8),
+ /// A Map is a type that an association between a key and a value.
+ ///
+ /// The key and value types are not constrained, but keys should be
+ /// hashable and unique.
+ ///
+ /// In a field with Map type, the field has a child Struct field, which then
+ /// has two children: key type and the second the value type. The names of the
+ /// child fields may be respectively "entries", "key", and "value", but this is
+ /// not enforced.
+ Map(NativeFieldRef),
Review Comment:
Actually there is a bit of a problem translating DataType::Map to NativeType::Map since there is no way of retrieving the key and the value because of the "but this is not enforced." regarding the naming of the struct field types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2437095868
> During logical planning (and/or execution if we implement runtime adaptiveness) you would have instead type information for all the function parameters and you could decide to which type it makes more sense to cast.
I'm not clear about this part. I assume that we only have Logical::Int and DataType::Int64 as the function input, and we also know the the function accept Logical::String. Is there any other information we know? Otherwise, I think we have a question that which DataType::Utf8 or DataType::Utf8View we need to cast to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2437275323
> since we don't know what is the backing native type of it
Isn't this the goal on having `native()` on [every LogicalType](https://github.com/notfilippo/datafusion/blob/7ed789176369c30972a41c76aaa32e3635a13222/datafusion/common/src/types/logical.rs#L78) (even user defined ones)? Anyway we could define a default mapping for NativeTypes and let user defined types override that behaviour via the trait.
Something like:
```rust
pub trait LogicalType: Sync + Send {
fn native(&self) -> &NativeType;
fn signature(&self) -> TypeSignature<'_>;
fn default_cast(&self) -> &[DataType] {
self.native().default_cast()
}
}
```
and then:
```rust
impl LogicalType for NativeType {
fn native(&self) -> &NativeType {
self
}
fn signature(&self) -> TypeSignature<'_> {
TypeSignature::Native(self)
}
fn default_cast(&self) -> &[DataType] {
use NativeType::*;
match self {
Utf8 => &[DataType::Utf8View, DataType::LargeUtf8, DataType::Utf8],
_ => todo!()
}
}
}
```
> The order is Utf8View -> LargeUtf8 -> Utf8.
So this means that if you received an Int64 then you first try to coerce to Utf8View and in what cases would you fall back to the LargeUtf8 and Utf8?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1816923693
##########
datafusion/common/src/types/native.rs:
##########
@@ -188,6 +190,147 @@ impl LogicalType for NativeType {
fn signature(&self) -> TypeSignature<'_> {
TypeSignature::Native(self)
}
+
+ fn default_cast_for(&self, origin: &DataType) -> Result<DataType> {
+ use DataType::*;
+
+ fn default_field_cast(to: &LogicalField, from: &Field) -> Result<FieldRef> {
+ Ok(Arc::new(Field::new(
+ to.name.clone(),
+ to.logical_type.default_cast_for(from.data_type())?,
+ to.nullable,
+ )))
+ }
+
+ Ok(match (self, origin) {
+ (Self::Null, _) => Null,
+ (Self::Boolean, _) => Boolean,
+ (Self::Int8, _) => Int8,
+ (Self::Int16, _) => Int16,
+ (Self::Int32, _) => Int32,
+ (Self::Int64, _) => Int64,
+ (Self::UInt8, _) => UInt8,
+ (Self::UInt16, _) => UInt16,
+ (Self::UInt32, _) => UInt32,
+ (Self::UInt64, _) => UInt64,
+ (Self::Float16, _) => Float16,
+ (Self::Float32, _) => Float32,
+ (Self::Float64, _) => Float64,
+ (Self::Decimal(p, s), _) if p <= &38 => Decimal128(*p, *s),
+ (Self::Decimal(p, s), _) => Decimal256(*p, *s),
+ (Self::Timestamp(tu, tz), _) => Timestamp(*tu, tz.clone()),
+ (Self::Date, _) => Date32,
+ (Self::Time(tu), _) => match tu {
+ TimeUnit::Second | TimeUnit::Millisecond => Time32(*tu),
+ TimeUnit::Microsecond | TimeUnit::Nanosecond => Time64(*tu),
+ },
+ (Self::Duration(tu), _) => Duration(*tu),
+ (Self::Interval(iu), _) => Interval(*iu),
+ (Self::Binary, LargeUtf8) => LargeBinary,
+ (Self::Binary, Utf8View) => BinaryView,
+ (Self::Binary, _) => Binary,
+ (Self::FixedSizeBinary(size), _) => FixedSizeBinary(*size),
+ (Self::Utf8, LargeBinary) => LargeUtf8,
+ (Self::Utf8, BinaryView) => Utf8View,
+ (Self::Utf8, _) => Utf8,
Review Comment:
Good point about the `can_cast_type`. Fixing soon!
> It seems they need to copy the whole default_cast_for and change it for their logical type.
Well they can override the function, put their specific case at the top and on the default (`(_, _) =>`) matching pattern they can just call the NativeType implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "berkaysynnada (via GitHub)" <gi...@apache.org>.
berkaysynnada commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2435194014
Has the [logical type branch](https://github.com/apache/datafusion/tree/logical-types) reached its final state, and are we starting the migration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2437061185
@jayzhan211 I think this approach goes against the problems that logical types try to solve. Specifically I think that we could (and should) define some kind of casting rules for a logical types (`NativeType::*` -> `String`, `Int*` <-> `Float*`, `JSON` -> `String`) via a method to be implemented in the LogicalType trait (`can_cast_to(&self, rhs: &dyn LogicalType)`). Then during physical planning when a function that takes a Logical string receives in input a Logical Int8 it can only check if the logical cast is valid. During logical planning (and/or execution if we implement runtime adaptiveness) you would have instead type information for all the function parameters and you could decide to which type it makes more sense to cast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1807176173
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
Review Comment:
`impl From<DataType> for NativeType` is enough for native type since we can know whether the ArrayRef matches the LogicalType we have. But for LogicalType::UserDefined, I think we need to define what kind of `DataType` it could be decoded to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1810509349
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,328 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{ops::Deref, sync::Arc};
+
+use arrow_schema::{DataType, Field, Fields, IntervalUnit, TimeUnit, UnionFields};
+
+use super::{LogicalType, TypeSignature};
+
+/// A record of a native type, its name and its nullability.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct NativeField {
+ name: String,
+ native_type: NativeType,
+ nullable: bool,
+}
+
+impl NativeField {
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ pub fn native_type(&self) -> &NativeType {
+ &self.native_type
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.nullable
+ }
+}
Review Comment:
Why not simply make these fields `pub`?
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// Signature that uniquely identifies a type among other types.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeSignature<'a> {
+ /// Represents a built-in native type.
+ Native(&'a NativeType),
+ /// Represents an arrow-compatible extension type.
+ /// (<https://arrow.apache.org/docs/format/Columnar.html#extension-types>)
+ ///
+ /// The `name` should contain the same value as 'ARROW:extension:name'.
+ Extension {
+ name: &'a str,
+ parameters: &'a [TypeParameter<'a>],
+ },
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeParameter<'a> {
+ Type(TypeSignature<'a>),
+ Number(i128),
+}
Review Comment:
As someone unfamiliar with the entire discussion of this feature, it's not clear to me from reading the above link to the extension-types of arrow how these type parameters map to the metadata described in the link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2435533799
I might take a look at it this weekend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1818018478
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,399 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{
+ LogicalField, LogicalFieldRef, LogicalFields, LogicalType, LogicalUnionFields,
+ TypeSignature,
+};
+use crate::error::{Result, _internal_err};
+use arrow::compute::can_cast_types;
+use arrow_schema::{
+ DataType, Field, FieldRef, Fields, IntervalUnit, TimeUnit, UnionFields,
+};
+use std::sync::Arc;
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
Review Comment:
I was confused about the name of `NativeType` before I understood the context. I thought the `NativeType` is a mirror of the Arrow DataType but it's not, it means the DataFusion type.
I think maybe we can just call it `Type` or `SqlType`. If I don't misunderstand it, it will be the core type system of DataFusion, right?
By the way, we have another type list in the SQL parser for the SQL API.
https://github.com/apache/datafusion-sqlparser-rs/blob/8e0d26abb3c4dda2fa091dfd9611be1dbdc14755/src/ast/data_type.rs#L36
I'm not sure if we could integrate both of them. Just mention that we need to maintain 3-level type mapping in the future. (SQL -> DataFusion -> Arrow)
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,399 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{
+ LogicalField, LogicalFieldRef, LogicalFields, LogicalType, LogicalUnionFields,
+ TypeSignature,
+};
+use crate::error::{Result, _internal_err};
+use arrow::compute::can_cast_types;
+use arrow_schema::{
+ DataType, Field, FieldRef, Fields, IntervalUnit, TimeUnit, UnionFields,
+};
+use std::sync::Arc;
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean type representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ Time(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ String,
+ /// A list of some logical data type with variable length.
+ List(LogicalFieldRef),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(LogicalFieldRef, i32),
+ /// A nested type that contains a number of sub-fields.
+ Struct(LogicalFields),
+ /// A nested type that can represent slots of differing types.
+ Union(LogicalUnionFields),
+ /// Decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal(u8, i8),
+ /// A Map is a type that an association between a key and a value.
+ ///
+ /// The key and value types are not constrained, but keys should be
+ /// hashable and unique.
+ ///
+ /// In a field with Map type, key type and the second the value type. The names of the
+ /// child fields may be respectively "entries", "key", and "value", but this is
+ /// not enforced.
+ Map(LogicalFieldRef),
+}
+
+impl LogicalType for NativeType {
+ fn native(&self) -> &NativeType {
+ self
+ }
+
+ fn signature(&self) -> TypeSignature<'_> {
+ TypeSignature::Native(self)
+ }
+
+ fn default_cast_for(&self, origin: &DataType) -> Result<DataType> {
+ use DataType::*;
+
+ fn default_field_cast(to: &LogicalField, from: &Field) -> Result<FieldRef> {
+ Ok(Arc::new(Field::new(
+ to.name.clone(),
+ to.logical_type.default_cast_for(from.data_type())?,
+ to.nullable,
+ )))
+ }
+
+ Ok(match (self, origin) {
+ (Self::Null, _) => Null,
+ (Self::Boolean, _) => Boolean,
+ (Self::Int8, _) => Int8,
+ (Self::Int16, _) => Int16,
+ (Self::Int32, _) => Int32,
+ (Self::Int64, _) => Int64,
+ (Self::UInt8, _) => UInt8,
+ (Self::UInt16, _) => UInt16,
+ (Self::UInt32, _) => UInt32,
+ (Self::UInt64, _) => UInt64,
+ (Self::Float16, _) => Float16,
+ (Self::Float32, _) => Float32,
+ (Self::Float64, _) => Float64,
+ (Self::Decimal(p, s), _) if p <= &38 => Decimal128(*p, *s),
+ (Self::Decimal(p, s), _) => Decimal256(*p, *s),
+ (Self::Timestamp(tu, tz), _) => Timestamp(*tu, tz.clone()),
+ (Self::Date, _) => Date32,
+ (Self::Time(tu), _) => match tu {
+ TimeUnit::Second | TimeUnit::Millisecond => Time32(*tu),
+ TimeUnit::Microsecond | TimeUnit::Nanosecond => Time64(*tu),
+ },
+ (Self::Duration(tu), _) => Duration(*tu),
+ (Self::Interval(iu), _) => Interval(*iu),
+ (Self::Binary, LargeUtf8) => LargeBinary,
+ (Self::Binary, Utf8View) => BinaryView,
+ (Self::Binary, data_type) if can_cast_types(data_type, &BinaryView) => {
+ BinaryView
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &LargeBinary) => {
+ LargeBinary
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &Binary) => Binary,
+ (Self::FixedSizeBinary(size), _) => FixedSizeBinary(*size),
+ (Self::String, LargeBinary) => LargeUtf8,
+ (Self::String, BinaryView) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8View) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &LargeUtf8) => {
+ LargeUtf8
+ }
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8) => Utf8,
+ (Self::List(to_field), List(from_field) | FixedSizeList(from_field, _)) => {
+ List(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeList(from_field)) => {
+ LargeList(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), ListView(from_field)) => {
+ ListView(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeListView(from_field)) => {
+ LargeListView(default_field_cast(to_field, from_field)?)
+ }
+ // List array where each element is a len 1 list of the origin type
+ (Self::List(field), _) => List(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ ))),
+ (
+ Self::FixedSizeList(to_field, to_size),
+ FixedSizeList(from_field, from_size),
+ ) if from_size == to_size => {
+ FixedSizeList(default_field_cast(to_field, from_field)?, *to_size)
+ }
+ (
+ Self::FixedSizeList(to_field, size),
+ List(from_field)
+ | LargeList(from_field)
+ | ListView(from_field)
+ | LargeListView(from_field),
+ ) => FixedSizeList(default_field_cast(to_field, from_field)?, *size),
Review Comment:
We will check if the size is equal when both logical and physical are `FixedSizeList`. It makes sense to me. However, in this case (logical fixed size to physical non-fixed size), we can't get the length information from the physical side. I guess it will cause some problems when the size of the logical one is less than the physical size (e.g. data loss).
Will we check the real length when the data are processed? If we need to check every record, does it cause some effort for the performance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1818073887
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,399 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{
+ LogicalField, LogicalFieldRef, LogicalFields, LogicalType, LogicalUnionFields,
+ TypeSignature,
+};
+use crate::error::{Result, _internal_err};
+use arrow::compute::can_cast_types;
+use arrow_schema::{
+ DataType, Field, FieldRef, Fields, IntervalUnit, TimeUnit, UnionFields,
+};
+use std::sync::Arc;
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean type representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ Time(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ String,
+ /// A list of some logical data type with variable length.
+ List(LogicalFieldRef),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(LogicalFieldRef, i32),
+ /// A nested type that contains a number of sub-fields.
+ Struct(LogicalFields),
+ /// A nested type that can represent slots of differing types.
+ Union(LogicalUnionFields),
+ /// Decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal(u8, i8),
+ /// A Map is a type that an association between a key and a value.
+ ///
+ /// The key and value types are not constrained, but keys should be
+ /// hashable and unique.
+ ///
+ /// In a field with Map type, key type and the second the value type. The names of the
+ /// child fields may be respectively "entries", "key", and "value", but this is
+ /// not enforced.
+ Map(LogicalFieldRef),
+}
+
+impl LogicalType for NativeType {
+ fn native(&self) -> &NativeType {
+ self
+ }
+
+ fn signature(&self) -> TypeSignature<'_> {
+ TypeSignature::Native(self)
+ }
+
+ fn default_cast_for(&self, origin: &DataType) -> Result<DataType> {
+ use DataType::*;
+
+ fn default_field_cast(to: &LogicalField, from: &Field) -> Result<FieldRef> {
+ Ok(Arc::new(Field::new(
+ to.name.clone(),
+ to.logical_type.default_cast_for(from.data_type())?,
+ to.nullable,
+ )))
+ }
+
+ Ok(match (self, origin) {
+ (Self::Null, _) => Null,
+ (Self::Boolean, _) => Boolean,
+ (Self::Int8, _) => Int8,
+ (Self::Int16, _) => Int16,
+ (Self::Int32, _) => Int32,
+ (Self::Int64, _) => Int64,
+ (Self::UInt8, _) => UInt8,
+ (Self::UInt16, _) => UInt16,
+ (Self::UInt32, _) => UInt32,
+ (Self::UInt64, _) => UInt64,
+ (Self::Float16, _) => Float16,
+ (Self::Float32, _) => Float32,
+ (Self::Float64, _) => Float64,
+ (Self::Decimal(p, s), _) if p <= &38 => Decimal128(*p, *s),
+ (Self::Decimal(p, s), _) => Decimal256(*p, *s),
+ (Self::Timestamp(tu, tz), _) => Timestamp(*tu, tz.clone()),
+ (Self::Date, _) => Date32,
+ (Self::Time(tu), _) => match tu {
+ TimeUnit::Second | TimeUnit::Millisecond => Time32(*tu),
+ TimeUnit::Microsecond | TimeUnit::Nanosecond => Time64(*tu),
+ },
+ (Self::Duration(tu), _) => Duration(*tu),
+ (Self::Interval(iu), _) => Interval(*iu),
+ (Self::Binary, LargeUtf8) => LargeBinary,
+ (Self::Binary, Utf8View) => BinaryView,
+ (Self::Binary, data_type) if can_cast_types(data_type, &BinaryView) => {
+ BinaryView
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &LargeBinary) => {
+ LargeBinary
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &Binary) => Binary,
+ (Self::FixedSizeBinary(size), _) => FixedSizeBinary(*size),
+ (Self::String, LargeBinary) => LargeUtf8,
+ (Self::String, BinaryView) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8View) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &LargeUtf8) => {
+ LargeUtf8
+ }
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8) => Utf8,
+ (Self::List(to_field), List(from_field) | FixedSizeList(from_field, _)) => {
+ List(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeList(from_field)) => {
+ LargeList(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), ListView(from_field)) => {
+ ListView(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeListView(from_field)) => {
+ LargeListView(default_field_cast(to_field, from_field)?)
+ }
+ // List array where each element is a len 1 list of the origin type
+ (Self::List(field), _) => List(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ ))),
+ (
+ Self::FixedSizeList(to_field, to_size),
+ FixedSizeList(from_field, from_size),
+ ) if from_size == to_size => {
+ FixedSizeList(default_field_cast(to_field, from_field)?, *to_size)
+ }
+ (
+ Self::FixedSizeList(to_field, size),
+ List(from_field)
+ | LargeList(from_field)
+ | ListView(from_field)
+ | LargeListView(from_field),
+ ) => FixedSizeList(default_field_cast(to_field, from_field)?, *size),
Review Comment:
oh. I found I misunderstood the direction. It's a case from `List` to `FixedSizeList`. Indeed, this behavior is safe. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2435305044
I am traveling currently but will have some bandwidth to take a look early next week if noone beats me to it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1795614543
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,41 @@
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
Review Comment:
Documentation missing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1798843685
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
Review Comment:
Given say arrow Int64 data, i want to know whether these is numbers, timestamp, time, date or something else (eg user-defined enum). The fact that any of these hypothetical logical types could be stored as Int64 doesn't help me know. Asking logical type "could you please decode this arrow type?" doesn't help me know.
Thus, going from arrow type to logical type is not an option. We simply need to know what logical type this should be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1825316074
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,399 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{
+ LogicalField, LogicalFieldRef, LogicalFields, LogicalType, LogicalUnionFields,
+ TypeSignature,
+};
+use crate::error::{Result, _internal_err};
+use arrow::compute::can_cast_types;
+use arrow_schema::{
+ DataType, Field, FieldRef, Fields, IntervalUnit, TimeUnit, UnionFields,
+};
+use std::sync::Arc;
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean type representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ Time(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ String,
+ /// A list of some logical data type with variable length.
+ List(LogicalFieldRef),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(LogicalFieldRef, i32),
+ /// A nested type that contains a number of sub-fields.
+ Struct(LogicalFields),
+ /// A nested type that can represent slots of differing types.
+ Union(LogicalUnionFields),
+ /// Decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal(u8, i8),
+ /// A Map is a type that an association between a key and a value.
+ ///
+ /// The key and value types are not constrained, but keys should be
+ /// hashable and unique.
+ ///
+ /// In a field with Map type, key type and the second the value type. The names of the
+ /// child fields may be respectively "entries", "key", and "value", but this is
+ /// not enforced.
+ Map(LogicalFieldRef),
+}
+
+impl LogicalType for NativeType {
+ fn native(&self) -> &NativeType {
+ self
+ }
+
+ fn signature(&self) -> TypeSignature<'_> {
+ TypeSignature::Native(self)
+ }
+
+ fn default_cast_for(&self, origin: &DataType) -> Result<DataType> {
+ use DataType::*;
+
+ fn default_field_cast(to: &LogicalField, from: &Field) -> Result<FieldRef> {
+ Ok(Arc::new(Field::new(
+ to.name.clone(),
+ to.logical_type.default_cast_for(from.data_type())?,
+ to.nullable,
+ )))
+ }
+
+ Ok(match (self, origin) {
+ (Self::Null, _) => Null,
+ (Self::Boolean, _) => Boolean,
+ (Self::Int8, _) => Int8,
+ (Self::Int16, _) => Int16,
+ (Self::Int32, _) => Int32,
+ (Self::Int64, _) => Int64,
+ (Self::UInt8, _) => UInt8,
+ (Self::UInt16, _) => UInt16,
+ (Self::UInt32, _) => UInt32,
+ (Self::UInt64, _) => UInt64,
+ (Self::Float16, _) => Float16,
+ (Self::Float32, _) => Float32,
+ (Self::Float64, _) => Float64,
+ (Self::Decimal(p, s), _) if p <= &38 => Decimal128(*p, *s),
+ (Self::Decimal(p, s), _) => Decimal256(*p, *s),
+ (Self::Timestamp(tu, tz), _) => Timestamp(*tu, tz.clone()),
+ (Self::Date, _) => Date32,
+ (Self::Time(tu), _) => match tu {
+ TimeUnit::Second | TimeUnit::Millisecond => Time32(*tu),
+ TimeUnit::Microsecond | TimeUnit::Nanosecond => Time64(*tu),
+ },
+ (Self::Duration(tu), _) => Duration(*tu),
+ (Self::Interval(iu), _) => Interval(*iu),
+ (Self::Binary, LargeUtf8) => LargeBinary,
+ (Self::Binary, Utf8View) => BinaryView,
+ (Self::Binary, data_type) if can_cast_types(data_type, &BinaryView) => {
+ BinaryView
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &LargeBinary) => {
+ LargeBinary
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &Binary) => Binary,
+ (Self::FixedSizeBinary(size), _) => FixedSizeBinary(*size),
+ (Self::String, LargeBinary) => LargeUtf8,
+ (Self::String, BinaryView) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8View) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &LargeUtf8) => {
+ LargeUtf8
+ }
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8) => Utf8,
+ (Self::List(to_field), List(from_field) | FixedSizeList(from_field, _)) => {
+ List(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeList(from_field)) => {
+ LargeList(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), ListView(from_field)) => {
+ ListView(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeListView(from_field)) => {
+ LargeListView(default_field_cast(to_field, from_field)?)
+ }
+ // List array where each element is a len 1 list of the origin type
+ (Self::List(field), _) => List(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ ))),
+ (
+ Self::FixedSizeList(to_field, to_size),
+ FixedSizeList(from_field, from_size),
+ ) if from_size == to_size => {
+ FixedSizeList(default_field_cast(to_field, from_field)?, *to_size)
+ }
+ (
+ Self::FixedSizeList(to_field, size),
+ List(from_field)
+ | LargeList(from_field)
+ | ListView(from_field)
+ | LargeListView(from_field),
+ ) => FixedSizeList(default_field_cast(to_field, from_field)?, *size),
+ // FixedSizeList array where each element is a len 1 list of the origin type
+ (Self::FixedSizeList(field, size), _) => FixedSizeList(
+ Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ )),
+ *size,
+ ),
+ // From https://github.com/apache/arrow-rs/blob/56525efbd5f37b89d1b56aa51709cab9f81bc89e/arrow-cast/src/cast/mod.rs#L189-L196
+ (Self::Struct(to_fields), Struct(from_fields))
+ if from_fields.len() == to_fields.len() =>
+ {
+ Struct(
+ from_fields
+ .iter()
+ .zip(to_fields.iter())
+ .map(|(from, to)| default_field_cast(to, from))
+ .collect::<Result<Fields>>()?,
+ )
+ }
+ (Self::Struct(to_fields), Null) => Struct(
+ to_fields
+ .iter()
+ .map(|field| {
+ Ok(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(&Null)?,
+ field.nullable,
+ )))
+ })
+ .collect::<Result<Fields>>()?,
+ ),
+ (Self::Map(to_field), Map(from_field, sorted)) => {
+ Map(default_field_cast(to_field, from_field)?, *sorted)
+ }
+ (Self::Map(field), Null) => Map(
+ Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(&Null)?,
+ field.nullable,
+ )),
+ false,
+ ),
+ (Self::Union(to_fields), Union(from_fields, mode))
+ if from_fields.len() == to_fields.len() =>
+ {
+ Union(
+ from_fields
+ .iter()
+ .zip(to_fields.iter())
+ .map(|((_, from), (i, to))| {
+ Ok((*i, default_field_cast(to, from)?))
+ })
+ .collect::<Result<UnionFields>>()?,
+ *mode,
+ )
+ }
+ _ => {
+ return _internal_err!(
+ "Unavailable default cast for native type {:?} from physical type {:?}",
+ self,
+ origin
+ )
+ }
+ })
+ }
+}
+
+// The following From<DataType>, From<Field>, ... implementations are temporary
+// mapping solutions to provide backwards compatibility while transitioning from
+// the purely physical system to a logical / physical system.
+
+impl From<DataType> for NativeType {
Review Comment:
Think of it as DataFusion’s type, or its built-in LogicalType. The term NativeType aligns with the concept of native types in Rust, so I believe it’s the preferred name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2437235307
> Not sure if there is any added benefit of letting the logical type choose it's sequence of casts instead of providing just one fallback type but maybe I'm missing something.
I have no string opinion on this too, but since the mapping is strongly related to LogicalType itself, for the point of customization, I think it makes sense to let the user define the types and the preference they want. Although this maybe a premature optimization that no one really use it.
> Currently in DataFusion if a signature accepts DataType::Utf8, LargeUtf8 and Utf8View and we receive DataType::Int64 what do we do? And what if the function accepts DataType::Utf8, LargeUtf8 but not Utf8View?
The order is Utf8View -> LargeUtf8 -> Utf8. I think most of the function accepts those 3 together, some doesn't support Utf8View yet, but it is just not yet implemeneted, in theory it should support 3 of them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1816904255
##########
datafusion/common/src/types/native.rs:
##########
@@ -188,6 +190,147 @@ impl LogicalType for NativeType {
fn signature(&self) -> TypeSignature<'_> {
TypeSignature::Native(self)
}
+
+ fn default_cast_for(&self, origin: &DataType) -> Result<DataType> {
+ use DataType::*;
+
+ fn default_field_cast(to: &LogicalField, from: &Field) -> Result<FieldRef> {
+ Ok(Arc::new(Field::new(
+ to.name.clone(),
+ to.logical_type.default_cast_for(from.data_type())?,
+ to.nullable,
+ )))
+ }
+
+ Ok(match (self, origin) {
+ (Self::Null, _) => Null,
+ (Self::Boolean, _) => Boolean,
+ (Self::Int8, _) => Int8,
+ (Self::Int16, _) => Int16,
+ (Self::Int32, _) => Int32,
+ (Self::Int64, _) => Int64,
+ (Self::UInt8, _) => UInt8,
+ (Self::UInt16, _) => UInt16,
+ (Self::UInt32, _) => UInt32,
+ (Self::UInt64, _) => UInt64,
+ (Self::Float16, _) => Float16,
+ (Self::Float32, _) => Float32,
+ (Self::Float64, _) => Float64,
+ (Self::Decimal(p, s), _) if p <= &38 => Decimal128(*p, *s),
+ (Self::Decimal(p, s), _) => Decimal256(*p, *s),
+ (Self::Timestamp(tu, tz), _) => Timestamp(*tu, tz.clone()),
+ (Self::Date, _) => Date32,
+ (Self::Time(tu), _) => match tu {
+ TimeUnit::Second | TimeUnit::Millisecond => Time32(*tu),
+ TimeUnit::Microsecond | TimeUnit::Nanosecond => Time64(*tu),
+ },
+ (Self::Duration(tu), _) => Duration(*tu),
+ (Self::Interval(iu), _) => Interval(*iu),
+ (Self::Binary, LargeUtf8) => LargeBinary,
+ (Self::Binary, Utf8View) => BinaryView,
+ (Self::Binary, _) => Binary,
+ (Self::FixedSizeBinary(size), _) => FixedSizeBinary(*size),
+ (Self::Utf8, LargeBinary) => LargeUtf8,
+ (Self::Utf8, BinaryView) => Utf8View,
+ (Self::Utf8, _) => Utf8,
Review Comment:
Some initial thoughts
First, I think utf8view is the default type for string.
In my proposed way,
I think the mapping should be based on `arrow::can_cast_to` so if the 1st target type is not supported we go to the next one.
But in this design, each type has a single target type to cast to. Is the default type always castable by arrow's `can_cast_to`? Maybe we can just add the check inside. Like
```rust
(Self::Utf8, data_type) if can_cast_type(data_type, Utf8View)=> Utf8View,
(Self::Utf8, data_type) if can_cast_type(data_type, LargeUtf8)=> LargeUtf8,
(Self::Utf8, data_type) if can_cast_type(data_type, Utf8)=> Utf8,
```
Another issue is whether the default type is great for every use case, for example, I think utf8view should be the default type but is there any usecase that require utf8 instead? If yes, how can they modify it? It seems they need to copy the whole `default_cast_for` and change it for their logical type. (We can assume no such case for now, but just to point out that it may not work if there is such a case)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2439547722
> I have no further concern so far, we can always add more function or modify it when we meet the use case.
Thank you for reviewing and helping this project along @jayzhan211 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1800916136
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,124 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeSignature<'a> {
Review Comment:
TODO: Implement a __better__ display
`Extension { name: "banana", parameters: [Number(1)] }` -> `banana(1)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2422215220
@alamb @andygrove @comphead please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1817701287
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,391 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{
+ LogicalField, LogicalFieldRef, LogicalFields, LogicalType, LogicalUnionFields,
+ TypeSignature,
+};
+use crate::error::{Result, _internal_err};
+use arrow::compute::can_cast_types;
+use arrow_schema::{
+ DataType, Field, FieldRef, Fields, IntervalUnit, TimeUnit, UnionFields,
+};
+use std::sync::Arc;
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean type representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ Time(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ Utf8,
Review Comment:
Is `String` a better name, similar to rust's string type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1817702282
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,391 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{
+ LogicalField, LogicalFieldRef, LogicalFields, LogicalType, LogicalUnionFields,
+ TypeSignature,
+};
+use crate::error::{Result, _internal_err};
+use arrow::compute::can_cast_types;
+use arrow_schema::{
+ DataType, Field, FieldRef, Fields, IntervalUnit, TimeUnit, UnionFields,
+};
+use std::sync::Arc;
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean type representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ Time(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ Utf8,
+ /// A list of some logical data type with variable length.
+ List(LogicalFieldRef),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(LogicalFieldRef, i32),
+ /// A nested type that contains a number of sub-fields.
+ Struct(LogicalFields),
+ /// A nested type that can represent slots of differing types.
+ Union(LogicalUnionFields),
+ /// Decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal(u8, i8),
+ /// A Map is a type that an association between a key and a value.
+ ///
+ /// The key and value types are not constrained, but keys should be
+ /// hashable and unique.
+ ///
+ /// In a field with Map type, key type and the second the value type. The names of the
+ /// child fields may be respectively "entries", "key", and "value", but this is
+ /// not enforced.
+ Map(LogicalFieldRef),
+}
+
+impl LogicalType for NativeType {
+ fn native(&self) -> &NativeType {
+ self
+ }
+
+ fn signature(&self) -> TypeSignature<'_> {
+ TypeSignature::Native(self)
+ }
+
+ fn default_cast_for(&self, origin: &DataType) -> Result<DataType> {
+ use DataType::*;
+
+ fn default_field_cast(to: &LogicalField, from: &Field) -> Result<FieldRef> {
+ Ok(Arc::new(Field::new(
+ to.name.clone(),
+ to.logical_type.default_cast_for(from.data_type())?,
+ to.nullable,
+ )))
+ }
+
+ Ok(match (self, origin) {
+ (Self::Null, _) => Null,
+ (Self::Boolean, _) => Boolean,
+ (Self::Int8, _) => Int8,
+ (Self::Int16, _) => Int16,
+ (Self::Int32, _) => Int32,
+ (Self::Int64, _) => Int64,
+ (Self::UInt8, _) => UInt8,
+ (Self::UInt16, _) => UInt16,
+ (Self::UInt32, _) => UInt32,
+ (Self::UInt64, _) => UInt64,
+ (Self::Float16, _) => Float16,
+ (Self::Float32, _) => Float32,
+ (Self::Float64, _) => Float64,
+ (Self::Decimal(p, s), _) if p <= &38 => Decimal128(*p, *s),
+ (Self::Decimal(p, s), _) => Decimal256(*p, *s),
+ (Self::Timestamp(tu, tz), _) => Timestamp(*tu, tz.clone()),
+ (Self::Date, _) => Date32,
+ (Self::Time(tu), _) => match tu {
+ TimeUnit::Second | TimeUnit::Millisecond => Time32(*tu),
+ TimeUnit::Microsecond | TimeUnit::Nanosecond => Time64(*tu),
+ },
+ (Self::Duration(tu), _) => Duration(*tu),
+ (Self::Interval(iu), _) => Interval(*iu),
+ (Self::Binary, LargeUtf8) => LargeBinary,
+ (Self::Binary, Utf8View) => BinaryView,
+ (Self::Binary, data_type) if can_cast_types(data_type, &BinaryView) => {
+ BinaryView
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &LargeBinary) => {
+ LargeBinary
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &Binary) => Binary,
+ (Self::FixedSizeBinary(size), _) => FixedSizeBinary(*size),
+ (Self::Utf8, LargeBinary) => LargeUtf8,
+ (Self::Utf8, BinaryView) => Utf8View,
+ (Self::Utf8, data_type) if can_cast_types(data_type, &Utf8View) => Utf8View,
+ (Self::Utf8, data_type) if can_cast_types(data_type, &LargeUtf8) => LargeUtf8,
+ (Self::Utf8, data_type) if can_cast_types(data_type, &Utf8) => Utf8,
+ (Self::List(to_field), List(from_field) | FixedSizeList(from_field, _)) => {
+ List(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeList(from_field)) => {
+ LargeList(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), ListView(from_field)) => {
+ ListView(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeListView(from_field)) => {
+ LargeListView(default_field_cast(to_field, from_field)?)
+ }
+ // List array where each element is a len 1 list of the origin type
+ (Self::List(field), _) => List(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ ))),
+ (
+ Self::FixedSizeList(to_field, to_size),
+ FixedSizeList(from_field, from_size),
+ ) if from_size == to_size => {
+ FixedSizeList(default_field_cast(to_field, from_field)?, *to_size)
+ }
+ (
+ Self::FixedSizeList(to_field, size),
+ List(from_field)
+ | LargeList(from_field)
+ | ListView(from_field)
+ | LargeListView(from_field),
+ ) => FixedSizeList(default_field_cast(to_field, from_field)?, *size),
+ // FixedSizeList array where each element is a len 1 list of the origin type
+ (Self::FixedSizeList(field, size), _) => FixedSizeList(
+ Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ )),
+ *size,
+ ),
+ // From https://github.com/apache/arrow-rs/blob/56525efbd5f37b89d1b56aa51709cab9f81bc89e/arrow-cast/src/cast/mod.rs#L189-L196
+ (Self::Struct(to_fields), Struct(from_fields))
+ if from_fields.len() == to_fields.len() =>
+ {
+ Struct(
+ from_fields
+ .iter()
+ .zip(to_fields.iter())
+ .map(|(from, to)| default_field_cast(to, from))
+ .collect::<Result<Fields>>()?,
+ )
+ }
+ (Self::Struct(to_fields), Null) => Struct(
+ to_fields
+ .iter()
+ .map(|field| {
+ Ok(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(&Null)?,
+ field.nullable,
+ )))
+ })
+ .collect::<Result<Fields>>()?,
+ ),
+ (Self::Map(to_field), Map(from_field, sorted)) => {
+ Map(default_field_cast(to_field, from_field)?, *sorted)
+ }
+ (Self::Map(field), Null) => Map(
+ Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(&Null)?,
+ field.nullable,
+ )),
+ false,
+ ),
+ (Self::Union(to_fields), Union(from_fields, mode))
+ if from_fields.len() == to_fields.len() =>
+ {
+ Union(
+ from_fields
+ .iter()
+ .zip(to_fields.iter())
+ .map(|((_, from), (i, to))| {
+ Ok((*i, default_field_cast(to, from)?))
+ })
+ .collect::<Result<UnionFields>>()?,
+ *mode,
+ )
+ }
+ _ => {
+ return _internal_err!(
+ "Unavailable default cast for native type {:?} from physical type {:?}",
+ self,
+ origin
+ )
+ }
+ })
+ }
+}
+
+// The following From<DataType>, From<Field>, ... implementations are temporary
+// mapping solutions to provide backwards compatibility while transitioning from
+// the purely physical system to a logical / physical system.
+
+impl From<DataType> for NativeType {
Review Comment:
We can add this for convenience, since &DataType is also used wildly
```rust
impl From<&DataType> for NativeType {
fn from(value: &DataType) -> Self {
value.clone().into()
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2437951678
@jayzhan211 I have a working prototype of the ideas we discussed above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 merged PR #12853:
URL: https://github.com/apache/datafusion/pull/12853
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2439648157
Thanks a lot for the review @jayzhan211 . I've added all the suggestion mentioned :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2435211194
> Has the [logical type branch](https://github.com/apache/datafusion/tree/logical-types?rgh-link-date=2024-10-24T12%3A43%3A34Z) reached its final state, and are we starting the migration?
Not really. But it seems like this part of the change is already needed to unblock [other](https://github.com/apache/datafusion/issues/12622#issuecomment-2398560315) [tasks](https://github.com/apache/datafusion/issues/12622#issuecomment-2401873724).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1810651532
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// Signature that uniquely identifies a type among other types.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeSignature<'a> {
+ /// Represents a built-in native type.
+ Native(&'a NativeType),
+ /// Represents an arrow-compatible extension type.
+ /// (<https://arrow.apache.org/docs/format/Columnar.html#extension-types>)
+ ///
+ /// The `name` should contain the same value as 'ARROW:extension:name'.
+ Extension {
+ name: &'a str,
+ parameters: &'a [TypeParameter<'a>],
+ },
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeParameter<'a> {
+ Type(TypeSignature<'a>),
+ Number(i128),
+}
Review Comment:
Ideally everything could be encoded in the name using `ARROW:extension:name` but we might leave the option handle parameters in a more structured manner to allow better matching when executing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1796251479
##########
datafusion/common/src/types/builtin.rs:
##########
@@ -0,0 +1,39 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{LogicalType, NativeType};
+
+#[derive(Debug)]
+pub struct BuiltinType {
+ native: NativeType,
+}
+
+impl LogicalType for BuiltinType {
+ fn native(&self) -> &NativeType {
+ &self.native
+ }
+
+ fn name(&self) -> Option<&str> {
+ None
Review Comment:
Do we need name for native type as well
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
Review Comment:
I previously propose `can_decode_to(DataType) -> bool`, so given logical type and DataType, we can know whether they are paired.
How can we do the equivalent check by the current design?
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,237 @@
+use std::sync::Arc;
+
+use arrow_schema::{DataType, IntervalUnit, TimeUnit};
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean datatype representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed 32-bit date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed 32-bit time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ /// Must be either seconds or milliseconds.
+ Time32(TimeUnit),
+ /// A signed 64-bit time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ /// Must be either microseconds or nanoseconds.
+ Time64(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ Utf8,
+ /// A list of some logical data type with variable length.
+ List(Box<NativeType>),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(Box<NativeType>, i32),
+ /// A nested datatype that contains a number of sub-fields.
+ Struct(Box<[(String, NativeType)]>),
Review Comment:
I think we need name, datatype and nullable for them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1807176173
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
Review Comment:
`impl From<DataType> for NativeType` is enough for native type since we can know whether the ArrayRef matches the LogicalType we have. But for LogicalType::UserDefined, I think we need to define what kind of `DataType` it could be decoded to.
We can figure this out if we meet any practical usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1825938851
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,399 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{
+ LogicalField, LogicalFieldRef, LogicalFields, LogicalType, LogicalUnionFields,
+ TypeSignature,
+};
+use crate::error::{Result, _internal_err};
+use arrow::compute::can_cast_types;
+use arrow_schema::{
+ DataType, Field, FieldRef, Fields, IntervalUnit, TimeUnit, UnionFields,
+};
+use std::sync::Arc;
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean type representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ Time(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ String,
+ /// A list of some logical data type with variable length.
+ List(LogicalFieldRef),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(LogicalFieldRef, i32),
+ /// A nested type that contains a number of sub-fields.
+ Struct(LogicalFields),
+ /// A nested type that can represent slots of differing types.
+ Union(LogicalUnionFields),
+ /// Decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal(u8, i8),
+ /// A Map is a type that an association between a key and a value.
+ ///
+ /// The key and value types are not constrained, but keys should be
+ /// hashable and unique.
+ ///
+ /// In a field with Map type, key type and the second the value type. The names of the
+ /// child fields may be respectively "entries", "key", and "value", but this is
+ /// not enforced.
+ Map(LogicalFieldRef),
+}
+
+impl LogicalType for NativeType {
+ fn native(&self) -> &NativeType {
+ self
+ }
+
+ fn signature(&self) -> TypeSignature<'_> {
+ TypeSignature::Native(self)
+ }
+
+ fn default_cast_for(&self, origin: &DataType) -> Result<DataType> {
+ use DataType::*;
+
+ fn default_field_cast(to: &LogicalField, from: &Field) -> Result<FieldRef> {
+ Ok(Arc::new(Field::new(
+ to.name.clone(),
+ to.logical_type.default_cast_for(from.data_type())?,
+ to.nullable,
+ )))
+ }
+
+ Ok(match (self, origin) {
+ (Self::Null, _) => Null,
+ (Self::Boolean, _) => Boolean,
+ (Self::Int8, _) => Int8,
+ (Self::Int16, _) => Int16,
+ (Self::Int32, _) => Int32,
+ (Self::Int64, _) => Int64,
+ (Self::UInt8, _) => UInt8,
+ (Self::UInt16, _) => UInt16,
+ (Self::UInt32, _) => UInt32,
+ (Self::UInt64, _) => UInt64,
+ (Self::Float16, _) => Float16,
+ (Self::Float32, _) => Float32,
+ (Self::Float64, _) => Float64,
+ (Self::Decimal(p, s), _) if p <= &38 => Decimal128(*p, *s),
+ (Self::Decimal(p, s), _) => Decimal256(*p, *s),
+ (Self::Timestamp(tu, tz), _) => Timestamp(*tu, tz.clone()),
+ (Self::Date, _) => Date32,
+ (Self::Time(tu), _) => match tu {
+ TimeUnit::Second | TimeUnit::Millisecond => Time32(*tu),
+ TimeUnit::Microsecond | TimeUnit::Nanosecond => Time64(*tu),
+ },
+ (Self::Duration(tu), _) => Duration(*tu),
+ (Self::Interval(iu), _) => Interval(*iu),
+ (Self::Binary, LargeUtf8) => LargeBinary,
+ (Self::Binary, Utf8View) => BinaryView,
+ (Self::Binary, data_type) if can_cast_types(data_type, &BinaryView) => {
+ BinaryView
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &LargeBinary) => {
+ LargeBinary
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &Binary) => Binary,
+ (Self::FixedSizeBinary(size), _) => FixedSizeBinary(*size),
+ (Self::String, LargeBinary) => LargeUtf8,
+ (Self::String, BinaryView) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8View) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &LargeUtf8) => {
+ LargeUtf8
+ }
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8) => Utf8,
+ (Self::List(to_field), List(from_field) | FixedSizeList(from_field, _)) => {
+ List(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeList(from_field)) => {
+ LargeList(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), ListView(from_field)) => {
+ ListView(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeListView(from_field)) => {
+ LargeListView(default_field_cast(to_field, from_field)?)
+ }
+ // List array where each element is a len 1 list of the origin type
+ (Self::List(field), _) => List(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ ))),
+ (
+ Self::FixedSizeList(to_field, to_size),
+ FixedSizeList(from_field, from_size),
+ ) if from_size == to_size => {
+ FixedSizeList(default_field_cast(to_field, from_field)?, *to_size)
+ }
+ (
+ Self::FixedSizeList(to_field, size),
+ List(from_field)
+ | LargeList(from_field)
+ | ListView(from_field)
+ | LargeListView(from_field),
+ ) => FixedSizeList(default_field_cast(to_field, from_field)?, *size),
+ // FixedSizeList array where each element is a len 1 list of the origin type
+ (Self::FixedSizeList(field, size), _) => FixedSizeList(
+ Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ )),
+ *size,
+ ),
+ // From https://github.com/apache/arrow-rs/blob/56525efbd5f37b89d1b56aa51709cab9f81bc89e/arrow-cast/src/cast/mod.rs#L189-L196
+ (Self::Struct(to_fields), Struct(from_fields))
+ if from_fields.len() == to_fields.len() =>
+ {
+ Struct(
+ from_fields
+ .iter()
+ .zip(to_fields.iter())
+ .map(|(from, to)| default_field_cast(to, from))
+ .collect::<Result<Fields>>()?,
+ )
+ }
+ (Self::Struct(to_fields), Null) => Struct(
+ to_fields
+ .iter()
+ .map(|field| {
+ Ok(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(&Null)?,
+ field.nullable,
+ )))
+ })
+ .collect::<Result<Fields>>()?,
+ ),
+ (Self::Map(to_field), Map(from_field, sorted)) => {
+ Map(default_field_cast(to_field, from_field)?, *sorted)
+ }
+ (Self::Map(field), Null) => Map(
+ Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(&Null)?,
+ field.nullable,
+ )),
+ false,
+ ),
+ (Self::Union(to_fields), Union(from_fields, mode))
+ if from_fields.len() == to_fields.len() =>
+ {
+ Union(
+ from_fields
+ .iter()
+ .zip(to_fields.iter())
+ .map(|((_, from), (i, to))| {
+ Ok((*i, default_field_cast(to, from)?))
+ })
+ .collect::<Result<UnionFields>>()?,
+ *mode,
+ )
+ }
+ _ => {
+ return _internal_err!(
+ "Unavailable default cast for native type {:?} from physical type {:?}",
+ self,
+ origin
+ )
+ }
+ })
+ }
+}
+
+// The following From<DataType>, From<Field>, ... implementations are temporary
+// mapping solutions to provide backwards compatibility while transitioning from
+// the purely physical system to a logical / physical system.
+
+impl From<DataType> for NativeType {
Review Comment:
If I read the structure doc
```
/// Representation of a type that DataFusion can handle natively. It is a subset
/// of the physical variants in Arrow's native [`DataType`].
```
its is not a Rust native types its more related to Arrow Types which eventually come to rust types. I feel we need to clarify this a little bit, but I cannot come up with the better phrase right now, lets leave it for follow up PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1800893742
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,124 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeSignature<'a> {
+ /// Represents a built-in native type.
+ Native(&'a NativeType),
+ /// Represents an arrow-compatible extension type.
+ /// (https://arrow.apache.org/docs/format/Columnar.html#extension-types)
+ ///
+ /// The `name` should contain the same value as 'ARROW:extension:name'.
+ Extension {
+ name: &'a str,
+ parameters: &'a [TypeParameter<'a>],
+ },
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeParameter<'a> {
+ Type(TypeSignature<'a>),
+ Number(i128),
+}
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: Sync + Send {
+ fn native(&self) -> &NativeType;
+ fn signature(&self) -> TypeSignature<'_>;
+}
+
+impl fmt::Debug for dyn LogicalType {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_tuple("LogicalType")
+ .field(&self.signature())
+ .field(&self.native())
+ .finish()
+ }
+}
+
+impl PartialEq for dyn LogicalType {
+ fn eq(&self, other: &Self) -> bool {
+ self.native().eq(other.native()) && self.signature().eq(&other.signature())
+ }
+}
+
+impl Eq for dyn LogicalType {}
+
+impl PartialOrd for dyn LogicalType {
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+ Some(self.cmp(other))
+ }
+}
+
+impl Ord for dyn LogicalType {
+ fn cmp(&self, other: &Self) -> Ordering {
+ self.signature()
+ .cmp(&other.signature())
+ .then(self.native().cmp(other.native()))
+ }
+}
+
+impl Hash for dyn LogicalType {
+ fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+ self.signature().hash(state);
+ self.native().hash(state);
+ }
+}
+
+#[cfg(test)]
+mod test {
+ #![allow(dead_code)]
+
+ use super::{LogicalType, TypeParameter, TypeSignature};
+ use crate::types::NativeType;
+
+ struct MagicalType {}
+
+ impl LogicalType for MagicalType {
+ fn native(&self) -> &NativeType {
+ &NativeType::Utf8
+ }
+
+ fn signature(&self) -> TypeSignature<'_> {
+ TypeSignature::Extension {
+ name: "MagicalType",
+ parameters: &[TypeParameter::Type(TypeSignature::Native(
+ &NativeType::Boolean,
+ ))],
+ }
+ }
+ }
+
+ fn test(logical_type: &dyn LogicalType) {
+ match logical_type.signature() {
+ TypeSignature::Extension {
+ name: "MagicalType",
+ parameters:
+ [TypeParameter::Type(TypeSignature::Native(NativeType::Boolean))],
+ } => {}
+ TypeSignature::Native(NativeType::Binary) => todo!(),
+ _ => unimplemented!(),
+ };
+ }
Review Comment:
I've added a more structured `TypeSignature`. I've encountered some problems while iterating for a design of this structure since I wanted to balance ease of handling and information contained. I think I've ended up with a good baseline that can be easily pattern matched and limits the copying of information.
Above you can see an example of this signature in action.
ptal @findepi @jayzhan211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2422852819
I dont remember the roots, so wondering, can we investigate and use a single type system which is Arrow Types and get rid of other types. At the end of the day it is based on Arrow Types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2443962047
Shall we merge this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1825316524
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::NativeType;
+use crate::error::Result;
+use arrow_schema::DataType;
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+/// Signature that uniquely identifies a type among other types.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeSignature<'a> {
+ /// Represents a built-in native type.
+ Native(&'a NativeType),
+ /// Represents an arrow-compatible extension type.
+ /// (<https://arrow.apache.org/docs/format/Columnar.html#extension-types>)
+ ///
+ /// The `name` should contain the same value as 'ARROW:extension:name'.
+ Extension {
Review Comment:
It is because arrow has extension type too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1824830156
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,399 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{
+ LogicalField, LogicalFieldRef, LogicalFields, LogicalType, LogicalUnionFields,
+ TypeSignature,
+};
+use crate::error::{Result, _internal_err};
+use arrow::compute::can_cast_types;
+use arrow_schema::{
+ DataType, Field, FieldRef, Fields, IntervalUnit, TimeUnit, UnionFields,
+};
+use std::sync::Arc;
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean type representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ Time(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ String,
+ /// A list of some logical data type with variable length.
+ List(LogicalFieldRef),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(LogicalFieldRef, i32),
+ /// A nested type that contains a number of sub-fields.
+ Struct(LogicalFields),
+ /// A nested type that can represent slots of differing types.
+ Union(LogicalUnionFields),
+ /// Decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal(u8, i8),
+ /// A Map is a type that an association between a key and a value.
+ ///
+ /// The key and value types are not constrained, but keys should be
+ /// hashable and unique.
+ ///
+ /// In a field with Map type, key type and the second the value type. The names of the
+ /// child fields may be respectively "entries", "key", and "value", but this is
+ /// not enforced.
+ Map(LogicalFieldRef),
+}
+
+impl LogicalType for NativeType {
+ fn native(&self) -> &NativeType {
+ self
+ }
+
+ fn signature(&self) -> TypeSignature<'_> {
+ TypeSignature::Native(self)
+ }
+
+ fn default_cast_for(&self, origin: &DataType) -> Result<DataType> {
+ use DataType::*;
+
+ fn default_field_cast(to: &LogicalField, from: &Field) -> Result<FieldRef> {
+ Ok(Arc::new(Field::new(
+ to.name.clone(),
+ to.logical_type.default_cast_for(from.data_type())?,
+ to.nullable,
+ )))
+ }
+
+ Ok(match (self, origin) {
+ (Self::Null, _) => Null,
+ (Self::Boolean, _) => Boolean,
+ (Self::Int8, _) => Int8,
+ (Self::Int16, _) => Int16,
+ (Self::Int32, _) => Int32,
+ (Self::Int64, _) => Int64,
+ (Self::UInt8, _) => UInt8,
+ (Self::UInt16, _) => UInt16,
+ (Self::UInt32, _) => UInt32,
+ (Self::UInt64, _) => UInt64,
+ (Self::Float16, _) => Float16,
+ (Self::Float32, _) => Float32,
+ (Self::Float64, _) => Float64,
+ (Self::Decimal(p, s), _) if p <= &38 => Decimal128(*p, *s),
+ (Self::Decimal(p, s), _) => Decimal256(*p, *s),
+ (Self::Timestamp(tu, tz), _) => Timestamp(*tu, tz.clone()),
+ (Self::Date, _) => Date32,
+ (Self::Time(tu), _) => match tu {
+ TimeUnit::Second | TimeUnit::Millisecond => Time32(*tu),
+ TimeUnit::Microsecond | TimeUnit::Nanosecond => Time64(*tu),
+ },
+ (Self::Duration(tu), _) => Duration(*tu),
+ (Self::Interval(iu), _) => Interval(*iu),
+ (Self::Binary, LargeUtf8) => LargeBinary,
+ (Self::Binary, Utf8View) => BinaryView,
+ (Self::Binary, data_type) if can_cast_types(data_type, &BinaryView) => {
+ BinaryView
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &LargeBinary) => {
+ LargeBinary
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &Binary) => Binary,
+ (Self::FixedSizeBinary(size), _) => FixedSizeBinary(*size),
+ (Self::String, LargeBinary) => LargeUtf8,
+ (Self::String, BinaryView) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8View) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &LargeUtf8) => {
+ LargeUtf8
+ }
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8) => Utf8,
+ (Self::List(to_field), List(from_field) | FixedSizeList(from_field, _)) => {
+ List(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeList(from_field)) => {
+ LargeList(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), ListView(from_field)) => {
+ ListView(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeListView(from_field)) => {
+ LargeListView(default_field_cast(to_field, from_field)?)
+ }
+ // List array where each element is a len 1 list of the origin type
+ (Self::List(field), _) => List(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ ))),
+ (
+ Self::FixedSizeList(to_field, to_size),
+ FixedSizeList(from_field, from_size),
+ ) if from_size == to_size => {
+ FixedSizeList(default_field_cast(to_field, from_field)?, *to_size)
+ }
+ (
+ Self::FixedSizeList(to_field, size),
+ List(from_field)
+ | LargeList(from_field)
+ | ListView(from_field)
+ | LargeListView(from_field),
+ ) => FixedSizeList(default_field_cast(to_field, from_field)?, *size),
+ // FixedSizeList array where each element is a len 1 list of the origin type
+ (Self::FixedSizeList(field, size), _) => FixedSizeList(
+ Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ )),
+ *size,
+ ),
+ // From https://github.com/apache/arrow-rs/blob/56525efbd5f37b89d1b56aa51709cab9f81bc89e/arrow-cast/src/cast/mod.rs#L189-L196
+ (Self::Struct(to_fields), Struct(from_fields))
+ if from_fields.len() == to_fields.len() =>
+ {
+ Struct(
+ from_fields
+ .iter()
+ .zip(to_fields.iter())
+ .map(|(from, to)| default_field_cast(to, from))
+ .collect::<Result<Fields>>()?,
+ )
+ }
+ (Self::Struct(to_fields), Null) => Struct(
+ to_fields
+ .iter()
+ .map(|field| {
+ Ok(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(&Null)?,
+ field.nullable,
+ )))
+ })
+ .collect::<Result<Fields>>()?,
+ ),
+ (Self::Map(to_field), Map(from_field, sorted)) => {
+ Map(default_field_cast(to_field, from_field)?, *sorted)
+ }
+ (Self::Map(field), Null) => Map(
+ Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(&Null)?,
+ field.nullable,
+ )),
+ false,
+ ),
+ (Self::Union(to_fields), Union(from_fields, mode))
+ if from_fields.len() == to_fields.len() =>
+ {
+ Union(
+ from_fields
+ .iter()
+ .zip(to_fields.iter())
+ .map(|((_, from), (i, to))| {
+ Ok((*i, default_field_cast(to, from)?))
+ })
+ .collect::<Result<UnionFields>>()?,
+ *mode,
+ )
+ }
+ _ => {
+ return _internal_err!(
+ "Unavailable default cast for native type {:?} from physical type {:?}",
+ self,
+ origin
+ )
+ }
+ })
+ }
+}
+
+// The following From<DataType>, From<Field>, ... implementations are temporary
+// mapping solutions to provide backwards compatibility while transitioning from
+// the purely physical system to a logical / physical system.
+
+impl From<DataType> for NativeType {
Review Comment:
NativeType name might be confusing or we need more docs? I'm reading the code and first comes to my mind Arrow Types or maybe rust native types?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1799352157
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
Review Comment:
I think the idea is that we have LogicalType already. In logical level, they are either LogicalNumber, LogicalTimestamp or LogicalDate, and we can differ them in logical level. They can also decode as i64, i32 in physical level. So asking logical type "could you please decode this arrow type?" is to tell the relationship between logical type and physical type. We don't need to know whether the arrow i64 is number or timestamp, because we already know that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1800620331
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
+ fn name(&self) -> Option<&str>;
+}
+
+impl PartialEq for dyn LogicalType {
+ fn eq(&self, other: &Self) -> bool {
+ self.native().eq(other.native()) && self.name().eq(&other.name())
+ }
+}
+
+impl Eq for dyn LogicalType {}
+
+impl PartialOrd for dyn LogicalType {
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+ Some(self.cmp(other))
+ }
+}
+
+impl Ord for dyn LogicalType {
Review Comment:
I've implemented the same capabilities as `DataType`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1810614888
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// Signature that uniquely identifies a type among other types.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeSignature<'a> {
+ /// Represents a built-in native type.
+ Native(&'a NativeType),
+ /// Represents an arrow-compatible extension type.
+ /// (<https://arrow.apache.org/docs/format/Columnar.html#extension-types>)
+ ///
+ /// The `name` should contain the same value as 'ARROW:extension:name'.
+ Extension {
+ name: &'a str,
Review Comment:
should be owned String, so that `TypeSignature` can live independently on its own
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,328 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{ops::Deref, sync::Arc};
+
+use arrow_schema::{DataType, Field, Fields, IntervalUnit, TimeUnit, UnionFields};
+
+use super::{LogicalType, TypeSignature};
+
+/// A record of a native type, its name and its nullability.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct NativeField {
+ name: String,
+ native_type: NativeType,
+ nullable: bool,
+}
+
+impl NativeField {
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ pub fn native_type(&self) -> &NativeType {
+ &self.native_type
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.nullable
+ }
+}
+
+/// A reference counted [`NativeField`].
+pub type NativeFieldRef = Arc<NativeField>;
+
+/// A cheaply cloneable, owned collection of [`NativeFieldRef`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct NativeFields(Arc<[NativeFieldRef]>);
+
+impl Deref for NativeFields {
+ type Target = [NativeFieldRef];
+
+ fn deref(&self) -> &Self::Target {
+ self.0.as_ref()
+ }
+}
+
+/// A cheaply cloneable, owned collection of [`NativeFieldRef`] and their
+/// corresponding type ids.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct NativeUnionFields(Arc<[(i8, NativeFieldRef)]>);
+
+impl Deref for NativeUnionFields {
+ type Target = [(i8, NativeFieldRef)];
+
+ fn deref(&self) -> &Self::Target {
+ self.0.as_ref()
+ }
+}
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean type representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ Time(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ Utf8,
+ /// A list of some logical data type with variable length.
+ List(NativeFieldRef),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(NativeFieldRef, i32),
+ /// A nested type that contains a number of sub-fields.
+ Struct(NativeFields),
+ /// A nested type that can represent slots of differing types.
+ Union(NativeUnionFields),
+ /// Decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal(u8, i8),
+ /// A Map is a type that an association between a key and a value.
+ ///
+ /// The key and value types are not constrained, but keys should be
+ /// hashable and unique.
+ ///
+ /// In a field with Map type, the field has a child Struct field, which then
+ /// has two children: key type and the second the value type. The names of the
+ /// child fields may be respectively "entries", "key", and "value", but this is
+ /// not enforced.
+ Map(NativeFieldRef),
Review Comment:
as in https://github.com/apache/datafusion/pull/12853#discussion_r1798849440
Map should be something like `Map(key_type, value_type)`
value_type should be any type, not necessarily native.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1809032543
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
Review Comment:
Well for any user defined logical type you still know the backing `native` type (via the `native()` method), so you should be able to use the same logic to know if your DataType can represent that logical type.
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
Review Comment:
For any user defined logical type you still know the backing `native` type (via the `native()` method), so you should be able to use the same logic to know if your DataType can represent that logical type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1817701192
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::NativeType;
+use crate::error::Result;
+use arrow_schema::DataType;
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+/// Signature that uniquely identifies a type among other types.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeSignature<'a> {
+ /// Represents a built-in native type.
+ Native(&'a NativeType),
+ /// Represents an arrow-compatible extension type.
+ /// (<https://arrow.apache.org/docs/format/Columnar.html#extension-types>)
+ ///
+ /// The `name` should contain the same value as 'ARROW:extension:name'.
+ Extension {
+ name: &'a str,
+ parameters: &'a [TypeParameter<'a>],
+ },
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeParameter<'a> {
+ Type(TypeSignature<'a>),
+ Number(i128),
+}
+
+/// A reference counted [`LogicalType`].
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
Review Comment:
I think we need function to get `LogicalTypeRef` as a singleton overall the datafusion.
Something like
```rust
// Singleton instances
// TODO: Replace with LazyLock
pub static LOGICAL_STRING: OnceLock<LogicalTypeRef> = OnceLock::new();
pub static LOGICAL_FLOAT16: OnceLock<LogicalTypeRef> = OnceLock::new();
// Usage functions
pub fn logical_string() -> LogicalTypeRef {
Arc::clone(LOGICAL_STRING.get_or_init(|| Arc::new(NativeType::Utf8)))
}
pub fn logical_float16() -> LogicalTypeRef {
Arc::clone(LOGICAL_FLOAT16.get_or_init(|| Arc::new(NativeType::Float16)))
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1818059422
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,399 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{
+ LogicalField, LogicalFieldRef, LogicalFields, LogicalType, LogicalUnionFields,
+ TypeSignature,
+};
+use crate::error::{Result, _internal_err};
+use arrow::compute::can_cast_types;
+use arrow_schema::{
+ DataType, Field, FieldRef, Fields, IntervalUnit, TimeUnit, UnionFields,
+};
+use std::sync::Arc;
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean type representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ Time(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ String,
+ /// A list of some logical data type with variable length.
+ List(LogicalFieldRef),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(LogicalFieldRef, i32),
+ /// A nested type that contains a number of sub-fields.
+ Struct(LogicalFields),
+ /// A nested type that can represent slots of differing types.
+ Union(LogicalUnionFields),
+ /// Decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal(u8, i8),
+ /// A Map is a type that an association between a key and a value.
+ ///
+ /// The key and value types are not constrained, but keys should be
+ /// hashable and unique.
+ ///
+ /// In a field with Map type, key type and the second the value type. The names of the
+ /// child fields may be respectively "entries", "key", and "value", but this is
+ /// not enforced.
+ Map(LogicalFieldRef),
+}
+
+impl LogicalType for NativeType {
+ fn native(&self) -> &NativeType {
+ self
+ }
+
+ fn signature(&self) -> TypeSignature<'_> {
+ TypeSignature::Native(self)
+ }
+
+ fn default_cast_for(&self, origin: &DataType) -> Result<DataType> {
+ use DataType::*;
+
+ fn default_field_cast(to: &LogicalField, from: &Field) -> Result<FieldRef> {
+ Ok(Arc::new(Field::new(
+ to.name.clone(),
+ to.logical_type.default_cast_for(from.data_type())?,
+ to.nullable,
+ )))
+ }
+
+ Ok(match (self, origin) {
+ (Self::Null, _) => Null,
+ (Self::Boolean, _) => Boolean,
+ (Self::Int8, _) => Int8,
+ (Self::Int16, _) => Int16,
+ (Self::Int32, _) => Int32,
+ (Self::Int64, _) => Int64,
+ (Self::UInt8, _) => UInt8,
+ (Self::UInt16, _) => UInt16,
+ (Self::UInt32, _) => UInt32,
+ (Self::UInt64, _) => UInt64,
+ (Self::Float16, _) => Float16,
+ (Self::Float32, _) => Float32,
+ (Self::Float64, _) => Float64,
+ (Self::Decimal(p, s), _) if p <= &38 => Decimal128(*p, *s),
+ (Self::Decimal(p, s), _) => Decimal256(*p, *s),
+ (Self::Timestamp(tu, tz), _) => Timestamp(*tu, tz.clone()),
+ (Self::Date, _) => Date32,
+ (Self::Time(tu), _) => match tu {
+ TimeUnit::Second | TimeUnit::Millisecond => Time32(*tu),
+ TimeUnit::Microsecond | TimeUnit::Nanosecond => Time64(*tu),
+ },
+ (Self::Duration(tu), _) => Duration(*tu),
+ (Self::Interval(iu), _) => Interval(*iu),
+ (Self::Binary, LargeUtf8) => LargeBinary,
+ (Self::Binary, Utf8View) => BinaryView,
+ (Self::Binary, data_type) if can_cast_types(data_type, &BinaryView) => {
+ BinaryView
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &LargeBinary) => {
+ LargeBinary
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &Binary) => Binary,
+ (Self::FixedSizeBinary(size), _) => FixedSizeBinary(*size),
+ (Self::String, LargeBinary) => LargeUtf8,
+ (Self::String, BinaryView) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8View) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &LargeUtf8) => {
+ LargeUtf8
+ }
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8) => Utf8,
+ (Self::List(to_field), List(from_field) | FixedSizeList(from_field, _)) => {
+ List(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeList(from_field)) => {
+ LargeList(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), ListView(from_field)) => {
+ ListView(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeListView(from_field)) => {
+ LargeListView(default_field_cast(to_field, from_field)?)
+ }
+ // List array where each element is a len 1 list of the origin type
+ (Self::List(field), _) => List(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ ))),
+ (
+ Self::FixedSizeList(to_field, to_size),
+ FixedSizeList(from_field, from_size),
+ ) if from_size == to_size => {
+ FixedSizeList(default_field_cast(to_field, from_field)?, *to_size)
+ }
+ (
+ Self::FixedSizeList(to_field, size),
+ List(from_field)
+ | LargeList(from_field)
+ | ListView(from_field)
+ | LargeListView(from_field),
+ ) => FixedSizeList(default_field_cast(to_field, from_field)?, *size),
Review Comment:
Logical fixed list to Physical list is equivalent to casting from fixed list to list in arrow::cast. If IIRC, it is safe to cast from fixed list to list, but not in the reverse direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1818058563
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,399 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{
+ LogicalField, LogicalFieldRef, LogicalFields, LogicalType, LogicalUnionFields,
+ TypeSignature,
+};
+use crate::error::{Result, _internal_err};
+use arrow::compute::can_cast_types;
+use arrow_schema::{
+ DataType, Field, FieldRef, Fields, IntervalUnit, TimeUnit, UnionFields,
+};
+use std::sync::Arc;
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
Review Comment:
I think Type is a bit general and SqlType is confused with SqlType in parser.
It makes sense to have three kinds of types in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1813064960
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,328 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{ops::Deref, sync::Arc};
+
+use arrow_schema::{DataType, Field, Fields, IntervalUnit, TimeUnit, UnionFields};
+
+use super::{LogicalType, TypeSignature};
+
+/// A record of a native type, its name and its nullability.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct NativeField {
+ name: String,
+ native_type: NativeType,
+ nullable: bool,
+}
+
+impl NativeField {
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ pub fn native_type(&self) -> &NativeType {
+ &self.native_type
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.nullable
+ }
+}
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2436897770
@notfilippo I think we need a way to know the physical types that we are able to casted to given the Logical type.
For example, if we have signature which expect Logical::String, and it accepts not only Utf8, LargeUtf8, Utf8View string family but also numeric int, we need to cast the int to string so we need to know the _physical types_ given a logical type. We don't know which DataType we should cast to if we only know there is LogicalType::String
Therefore, I think we need to add another function that returns the physical types for LogicalType. Note that the order matters if we would like the cast to Utf8View, LargeUtf8 and then Utf8 in order.
Something like
```rust
fn decode_types(&self) -> Vec<DataType> {
vec![DataType::Utf8View, DataType::LargeUtf8, DataType::Utf8]
}
```
```
postgres=# select concat('string', 123);
concat
-----------
string123
(1 row)
```
is one of the example in postgres that cast a non-string to the stirng
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1810649493
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,328 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{ops::Deref, sync::Arc};
+
+use arrow_schema::{DataType, Field, Fields, IntervalUnit, TimeUnit, UnionFields};
+
+use super::{LogicalType, TypeSignature};
+
+/// A record of a native type, its name and its nullability.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct NativeField {
+ name: String,
+ native_type: NativeType,
+ nullable: bool,
+}
+
+impl NativeField {
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ pub fn native_type(&self) -> &NativeType {
+ &self.native_type
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.nullable
+ }
+}
Review Comment:
I wanted to use the same access pattern as arrow's Field object but definitely a fair point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1825935272
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::NativeType;
+use crate::error::Result;
+use arrow_schema::DataType;
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+/// Signature that uniquely identifies a type among other types.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeSignature<'a> {
+ /// Represents a built-in native type.
+ Native(&'a NativeType),
+ /// Represents an arrow-compatible extension type.
+ /// (<https://arrow.apache.org/docs/format/Columnar.html#extension-types>)
+ ///
+ /// The `name` should contain the same value as 'ARROW:extension:name'.
+ Extension {
Review Comment:
Got there is a link to extension Arrow types above, it should be enough
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2450322686
If there are no more comments on this PR, I think we can merge it tomorrow.
@alamb @jayzhan211 @ozankabak - what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1824830156
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,399 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{
+ LogicalField, LogicalFieldRef, LogicalFields, LogicalType, LogicalUnionFields,
+ TypeSignature,
+};
+use crate::error::{Result, _internal_err};
+use arrow::compute::can_cast_types;
+use arrow_schema::{
+ DataType, Field, FieldRef, Fields, IntervalUnit, TimeUnit, UnionFields,
+};
+use std::sync::Arc;
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean type representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ Time(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ String,
+ /// A list of some logical data type with variable length.
+ List(LogicalFieldRef),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(LogicalFieldRef, i32),
+ /// A nested type that contains a number of sub-fields.
+ Struct(LogicalFields),
+ /// A nested type that can represent slots of differing types.
+ Union(LogicalUnionFields),
+ /// Decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal(u8, i8),
+ /// A Map is a type that an association between a key and a value.
+ ///
+ /// The key and value types are not constrained, but keys should be
+ /// hashable and unique.
+ ///
+ /// In a field with Map type, key type and the second the value type. The names of the
+ /// child fields may be respectively "entries", "key", and "value", but this is
+ /// not enforced.
+ Map(LogicalFieldRef),
+}
+
+impl LogicalType for NativeType {
+ fn native(&self) -> &NativeType {
+ self
+ }
+
+ fn signature(&self) -> TypeSignature<'_> {
+ TypeSignature::Native(self)
+ }
+
+ fn default_cast_for(&self, origin: &DataType) -> Result<DataType> {
+ use DataType::*;
+
+ fn default_field_cast(to: &LogicalField, from: &Field) -> Result<FieldRef> {
+ Ok(Arc::new(Field::new(
+ to.name.clone(),
+ to.logical_type.default_cast_for(from.data_type())?,
+ to.nullable,
+ )))
+ }
+
+ Ok(match (self, origin) {
+ (Self::Null, _) => Null,
+ (Self::Boolean, _) => Boolean,
+ (Self::Int8, _) => Int8,
+ (Self::Int16, _) => Int16,
+ (Self::Int32, _) => Int32,
+ (Self::Int64, _) => Int64,
+ (Self::UInt8, _) => UInt8,
+ (Self::UInt16, _) => UInt16,
+ (Self::UInt32, _) => UInt32,
+ (Self::UInt64, _) => UInt64,
+ (Self::Float16, _) => Float16,
+ (Self::Float32, _) => Float32,
+ (Self::Float64, _) => Float64,
+ (Self::Decimal(p, s), _) if p <= &38 => Decimal128(*p, *s),
+ (Self::Decimal(p, s), _) => Decimal256(*p, *s),
+ (Self::Timestamp(tu, tz), _) => Timestamp(*tu, tz.clone()),
+ (Self::Date, _) => Date32,
+ (Self::Time(tu), _) => match tu {
+ TimeUnit::Second | TimeUnit::Millisecond => Time32(*tu),
+ TimeUnit::Microsecond | TimeUnit::Nanosecond => Time64(*tu),
+ },
+ (Self::Duration(tu), _) => Duration(*tu),
+ (Self::Interval(iu), _) => Interval(*iu),
+ (Self::Binary, LargeUtf8) => LargeBinary,
+ (Self::Binary, Utf8View) => BinaryView,
+ (Self::Binary, data_type) if can_cast_types(data_type, &BinaryView) => {
+ BinaryView
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &LargeBinary) => {
+ LargeBinary
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &Binary) => Binary,
+ (Self::FixedSizeBinary(size), _) => FixedSizeBinary(*size),
+ (Self::String, LargeBinary) => LargeUtf8,
+ (Self::String, BinaryView) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8View) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &LargeUtf8) => {
+ LargeUtf8
+ }
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8) => Utf8,
+ (Self::List(to_field), List(from_field) | FixedSizeList(from_field, _)) => {
+ List(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeList(from_field)) => {
+ LargeList(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), ListView(from_field)) => {
+ ListView(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeListView(from_field)) => {
+ LargeListView(default_field_cast(to_field, from_field)?)
+ }
+ // List array where each element is a len 1 list of the origin type
+ (Self::List(field), _) => List(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ ))),
+ (
+ Self::FixedSizeList(to_field, to_size),
+ FixedSizeList(from_field, from_size),
+ ) if from_size == to_size => {
+ FixedSizeList(default_field_cast(to_field, from_field)?, *to_size)
+ }
+ (
+ Self::FixedSizeList(to_field, size),
+ List(from_field)
+ | LargeList(from_field)
+ | ListView(from_field)
+ | LargeListView(from_field),
+ ) => FixedSizeList(default_field_cast(to_field, from_field)?, *size),
+ // FixedSizeList array where each element is a len 1 list of the origin type
+ (Self::FixedSizeList(field, size), _) => FixedSizeList(
+ Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ )),
+ *size,
+ ),
+ // From https://github.com/apache/arrow-rs/blob/56525efbd5f37b89d1b56aa51709cab9f81bc89e/arrow-cast/src/cast/mod.rs#L189-L196
+ (Self::Struct(to_fields), Struct(from_fields))
+ if from_fields.len() == to_fields.len() =>
+ {
+ Struct(
+ from_fields
+ .iter()
+ .zip(to_fields.iter())
+ .map(|(from, to)| default_field_cast(to, from))
+ .collect::<Result<Fields>>()?,
+ )
+ }
+ (Self::Struct(to_fields), Null) => Struct(
+ to_fields
+ .iter()
+ .map(|field| {
+ Ok(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(&Null)?,
+ field.nullable,
+ )))
+ })
+ .collect::<Result<Fields>>()?,
+ ),
+ (Self::Map(to_field), Map(from_field, sorted)) => {
+ Map(default_field_cast(to_field, from_field)?, *sorted)
+ }
+ (Self::Map(field), Null) => Map(
+ Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(&Null)?,
+ field.nullable,
+ )),
+ false,
+ ),
+ (Self::Union(to_fields), Union(from_fields, mode))
+ if from_fields.len() == to_fields.len() =>
+ {
+ Union(
+ from_fields
+ .iter()
+ .zip(to_fields.iter())
+ .map(|((_, from), (i, to))| {
+ Ok((*i, default_field_cast(to, from)?))
+ })
+ .collect::<Result<UnionFields>>()?,
+ *mode,
+ )
+ }
+ _ => {
+ return _internal_err!(
+ "Unavailable default cast for native type {:?} from physical type {:?}",
+ self,
+ origin
+ )
+ }
+ })
+ }
+}
+
+// The following From<DataType>, From<Field>, ... implementations are temporary
+// mapping solutions to provide backwards compatibility while transitioning from
+// the purely physical system to a logical / physical system.
+
+impl From<DataType> for NativeType {
Review Comment:
NativeType name might be confusing or we need more docs? I'm reading the code and first comes to my mind Arrow Types or maybe rust native types? I mean the name should be sufficient to understand what kind of native it is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1795618569
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,41 @@
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
+ fn name(&self) -> Option<&str>;
Review Comment:
As @findepi proposed (https://github.com/apache/datafusion/issues/11513#issuecomment-2401814374) maybe a more structured approach is needed.
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,41 @@
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
+ fn name(&self) -> Option<&str>;
+}
+
+impl PartialEq for dyn LogicalType {
+ fn eq(&self, other: &Self) -> bool {
+ self.native().eq(other.native()) && self.name().eq(&other.name())
+ }
+}
+
+impl Eq for dyn LogicalType {}
+
+impl PartialOrd for dyn LogicalType {
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+ Some(self.cmp(other))
+ }
+}
+
+impl Ord for dyn LogicalType {
+ fn cmp(&self, other: &Self) -> Ordering {
+ self.name()
+ .cmp(&other.name())
+ .then(self.native().cmp(other.native()))
+ }
+}
+
+impl Hash for dyn LogicalType {
+ fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+ self.name().hash(state);
+ self.native().hash(state);
+ }
+}
Review Comment:
Should we implement these traits manually or should we leave to the implementors?
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,237 @@
+use std::sync::Arc;
+
+use arrow_schema::{DataType, IntervalUnit, TimeUnit};
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean datatype representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed 32-bit date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed 32-bit time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ /// Must be either seconds or milliseconds.
+ Time32(TimeUnit),
+ /// A signed 64-bit time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ /// Must be either microseconds or nanoseconds.
+ Time64(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ Utf8,
+ /// A list of some logical data type with variable length.
+ List(Box<NativeType>),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(Box<NativeType>, i32),
+ /// A nested datatype that contains a number of sub-fields.
+ Struct(Box<[(String, NativeType)]>),
Review Comment:
Should we track nullability of the variants that in `DataType` use the `Field` object (data_type + nullability + metadata + other) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1795614543
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,41 @@
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
Review Comment:
TODO: Documentation missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2424251691
> dont remember the roots, so wondering, can we investigate and use a single type system which is Arrow Types and get rid of other types. At the end of the day it is based on Arrow Types.
I can try to summarise the discussion that has happened in the last months regarding this proposal.
Currently DataFusion:
- Doesn’t support extension types, both in the logical sense (e.g. JSON) and in the physical sense (e.g. a logical string natively is Utf8, LargeUtf8, Utf8View but ideally a user might want to also define a special physical type like List(u8) to be logically equivalent to a string)
- Has a lot of redundant code to handle logically equivalent values during logical planning (i.e. ScalarValue, function signatures)
- Doesn’t support any form of runtime adaptability and assumes that all record batches as input are of the same exact schema while potentially having all the infrastructure needed to be able to support late coercion of logically equivalent values during physical execution (which seems to be a problem that comet is also looking to solve)
while “arrow datatype everywhere” is definitely working for DataFusion currently, my opinion is that this is a needed step towards extensibility and it will help enterprise users looking to migrate their existing engine, custom file format and types to DataFusion, efficiently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2444044837
> Shall we merge this PR?
I think there is enough agreement to start working on the other pieces of the puzzle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1818059422
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,399 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{
+ LogicalField, LogicalFieldRef, LogicalFields, LogicalType, LogicalUnionFields,
+ TypeSignature,
+};
+use crate::error::{Result, _internal_err};
+use arrow::compute::can_cast_types;
+use arrow_schema::{
+ DataType, Field, FieldRef, Fields, IntervalUnit, TimeUnit, UnionFields,
+};
+use std::sync::Arc;
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean type representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ Time(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ String,
+ /// A list of some logical data type with variable length.
+ List(LogicalFieldRef),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(LogicalFieldRef, i32),
+ /// A nested type that contains a number of sub-fields.
+ Struct(LogicalFields),
+ /// A nested type that can represent slots of differing types.
+ Union(LogicalUnionFields),
+ /// Decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal(u8, i8),
+ /// A Map is a type that an association between a key and a value.
+ ///
+ /// The key and value types are not constrained, but keys should be
+ /// hashable and unique.
+ ///
+ /// In a field with Map type, key type and the second the value type. The names of the
+ /// child fields may be respectively "entries", "key", and "value", but this is
+ /// not enforced.
+ Map(LogicalFieldRef),
+}
+
+impl LogicalType for NativeType {
+ fn native(&self) -> &NativeType {
+ self
+ }
+
+ fn signature(&self) -> TypeSignature<'_> {
+ TypeSignature::Native(self)
+ }
+
+ fn default_cast_for(&self, origin: &DataType) -> Result<DataType> {
+ use DataType::*;
+
+ fn default_field_cast(to: &LogicalField, from: &Field) -> Result<FieldRef> {
+ Ok(Arc::new(Field::new(
+ to.name.clone(),
+ to.logical_type.default_cast_for(from.data_type())?,
+ to.nullable,
+ )))
+ }
+
+ Ok(match (self, origin) {
+ (Self::Null, _) => Null,
+ (Self::Boolean, _) => Boolean,
+ (Self::Int8, _) => Int8,
+ (Self::Int16, _) => Int16,
+ (Self::Int32, _) => Int32,
+ (Self::Int64, _) => Int64,
+ (Self::UInt8, _) => UInt8,
+ (Self::UInt16, _) => UInt16,
+ (Self::UInt32, _) => UInt32,
+ (Self::UInt64, _) => UInt64,
+ (Self::Float16, _) => Float16,
+ (Self::Float32, _) => Float32,
+ (Self::Float64, _) => Float64,
+ (Self::Decimal(p, s), _) if p <= &38 => Decimal128(*p, *s),
+ (Self::Decimal(p, s), _) => Decimal256(*p, *s),
+ (Self::Timestamp(tu, tz), _) => Timestamp(*tu, tz.clone()),
+ (Self::Date, _) => Date32,
+ (Self::Time(tu), _) => match tu {
+ TimeUnit::Second | TimeUnit::Millisecond => Time32(*tu),
+ TimeUnit::Microsecond | TimeUnit::Nanosecond => Time64(*tu),
+ },
+ (Self::Duration(tu), _) => Duration(*tu),
+ (Self::Interval(iu), _) => Interval(*iu),
+ (Self::Binary, LargeUtf8) => LargeBinary,
+ (Self::Binary, Utf8View) => BinaryView,
+ (Self::Binary, data_type) if can_cast_types(data_type, &BinaryView) => {
+ BinaryView
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &LargeBinary) => {
+ LargeBinary
+ }
+ (Self::Binary, data_type) if can_cast_types(data_type, &Binary) => Binary,
+ (Self::FixedSizeBinary(size), _) => FixedSizeBinary(*size),
+ (Self::String, LargeBinary) => LargeUtf8,
+ (Self::String, BinaryView) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8View) => Utf8View,
+ (Self::String, data_type) if can_cast_types(data_type, &LargeUtf8) => {
+ LargeUtf8
+ }
+ (Self::String, data_type) if can_cast_types(data_type, &Utf8) => Utf8,
+ (Self::List(to_field), List(from_field) | FixedSizeList(from_field, _)) => {
+ List(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeList(from_field)) => {
+ LargeList(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), ListView(from_field)) => {
+ ListView(default_field_cast(to_field, from_field)?)
+ }
+ (Self::List(to_field), LargeListView(from_field)) => {
+ LargeListView(default_field_cast(to_field, from_field)?)
+ }
+ // List array where each element is a len 1 list of the origin type
+ (Self::List(field), _) => List(Arc::new(Field::new(
+ field.name.clone(),
+ field.logical_type.default_cast_for(origin)?,
+ field.nullable,
+ ))),
+ (
+ Self::FixedSizeList(to_field, to_size),
+ FixedSizeList(from_field, from_size),
+ ) if from_size == to_size => {
+ FixedSizeList(default_field_cast(to_field, from_field)?, *to_size)
+ }
+ (
+ Self::FixedSizeList(to_field, size),
+ List(from_field)
+ | LargeList(from_field)
+ | ListView(from_field)
+ | LargeListView(from_field),
+ ) => FixedSizeList(default_field_cast(to_field, from_field)?, *size),
Review Comment:
Logical fixed list to Physical list is casting from fixed list to list in arrow::cast. If IIRC, it is safe to cast from fixed list to list, but not in the reverse direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2437397665
> So this means that if you received an Int64 then you first try to coerce to Utf8View and in what cases would you fall back to the LargeUtf8 and Utf8?
Good question, maybe we just need a single type and cast to that one. If we can't it probably mean they don't belong to the same logical type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1795614543
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,41 @@
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
Review Comment:
TODO: Documentation missing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1798841086
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,41 @@
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
+ fn name(&self) -> Option<&str>;
Review Comment:
A more structured approach will be needed if we want to reasonably serialize generic types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1798849440
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,254 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_schema::{DataType, IntervalUnit, TimeUnit};
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean datatype representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed 32-bit date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed 32-bit time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ /// Must be either seconds or milliseconds.
+ Time32(TimeUnit),
+ /// A signed 64-bit time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ /// Must be either microseconds or nanoseconds.
+ Time64(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ Utf8,
+ /// A list of some logical data type with variable length.
+ List(Box<NativeType>),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(Box<NativeType>, i32),
+ /// A nested datatype that contains a number of sub-fields.
+ Struct(Box<[(String, NativeType)]>),
+ /// A nested datatype that can represent slots of differing types.
+ Union(Box<[(i8, NativeType)]>),
+ /// Exact 128-bit width decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal128(u8, i8),
+ /// Exact 256-bit width decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal256(u8, i8),
+ /// A Map is a logical nested type that is represented as
+ ///
+ /// `List<entries: Struct<key: K, value: V>>`
+ ///
+ /// The keys and values are each respectively contiguous.
+ /// The key and value types are not constrained, but keys should be
+ /// hashable and unique.
+ /// Whether the keys are sorted can be set in the `bool` after the `Field`.
+ ///
+ /// In a field with Map type, the field has a child Struct field, which then
+ /// has two children: key type and the second the value type. The names of the
+ /// child fields may be respectively "entries", "key", and "value", but this is
+ /// not enforced.
+ Map(Box<NativeType>),
Review Comment:
Map should be something like `Map(Box<NativeType>, Box<NativeType>)`
in arrow it's `Map(FieldRef` and the field has to be a struct with key/value fields.
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,254 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_schema::{DataType, IntervalUnit, TimeUnit};
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean datatype representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed 32-bit date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed 32-bit time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ /// Must be either seconds or milliseconds.
+ Time32(TimeUnit),
+ /// A signed 64-bit time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ /// Must be either microseconds or nanoseconds.
+ Time64(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ Utf8,
+ /// A list of some logical data type with variable length.
+ List(Box<NativeType>),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(Box<NativeType>, i32),
+ /// A nested datatype that contains a number of sub-fields.
+ Struct(Box<[(String, NativeType)]>),
+ /// A nested datatype that can represent slots of differing types.
+ Union(Box<[(i8, NativeType)]>),
+ /// Exact 128-bit width decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal128(u8, i8),
+ /// Exact 256-bit width decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal256(u8, i8),
Review Comment:
Do we need both decimal forms?
the runtime representation is a function of precision anyway
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,254 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_schema::{DataType, IntervalUnit, TimeUnit};
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean datatype representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed 32-bit date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed 32-bit time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ /// Must be either seconds or milliseconds.
+ Time32(TimeUnit),
+ /// A signed 64-bit time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ /// Must be either microseconds or nanoseconds.
+ Time64(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ Utf8,
+ /// A list of some logical data type with variable length.
+ List(Box<NativeType>),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(Box<NativeType>, i32),
+ /// A nested datatype that contains a number of sub-fields.
+ Struct(Box<[(String, NativeType)]>),
+ /// A nested datatype that can represent slots of differing types.
+ Union(Box<[(i8, NativeType)]>),
+ /// Exact 128-bit width decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal128(u8, i8),
+ /// Exact 256-bit width decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal256(u8, i8),
+ /// A Map is a logical nested type that is represented as
+ ///
+ /// `List<entries: Struct<key: K, value: V>>`
Review Comment:
the Map's docs needs to be updated. they still refer to arrow DataType's FieldRef-based way of defining a map
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
Review Comment:
```suggestion
pub trait LogicalType: Sync + Send {
```
- require impls to be thread-safe
- Debug is somethig we could perhaps impl here (especially that we impl Eq -- an implementation cannot have any special properties)
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
+ fn name(&self) -> Option<&str>;
+}
+
+impl PartialEq for dyn LogicalType {
+ fn eq(&self, other: &Self) -> bool {
+ self.native().eq(other.native()) && self.name().eq(&other.name())
Review Comment:
name should be unique.
it should be an error for two types to have same name, but different native
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
+ fn name(&self) -> Option<&str>;
Review Comment:
Especially given this returns Option, but also because of how it's used in Eq, let's document implementation requirements here.
I suppose we will end up having code like
```rust
if type.name().is_none() {
// it's builtin
match type.native() {
....
}
}
```
but then, if builtins have really special treatment, we really could model them better:
```rust
enum LogicalType {
Builtin(NativeType)
Extension(String, NativeType)
}
```
This would be strictly better than current code, but I don't think it would actually work OK. In the future type will need to provide stuff like comparison of values, or hash of values (unless they are simply resolved as function calls -- not sure how this would work for sorting, but maybe it would)
If we don't want the builtin types to have as much special treatment, then we should have the name mandatory and perhaps structured as in https://github.com/apache/datafusion/issues/11513#issuecomment-2401814374 , see also https://github.com/findepi/datafusion/blob/findepi/types/datafusion/common/src/types/type.rs and surrounding files
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,41 @@
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
+ fn name(&self) -> Option<&str>;
+}
+
+impl PartialEq for dyn LogicalType {
+ fn eq(&self, other: &Self) -> bool {
+ self.native().eq(other.native()) && self.name().eq(&other.name())
+ }
+}
+
+impl Eq for dyn LogicalType {}
+
+impl PartialOrd for dyn LogicalType {
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+ Some(self.cmp(other))
+ }
+}
+
+impl Ord for dyn LogicalType {
+ fn cmp(&self, other: &Self) -> Ordering {
+ self.name()
+ .cmp(&other.name())
+ .then(self.native().cmp(other.native()))
+ }
+}
+
+impl Hash for dyn LogicalType {
+ fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+ self.name().hash(state);
+ self.native().hash(state);
+ }
+}
Review Comment:
That's a good question. It all comes form Eq for me.
If we implement Eq by name (or name + native), then we disallow implementations from having any attributes that could affect their semantics (ie an implementation may have a field caching something, but it should effectively be a function of the name). And then we can implement other traits for the type as well.
And in fact this makes sense. We need a way to identify a type. We will store this information in the field metadata, so either type needs to be serializable, or it's name needs to be serializable and resolvable. The latter sounds like the way to go, especially given extension types (https://github.com/apache/datafusion/issues/12644), which won't be known at compile time
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,254 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_schema::{DataType, IntervalUnit, TimeUnit};
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean datatype representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed 32-bit date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed 32-bit time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ /// Must be either seconds or milliseconds.
+ Time32(TimeUnit),
+ /// A signed 64-bit time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ /// Must be either microseconds or nanoseconds.
+ Time64(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ Utf8,
+ /// A list of some logical data type with variable length.
+ List(Box<NativeType>),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(Box<NativeType>, i32),
+ /// A nested datatype that contains a number of sub-fields.
+ Struct(Box<[(String, NativeType)]>),
+ /// A nested datatype that can represent slots of differing types.
+ Union(Box<[(i8, NativeType)]>),
+ /// Exact 128-bit width decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal128(u8, i8),
+ /// Exact 256-bit width decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal256(u8, i8),
+ /// A Map is a logical nested type that is represented as
+ ///
+ /// `List<entries: Struct<key: K, value: V>>`
+ ///
+ /// The keys and values are each respectively contiguous.
+ /// The key and value types are not constrained, but keys should be
+ /// hashable and unique.
+ /// Whether the keys are sorted can be set in the `bool` after the `Field`.
+ ///
+ /// In a field with Map type, the field has a child Struct field, which then
+ /// has two children: key type and the second the value type. The names of the
+ /// child fields may be respectively "entries", "key", and "value", but this is
+ /// not enforced.
+ Map(Box<NativeType>),
+}
+
+impl From<DataType> for NativeType {
+ fn from(value: DataType) -> Self {
Review Comment:
Per https://github.com/apache/datafusion/pull/12853#discussion_r1796246184, i hope we won't need this in the long run. We could maybe capture this intent as a cod comment.
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,254 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_schema::{DataType, IntervalUnit, TimeUnit};
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean datatype representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed 32-bit date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed 32-bit time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ /// Must be either seconds or milliseconds.
+ Time32(TimeUnit),
+ /// A signed 64-bit time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ /// Must be either microseconds or nanoseconds.
+ Time64(TimeUnit),
Review Comment:
Do we need both? The runtime representation could cheaply depend on TimeUnit (as the docs says), without this having to be two types.
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
+ fn name(&self) -> Option<&str>;
+}
+
+impl PartialEq for dyn LogicalType {
+ fn eq(&self, other: &Self) -> bool {
+ self.native().eq(other.native()) && self.name().eq(&other.name())
+ }
+}
+
+impl Eq for dyn LogicalType {}
+
+impl PartialOrd for dyn LogicalType {
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+ Some(self.cmp(other))
+ }
+}
+
+impl Ord for dyn LogicalType {
Review Comment:
do we need to sort types?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1799352157
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
Review Comment:
I think the idea is that we have LogicalType already. LogicalNumber, LogicalTimestamp, LogicalDate, we can differ them in logical level already, but they can also decode as i64, i32 in physical level.
Asking logical type "could you please decode this arrow type?" is to tell the relationship between logical type and physical type. We don't need to know whether the arrow i64 is number or timestamp, because we already know that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1800893742
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,124 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeSignature<'a> {
+ /// Represents a built-in native type.
+ Native(&'a NativeType),
+ /// Represents an arrow-compatible extension type.
+ /// (https://arrow.apache.org/docs/format/Columnar.html#extension-types)
+ ///
+ /// The `name` should contain the same value as 'ARROW:extension:name'.
+ Extension {
+ name: &'a str,
+ parameters: &'a [TypeParameter<'a>],
+ },
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeParameter<'a> {
+ Type(TypeSignature<'a>),
+ Number(i128),
+}
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: Sync + Send {
+ fn native(&self) -> &NativeType;
+ fn signature(&self) -> TypeSignature<'_>;
+}
+
+impl fmt::Debug for dyn LogicalType {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_tuple("LogicalType")
+ .field(&self.signature())
+ .field(&self.native())
+ .finish()
+ }
+}
+
+impl PartialEq for dyn LogicalType {
+ fn eq(&self, other: &Self) -> bool {
+ self.native().eq(other.native()) && self.signature().eq(&other.signature())
+ }
+}
+
+impl Eq for dyn LogicalType {}
+
+impl PartialOrd for dyn LogicalType {
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+ Some(self.cmp(other))
+ }
+}
+
+impl Ord for dyn LogicalType {
+ fn cmp(&self, other: &Self) -> Ordering {
+ self.signature()
+ .cmp(&other.signature())
+ .then(self.native().cmp(other.native()))
+ }
+}
+
+impl Hash for dyn LogicalType {
+ fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+ self.signature().hash(state);
+ self.native().hash(state);
+ }
+}
+
+#[cfg(test)]
+mod test {
+ #![allow(dead_code)]
+
+ use super::{LogicalType, TypeParameter, TypeSignature};
+ use crate::types::NativeType;
+
+ struct MagicalType {}
+
+ impl LogicalType for MagicalType {
+ fn native(&self) -> &NativeType {
+ &NativeType::Utf8
+ }
+
+ fn signature(&self) -> TypeSignature<'_> {
+ TypeSignature::Extension {
+ name: "MagicalType",
+ parameters: &[TypeParameter::Type(TypeSignature::Native(
+ &NativeType::Boolean,
+ ))],
+ }
+ }
+ }
+
+ fn test(logical_type: &dyn LogicalType) {
+ match logical_type.signature() {
+ TypeSignature::Extension {
+ name: "MagicalType",
+ parameters:
+ [TypeParameter::Type(TypeSignature::Native(NativeType::Boolean))],
+ } => {}
+ TypeSignature::Native(NativeType::Binary) => todo!(),
+ _ => unimplemented!(),
+ };
+ }
Review Comment:
I've added a more structured `TypeSignature`. I've encountered some problems while iterating for a design of this structure since I wanted to balance ease of handling and information contained. I think I've ended up with a good baseline that can be easily pattern matched and limits the copying of information. ptal @findepi @jayzhan211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1800918056
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
Review Comment:
I'm not sure I can follow. @jayzhan211 -- can you write a small practical example? I want to make sure I understand the use case. Thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2435161877
cc @goldmedal @ozankabak maybe you can take a look at this PR as well if you have bandwidth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1810648029
##########
datafusion/common/src/types/native.rs:
##########
@@ -0,0 +1,328 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{ops::Deref, sync::Arc};
+
+use arrow_schema::{DataType, Field, Fields, IntervalUnit, TimeUnit, UnionFields};
+
+use super::{LogicalType, TypeSignature};
+
+/// A record of a native type, its name and its nullability.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct NativeField {
+ name: String,
+ native_type: NativeType,
+ nullable: bool,
+}
+
+impl NativeField {
+ pub fn name(&self) -> &str {
+ &self.name
+ }
+
+ pub fn native_type(&self) -> &NativeType {
+ &self.native_type
+ }
+
+ pub fn nullable(&self) -> bool {
+ self.nullable
+ }
+}
+
+/// A reference counted [`NativeField`].
+pub type NativeFieldRef = Arc<NativeField>;
+
+/// A cheaply cloneable, owned collection of [`NativeFieldRef`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct NativeFields(Arc<[NativeFieldRef]>);
+
+impl Deref for NativeFields {
+ type Target = [NativeFieldRef];
+
+ fn deref(&self) -> &Self::Target {
+ self.0.as_ref()
+ }
+}
+
+/// A cheaply cloneable, owned collection of [`NativeFieldRef`] and their
+/// corresponding type ids.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct NativeUnionFields(Arc<[(i8, NativeFieldRef)]>);
+
+impl Deref for NativeUnionFields {
+ type Target = [(i8, NativeFieldRef)];
+
+ fn deref(&self) -> &Self::Target {
+ self.0.as_ref()
+ }
+}
+
+/// Representation of a type that DataFusion can handle natively. It is a subset
+/// of the physical variants in Arrow's native [`DataType`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum NativeType {
+ /// Null type
+ Null,
+ /// A boolean type representing the values `true` and `false`.
+ Boolean,
+ /// A signed 8-bit integer.
+ Int8,
+ /// A signed 16-bit integer.
+ Int16,
+ /// A signed 32-bit integer.
+ Int32,
+ /// A signed 64-bit integer.
+ Int64,
+ /// An unsigned 8-bit integer.
+ UInt8,
+ /// An unsigned 16-bit integer.
+ UInt16,
+ /// An unsigned 32-bit integer.
+ UInt32,
+ /// An unsigned 64-bit integer.
+ UInt64,
+ /// A 16-bit floating point number.
+ Float16,
+ /// A 32-bit floating point number.
+ Float32,
+ /// A 64-bit floating point number.
+ Float64,
+ /// A timestamp with an optional timezone.
+ ///
+ /// Time is measured as a Unix epoch, counting the seconds from
+ /// 00:00:00.000 on 1 January 1970, excluding leap seconds,
+ /// as a signed 64-bit integer.
+ ///
+ /// The time zone is a string indicating the name of a time zone, one of:
+ ///
+ /// * As used in the Olson time zone database (the "tz database" or
+ /// "tzdata"), such as "America/New_York"
+ /// * An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
+ ///
+ /// Timestamps with a non-empty timezone
+ /// ------------------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in the *UTC* timezone
+ /// (the Unix epoch), regardless of the Timestamp's own timezone.
+ ///
+ /// Therefore, timestamp values with a non-empty timezone correspond to
+ /// physical points in time together with some additional information about
+ /// how the data was obtained and/or how to display it (the timezone).
+ ///
+ /// For example, the timestamp value 0 with the timezone string "Europe/Paris"
+ /// corresponds to "January 1st 1970, 00h00" in the UTC timezone, but the
+ /// application may prefer to display it as "January 1st 1970, 01h00" in
+ /// the Europe/Paris timezone (which is the same physical point in time).
+ ///
+ /// One consequence is that timestamp values with a non-empty timezone
+ /// can be compared and ordered directly, since they all share the same
+ /// well-known point of reference (the Unix epoch).
+ ///
+ /// Timestamps with an unset / empty timezone
+ /// -----------------------------------------
+ ///
+ /// If a Timestamp column has no timezone value, its epoch is
+ /// 1970-01-01 00:00:00 (January 1st 1970, midnight) in an *unknown* timezone.
+ ///
+ /// Therefore, timestamp values without a timezone cannot be meaningfully
+ /// interpreted as physical points in time, but only as calendar / clock
+ /// indications ("wall clock time") in an unspecified timezone.
+ ///
+ /// For example, the timestamp value 0 with an empty timezone string
+ /// corresponds to "January 1st 1970, 00h00" in an unknown timezone: there
+ /// is not enough information to interpret it as a well-defined physical
+ /// point in time.
+ ///
+ /// One consequence is that timestamp values without a timezone cannot
+ /// be reliably compared or ordered, since they may have different points of
+ /// reference. In particular, it is *not* possible to interpret an unset
+ /// or empty timezone as the same as "UTC".
+ ///
+ /// Conversion between timezones
+ /// ----------------------------
+ ///
+ /// If a Timestamp column has a non-empty timezone, changing the timezone
+ /// to a different non-empty value is a metadata-only operation:
+ /// the timestamp values need not change as their point of reference remains
+ /// the same (the Unix epoch).
+ ///
+ /// However, if a Timestamp column has no timezone value, changing it to a
+ /// non-empty value requires to think about the desired semantics.
+ /// One possibility is to assume that the original timestamp values are
+ /// relative to the epoch of the timezone being set; timestamp values should
+ /// then adjusted to the Unix epoch (for example, changing the timezone from
+ /// empty to "Europe/Paris" would require converting the timestamp values
+ /// from "Europe/Paris" to "UTC", which seems counter-intuitive but is
+ /// nevertheless correct).
+ ///
+ /// ```
+ /// # use arrow_schema::{DataType, TimeUnit};
+ /// DataType::Timestamp(TimeUnit::Second, None);
+ /// DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
+ /// DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));
+ /// ```
+ Timestamp(TimeUnit, Option<Arc<str>>),
+ /// A signed date representing the elapsed time since UNIX epoch (1970-01-01)
+ /// in days.
+ Date,
+ /// A signed time representing the elapsed time since midnight in the unit of `TimeUnit`.
+ Time(TimeUnit),
+ /// Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
+ Duration(TimeUnit),
+ /// A "calendar" interval which models types that don't necessarily
+ /// have a precise duration without the context of a base timestamp (e.g.
+ /// days can differ in length during day light savings time transitions).
+ Interval(IntervalUnit),
+ /// Opaque binary data of variable length.
+ Binary,
+ /// Opaque binary data of fixed size.
+ /// Enum parameter specifies the number of bytes per value.
+ FixedSizeBinary(i32),
+ /// A variable-length string in Unicode with UTF-8 encoding.
+ Utf8,
+ /// A list of some logical data type with variable length.
+ List(NativeFieldRef),
+ /// A list of some logical data type with fixed length.
+ FixedSizeList(NativeFieldRef, i32),
+ /// A nested type that contains a number of sub-fields.
+ Struct(NativeFields),
+ /// A nested type that can represent slots of differing types.
+ Union(NativeUnionFields),
+ /// Decimal value with precision and scale
+ ///
+ /// * precision is the total number of digits
+ /// * scale is the number of digits past the decimal
+ ///
+ /// For example the number 123.45 has precision 5 and scale 2.
+ ///
+ /// In certain situations, scale could be negative number. For
+ /// negative scale, it is the number of padding 0 to the right
+ /// of the digits.
+ ///
+ /// For example the number 12300 could be treated as a decimal
+ /// has precision 3 and scale -2.
+ Decimal(u8, i8),
+ /// A Map is a type that an association between a key and a value.
+ ///
+ /// The key and value types are not constrained, but keys should be
+ /// hashable and unique.
+ ///
+ /// In a field with Map type, the field has a child Struct field, which then
+ /// has two children: key type and the second the value type. The names of the
+ /// child fields may be respectively "entries", "key", and "value", but this is
+ /// not enforced.
+ Map(NativeFieldRef),
Review Comment:
That's a good point. It should be `LogicalTypeRef` here and in the `NativeField` type then (which I guess will become the `LogicalField` type).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1810645738
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// Signature that uniquely identifies a type among other types.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeSignature<'a> {
+ /// Represents a built-in native type.
+ Native(&'a NativeType),
+ /// Represents an arrow-compatible extension type.
+ /// (<https://arrow.apache.org/docs/format/Columnar.html#extension-types>)
+ ///
+ /// The `name` should contain the same value as 'ARROW:extension:name'.
+ Extension {
+ name: &'a str,
Review Comment:
TypeSignature uses lifetimes just to make pattern matching easier when dealing with parameters (or lack thereof).
```rust
pub fn important_stuff(logical_type: &dyn LogicalType) {
match logical_type.signature() {
TypeSignature::Native(NativeType::Utf8) => todo!(),
TypeSignature::Extension {
name: "JSON", // <-- changing from &'a str to String leads to: expected `String`, but found `&str`,
parameters: &[]
} => todo!(),
_ => unimplemented!()
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2457721548
Epic work!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2453277957
I think this is ready to merge. Additional improvements can be made in a follow-up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1824823684
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::NativeType;
+use crate::error::Result;
+use arrow_schema::DataType;
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+/// Signature that uniquely identifies a type among other types.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeSignature<'a> {
+ /// Represents a built-in native type.
+ Native(&'a NativeType),
+ /// Represents an arrow-compatible extension type.
+ /// (<https://arrow.apache.org/docs/format/Columnar.html#extension-types>)
+ ///
+ /// The `name` should contain the same value as 'ARROW:extension:name'.
+ Extension {
Review Comment:
maybe we can add the comment why the extension is needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1800898015
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
+ fn name(&self) -> Option<&str>;
+}
+
+impl PartialEq for dyn LogicalType {
+ fn eq(&self, other: &Self) -> bool {
+ self.native().eq(other.native()) && self.name().eq(&other.name())
Review Comment:
`panic`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [logical-types] add NativeType and LogicalType [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2410274263
cc @sadboy @milevin @wolfram-s @wizardxz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1800898015
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+use super::NativeType;
+
+/// A reference counted [`LogicalType`]
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
+
+pub trait LogicalType: fmt::Debug {
+ fn native(&self) -> &NativeType;
+ fn name(&self) -> Option<&str>;
+}
+
+impl PartialEq for dyn LogicalType {
+ fn eq(&self, other: &Self) -> bool {
+ self.native().eq(other.native()) && self.name().eq(&other.name())
Review Comment:
`panic` on the partial equal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2437509899
> But that also means we will always cast to the large type like largeutf8 or largeutf8view?( I think we will have one in the future)
Yeah, for this particular edge case that is the only safe and sensible option I can think of.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#discussion_r1817701192
##########
datafusion/common/src/types/logical.rs:
##########
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::NativeType;
+use crate::error::Result;
+use arrow_schema::DataType;
+use core::fmt;
+use std::{cmp::Ordering, hash::Hash, sync::Arc};
+
+/// Signature that uniquely identifies a type among other types.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeSignature<'a> {
+ /// Represents a built-in native type.
+ Native(&'a NativeType),
+ /// Represents an arrow-compatible extension type.
+ /// (<https://arrow.apache.org/docs/format/Columnar.html#extension-types>)
+ ///
+ /// The `name` should contain the same value as 'ARROW:extension:name'.
+ Extension {
+ name: &'a str,
+ parameters: &'a [TypeParameter<'a>],
+ },
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum TypeParameter<'a> {
+ Type(TypeSignature<'a>),
+ Number(i128),
+}
+
+/// A reference counted [`LogicalType`].
+pub type LogicalTypeRef = Arc<dyn LogicalType>;
Review Comment:
I think we need function to get `LogicalTypeRef` as a singleton overall the datafusion.
Something like
```rust
// Singleton instances
// TODO: Replace with LazyLock
pub static LOGICAL_STRING: OnceLock<LogicalTypeRef> = OnceLock::new();
pub static LOGICAL_FLOAT16: OnceLock<LogicalTypeRef> = OnceLock::new();
// Usage functions
pub fn logical_string() -> LogicalTypeRef {
Arc::clone(LOGICAL_STRING.get_or_init(|| Arc::new(String)))
}
pub fn logical_float16() -> LogicalTypeRef {
Arc::clone(LOGICAL_FLOAT16.get_or_init(|| Arc::new(Float16)))
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(logical-types): add NativeType and LogicalType [datafusion]
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on PR #12853:
URL: https://github.com/apache/datafusion/pull/12853#issuecomment-2444199553
@findepi - what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org