You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ZENOTME (via GitHub)" <gi...@apache.org> on 2023/08/28 12:53:37 UTC
[GitHub] [iceberg-rust] ZENOTME opened a new pull request, #42: feat: support transform function
ZENOTME opened a new pull request, #42:
URL: https://github.com/apache/iceberg-rust/pull/42
This PR add transform function trait and following implementation:
1. identity
2. void
3. temporal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1308509475
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,329 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::TransformFunction;
+use crate::Result;
+use arrow::array::{Array, TimestampMicrosecondArray};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+const BASE_YEAR: i32 = 1970;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).expect("Should not call transform in Year with non-date type");
Review Comment:
In our experience in icelake, the transform function only used in partition writer.🤔 But I'm not sure whether that change in future. Return Unexpected seems more reasonable beacause at least for now, partition writer encounter unsupport type will return a error.
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,329 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::TransformFunction;
+use crate::Result;
+use arrow::array::{Array, TimestampMicrosecondArray};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+const BASE_YEAR: i32 = 1970;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).expect("Should not call transform in Year with non-date type");
Review Comment:
In our experience in icelake, the transform function only used in partition writer.🤔 But I'm not sure whether that change in future. Return Unexpected seems more reasonable beacause at least for now, partition writer encounter unsupport type will return a error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#issuecomment-1703709042
Hi, I have [implemented the bucket and truncate transform in icelake](https://github.com/icelake-io/icelake/pull/173) and I'm ready to port them. Do you prefer to port them in this PR or another PR. I'm concern whether that cause this PR so big that hard to review. Anyway, I'm ok with both way and it's up to you. @liurenjie1024 @Xuanwo @JanKaul @nastra @Fokko
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] liurenjie1024 commented on a diff in pull request #42: feat: support transform function
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1311147980
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
Review Comment:
For writer, I don't think we should provide it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] liurenjie1024 commented on a diff in pull request #42: feat: support transform function
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1311128206
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
Review Comment:
For example when we want to execute `select * from t where a = 1 and b=2`, and table `t` is partitioned by a, we can prune unnecessary partitions by calculating `a=1`. In this case we need to use iceberg value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1324540344
##########
crates/iceberg/src/transform/truncate.rs:
##########
@@ -0,0 +1,177 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::ArrayRef;
+use arrow_schema::DataType;
+
+use crate::Error;
+
+use super::TransformFunction;
+
+pub struct Truncate {
+ width: u32,
+}
+
+impl Truncate {
+ pub fn new(width: u32) -> Self {
+ Self { width }
+ }
+}
+
+impl TransformFunction for Truncate {
+ fn transform(&self, input: ArrayRef) -> crate::Result<ArrayRef> {
+ match input.data_type() {
+ DataType::Int32 => {
+ let width: i32 = self.width.try_into().map_err(|_| {
+ Error::new(
+ crate::ErrorKind::DataInvalid,
+ "width is failed to convert to i32 when truncate Int32Array",
+ )
+ })?;
+ let res: arrow_array::Int32Array = input
+ .as_any()
+ .downcast_ref::<arrow_array::Int32Array>()
+ .unwrap()
+ .unary(|v| v - v.rem_euclid(width));
+ Ok(Arc::new(res))
+ }
+ DataType::Int64 => {
+ let width = self.width as i64;
+ let res: arrow_array::Int64Array = input
+ .as_any()
+ .downcast_ref::<arrow_array::Int64Array>()
+ .unwrap()
+ .unary(|v| v - (((v % width) + width) % width));
+ Ok(Arc::new(res))
+ }
+ DataType::Decimal128(precision, scale) => {
+ let width = self.width as i128;
+ let res: arrow_array::Decimal128Array = input
+ .as_any()
+ .downcast_ref::<arrow_array::Decimal128Array>()
+ .unwrap()
+ .unary(|v| v - (((v % width) + width) % width))
+ .with_precision_and_scale(*precision, *scale)
+ .map_err(|err| Error::new(crate::ErrorKind::Unexpected, format!("{err}")))?;
+ Ok(Arc::new(res))
+ }
+ DataType::Utf8 => {
+ let len = self.width as usize;
+ let res: arrow_array::StringArray = arrow_array::StringArray::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::StringArray>()
+ .unwrap()
+ .iter()
+ .map(|v| v.map(|v| &v[..len])),
Review Comment:
Thanks for your info!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1325276422
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
Review Comment:
> When a file is being written, Iceberg keeps the upper and lower bounds of each of the columns. By default, a truncate(16) transform is applied before storing the metrics in the manifest.
Did you mean [the lower bound(upper bound) in field summary](https://iceberg.apache.org/spec/#manifests:~:text=NaN%20value%20for%20the%20field-,optional,field%2C%20or%20null%20if%20all%20values%20are%20null%20or%20NaN%20%5B2%5D,-optional)? So it's the lower bound of original column and then apply transform to it `transform(lower_bound(col))` rather than lower bound of transform column `lower_bound(transform(col))`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] liurenjie1024 commented on a diff in pull request #42: feat: support transform function
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1308557984
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -28,24 +28,24 @@ use arrow::{
use chrono::Datelike;
use std::sync::Arc;
-/// 719163 is the number of days from 0000-01-01 to 1970-01-01
-const EPOCH_DAY_FROM_CE: i32 = 719163;
-const DAY_PER_SECOND: f64 = 0.0000115741;
-const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
-const BASE_YEAR: i32 = 1970;
+const DAY_FROM_UNIX_EPOCH: i32 = 719163;
Review Comment:
Add comments for all these constants?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1311093814
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
Review Comment:
Currently this transform logic is specific for arrow record batch. Maybe we need to consider a interface more generic?
Actually I think the partition writer is used by external system, and arrow is a good middle layer between iceberg and external system. If we want to consider iceberg value, we need to take a effor to maintain convert of iceberg value. Is there benefit for using iceberg value?🤔
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
Review Comment:
Currently this transform logic is specific for arrow record batch. Maybe we need to consider a interface more generic?
Actually I think the partition writer is used by external system, and arrow is a good middle layer between iceberg and external system. If we want to consider iceberg value, we need to take a effor to maintain convert of iceberg value. Is there benefit for using iceberg value?🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] JanKaul commented on a diff in pull request #42: feat: support transform function
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1319404484
##########
crates/iceberg/src/transform/truncate.rs:
##########
@@ -0,0 +1,177 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::ArrayRef;
+use arrow_schema::DataType;
+
+use crate::Error;
+
+use super::TransformFunction;
+
+pub struct Truncate {
+ width: u32,
+}
+
+impl Truncate {
+ pub fn new(width: u32) -> Self {
+ Self { width }
+ }
+}
+
+impl TransformFunction for Truncate {
+ fn transform(&self, input: ArrayRef) -> crate::Result<ArrayRef> {
+ match input.data_type() {
+ DataType::Int32 => {
+ let width: i32 = self.width.try_into().map_err(|_| {
+ Error::new(
+ crate::ErrorKind::DataInvalid,
+ "width is failed to convert to i32 when truncate Int32Array",
+ )
+ })?;
+ let res: arrow_array::Int32Array = input
+ .as_any()
+ .downcast_ref::<arrow_array::Int32Array>()
+ .unwrap()
+ .unary(|v| v - (((v % width) + width) % width));
Review Comment:
We could use `v - v.rem_euclid(width)` instead of `v - (((v % width) + width) % width)`. But it's also fine as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1308201655
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,669 @@
+use super::TransformFunction;
+use crate::{Error, Result};
+use arrow::array::{
+ Array, Date64Array, TimestampMicrosecondArray, TimestampMillisecondArray,
+ TimestampNanosecondArray, TimestampSecondArray,
+};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).map_err(|err| {
Review Comment:
Should we also add check before it for unsuport type like `Date64` @liurenjie1024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1311087157
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
+}
+
+/// BoxedTransformFunction is a boxed trait object of TransformFunction.
+pub type BoxedTransformFunction = Box<dyn TransformFunction>;
+
+/// create_transform_function creates a boxed trait object of TransformFunction from a Transform.
+pub fn create_transform_function(transform: &Transform) -> Result<BoxedTransformFunction> {
+ match transform {
+ Transform::Identity => Ok(Box::new(identity::Identity {})),
+ Transform::Void => Ok(Box::new(void::Void {})),
+ Transform::Year => Ok(Box::new(temporal::Year {})),
+ Transform::Month => Ok(Box::new(temporal::Month {})),
+ Transform::Day => Ok(Box::new(temporal::Day {})),
+ Transform::Hour => Ok(Box::new(temporal::Hour {})),
+ _ => Err(crate::error::Error::new(
+ crate::ErrorKind::FeatureUnsupported,
+ format!("Transform {:?} is not implemented", transform),
+ )),
+ }
Review Comment:
Yes, this PR is mainly the framework of transform. Actually I've implement the Bucket in icelake.Is ok to implement them together in next PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] Fokko commented on a diff in pull request #42: feat: support transform function
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1310188384
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
+}
+
+/// BoxedTransformFunction is a boxed trait object of TransformFunction.
+pub type BoxedTransformFunction = Box<dyn TransformFunction>;
+
+/// create_transform_function creates a boxed trait object of TransformFunction from a Transform.
+pub fn create_transform_function(transform: &Transform) -> Result<BoxedTransformFunction> {
+ match transform {
+ Transform::Identity => Ok(Box::new(identity::Identity {})),
+ Transform::Void => Ok(Box::new(void::Void {})),
+ Transform::Year => Ok(Box::new(temporal::Year {})),
+ Transform::Month => Ok(Box::new(temporal::Month {})),
+ Transform::Day => Ok(Box::new(temporal::Day {})),
+ Transform::Hour => Ok(Box::new(temporal::Hour {})),
+ _ => Err(crate::error::Error::new(
+ crate::ErrorKind::FeatureUnsupported,
+ format!("Transform {:?} is not implemented", transform),
+ )),
+ }
Review Comment:
`Truncate` and `Bucket` are missing
##########
crates/iceberg/Cargo.toml:
##########
@@ -28,6 +28,7 @@ keywords = ["iceberg"]
[dependencies]
apache-avro = "0.15.0"
+arrow = { version = ">=46" }
Review Comment:
In PyIceberg we make Arrow an optional dependency because it is so big, idk if the rust implementation has the same problem.
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,343 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::TransformFunction;
+use crate::{Error, ErrorKind, Result};
+use arrow::array::{Array, TimestampMicrosecondArray};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// The number of days since unix epoch.
+const DAY_SINCE_UNIX_EPOCH: i32 = 719163;
+/// Hour in one second.
+const HOUR_PER_SECOND: f64 = 1.0_f64 / 3600.0_f64;
+/// Day in one second.
+const DAY_PER_SECOND: f64 = 1.0_f64 / 24.0_f64 / 3600.0_f64;
+/// Year of unix epoch.
+const UNIX_EPOCH_YEAR: i32 = 1970;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array =
+ year_dyn(&input).map_err(|err| Error::new(ErrorKind::Unexpected, format!("{err}")))?;
+ Ok(Arc::<Int32Array>::new(
+ array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| v - UNIX_EPOCH_YEAR),
+ ))
+ }
+}
+
+/// Extract a date or timestamp month, as months from 1970-01-01
+pub struct Month;
+
+impl TransformFunction for Month {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let year_array =
+ year_dyn(&input).map_err(|err| Error::new(ErrorKind::Unexpected, format!("{err}")))?;
+ let year_array: Int32Array = year_array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| 12 * (v - UNIX_EPOCH_YEAR));
+ let month_array =
+ month_dyn(&input).map_err(|err| Error::new(ErrorKind::Unexpected, format!("{err}")))?;
+ Ok(Arc::<Int32Array>::new(
+ binary(
+ month_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ year_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ // Compute month from 1970-01-01, so minus 1 here.
+ |a, b| a + b - 1,
+ )
+ .unwrap(),
+ ))
+ }
+}
+
+/// Extract a date or timestamp day, as days from 1970-01-01
+pub struct Day;
+
+impl TransformFunction for Day {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let res: Int32Array = match input.data_type() {
+ DataType::Timestamp(datatypes::TimeUnit::Microsecond, _) => input
+ .as_any()
+ .downcast_ref::<TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 / 1000.0 / 1000.0 * DAY_PER_SECOND) as i32 }),
+ DataType::Date32 => {
+ input
+ .as_any()
+ .downcast_ref::<Date32Array>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ datatypes::Date32Type::to_naive_date(v).num_days_from_ce()
+ - DAY_SINCE_UNIX_EPOCH
+ })
+ }
+ _ => {
+ return Err(Error::new(
+ ErrorKind::Unexpected,
+ format!(
+ "Should not call internally for unsupport data type {:?}",
+ input.data_type()
+ ),
+ ))
+ }
+ };
+ Ok(Arc::new(res))
+ }
+}
+
+/// Extract a timestamp hour, as hours from 1970-01-01 00:00:00
+pub struct Hour;
+
+impl TransformFunction for Hour {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let res: Int32Array = match input.data_type() {
+ DataType::Timestamp(datatypes::TimeUnit::Microsecond, _) => input
+ .as_any()
+ .downcast_ref::<TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 * HOUR_PER_SECOND / 1000.0 / 1000.0) as i32 }),
+ _ => {
+ return Err(Error::new(
+ ErrorKind::Unexpected,
+ format!(
+ "Should not call internally for unsupport data type {:?}",
+ input.data_type()
+ ),
+ ))
+ }
+ };
+ Ok(Arc::new(res))
+ }
+}
+
+#[cfg(test)]
+mod test {
+ use arrow::array::{ArrayRef, Date32Array, Int32Array, TimestampMicrosecondArray};
+ use chrono::NaiveDate;
+ use std::sync::Arc;
+
+ use crate::transform::TransformFunction;
+
+ #[test]
+ fn test_transform_years() {
+ let year = super::Year;
+ let ori_date = vec![
+ NaiveDate::from_ymd_opt(1970, 1, 1).unwrap(),
Review Comment:
Can we also add `timestamp`, `timestamptz`?
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,343 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::TransformFunction;
+use crate::{Error, ErrorKind, Result};
+use arrow::array::{Array, TimestampMicrosecondArray};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// The number of days since unix epoch.
+const DAY_SINCE_UNIX_EPOCH: i32 = 719163;
+/// Hour in one second.
+const HOUR_PER_SECOND: f64 = 1.0_f64 / 3600.0_f64;
+/// Day in one second.
+const DAY_PER_SECOND: f64 = 1.0_f64 / 24.0_f64 / 3600.0_f64;
+/// Year of unix epoch.
+const UNIX_EPOCH_YEAR: i32 = 1970;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array =
+ year_dyn(&input).map_err(|err| Error::new(ErrorKind::Unexpected, format!("{err}")))?;
+ Ok(Arc::<Int32Array>::new(
+ array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| v - UNIX_EPOCH_YEAR),
+ ))
+ }
+}
+
+/// Extract a date or timestamp month, as months from 1970-01-01
+pub struct Month;
+
+impl TransformFunction for Month {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let year_array =
+ year_dyn(&input).map_err(|err| Error::new(ErrorKind::Unexpected, format!("{err}")))?;
+ let year_array: Int32Array = year_array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| 12 * (v - UNIX_EPOCH_YEAR));
+ let month_array =
+ month_dyn(&input).map_err(|err| Error::new(ErrorKind::Unexpected, format!("{err}")))?;
+ Ok(Arc::<Int32Array>::new(
+ binary(
+ month_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ year_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ // Compute month from 1970-01-01, so minus 1 here.
+ |a, b| a + b - 1,
+ )
+ .unwrap(),
+ ))
+ }
+}
+
+/// Extract a date or timestamp day, as days from 1970-01-01
+pub struct Day;
+
+impl TransformFunction for Day {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let res: Int32Array = match input.data_type() {
+ DataType::Timestamp(datatypes::TimeUnit::Microsecond, _) => input
+ .as_any()
+ .downcast_ref::<TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 / 1000.0 / 1000.0 * DAY_PER_SECOND) as i32 }),
+ DataType::Date32 => {
+ input
+ .as_any()
+ .downcast_ref::<Date32Array>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ datatypes::Date32Type::to_naive_date(v).num_days_from_ce()
+ - DAY_SINCE_UNIX_EPOCH
+ })
+ }
+ _ => {
+ return Err(Error::new(
+ ErrorKind::Unexpected,
+ format!(
+ "Should not call internally for unsupport data type {:?}",
+ input.data_type()
+ ),
+ ))
+ }
+ };
+ Ok(Arc::new(res))
+ }
+}
+
+/// Extract a timestamp hour, as hours from 1970-01-01 00:00:00
+pub struct Hour;
+
+impl TransformFunction for Hour {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let res: Int32Array = match input.data_type() {
+ DataType::Timestamp(datatypes::TimeUnit::Microsecond, _) => input
+ .as_any()
+ .downcast_ref::<TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 * HOUR_PER_SECOND / 1000.0 / 1000.0) as i32 }),
+ _ => {
+ return Err(Error::new(
+ ErrorKind::Unexpected,
+ format!(
+ "Should not call internally for unsupport data type {:?}",
+ input.data_type()
+ ),
+ ))
+ }
+ };
+ Ok(Arc::new(res))
+ }
+}
+
+#[cfg(test)]
+mod test {
+ use arrow::array::{ArrayRef, Date32Array, Int32Array, TimestampMicrosecondArray};
+ use chrono::NaiveDate;
+ use std::sync::Arc;
+
+ use crate::transform::TransformFunction;
+
+ #[test]
+ fn test_transform_years() {
Review Comment:
These tests are great, thanks!
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
Review Comment:
I'm not a rust'er, but is this passing in an array?
Maybe some context on how this is implemented in Python (and the Java implementation also followed this path eventually). In PyIceberg we initialize a transform, but that one might be unbound. A common place to find a transform is in the partition spec: https://iceberg.apache.org/spec/#partition-specs
Here we have a reference to a field by the `source-id`. This means that we don't know yet what kind the input is. For example, you can bucket a string, uuid, long, int, datetime, etc. Once we bind the transform to the schema, we return the actual function as a callable (we pass a function):
```python
def transform(self, source: IcebergType, bucket: bool = True) -> Callable[[Optional[Any]], Optional[int]]:
source_type = type(source)
if source_type in {IntegerType, LongType, DateType, TimeType, TimestampType, TimestamptzType}:
def hash_func(v: Any) -> int:
return mmh3.hash(struct.pack("<q", v))
elif source_type == DecimalType:
def hash_func(v: Any) -> int:
return mmh3.hash(decimal_to_bytes(v))
elif source_type in {StringType, FixedType, BinaryType}:
def hash_func(v: Any) -> int:
return mmh3.hash(v)
elif source_type == UUIDType:
def hash_func(v: Any) -> int:
if isinstance(v, UUID):
return mmh3.hash(v.bytes)
return mmh3.hash(v)
else:
raise ValueError(f"Unknown type {source}")
if bucket:
return lambda v: (hash_func(v) & IntegerType.max) % self._num_buckets if v else None
return hash_func
```
This way we can initialize the transform, even when we don't know the type of the source type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] Xuanwo commented on a diff in pull request #42: feat: support transform function
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1308492711
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,329 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::TransformFunction;
+use crate::Result;
+use arrow::array::{Array, TimestampMicrosecondArray};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+const BASE_YEAR: i32 = 1970;
Review Comment:
Although the name itself is clear, I believe adding a brief explanation of its meaning would be beneficial.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1333783538
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
Review Comment:
> But I do think transform functions accepting single value would also be useful, maybe we can implement it in following pr?
+1. And it's better to integrate transform array and single value, I can do it when implementation of single value version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] liurenjie1024 commented on pull request #42: feat: support transform function
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#issuecomment-1704509109
> Hi, I have [implemented the bucket and truncate transform in icelake](https://github.com/icelake-io/icelake/pull/173) and I'm ready to port them. Do you prefer to port them in this PR or another PR. I'm concern whether that cause this PR so big that hard to review. Anyway, I'm ok with both way and it's up to you. @liurenjie1024 @Xuanwo @JanKaul @nastra @Fokko
Both are good to me. I prefer to put them in one pr so that it's easier to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] liurenjie1024 commented on a diff in pull request #42: feat: support transform function
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1308471018
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,329 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::TransformFunction;
+use crate::Result;
+use arrow::array::{Array, TimestampMicrosecondArray};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+const BASE_YEAR: i32 = 1970;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).expect("Should not call transform in Year with non-date type");
+ Ok(Arc::<Int32Array>::new(
+ array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| v - BASE_YEAR),
+ ))
+ }
+}
+
+/// Extract a date or timestamp month, as months from 1970-01-01
+pub struct Month;
+
+impl TransformFunction for Month {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let year_array =
+ year_dyn(&input).expect("Should not call transform in Month with non-date type");
Review Comment:
As above.
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,329 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::TransformFunction;
+use crate::Result;
+use arrow::array::{Array, TimestampMicrosecondArray};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+const BASE_YEAR: i32 = 1970;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).expect("Should not call transform in Year with non-date type");
Review Comment:
Hmm, should we really make it panic here? As a public interface, I don't think it's good practice to panic, it's better to return InvalidData error? cc @ZENOTME @Xuanwo
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,329 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::TransformFunction;
+use crate::Result;
+use arrow::array::{Array, TimestampMicrosecondArray};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+const BASE_YEAR: i32 = 1970;
Review Comment:
```suggestion
const EPOCH_YEAR: i32 = 1970;
```
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,329 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::TransformFunction;
+use crate::Result;
+use arrow::array::{Array, TimestampMicrosecondArray};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+const BASE_YEAR: i32 = 1970;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).expect("Should not call transform in Year with non-date type");
+ Ok(Arc::<Int32Array>::new(
+ array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| v - BASE_YEAR),
+ ))
+ }
+}
+
+/// Extract a date or timestamp month, as months from 1970-01-01
+pub struct Month;
+
+impl TransformFunction for Month {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let year_array =
+ year_dyn(&input).expect("Should not call transform in Month with non-date type");
+ let year_array: Int32Array = year_array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| 12 * (v - BASE_YEAR));
+ let month_array =
+ month_dyn(&input).expect("Should not call transform in Month with non-date type");
+ Ok(Arc::<Int32Array>::new(
+ binary(
+ month_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ year_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ // Compute month from 1970-01-01, so minus 1 here.
+ |a, b| a + b - 1,
+ )
+ .unwrap(),
+ ))
+ }
+}
+
+/// Extract a date or timestamp day, as days from 1970-01-01
+pub struct Day;
+
+impl TransformFunction for Day {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let res: Int32Array = match input.data_type() {
+ DataType::Timestamp(datatypes::TimeUnit::Microsecond, _) => input
+ .as_any()
+ .downcast_ref::<TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 / 1000.0 / 1000.0 * DAY_PER_SECOND) as i32 }),
+ DataType::Date32 => {
+ input
+ .as_any()
+ .downcast_ref::<Date32Array>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ datatypes::Date32Type::to_naive_date(v).num_days_from_ce()
+ - EPOCH_DAY_FROM_CE
+ })
+ }
+ _ => unreachable!(
+ "Should not call transform in Day with type {:?}",
+ input.data_type()
+ ),
+ };
+ Ok(Arc::new(res))
+ }
+}
+
+/// Extract a timestamp hour, as hours from 1970-01-01 00:00:00
+pub struct Hour;
+
+impl TransformFunction for Hour {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let res: Int32Array = match input.data_type() {
+ DataType::Timestamp(datatypes::TimeUnit::Microsecond, _) => input
+ .as_any()
+ .downcast_ref::<TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 * HOUR_PER_SECOND / 1000.0 / 1000.0) as i32 }),
+ _ => unreachable!(
Review Comment:
As above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] Xuanwo commented on a diff in pull request #42: feat: support transform function
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1308200120
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,669 @@
+use super::TransformFunction;
+use crate::{Error, Result};
+use arrow::array::{
+ Array, Date64Array, TimestampMicrosecondArray, TimestampMillisecondArray,
+ TimestampNanosecondArray, TimestampSecondArray,
+};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).map_err(|err| {
+ Error::new(
+ crate::ErrorKind::ArrowCompute,
Review Comment:
Using `unreachable!()` at place where users should never reached is a good idea to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1325276422
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
Review Comment:
> When a file is being written, Iceberg keeps the upper and lower bounds of each of the columns. By default, a truncate(16) transform is applied before storing the metrics in the manifest.
Did you mean [the lower bound(upper bound) in field summary](https://iceberg.apache.org/spec/#manifests:~:text=NaN%20value%20for%20the%20field-,optional,field%2C%20or%20null%20if%20all%20values%20are%20null%20or%20NaN%20%5B2%5D,-optional)? So it's the lower bound of original column and then apply transform to it `transform(lower_bound(col))` rather than lower bound of transform column `lower_bound(transform(col))`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] liurenjie1024 commented on a diff in pull request #42: feat: support transform function
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1311060057
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
+}
+
+/// BoxedTransformFunction is a boxed trait object of TransformFunction.
+pub type BoxedTransformFunction = Box<dyn TransformFunction>;
+
+/// create_transform_function creates a boxed trait object of TransformFunction from a Transform.
+pub fn create_transform_function(transform: &Transform) -> Result<BoxedTransformFunction> {
+ match transform {
+ Transform::Identity => Ok(Box::new(identity::Identity {})),
+ Transform::Void => Ok(Box::new(void::Void {})),
+ Transform::Year => Ok(Box::new(temporal::Year {})),
+ Transform::Month => Ok(Box::new(temporal::Month {})),
+ Transform::Day => Ok(Box::new(temporal::Day {})),
+ Transform::Hour => Ok(Box::new(temporal::Hour {})),
+ _ => Err(crate::error::Error::new(
+ crate::ErrorKind::FeatureUnsupported,
+ format!("Transform {:?} is not implemented", transform),
+ )),
+ }
Review Comment:
I think it's left to implement in following pr? cc @ZENOTME
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] liurenjie1024 commented on a diff in pull request #42: feat: support transform function
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1308133680
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,669 @@
+use super::TransformFunction;
+use crate::{Error, Result};
+use arrow::array::{
+ Array, Date64Array, TimestampMicrosecondArray, TimestampMillisecondArray,
+ TimestampNanosecondArray, TimestampSecondArray,
+};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).map_err(|err| {
+ Error::new(
+ crate::ErrorKind::ArrowCompute,
+ format!("error in transformfunction: {}", err),
+ )
+ })?;
+ Ok(Arc::<Int32Array>::new(
+ array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| v - 1970),
Review Comment:
Please replace `1970` with some constants.
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,669 @@
+use super::TransformFunction;
+use crate::{Error, Result};
+use arrow::array::{
+ Array, Date64Array, TimestampMicrosecondArray, TimestampMillisecondArray,
+ TimestampNanosecondArray, TimestampSecondArray,
+};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).map_err(|err| {
+ Error::new(
+ crate::ErrorKind::ArrowCompute,
+ format!("error in transformfunction: {}", err),
+ )
+ })?;
+ Ok(Arc::<Int32Array>::new(
+ array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| v - 1970),
+ ))
+ }
+}
+
+/// Extract a date or timestamp month, as months from 1970-01-01
+pub struct Month;
+
+impl TransformFunction for Month {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let year_array = year_dyn(&input)
+ .map_err(|err| Error::new(crate::ErrorKind::ArrowCompute, format!("{err}")))?;
+ let year_array: Int32Array = year_array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| 12 * (v - 1970));
Review Comment:
As above.
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,669 @@
+use super::TransformFunction;
+use crate::{Error, Result};
+use arrow::array::{
+ Array, Date64Array, TimestampMicrosecondArray, TimestampMillisecondArray,
+ TimestampNanosecondArray, TimestampSecondArray,
+};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).map_err(|err| {
+ Error::new(
+ crate::ErrorKind::ArrowCompute,
+ format!("error in transformfunction: {}", err),
+ )
+ })?;
+ Ok(Arc::<Int32Array>::new(
+ array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| v - 1970),
+ ))
+ }
+}
+
+/// Extract a date or timestamp month, as months from 1970-01-01
+pub struct Month;
+
+impl TransformFunction for Month {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let year_array = year_dyn(&input)
+ .map_err(|err| Error::new(crate::ErrorKind::ArrowCompute, format!("{err}")))?;
+ let year_array: Int32Array = year_array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| 12 * (v - 1970));
+ let month_array = month_dyn(&input)
+ .map_err(|err| Error::new(crate::ErrorKind::ArrowCompute, format!("{err}")))?;
+ Ok(Arc::<Int32Array>::new(
+ binary(
+ month_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ year_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ // Compute month from 1970-01-01, so minus 1 here.
+ |a, b| a + b - 1,
+ )
+ .unwrap(),
+ ))
+ }
+}
+
+/// Extract a date or timestamp day, as days from 1970-01-01
+pub struct Day;
+
+impl TransformFunction for Day {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let res: Int32Array = match input.data_type() {
+ DataType::Timestamp(unit, _) => match unit {
+ datatypes::TimeUnit::Second => input
+ .as_any()
+ .downcast_ref::<TimestampSecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 * DAY_PER_SECOND) as i32 }),
+ datatypes::TimeUnit::Millisecond => input
+ .as_any()
+ .downcast_ref::<TimestampMillisecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 / 1000.0 * DAY_PER_SECOND) as i32 }),
+ datatypes::TimeUnit::Microsecond => input
+ .as_any()
+ .downcast_ref::<TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 / 1000.0 / 1000.0 * DAY_PER_SECOND) as i32 }),
+ datatypes::TimeUnit::Nanosecond => input
+ .as_any()
+ .downcast_ref::<TimestampNanosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ (v as f64 / 1000.0 / 1000.0 / 1000.0 * DAY_PER_SECOND) as i32
+ }),
+ },
+ DataType::Date32 => {
+ input
+ .as_any()
+ .downcast_ref::<Date32Array>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ datatypes::Date32Type::to_naive_date(v).num_days_from_ce()
+ - EPOCH_DAY_FROM_CE
+ })
+ }
+ DataType::Date64 => {
+ input
+ .as_any()
+ .downcast_ref::<Date64Array>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ datatypes::Date64Type::to_naive_date(v).num_days_from_ce()
+ - EPOCH_DAY_FROM_CE
+ })
+ }
+ _ => unreachable!(
+ "Should not call transform in Day with type {:?}",
+ input.data_type()
+ ),
+ };
+ Ok(Arc::new(res))
Review Comment:
Missing timestamp with timezone?
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,669 @@
+use super::TransformFunction;
+use crate::{Error, Result};
+use arrow::array::{
+ Array, Date64Array, TimestampMicrosecondArray, TimestampMillisecondArray,
+ TimestampNanosecondArray, TimestampSecondArray,
+};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).map_err(|err| {
+ Error::new(
+ crate::ErrorKind::ArrowCompute,
+ format!("error in transformfunction: {}", err),
+ )
+ })?;
+ Ok(Arc::<Int32Array>::new(
+ array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| v - 1970),
+ ))
+ }
+}
+
+/// Extract a date or timestamp month, as months from 1970-01-01
+pub struct Month;
+
+impl TransformFunction for Month {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let year_array = year_dyn(&input)
+ .map_err(|err| Error::new(crate::ErrorKind::ArrowCompute, format!("{err}")))?;
+ let year_array: Int32Array = year_array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| 12 * (v - 1970));
+ let month_array = month_dyn(&input)
+ .map_err(|err| Error::new(crate::ErrorKind::ArrowCompute, format!("{err}")))?;
+ Ok(Arc::<Int32Array>::new(
+ binary(
+ month_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ year_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ // Compute month from 1970-01-01, so minus 1 here.
+ |a, b| a + b - 1,
+ )
+ .unwrap(),
+ ))
+ }
+}
+
+/// Extract a date or timestamp day, as days from 1970-01-01
+pub struct Day;
+
+impl TransformFunction for Day {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let res: Int32Array = match input.data_type() {
+ DataType::Timestamp(unit, _) => match unit {
+ datatypes::TimeUnit::Second => input
+ .as_any()
+ .downcast_ref::<TimestampSecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 * DAY_PER_SECOND) as i32 }),
+ datatypes::TimeUnit::Millisecond => input
Review Comment:
Currently we iceberg only supports microseconds timestamp, we should remove unsupport data types.
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,669 @@
+use super::TransformFunction;
+use crate::{Error, Result};
+use arrow::array::{
+ Array, Date64Array, TimestampMicrosecondArray, TimestampMillisecondArray,
+ TimestampNanosecondArray, TimestampSecondArray,
+};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).map_err(|err| {
+ Error::new(
+ crate::ErrorKind::ArrowCompute,
+ format!("error in transformfunction: {}", err),
+ )
+ })?;
+ Ok(Arc::<Int32Array>::new(
+ array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| v - 1970),
+ ))
+ }
+}
+
+/// Extract a date or timestamp month, as months from 1970-01-01
+pub struct Month;
+
+impl TransformFunction for Month {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let year_array = year_dyn(&input)
+ .map_err(|err| Error::new(crate::ErrorKind::ArrowCompute, format!("{err}")))?;
+ let year_array: Int32Array = year_array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| 12 * (v - 1970));
+ let month_array = month_dyn(&input)
+ .map_err(|err| Error::new(crate::ErrorKind::ArrowCompute, format!("{err}")))?;
+ Ok(Arc::<Int32Array>::new(
+ binary(
+ month_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ year_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ // Compute month from 1970-01-01, so minus 1 here.
+ |a, b| a + b - 1,
+ )
+ .unwrap(),
+ ))
+ }
+}
+
+/// Extract a date or timestamp day, as days from 1970-01-01
+pub struct Day;
+
+impl TransformFunction for Day {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let res: Int32Array = match input.data_type() {
+ DataType::Timestamp(unit, _) => match unit {
+ datatypes::TimeUnit::Second => input
+ .as_any()
+ .downcast_ref::<TimestampSecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 * DAY_PER_SECOND) as i32 }),
+ datatypes::TimeUnit::Millisecond => input
+ .as_any()
+ .downcast_ref::<TimestampMillisecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 / 1000.0 * DAY_PER_SECOND) as i32 }),
+ datatypes::TimeUnit::Microsecond => input
+ .as_any()
+ .downcast_ref::<TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 / 1000.0 / 1000.0 * DAY_PER_SECOND) as i32 }),
+ datatypes::TimeUnit::Nanosecond => input
+ .as_any()
+ .downcast_ref::<TimestampNanosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ (v as f64 / 1000.0 / 1000.0 / 1000.0 * DAY_PER_SECOND) as i32
+ }),
+ },
+ DataType::Date32 => {
+ input
+ .as_any()
+ .downcast_ref::<Date32Array>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ datatypes::Date32Type::to_naive_date(v).num_days_from_ce()
+ - EPOCH_DAY_FROM_CE
+ })
+ }
+ DataType::Date64 => {
Review Comment:
Currenlty only 32bit dates are supported, remove this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] Xuanwo commented on a diff in pull request #42: feat: support transform function
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1308493879
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,329 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::TransformFunction;
+use crate::Result;
+use arrow::array::{Array, TimestampMicrosecondArray};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+const BASE_YEAR: i32 = 1970;
Review Comment:
For example, I spent some time to understand what's `DAY_PER_SECOND`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1311098421
##########
crates/iceberg/Cargo.toml:
##########
@@ -28,6 +28,7 @@ keywords = ["iceberg"]
[dependencies]
apache-avro = "0.15.0"
+arrow = { version = ">=46" }
Review Comment:
Seems compute module is not seperate as a crate and we have to import arrow if we want to use it.🥵 https://docs.rs/arrow/latest/arrow/compute/index.html
##########
crates/iceberg/Cargo.toml:
##########
@@ -28,6 +28,7 @@ keywords = ["iceberg"]
[dependencies]
apache-avro = "0.15.0"
+arrow = { version = ">=46" }
Review Comment:
Seems compute module is not seperate as a crate and we have to import arrow if we want to use it.🥵 https://docs.rs/arrow/latest/arrow/compute/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] Fokko commented on a diff in pull request #42: feat: support transform function
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1324166782
##########
crates/iceberg/src/transform/truncate.rs:
##########
@@ -0,0 +1,177 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::ArrayRef;
+use arrow_schema::DataType;
+
+use crate::Error;
+
+use super::TransformFunction;
+
+pub struct Truncate {
+ width: u32,
+}
+
+impl Truncate {
+ pub fn new(width: u32) -> Self {
+ Self { width }
+ }
+}
+
+impl TransformFunction for Truncate {
+ fn transform(&self, input: ArrayRef) -> crate::Result<ArrayRef> {
+ match input.data_type() {
+ DataType::Int32 => {
+ let width: i32 = self.width.try_into().map_err(|_| {
+ Error::new(
+ crate::ErrorKind::DataInvalid,
+ "width is failed to convert to i32 when truncate Int32Array",
+ )
+ })?;
+ let res: arrow_array::Int32Array = input
+ .as_any()
+ .downcast_ref::<arrow_array::Int32Array>()
+ .unwrap()
+ .unary(|v| v - v.rem_euclid(width));
+ Ok(Arc::new(res))
+ }
+ DataType::Int64 => {
+ let width = self.width as i64;
+ let res: arrow_array::Int64Array = input
+ .as_any()
+ .downcast_ref::<arrow_array::Int64Array>()
+ .unwrap()
+ .unary(|v| v - (((v % width) + width) % width));
+ Ok(Arc::new(res))
+ }
+ DataType::Decimal128(precision, scale) => {
+ let width = self.width as i128;
+ let res: arrow_array::Decimal128Array = input
+ .as_any()
+ .downcast_ref::<arrow_array::Decimal128Array>()
+ .unwrap()
+ .unary(|v| v - (((v % width) + width) % width))
+ .with_precision_and_scale(*precision, *scale)
+ .map_err(|err| Error::new(crate::ErrorKind::Unexpected, format!("{err}")))?;
+ Ok(Arc::new(res))
+ }
+ DataType::Utf8 => {
+ let len = self.width as usize;
+ let res: arrow_array::StringArray = arrow_array::StringArray::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::StringArray>()
+ .unwrap()
+ .iter()
+ .map(|v| v.map(|v| &v[..len])),
Review Comment:
Unfortunately we have to make sure that we don't break the Unicode encoding:
https://github.com/apache/iceberg/blob/master/api/src/main/java/org/apache/iceberg/util/UnicodeUtil.java#L36-L55
Here are some tests that might help: https://github.com/apache/iceberg/blob/master/core/src/test/java/org/apache/iceberg/TestMetricsTruncation.java#L145
##########
crates/iceberg/src/transform/bucket.rs:
##########
@@ -0,0 +1,227 @@
+use std::sync::Arc;
+
+use arrow_array::ArrayRef;
+use arrow_schema::{DataType, TimeUnit};
+
+use super::TransformFunction;
+
+pub struct Bucket {
+ mod_n: u32,
+}
+
+impl Bucket {
+ pub fn new(mod_n: u32) -> Self {
+ Self { mod_n }
+ }
+}
+
+impl Bucket {
+ /// When switch the hash function, we only need to change this function.
+ fn hash_bytes(mut v: &[u8]) -> i32 {
+ murmur3::murmur3_32(&mut v, 0).unwrap() as i32
+ }
+
+ fn hash_int(v: i32) -> i32 {
+ Self::hash_long(v as i64)
+ }
+
+ fn hash_long(v: i64) -> i32 {
+ Self::hash_bytes(v.to_le_bytes().as_slice())
+ }
+
+ /// v is days from unix epoch
+ fn hash_date(v: i32) -> i32 {
+ Self::hash_int(v)
+ }
+
+ /// v is microseconds from midnight
+ fn hash_time(v: i64) -> i32 {
+ Self::hash_long(v)
+ }
+
+ /// v is microseconds from unix epoch
+ fn hash_timestamp(v: i64) -> i32 {
+ Self::hash_long(v)
+ }
+
+ fn hash_str(s: &str) -> i32 {
+ Self::hash_bytes(s.as_bytes())
+ }
+
+ /// Decimal values are hashed using the minimum number of bytes required to hold the unscaled value as a two’s complement big-endian
+ /// ref: https://iceberg.apache.org/spec/#appendix-b-32-bit-hash-requirements
+ fn hash_decimal(v: i128) -> i32 {
+ let bytes = v.to_be_bytes();
+ if let Some(start) = bytes.iter().position(|&x| x != 0) {
+ Self::hash_bytes(&bytes[start..])
+ } else {
+ Self::hash_bytes(&[0])
+ }
+ }
+
+ /// def bucket_N(x) = (murmur3_x86_32_hash(x) & Integer.MAX_VALUE) % N
+ /// ref: https://iceberg.apache.org/spec/#partitioning
+ fn bucket_n(&self, v: i32) -> i32 {
+ (v & i32::MAX) % (self.mod_n as i32)
+ }
+}
+
+impl TransformFunction for Bucket {
+ fn transform(&self, input: ArrayRef) -> crate::Result<ArrayRef> {
+ let res: arrow_array::Int32Array = match input.data_type() {
+ DataType::Int32 => input
+ .as_any()
+ .downcast_ref::<arrow_array::Int32Array>()
+ .unwrap()
+ .unary(|v| self.bucket_n(Self::hash_int(v))),
+ DataType::Int64 => input
+ .as_any()
+ .downcast_ref::<arrow_array::Int64Array>()
+ .unwrap()
+ .unary(|v| self.bucket_n(Self::hash_long(v))),
+ DataType::Decimal128(_, _) => input
+ .as_any()
+ .downcast_ref::<arrow_array::Decimal128Array>()
+ .unwrap()
+ .unary(|v| self.bucket_n(Self::hash_decimal(v))),
+ DataType::Date32 => input
+ .as_any()
+ .downcast_ref::<arrow_array::Date32Array>()
+ .unwrap()
+ .unary(|v| self.bucket_n(Self::hash_date(v))),
+ DataType::Time64(TimeUnit::Microsecond) => input
+ .as_any()
+ .downcast_ref::<arrow_array::Time64MicrosecondArray>()
+ .unwrap()
+ .unary(|v| self.bucket_n(Self::hash_time(v))),
+ DataType::Timestamp(TimeUnit::Microsecond, _) => input
+ .as_any()
+ .downcast_ref::<arrow_array::TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| self.bucket_n(Self::hash_timestamp(v))),
+ DataType::Utf8 => arrow_array::Int32Array::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::StringArray>()
+ .unwrap()
+ .iter()
+ .map(|v| self.bucket_n(Self::hash_str(v.unwrap()))),
+ ),
+ DataType::LargeUtf8 => arrow_array::Int32Array::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::LargeStringArray>()
+ .unwrap()
+ .iter()
+ .map(|v| self.bucket_n(Self::hash_str(v.unwrap()))),
+ ),
+ DataType::Binary => arrow_array::Int32Array::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::BinaryArray>()
+ .unwrap()
+ .iter()
+ .map(|v| self.bucket_n(Self::hash_bytes(v.unwrap()))),
+ ),
+ DataType::LargeBinary => arrow_array::Int32Array::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::LargeBinaryArray>()
+ .unwrap()
+ .iter()
+ .map(|v| self.bucket_n(Self::hash_bytes(v.unwrap()))),
+ ),
+ DataType::FixedSizeBinary(_) => arrow_array::Int32Array::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::FixedSizeBinaryArray>()
+ .unwrap()
+ .iter()
+ .map(|v| self.bucket_n(Self::hash_bytes(v.unwrap()))),
+ ),
+ _ => unreachable!("Unsupported data type: {:?}", input.data_type()),
+ };
+ Ok(Arc::new(res))
+ }
+}
+
+#[cfg(test)]
+mod test {
+ use chrono::{DateTime, NaiveDate, NaiveDateTime, NaiveTime};
+
+ use super::Bucket;
+ #[test]
+ fn test_hash() {
Review Comment:
Thanks or adding these! 👍🏻
##########
crates/iceberg/src/transform/bucket.rs:
##########
@@ -0,0 +1,227 @@
+use std::sync::Arc;
+
+use arrow_array::ArrayRef;
+use arrow_schema::{DataType, TimeUnit};
+
+use super::TransformFunction;
+
+pub struct Bucket {
+ mod_n: u32,
+}
+
+impl Bucket {
+ pub fn new(mod_n: u32) -> Self {
+ Self { mod_n }
+ }
+}
+
+impl Bucket {
+ /// When switch the hash function, we only need to change this function.
+ fn hash_bytes(mut v: &[u8]) -> i32 {
+ murmur3::murmur3_32(&mut v, 0).unwrap() as i32
+ }
+
+ fn hash_int(v: i32) -> i32 {
+ Self::hash_long(v as i64)
+ }
+
+ fn hash_long(v: i64) -> i32 {
+ Self::hash_bytes(v.to_le_bytes().as_slice())
+ }
+
+ /// v is days from unix epoch
+ fn hash_date(v: i32) -> i32 {
+ Self::hash_int(v)
+ }
+
+ /// v is microseconds from midnight
+ fn hash_time(v: i64) -> i32 {
+ Self::hash_long(v)
+ }
+
+ /// v is microseconds from unix epoch
+ fn hash_timestamp(v: i64) -> i32 {
+ Self::hash_long(v)
+ }
+
+ fn hash_str(s: &str) -> i32 {
+ Self::hash_bytes(s.as_bytes())
+ }
+
+ /// Decimal values are hashed using the minimum number of bytes required to hold the unscaled value as a two’s complement big-endian
+ /// ref: https://iceberg.apache.org/spec/#appendix-b-32-bit-hash-requirements
+ fn hash_decimal(v: i128) -> i32 {
+ let bytes = v.to_be_bytes();
+ if let Some(start) = bytes.iter().position(|&x| x != 0) {
+ Self::hash_bytes(&bytes[start..])
+ } else {
+ Self::hash_bytes(&[0])
+ }
+ }
+
+ /// def bucket_N(x) = (murmur3_x86_32_hash(x) & Integer.MAX_VALUE) % N
+ /// ref: https://iceberg.apache.org/spec/#partitioning
+ fn bucket_n(&self, v: i32) -> i32 {
+ (v & i32::MAX) % (self.mod_n as i32)
+ }
+}
+
+impl TransformFunction for Bucket {
+ fn transform(&self, input: ArrayRef) -> crate::Result<ArrayRef> {
+ let res: arrow_array::Int32Array = match input.data_type() {
+ DataType::Int32 => input
+ .as_any()
+ .downcast_ref::<arrow_array::Int32Array>()
+ .unwrap()
+ .unary(|v| self.bucket_n(Self::hash_int(v))),
+ DataType::Int64 => input
+ .as_any()
+ .downcast_ref::<arrow_array::Int64Array>()
+ .unwrap()
+ .unary(|v| self.bucket_n(Self::hash_long(v))),
+ DataType::Decimal128(_, _) => input
+ .as_any()
+ .downcast_ref::<arrow_array::Decimal128Array>()
+ .unwrap()
+ .unary(|v| self.bucket_n(Self::hash_decimal(v))),
+ DataType::Date32 => input
+ .as_any()
+ .downcast_ref::<arrow_array::Date32Array>()
+ .unwrap()
+ .unary(|v| self.bucket_n(Self::hash_date(v))),
+ DataType::Time64(TimeUnit::Microsecond) => input
+ .as_any()
+ .downcast_ref::<arrow_array::Time64MicrosecondArray>()
+ .unwrap()
+ .unary(|v| self.bucket_n(Self::hash_time(v))),
+ DataType::Timestamp(TimeUnit::Microsecond, _) => input
+ .as_any()
+ .downcast_ref::<arrow_array::TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| self.bucket_n(Self::hash_timestamp(v))),
+ DataType::Utf8 => arrow_array::Int32Array::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::StringArray>()
+ .unwrap()
+ .iter()
+ .map(|v| self.bucket_n(Self::hash_str(v.unwrap()))),
+ ),
+ DataType::LargeUtf8 => arrow_array::Int32Array::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::LargeStringArray>()
+ .unwrap()
+ .iter()
+ .map(|v| self.bucket_n(Self::hash_str(v.unwrap()))),
+ ),
+ DataType::Binary => arrow_array::Int32Array::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::BinaryArray>()
+ .unwrap()
+ .iter()
+ .map(|v| self.bucket_n(Self::hash_bytes(v.unwrap()))),
+ ),
+ DataType::LargeBinary => arrow_array::Int32Array::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::LargeBinaryArray>()
+ .unwrap()
+ .iter()
+ .map(|v| self.bucket_n(Self::hash_bytes(v.unwrap()))),
+ ),
+ DataType::FixedSizeBinary(_) => arrow_array::Int32Array::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::FixedSizeBinaryArray>()
+ .unwrap()
+ .iter()
+ .map(|v| self.bucket_n(Self::hash_bytes(v.unwrap()))),
+ ),
+ _ => unreachable!("Unsupported data type: {:?}", input.data_type()),
+ };
+ Ok(Arc::new(res))
+ }
+}
+
+#[cfg(test)]
+mod test {
+ use chrono::{DateTime, NaiveDate, NaiveDateTime, NaiveTime};
+
+ use super::Bucket;
+ #[test]
+ fn test_hash() {
Review Comment:
Thanks or adding these! 👍🏻
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
Review Comment:
Transforms are used in many places.
> For example when we want to execute select * from t where a = 1 and b=2, and table t is partitioned by a, we can prune unnecessary partitions by calculating a=1. In this case, we need to use iceberg value.
This brings me back to my original question. It is just a single value, and vectorization would not help much. Wouldn't it be easier to operate it on a single value?
Keep in mind that transforms are also used when writing data. When a file is being written, Iceberg keeps the upper and lower bounds of each of the columns. By default, a `truncate(16)` transform is applied before storing the metrics in the manifest.
In PyIceberg, we use Arrow to write: https://github.com/apache/iceberg/pull/7831 Then we use Arrows' metadata collector to get the lower and upper bound of each row group. Which we then store in the manifest file. I'm not saying that it should be the same in 🦀 , but just for the context.
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
Review Comment:
Transforms are used in many places.
> For example when we want to execute select * from t where a = 1 and b=2, and table t is partitioned by a, we can prune unnecessary partitions by calculating a=1. In this case, we need to use iceberg value.
This brings me back to my original question. It is just a single value, and vectorization would not help much. Wouldn't it be easier to operate it on a single value?
Keep in mind that transforms are also used when writing data. When a file is being written, Iceberg keeps the upper and lower bounds of each of the columns. By default, a `truncate(16)` transform is applied before storing the metrics in the manifest.
In PyIceberg, we use Arrow to write: https://github.com/apache/iceberg/pull/7831 Then we use Arrows' metadata collector to get the lower and upper bound of each row group. Which we then store in the manifest file. I'm not saying that it should be the same in 🦀 , but just for the context.
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
+}
+
+/// BoxedTransformFunction is a boxed trait object of TransformFunction.
+pub type BoxedTransformFunction = Box<dyn TransformFunction>;
+
+/// create_transform_function creates a boxed trait object of TransformFunction from a Transform.
+pub fn create_transform_function(transform: &Transform) -> Result<BoxedTransformFunction> {
+ match transform {
+ Transform::Identity => Ok(Box::new(identity::Identity {})),
+ Transform::Void => Ok(Box::new(void::Void {})),
+ Transform::Year => Ok(Box::new(temporal::Year {})),
+ Transform::Month => Ok(Box::new(temporal::Month {})),
+ Transform::Day => Ok(Box::new(temporal::Day {})),
+ Transform::Hour => Ok(Box::new(temporal::Hour {})),
+ _ => Err(crate::error::Error::new(
+ crate::ErrorKind::FeatureUnsupported,
+ format!("Transform {:?} is not implemented", transform),
+ )),
+ }
Review Comment:
I prefer to have them all in one PR, thanks! 👍🏻
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
+}
+
+/// BoxedTransformFunction is a boxed trait object of TransformFunction.
+pub type BoxedTransformFunction = Box<dyn TransformFunction>;
+
+/// create_transform_function creates a boxed trait object of TransformFunction from a Transform.
+pub fn create_transform_function(transform: &Transform) -> Result<BoxedTransformFunction> {
+ match transform {
+ Transform::Identity => Ok(Box::new(identity::Identity {})),
+ Transform::Void => Ok(Box::new(void::Void {})),
+ Transform::Year => Ok(Box::new(temporal::Year {})),
+ Transform::Month => Ok(Box::new(temporal::Month {})),
+ Transform::Day => Ok(Box::new(temporal::Day {})),
+ Transform::Hour => Ok(Box::new(temporal::Hour {})),
+ _ => Err(crate::error::Error::new(
+ crate::ErrorKind::FeatureUnsupported,
+ format!("Transform {:?} is not implemented", transform),
+ )),
+ }
Review Comment:
I prefer to have them all in one PR, thanks! 👍🏻
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] Xuanwo commented on a diff in pull request #42: feat: support transform function
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1308159187
##########
crates/iceberg/src/error.rs:
##########
@@ -44,6 +44,10 @@ pub enum ErrorKind {
///
/// This error is returned when given iceberg feature is not supported.
FeatureUnsupported,
+ /// Arrow compute error.
+ ///
+ /// Error returned by compute function in arrow.
+ ArrowCompute,
Review Comment:
`ArrowCompute` is not an error code that can take actions. How about remove it or replace it by an actionable error code.
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,669 @@
+use super::TransformFunction;
+use crate::{Error, Result};
+use arrow::array::{
+ Array, Date64Array, TimestampMicrosecondArray, TimestampMillisecondArray,
+ TimestampNanosecondArray, TimestampSecondArray,
+};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).map_err(|err| {
+ Error::new(
+ crate::ErrorKind::ArrowCompute,
Review Comment:
`ArrowCompute` is not an error code that can take actions.
The acutal error here is `{name} does not support`: https://arrow.apache.org/rust/src/arrow_arith/temporal.rs.html#452. But arrow will wrap all compute time error into `ArrowError::Compute`, so there is no good way for us to extract them.
How about using `Unexpected` here? There is no action for our users to take.
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,669 @@
+use super::TransformFunction;
+use crate::{Error, Result};
+use arrow::array::{
+ Array, Date64Array, TimestampMicrosecondArray, TimestampMillisecondArray,
+ TimestampNanosecondArray, TimestampSecondArray,
+};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).map_err(|err| {
+ Error::new(
+ crate::ErrorKind::ArrowCompute,
+ format!("error in transformfunction: {}", err),
Review Comment:
```suggestion
format!("error in transform function: {}", err),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1308558571
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,329 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::TransformFunction;
+use crate::Result;
+use arrow::array::{Array, TimestampMicrosecondArray};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+const BASE_YEAR: i32 = 1970;
Review Comment:
I try to make it more clean and self-explain without comment. Please let me know is it still not clear enogh and need comment. @Xuanwo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#issuecomment-1695650618
cc @liurenjie1024 @Xuanwo @JanKaul @nastra @Fokko PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] liurenjie1024 commented on a diff in pull request #42: feat: support transform function
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1331297255
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
Review Comment:
Hi, @Fokko For more background on why we use an array can be found here: https://github.com/apache/iceberg-rust/issues/32#issuecomment-1697308710.
But I do think transform functions accepting single value would also be useful, maybe we can implement it in following pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1320926777
##########
crates/iceberg/src/transform/truncate.rs:
##########
@@ -0,0 +1,177 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::ArrayRef;
+use arrow_schema::DataType;
+
+use crate::Error;
+
+use super::TransformFunction;
+
+pub struct Truncate {
+ width: u32,
+}
+
+impl Truncate {
+ pub fn new(width: u32) -> Self {
+ Self { width }
+ }
+}
+
+impl TransformFunction for Truncate {
+ fn transform(&self, input: ArrayRef) -> crate::Result<ArrayRef> {
+ match input.data_type() {
+ DataType::Int32 => {
+ let width: i32 = self.width.try_into().map_err(|_| {
+ Error::new(
+ crate::ErrorKind::DataInvalid,
+ "width is failed to convert to i32 when truncate Int32Array",
+ )
+ })?;
+ let res: arrow_array::Int32Array = input
+ .as_any()
+ .downcast_ref::<arrow_array::Int32Array>()
+ .unwrap()
+ .unary(|v| v - (((v % width) + width) % width));
Review Comment:
Thanks! `v.rem_euclid` is more clear!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] Fokko commented on a diff in pull request #42: feat: support transform function
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1333384560
##########
crates/iceberg/src/transform/truncate.rs:
##########
@@ -0,0 +1,217 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::ArrayRef;
+use arrow_schema::DataType;
+
+use crate::Error;
+
+use super::TransformFunction;
+
+pub struct Truncate {
+ width: u32,
+}
+
+impl Truncate {
+ pub fn new(width: u32) -> Self {
+ Self { width }
+ }
+
+ fn truncate_str_by_char(s: &str, max_chars: usize) -> &str {
+ match s.char_indices().nth(max_chars) {
+ None => s,
+ Some((idx, _)) => &s[..idx],
+ }
+ }
+}
+
+impl TransformFunction for Truncate {
+ fn transform(&self, input: ArrayRef) -> crate::Result<ArrayRef> {
+ match input.data_type() {
+ DataType::Int32 => {
+ let width: i32 = self.width.try_into().map_err(|_| {
+ Error::new(
+ crate::ErrorKind::DataInvalid,
+ "width is failed to convert to i32 when truncate Int32Array",
+ )
+ })?;
+ let res: arrow_array::Int32Array = input
+ .as_any()
+ .downcast_ref::<arrow_array::Int32Array>()
+ .unwrap()
+ .unary(|v| v - v.rem_euclid(width));
+ Ok(Arc::new(res))
+ }
+ DataType::Int64 => {
+ let width = self.width as i64;
+ let res: arrow_array::Int64Array = input
+ .as_any()
+ .downcast_ref::<arrow_array::Int64Array>()
+ .unwrap()
+ .unary(|v| v - (((v % width) + width) % width));
+ Ok(Arc::new(res))
+ }
+ DataType::Decimal128(precision, scale) => {
+ let width = self.width as i128;
+ let res: arrow_array::Decimal128Array = input
+ .as_any()
+ .downcast_ref::<arrow_array::Decimal128Array>()
+ .unwrap()
+ .unary(|v| v - (((v % width) + width) % width))
+ .with_precision_and_scale(*precision, *scale)
+ .map_err(|err| Error::new(crate::ErrorKind::Unexpected, format!("{err}")))?;
+ Ok(Arc::new(res))
+ }
+ DataType::Utf8 => {
+ let len = self.width as usize;
+ let res: arrow_array::StringArray = arrow_array::StringArray::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::StringArray>()
+ .unwrap()
+ .iter()
+ .map(|v| v.map(|v| Self::truncate_str_by_char(v, len))),
+ );
+ Ok(Arc::new(res))
+ }
+ DataType::LargeUtf8 => {
+ let len = self.width as usize;
+ let res: arrow_array::LargeStringArray = arrow_array::LargeStringArray::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::LargeStringArray>()
+ .unwrap()
+ .iter()
+ .map(|v| v.map(|v| Self::truncate_str_by_char(v, len))),
+ );
+ Ok(Arc::new(res))
+ }
+ _ => unreachable!("Truncate transform only supports (int,long,decimal,string) types"),
+ }
+ }
+}
+
+#[cfg(test)]
+mod test {
+ use std::sync::Arc;
+
+ use arrow_array::{
+ builder::PrimitiveBuilder, types::Decimal128Type, Decimal128Array, Int32Array, Int64Array,
+ };
+
+ use crate::transform::TransformFunction;
+
+ // Test case ref from: https://iceberg.apache.org/spec/#truncate-transform-details
+ #[test]
+ fn test_truncate_simple() {
+ // test truncate int
+ let input = Arc::new(Int32Array::from(vec![1, -1]));
+ let res = super::Truncate::new(10).transform(input).unwrap();
+ assert_eq!(
+ res.as_any().downcast_ref::<Int32Array>().unwrap().value(0),
+ 0
+ );
+ assert_eq!(
+ res.as_any().downcast_ref::<Int32Array>().unwrap().value(1),
+ -10
+ );
+
+ // test truncate long
+ let input = Arc::new(Int64Array::from(vec![1, -1]));
+ let res = super::Truncate::new(10).transform(input).unwrap();
+ assert_eq!(
+ res.as_any().downcast_ref::<Int64Array>().unwrap().value(0),
+ 0
+ );
+ assert_eq!(
+ res.as_any().downcast_ref::<Int64Array>().unwrap().value(1),
+ -10
+ );
+
+ // test decimal
+ let mut buidler = PrimitiveBuilder::<Decimal128Type>::new()
+ .with_precision_and_scale(20, 2)
+ .unwrap();
+ buidler.append_value(1065);
+ let input = Arc::new(buidler.finish());
+ let res = super::Truncate::new(50).transform(input).unwrap();
+ assert_eq!(
+ res.as_any()
+ .downcast_ref::<Decimal128Array>()
+ .unwrap()
+ .value(0),
+ 1050
+ );
+
+ // test string
+ let input = Arc::new(arrow_array::StringArray::from(vec!["iceberg"]));
+ let res = super::Truncate::new(3).transform(input).unwrap();
+ assert_eq!(
+ res.as_any()
+ .downcast_ref::<arrow_array::StringArray>()
+ .unwrap()
+ .value(0),
+ "ice"
+ );
+
+ // test large string
+ let input = Arc::new(arrow_array::LargeStringArray::from(vec!["iceberg"]));
+ let res = super::Truncate::new(3).transform(input).unwrap();
+ assert_eq!(
+ res.as_any()
+ .downcast_ref::<arrow_array::LargeStringArray>()
+ .unwrap()
+ .value(0),
+ "ice"
+ );
+ }
+
+ #[test]
+ fn test_string_truncate() {
+ let test1 = "イロハニホヘト";
+ let test1_2_expected = "イロ";
+ assert_eq!(
+ super::Truncate::truncate_str_by_char(test1, 2),
+ test1_2_expected
+ );
+
+ let test1_3_expected = "イロハ";
+ assert_eq!(
+ super::Truncate::truncate_str_by_char(test1, 3),
+ test1_3_expected
+ );
+
+ let test2 = "щщаεはчωいにπάほхεろへσκζ";
Review Comment:
👍
##########
crates/iceberg/src/transform/truncate.rs:
##########
@@ -0,0 +1,217 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::ArrayRef;
+use arrow_schema::DataType;
+
+use crate::Error;
+
+use super::TransformFunction;
+
+pub struct Truncate {
+ width: u32,
+}
+
+impl Truncate {
+ pub fn new(width: u32) -> Self {
+ Self { width }
+ }
+
+ fn truncate_str_by_char(s: &str, max_chars: usize) -> &str {
+ match s.char_indices().nth(max_chars) {
+ None => s,
+ Some((idx, _)) => &s[..idx],
+ }
+ }
+}
+
+impl TransformFunction for Truncate {
+ fn transform(&self, input: ArrayRef) -> crate::Result<ArrayRef> {
+ match input.data_type() {
+ DataType::Int32 => {
+ let width: i32 = self.width.try_into().map_err(|_| {
+ Error::new(
+ crate::ErrorKind::DataInvalid,
+ "width is failed to convert to i32 when truncate Int32Array",
+ )
+ })?;
+ let res: arrow_array::Int32Array = input
+ .as_any()
+ .downcast_ref::<arrow_array::Int32Array>()
+ .unwrap()
+ .unary(|v| v - v.rem_euclid(width));
+ Ok(Arc::new(res))
+ }
+ DataType::Int64 => {
+ let width = self.width as i64;
+ let res: arrow_array::Int64Array = input
+ .as_any()
+ .downcast_ref::<arrow_array::Int64Array>()
+ .unwrap()
+ .unary(|v| v - (((v % width) + width) % width));
+ Ok(Arc::new(res))
+ }
+ DataType::Decimal128(precision, scale) => {
+ let width = self.width as i128;
+ let res: arrow_array::Decimal128Array = input
+ .as_any()
+ .downcast_ref::<arrow_array::Decimal128Array>()
+ .unwrap()
+ .unary(|v| v - (((v % width) + width) % width))
+ .with_precision_and_scale(*precision, *scale)
+ .map_err(|err| Error::new(crate::ErrorKind::Unexpected, format!("{err}")))?;
+ Ok(Arc::new(res))
+ }
+ DataType::Utf8 => {
+ let len = self.width as usize;
+ let res: arrow_array::StringArray = arrow_array::StringArray::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::StringArray>()
+ .unwrap()
+ .iter()
+ .map(|v| v.map(|v| Self::truncate_str_by_char(v, len))),
+ );
+ Ok(Arc::new(res))
+ }
+ DataType::LargeUtf8 => {
+ let len = self.width as usize;
+ let res: arrow_array::LargeStringArray = arrow_array::LargeStringArray::from_iter(
+ input
+ .as_any()
+ .downcast_ref::<arrow_array::LargeStringArray>()
+ .unwrap()
+ .iter()
+ .map(|v| v.map(|v| Self::truncate_str_by_char(v, len))),
+ );
+ Ok(Arc::new(res))
+ }
+ _ => unreachable!("Truncate transform only supports (int,long,decimal,string) types"),
+ }
+ }
+}
+
+#[cfg(test)]
+mod test {
+ use std::sync::Arc;
+
+ use arrow_array::{
+ builder::PrimitiveBuilder, types::Decimal128Type, Decimal128Array, Int32Array, Int64Array,
+ };
+
+ use crate::transform::TransformFunction;
+
+ // Test case ref from: https://iceberg.apache.org/spec/#truncate-transform-details
+ #[test]
+ fn test_truncate_simple() {
+ // test truncate int
+ let input = Arc::new(Int32Array::from(vec![1, -1]));
+ let res = super::Truncate::new(10).transform(input).unwrap();
+ assert_eq!(
+ res.as_any().downcast_ref::<Int32Array>().unwrap().value(0),
+ 0
+ );
+ assert_eq!(
+ res.as_any().downcast_ref::<Int32Array>().unwrap().value(1),
+ -10
+ );
+
+ // test truncate long
+ let input = Arc::new(Int64Array::from(vec![1, -1]));
+ let res = super::Truncate::new(10).transform(input).unwrap();
+ assert_eq!(
+ res.as_any().downcast_ref::<Int64Array>().unwrap().value(0),
+ 0
+ );
+ assert_eq!(
+ res.as_any().downcast_ref::<Int64Array>().unwrap().value(1),
+ -10
+ );
+
+ // test decimal
+ let mut buidler = PrimitiveBuilder::<Decimal128Type>::new()
+ .with_precision_and_scale(20, 2)
+ .unwrap();
+ buidler.append_value(1065);
+ let input = Arc::new(buidler.finish());
+ let res = super::Truncate::new(50).transform(input).unwrap();
+ assert_eq!(
+ res.as_any()
+ .downcast_ref::<Decimal128Array>()
+ .unwrap()
+ .value(0),
+ 1050
+ );
+
+ // test string
+ let input = Arc::new(arrow_array::StringArray::from(vec!["iceberg"]));
+ let res = super::Truncate::new(3).transform(input).unwrap();
+ assert_eq!(
+ res.as_any()
+ .downcast_ref::<arrow_array::StringArray>()
+ .unwrap()
+ .value(0),
+ "ice"
+ );
+
+ // test large string
+ let input = Arc::new(arrow_array::LargeStringArray::from(vec!["iceberg"]));
+ let res = super::Truncate::new(3).transform(input).unwrap();
+ assert_eq!(
+ res.as_any()
+ .downcast_ref::<arrow_array::LargeStringArray>()
+ .unwrap()
+ .value(0),
+ "ice"
+ );
+ }
+
+ #[test]
+ fn test_string_truncate() {
+ let test1 = "イロハニホヘト";
+ let test1_2_expected = "イロ";
+ assert_eq!(
+ super::Truncate::truncate_str_by_char(test1, 2),
+ test1_2_expected
+ );
+
+ let test1_3_expected = "イロハ";
+ assert_eq!(
+ super::Truncate::truncate_str_by_char(test1, 3),
+ test1_3_expected
+ );
+
+ let test2 = "щщаεはчωいにπάほхεろへσκζ";
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1308159095
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,669 @@
+use super::TransformFunction;
+use crate::{Error, Result};
+use arrow::array::{
+ Array, Date64Array, TimestampMicrosecondArray, TimestampMillisecondArray,
+ TimestampNanosecondArray, TimestampSecondArray,
+};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).map_err(|err| {
+ Error::new(
+ crate::ErrorKind::ArrowCompute,
+ format!("error in transformfunction: {}", err),
+ )
+ })?;
+ Ok(Arc::<Int32Array>::new(
+ array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| v - 1970),
+ ))
+ }
+}
+
+/// Extract a date or timestamp month, as months from 1970-01-01
+pub struct Month;
+
+impl TransformFunction for Month {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let year_array = year_dyn(&input)
+ .map_err(|err| Error::new(crate::ErrorKind::ArrowCompute, format!("{err}")))?;
+ let year_array: Int32Array = year_array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| 12 * (v - 1970));
+ let month_array = month_dyn(&input)
+ .map_err(|err| Error::new(crate::ErrorKind::ArrowCompute, format!("{err}")))?;
+ Ok(Arc::<Int32Array>::new(
+ binary(
+ month_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ year_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ // Compute month from 1970-01-01, so minus 1 here.
+ |a, b| a + b - 1,
+ )
+ .unwrap(),
+ ))
+ }
+}
+
+/// Extract a date or timestamp day, as days from 1970-01-01
+pub struct Day;
+
+impl TransformFunction for Day {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let res: Int32Array = match input.data_type() {
+ DataType::Timestamp(unit, _) => match unit {
+ datatypes::TimeUnit::Second => input
+ .as_any()
+ .downcast_ref::<TimestampSecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 * DAY_PER_SECOND) as i32 }),
+ datatypes::TimeUnit::Millisecond => input
+ .as_any()
+ .downcast_ref::<TimestampMillisecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 / 1000.0 * DAY_PER_SECOND) as i32 }),
+ datatypes::TimeUnit::Microsecond => input
+ .as_any()
+ .downcast_ref::<TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 / 1000.0 / 1000.0 * DAY_PER_SECOND) as i32 }),
+ datatypes::TimeUnit::Nanosecond => input
+ .as_any()
+ .downcast_ref::<TimestampNanosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ (v as f64 / 1000.0 / 1000.0 / 1000.0 * DAY_PER_SECOND) as i32
+ }),
+ },
+ DataType::Date32 => {
+ input
+ .as_any()
+ .downcast_ref::<Date32Array>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ datatypes::Date32Type::to_naive_date(v).num_days_from_ce()
+ - EPOCH_DAY_FROM_CE
+ })
+ }
+ DataType::Date64 => {
+ input
+ .as_any()
+ .downcast_ref::<Date64Array>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ datatypes::Date64Type::to_naive_date(v).num_days_from_ce()
+ - EPOCH_DAY_FROM_CE
+ })
+ }
+ _ => unreachable!(
+ "Should not call transform in Day with type {:?}",
+ input.data_type()
+ ),
+ };
+ Ok(Arc::new(res))
Review Comment:
> If a Timestamp column has a non-empty timezone value, its epoch is 1970-01-01 00:00:00 (January 1st 1970, midnight) in the UTC timezone (the Unix epoch), regardless of the Timestamp’s own timezone.
The timezone will not affect the epoch so we can use it directly. The result is always regard as in the UTC timezone.
https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html#timestamps-with-a-non-empty-timezone
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,669 @@
+use super::TransformFunction;
+use crate::{Error, Result};
+use arrow::array::{
+ Array, Date64Array, TimestampMicrosecondArray, TimestampMillisecondArray,
+ TimestampNanosecondArray, TimestampSecondArray,
+};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).map_err(|err| {
+ Error::new(
+ crate::ErrorKind::ArrowCompute,
+ format!("error in transformfunction: {}", err),
+ )
+ })?;
+ Ok(Arc::<Int32Array>::new(
+ array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| v - 1970),
+ ))
+ }
+}
+
+/// Extract a date or timestamp month, as months from 1970-01-01
+pub struct Month;
+
+impl TransformFunction for Month {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let year_array = year_dyn(&input)
+ .map_err(|err| Error::new(crate::ErrorKind::ArrowCompute, format!("{err}")))?;
+ let year_array: Int32Array = year_array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| 12 * (v - 1970));
+ let month_array = month_dyn(&input)
+ .map_err(|err| Error::new(crate::ErrorKind::ArrowCompute, format!("{err}")))?;
+ Ok(Arc::<Int32Array>::new(
+ binary(
+ month_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ year_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ // Compute month from 1970-01-01, so minus 1 here.
+ |a, b| a + b - 1,
+ )
+ .unwrap(),
+ ))
+ }
+}
+
+/// Extract a date or timestamp day, as days from 1970-01-01
+pub struct Day;
+
+impl TransformFunction for Day {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let res: Int32Array = match input.data_type() {
+ DataType::Timestamp(unit, _) => match unit {
+ datatypes::TimeUnit::Second => input
+ .as_any()
+ .downcast_ref::<TimestampSecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 * DAY_PER_SECOND) as i32 }),
+ datatypes::TimeUnit::Millisecond => input
+ .as_any()
+ .downcast_ref::<TimestampMillisecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 / 1000.0 * DAY_PER_SECOND) as i32 }),
+ datatypes::TimeUnit::Microsecond => input
+ .as_any()
+ .downcast_ref::<TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 / 1000.0 / 1000.0 * DAY_PER_SECOND) as i32 }),
+ datatypes::TimeUnit::Nanosecond => input
+ .as_any()
+ .downcast_ref::<TimestampNanosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ (v as f64 / 1000.0 / 1000.0 / 1000.0 * DAY_PER_SECOND) as i32
+ }),
+ },
+ DataType::Date32 => {
+ input
+ .as_any()
+ .downcast_ref::<Date32Array>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ datatypes::Date32Type::to_naive_date(v).num_days_from_ce()
+ - EPOCH_DAY_FROM_CE
+ })
+ }
+ DataType::Date64 => {
+ input
+ .as_any()
+ .downcast_ref::<Date64Array>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ datatypes::Date64Type::to_naive_date(v).num_days_from_ce()
+ - EPOCH_DAY_FROM_CE
+ })
+ }
+ _ => unreachable!(
+ "Should not call transform in Day with type {:?}",
+ input.data_type()
+ ),
+ };
+ Ok(Arc::new(res))
Review Comment:
> If a Timestamp column has a non-empty timezone value, its epoch is 1970-01-01 00:00:00 (January 1st 1970, midnight) in the UTC timezone (the Unix epoch), regardless of the Timestamp’s own timezone.
The timezone will not affect the epoch so we can use it directly. The result is always regard as in the UTC timezone.
https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html#timestamps-with-a-non-empty-timezone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] Xuanwo commented on a diff in pull request #42: feat: support transform function
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1308493879
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,329 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::TransformFunction;
+use crate::Result;
+use arrow::array::{Array, TimestampMicrosecondArray};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+const BASE_YEAR: i32 = 1970;
Review Comment:
For example, I spent some time to understand what's `DAY_PER_SECOND`.
Besides, `const` means rustc will help us calculate the value during compile time. So we can use `1_f64 / 24.0 / 3600.0` here to make it more clear.
If there are some limiation for the calculation (we do need the value to be `0.0000115741`), we can add the `1_f64 / 24.0 / 3600.0` as comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] liurenjie1024 commented on a diff in pull request #42: feat: support transform function
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1311128206
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
Review Comment:
For example when we want to execute `select * from t where a = 1 and b=2`, and table `t` is partitioned by a, we can prune unnecessary partitions by calculating `a=1`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1308198500
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,669 @@
+use super::TransformFunction;
+use crate::{Error, Result};
+use arrow::array::{
+ Array, Date64Array, TimestampMicrosecondArray, TimestampMillisecondArray,
+ TimestampNanosecondArray, TimestampSecondArray,
+};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).map_err(|err| {
+ Error::new(
+ crate::ErrorKind::ArrowCompute,
Review Comment:
Transform function is a internal function. For unsupport type, we should check it before call transform function and return error. We can panic directly if meet unsupport type in transform function. So can we replace it with `unreachable!` here.🤔
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,669 @@
+use super::TransformFunction;
+use crate::{Error, Result};
+use arrow::array::{
+ Array, Date64Array, TimestampMicrosecondArray, TimestampMillisecondArray,
+ TimestampNanosecondArray, TimestampSecondArray,
+};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).map_err(|err| {
+ Error::new(
+ crate::ErrorKind::ArrowCompute,
Review Comment:
Transform function is a internal function. For unsupport type, we should check it before call transform function and return error. We can panic directly if meet unsupport type in transform function. So can we replace it with `unreachable!` here.🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] liurenjie1024 commented on a diff in pull request #42: feat: support transform function
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1311063605
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
Review Comment:
There are two differences here:
1. The transform function is the actual implementation of transform logic, it's similar to `hash_func` above in python.
2. The bind process happens in following `create_transform_function`. The difference is that in rust we don't accept a `Type` parameter, since it's included in `ArrayRef`.
3. We implement the partition function in arrow array rather iceberg value. This is because currently the most important use case for us it to implement `PartitionedWriter`, which accepts arrow `RecordBatch`. And implement it on arrow array let use do it in vectorized approach(much faster than row based approach). I guess eventuall we will need to implement an other version in iceberg value and it would be useful in partition pruning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1311142573
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
Review Comment:
Sorry, I think my question should be whether we need to provide write interface use iceberg value. We should maintain iceberg value internally for patition and convert between arrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1308176568
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,669 @@
+use super::TransformFunction;
+use crate::{Error, Result};
+use arrow::array::{
+ Array, Date64Array, TimestampMicrosecondArray, TimestampMillisecondArray,
+ TimestampNanosecondArray, TimestampSecondArray,
+};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).map_err(|err| {
+ Error::new(
+ crate::ErrorKind::ArrowCompute,
Review Comment:
Sounds reasonable for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] Xuanwo commented on a diff in pull request #42: feat: support transform function
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1308491079
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,329 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::TransformFunction;
+use crate::Result;
+use arrow::array::{Array, TimestampMicrosecondArray};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// 719163 is the number of days from 0000-01-01 to 1970-01-01
+const EPOCH_DAY_FROM_CE: i32 = 719163;
+const DAY_PER_SECOND: f64 = 0.0000115741;
+const HOUR_PER_SECOND: f64 = 1_f64 / 3600.0;
+const BASE_YEAR: i32 = 1970;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array = year_dyn(&input).expect("Should not call transform in Year with non-date type");
Review Comment:
I'm ok if this place is **REALLY** `unreachable`, otherwise, return an error is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] liurenjie1024 commented on a diff in pull request #42: feat: support transform function
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1311059906
##########
crates/iceberg/Cargo.toml:
##########
@@ -28,6 +28,7 @@ keywords = ["iceberg"]
[dependencies]
apache-avro = "0.15.0"
+arrow = { version = ">=46" }
Review Comment:
Good point, we should use `arrow-array` only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] liurenjie1024 commented on a diff in pull request #42: feat: support transform function
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1311130728
##########
crates/iceberg/Cargo.toml:
##########
@@ -28,6 +28,7 @@ keywords = ["iceberg"]
[dependencies]
apache-avro = "0.15.0"
+arrow = { version = ">=46" }
Review Comment:
There are several arrow modules:

I guess the methods are in `arrow-arith`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1311151093
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,343 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::TransformFunction;
+use crate::{Error, ErrorKind, Result};
+use arrow::array::{Array, TimestampMicrosecondArray};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// The number of days since unix epoch.
+const DAY_SINCE_UNIX_EPOCH: i32 = 719163;
+/// Hour in one second.
+const HOUR_PER_SECOND: f64 = 1.0_f64 / 3600.0_f64;
+/// Day in one second.
+const DAY_PER_SECOND: f64 = 1.0_f64 / 24.0_f64 / 3600.0_f64;
+/// Year of unix epoch.
+const UNIX_EPOCH_YEAR: i32 = 1970;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array =
+ year_dyn(&input).map_err(|err| Error::new(ErrorKind::Unexpected, format!("{err}")))?;
+ Ok(Arc::<Int32Array>::new(
+ array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| v - UNIX_EPOCH_YEAR),
+ ))
+ }
+}
+
+/// Extract a date or timestamp month, as months from 1970-01-01
+pub struct Month;
+
+impl TransformFunction for Month {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let year_array =
+ year_dyn(&input).map_err(|err| Error::new(ErrorKind::Unexpected, format!("{err}")))?;
+ let year_array: Int32Array = year_array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| 12 * (v - UNIX_EPOCH_YEAR));
+ let month_array =
+ month_dyn(&input).map_err(|err| Error::new(ErrorKind::Unexpected, format!("{err}")))?;
+ Ok(Arc::<Int32Array>::new(
+ binary(
+ month_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ year_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ // Compute month from 1970-01-01, so minus 1 here.
+ |a, b| a + b - 1,
+ )
+ .unwrap(),
+ ))
+ }
+}
+
+/// Extract a date or timestamp day, as days from 1970-01-01
+pub struct Day;
+
+impl TransformFunction for Day {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let res: Int32Array = match input.data_type() {
+ DataType::Timestamp(datatypes::TimeUnit::Microsecond, _) => input
+ .as_any()
+ .downcast_ref::<TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 / 1000.0 / 1000.0 * DAY_PER_SECOND) as i32 }),
+ DataType::Date32 => {
+ input
+ .as_any()
+ .downcast_ref::<Date32Array>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ datatypes::Date32Type::to_naive_date(v).num_days_from_ce()
+ - DAY_SINCE_UNIX_EPOCH
+ })
+ }
+ _ => {
+ return Err(Error::new(
+ ErrorKind::Unexpected,
+ format!(
+ "Should not call internally for unsupport data type {:?}",
+ input.data_type()
+ ),
+ ))
+ }
+ };
+ Ok(Arc::new(res))
+ }
+}
+
+/// Extract a timestamp hour, as hours from 1970-01-01 00:00:00
+pub struct Hour;
+
+impl TransformFunction for Hour {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let res: Int32Array = match input.data_type() {
+ DataType::Timestamp(datatypes::TimeUnit::Microsecond, _) => input
+ .as_any()
+ .downcast_ref::<TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 * HOUR_PER_SECOND / 1000.0 / 1000.0) as i32 }),
+ _ => {
+ return Err(Error::new(
+ ErrorKind::Unexpected,
+ format!(
+ "Should not call internally for unsupport data type {:?}",
+ input.data_type()
+ ),
+ ))
+ }
+ };
+ Ok(Arc::new(res))
+ }
+}
+
+#[cfg(test)]
+mod test {
+ use arrow::array::{ArrayRef, Date32Array, Int32Array, TimestampMicrosecondArray};
+ use chrono::NaiveDate;
+ use std::sync::Arc;
+
+ use crate::transform::TransformFunction;
+
+ #[test]
+ fn test_transform_years() {
+ let year = super::Year;
+ let ori_date = vec![
+ NaiveDate::from_ymd_opt(1970, 1, 1).unwrap(),
Review Comment:
Yes. And I think here NaiveDateTime(timestamp) and DataTime(timestamptz) have the same effect. I add the NaiveDateTime to cover more case. 🤔 Please let me know if something I miss.
##########
crates/iceberg/src/transform/temporal.rs:
##########
@@ -0,0 +1,343 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::TransformFunction;
+use crate::{Error, ErrorKind, Result};
+use arrow::array::{Array, TimestampMicrosecondArray};
+use arrow::compute::binary;
+use arrow::datatypes;
+use arrow::datatypes::DataType;
+use arrow::{
+ array::{ArrayRef, Date32Array, Int32Array},
+ compute::{month_dyn, year_dyn},
+};
+use chrono::Datelike;
+use std::sync::Arc;
+
+/// The number of days since unix epoch.
+const DAY_SINCE_UNIX_EPOCH: i32 = 719163;
+/// Hour in one second.
+const HOUR_PER_SECOND: f64 = 1.0_f64 / 3600.0_f64;
+/// Day in one second.
+const DAY_PER_SECOND: f64 = 1.0_f64 / 24.0_f64 / 3600.0_f64;
+/// Year of unix epoch.
+const UNIX_EPOCH_YEAR: i32 = 1970;
+
+/// Extract a date or timestamp year, as years from 1970
+pub struct Year;
+
+impl TransformFunction for Year {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let array =
+ year_dyn(&input).map_err(|err| Error::new(ErrorKind::Unexpected, format!("{err}")))?;
+ Ok(Arc::<Int32Array>::new(
+ array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| v - UNIX_EPOCH_YEAR),
+ ))
+ }
+}
+
+/// Extract a date or timestamp month, as months from 1970-01-01
+pub struct Month;
+
+impl TransformFunction for Month {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let year_array =
+ year_dyn(&input).map_err(|err| Error::new(ErrorKind::Unexpected, format!("{err}")))?;
+ let year_array: Int32Array = year_array
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .unary(|v| 12 * (v - UNIX_EPOCH_YEAR));
+ let month_array =
+ month_dyn(&input).map_err(|err| Error::new(ErrorKind::Unexpected, format!("{err}")))?;
+ Ok(Arc::<Int32Array>::new(
+ binary(
+ month_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ year_array.as_any().downcast_ref::<Int32Array>().unwrap(),
+ // Compute month from 1970-01-01, so minus 1 here.
+ |a, b| a + b - 1,
+ )
+ .unwrap(),
+ ))
+ }
+}
+
+/// Extract a date or timestamp day, as days from 1970-01-01
+pub struct Day;
+
+impl TransformFunction for Day {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let res: Int32Array = match input.data_type() {
+ DataType::Timestamp(datatypes::TimeUnit::Microsecond, _) => input
+ .as_any()
+ .downcast_ref::<TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 / 1000.0 / 1000.0 * DAY_PER_SECOND) as i32 }),
+ DataType::Date32 => {
+ input
+ .as_any()
+ .downcast_ref::<Date32Array>()
+ .unwrap()
+ .unary(|v| -> i32 {
+ datatypes::Date32Type::to_naive_date(v).num_days_from_ce()
+ - DAY_SINCE_UNIX_EPOCH
+ })
+ }
+ _ => {
+ return Err(Error::new(
+ ErrorKind::Unexpected,
+ format!(
+ "Should not call internally for unsupport data type {:?}",
+ input.data_type()
+ ),
+ ))
+ }
+ };
+ Ok(Arc::new(res))
+ }
+}
+
+/// Extract a timestamp hour, as hours from 1970-01-01 00:00:00
+pub struct Hour;
+
+impl TransformFunction for Hour {
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
+ let res: Int32Array = match input.data_type() {
+ DataType::Timestamp(datatypes::TimeUnit::Microsecond, _) => input
+ .as_any()
+ .downcast_ref::<TimestampMicrosecondArray>()
+ .unwrap()
+ .unary(|v| -> i32 { (v as f64 * HOUR_PER_SECOND / 1000.0 / 1000.0) as i32 }),
+ _ => {
+ return Err(Error::new(
+ ErrorKind::Unexpected,
+ format!(
+ "Should not call internally for unsupport data type {:?}",
+ input.data_type()
+ ),
+ ))
+ }
+ };
+ Ok(Arc::new(res))
+ }
+}
+
+#[cfg(test)]
+mod test {
+ use arrow::array::{ArrayRef, Date32Array, Int32Array, TimestampMicrosecondArray};
+ use chrono::NaiveDate;
+ use std::sync::Arc;
+
+ use crate::transform::TransformFunction;
+
+ #[test]
+ fn test_transform_years() {
+ let year = super::Year;
+ let ori_date = vec![
+ NaiveDate::from_ymd_opt(1970, 1, 1).unwrap(),
Review Comment:
Yes. And I think here NaiveDateTime(timestamp) and DataTime(timestamptz) have the same effect. I add the NaiveDateTime to cover more case. 🤔 Please let me know if something I miss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1333783538
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
Review Comment:
Yes. And it's better to integrate transform array and single value, I can do it when implementation of single value version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] Fokko merged pull request #42: feat: support transform function
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] ZENOTME commented on a diff in pull request #42: feat: support transform function
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on code in PR #42:
URL: https://github.com/apache/iceberg-rust/pull/42#discussion_r1318165935
##########
crates/iceberg/src/transform/mod.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transform function used to compute partition values.
+use crate::{spec::Transform, Result};
+use arrow::array::ArrayRef;
+
+mod identity;
+mod temporal;
+mod void;
+
+/// TransformFunction is a trait that defines the interface for all transform functions.
+pub trait TransformFunction: Send {
+ /// transform will take an input array and transform it into a new array.
+ /// The implementation of this function will need to check and downcast the input to specific
+ /// type.
+ fn transform(&self, input: ArrayRef) -> Result<ArrayRef>;
+}
+
+/// BoxedTransformFunction is a boxed trait object of TransformFunction.
+pub type BoxedTransformFunction = Box<dyn TransformFunction>;
+
+/// create_transform_function creates a boxed trait object of TransformFunction from a Transform.
+pub fn create_transform_function(transform: &Transform) -> Result<BoxedTransformFunction> {
+ match transform {
+ Transform::Identity => Ok(Box::new(identity::Identity {})),
+ Transform::Void => Ok(Box::new(void::Void {})),
+ Transform::Year => Ok(Box::new(temporal::Year {})),
+ Transform::Month => Ok(Box::new(temporal::Month {})),
+ Transform::Day => Ok(Box::new(temporal::Day {})),
+ Transform::Hour => Ok(Box::new(temporal::Hour {})),
+ _ => Err(crate::error::Error::new(
+ crate::ErrorKind::FeatureUnsupported,
+ format!("Transform {:?} is not implemented", transform),
+ )),
+ }
Review Comment:
Have completed all transform function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org