You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "felipepessoto (via GitHub)" <gi...@apache.org> on 2026/09/04 01:37:10 UTC

[PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

felipepessoto opened a new pull request, #12967:
URL: https://github.com/apache/gluten/pull/12967

   ## What changes are proposed in this pull request?
   
   Enable native `TIMESTAMP_NTZ` aggregation in the Velox backend:
   
   - Allow aggregate and shuffle operators, plus the projection expressions used by Delta statistics collection, through the `TIMESTAMP_NTZ` fallback validator.
   - Accept `TimestampNTZType` aggregate buffers.
   - Preserve `TIMESTAMP_NTZ` as a distinct `tsntz` function-signature type and as Substrait `PrecisionTimestamp`.
   - Add native min/max and Delta statistics regression coverage, including nested `TIMESTAMP_NTZ` columns.
   
   This prevents Delta's local statistics `SortAggregateExec` from falling back to a vanilla Spark plan and addresses the `TIMESTAMP_NTZ` failure described in #12538. This change is independent of #12966.
   
   ## How was this patch tested?
   
   - `cpp/build/velox/tests/velox_plan_conversion_test --gtest_filter='VeloxSubstraitSignatureTest.*:VeloxToSubstraitTypeTest.*:VeloxSubstraitRoundTripTest.minMaxTimestampUtc'`
   - `./build/mvn test -Pspark-ut -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta -DwildcardSuites=org.apache.spark.sql.GlutenTimestampNtzAggregateSuite`
   - `./build/mvn test -Pspark-ut -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta -DwildcardSuites=org.apache.spark.sql.delta.GlutenDeltaStatsSuite`
   - `./build/mvn test -Pspark-ut -Pbackends-velox -Pspark-3.5 -Pscala-2.12 -Pjava-17 -Pdelta -DwildcardSuites=org.apache.spark.sql.delta.GlutenDeltaStatsSuite`
   - `./build/mvn test -Pspark-ut -Pbackends-velox -Pspark-3.5 -Pscala-2.12 -Pjava-17 -Pdelta -DwildcardSuites=org.apache.gluten.functions.DateFunctionsValidateSuite`
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: GitHub Copilot CLI 1.0.82
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5702575597

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5704729644

   Addressed the missing `IsNotNull` allowance from [this review](https://github.com/apache/gluten/pull/12967#pullrequestreview-5226857135) in [fb29777ab9](https://github.com/apache/gluten/commit/fb29777ab969ac9dd37aa12d2f2eb550c3de7306). It now follows the existing `IsNull` pattern in the NTZ projection allowlist.
   
   Added a regression over nullable NTZ input for `isnull`, `isnotnull`, and `if(isnotnull(ts), 1, 0)`. It checks the expected results and requires native projection when `enableTimestampNtzValidation=false`, while checking Spark fallback when the flag is `true`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r3977731425


##########
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.test.DeltaSQLCommandTest
+
+class GlutenDeltaStatsSuite extends DeltaSQLCommandTest {
+
+  import testImplicits._
+
+  test("collect TIMESTAMP_NTZ statistics natively") {
+    withSQLConf(DeltaSQLConf.DELTA_COLLECT_STATS.key -> "true") {
+      withTempDir {
+        dir =>
+          val path = dir.getCanonicalPath
+          val data = Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr(
+              "cast(input as timestamp_ntz) as ts",
+              "struct(cast(input as timestamp_ntz) as ts) as nested")
+
+          data.coalesce(1).write.format("delta").save(path)
+
+          val actual = spark.read.format("delta").load(path)
+          assert(actual.collect().toSet == data.collect().toSet)
+
+          val addFiles = DeltaLog.forTable(spark, path).update().allFiles.collect()
+          assert(addFiles.length == 1)
+          val stats = addFiles.head.stats
+          assert(stats != null)
+          val statsValues = Seq(stats)
+            .toDF("stats")
+            .selectExpr(
+              "get_json_object(stats, '$.minValues.ts')",
+              "get_json_object(stats, '$.minValues.nested.ts')",
+              "get_json_object(stats, '$.maxValues.ts')",
+              "get_json_object(stats, '$.maxValues.nested.ts')"
+            )
+            .head()
+          assert(
+            statsValues == Row(
+              "1969-12-31T23:59:59.999",
+              "1969-12-31T23:59:59.999",
+              "2024-01-01T00:00:00.123",
+              "2024-01-01T00:00:00.123"),
+            stats)

Review Comment:
   > The test could then assert that the hook was invoked and that the captured plan contains HashAggregateExecTransformer. 
   
   Hi @zhztheplayer, do you have any preference for this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5534523583

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5694551450

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r4077459913


##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenTimestampNtzAggregateSuite.scala:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql
+
+import org.apache.gluten.config.GlutenConfig
+import org.apache.gluten.execution.{HashAggregateExecBaseTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.execution.ProjectExec
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.functions.{max, min}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.TimestampNTZType
+
+import java.time.LocalDateTime
+
+class GlutenTimestampNtzAggregateSuite extends GlutenSQLTestsTrait {
+
+  import testImplicits._
+
+  testGluten("min and max") {
+    withSQLConf(
+      SQLConf.ANSI_ENABLED.key -> "false",
+      GlutenConfig.GLUTEN_ANSI_FALLBACK_ENABLED.key -> "false") {
+      withTempPath {
+        path =>
+          Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr("cast(input as timestamp_ntz) as ts")
+            .write
+            .parquet(path.getCanonicalPath)
+
+          val result = spark.read.parquet(path.getCanonicalPath).agg(min($"ts"), max($"ts"))
+          checkAnswer(
+            result,
+            Row(
+              LocalDateTime.parse("1969-12-31T23:59:59.999999"),
+              LocalDateTime.parse("2024-01-01T00:00:00.123456")))
+          assert(
+            getExecutedPlan(result).exists(_.isInstanceOf[HashAggregateExecBaseTransformer]),
+            result.queryExecution.executedPlan.treeString)
+      }
+    }
+  }
+
+  testGluten("min and max grouped by timestamp_ntz") {
+    withSQLConf(
+      SQLConf.ANSI_ENABLED.key -> "false",
+      SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
+      SQLConf.SESSION_LOCAL_TIMEZONE.key -> "America/Los_Angeles",
+      SQLConf.SHUFFLE_PARTITIONS.key -> "2",
+      GlutenConfig.GLUTEN_ANSI_FALLBACK_ENABLED.key -> "false",
+      "spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation" -> "false"
+    ) {
+      withTempPath {
+        path =>
+          val beforeEpoch = LocalDateTime.parse("1969-12-31T23:59:59.999999")
+          val afterEpoch = LocalDateTime.parse("1970-01-01T00:00:00.000001")
+          val firstKey = LocalDateTime.parse("2024-01-01T00:00:00.123456")
+          val secondKey = LocalDateTime.parse("2024-01-01T00:00:00.123457")
+          Seq(
+            (firstKey, beforeEpoch),
+            (firstKey, secondKey),
+            (firstKey, null),
+            (secondKey, afterEpoch),
+            (secondKey, firstKey),
+            (null, beforeEpoch),
+            (null, afterEpoch),
+            (beforeEpoch, null)
+          ).toDF("key", "ts")
+            .write
+            .parquet(path.getCanonicalPath)
+
+          val result = spark.read
+            .parquet(path.getCanonicalPath)
+            .groupBy($"key")
+            .agg(min($"ts"), max($"ts"))
+          checkAnswer(
+            result,
+            Seq(
+              Row(firstKey, beforeEpoch, secondKey),
+              Row(secondKey, afterEpoch, firstKey),
+              Row(null, beforeEpoch, afterEpoch),
+              Row(beforeEpoch, null, null)))
+          val aggregates = getExecutedPlan(result).collect {
+            case aggregate: BaseAggregateExec => aggregate
+          }
+          assert(aggregates.nonEmpty, result.queryExecution.executedPlan.treeString)
+          assert(
+            aggregates.forall {
+              case aggregate: HashAggregateExecBaseTransformer =>
+                aggregate.groupingExpressions.map(_.dataType) == Seq(TimestampNTZType)
+              case _ => false
+            },
+            result.queryExecution.executedPlan.treeString
+          )

Review Comment:
   The stronger assertion is intentional: this grouped min/max case should execute all aggregation stages natively while preserving the NTZ grouping type. It allows multiple stages and does not require a fixed node count. Collecting only native aggregates would hide a regression where one stage falls back to Spark, so I’d keep the non-empty check and assertion over every aggregate.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r4031435191


##########
cpp/velox/substrait/VeloxToSubstraitType.cc:
##########
@@ -31,6 +31,14 @@ const ::substrait::Type& VeloxToSubstraitTypeConvertor::toSubstraitType(
     substraitType->set_allocated_date(substraitDate);
     return *substraitType;
   }
+  if (type->equivalent(*velox::TIMESTAMP_UTC())) {
+    auto substraitPrecisionTimestamp =
+        google::protobuf::Arena::CreateMessage<::substrait::Type_PrecisionTimestamp>(&arena);
+    substraitPrecisionTimestamp->set_precision(6);
+    substraitPrecisionTimestamp->set_nullability(::substrait::Type_Nullability_NULLABILITY_NULLABLE);

Review Comment:
   I've separated the NTZ literal work into [apache/gluten#13041](https://github.com/apache/gluten/pull/13041), opened as a draft.
   
   The follow-up addresses literal field selection and decoding, logical-type preservation for constants and typed nulls, and the missing JVM literal wiring. It has dedicated native and Spark literal regressions.
   
   It is based directly on `main` and includes the NTZ type mapping it needs, so it can be reviewed and merged independently of this PR. This keeps the aggregation and Delta statistics changes here separate from the literal-conversion work.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5616749539

   @felipepessoto Yes, I added the `TIMESTAMP_UTC` type in Velox. It was designed to share the same physical representation as `TimestampType` while having different logical semantics. The motivation was to represent the timezone-adjustment semantics correctly while still allowing us to reuse the existing implementations where possible.
   
   My concern is that, since we haven’t yet verified that aggregate functions work correctly with `TIMESTAMP_UTC`, it may be better not to treat them as supported implicitly. Instead, we could explicitly validate the support in Velox by adding unit tests and updating the coverage documentation, even if no implementation changes turn out to be necessary.
   
   Would you be interested in helping add this support for `min`/`max` in Velox first? Thanks.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r3937596799


##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenTimestampNtzAggregateSuite.scala:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql
+
+import org.apache.gluten.config.GlutenConfig
+import org.apache.gluten.execution.{HashAggregateExecBaseTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.functions.{max, min}
+import org.apache.spark.sql.internal.SQLConf
+
+import java.time.LocalDateTime
+
+class GlutenTimestampNtzAggregateSuite extends GlutenSQLTestsTrait {
+
+  import testImplicits._
+
+  testGluten("min and max") {
+    withSQLConf(
+      SQLConf.ANSI_ENABLED.key -> "false",
+      GlutenConfig.GLUTEN_ANSI_FALLBACK_ENABLED.key -> "false") {
+      withTempPath {
+        path =>
+          Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr("cast(input as timestamp_ntz) as ts")
+            .write
+            .parquet(path.getCanonicalPath)
+
+          val result = spark.read.parquet(path.getCanonicalPath).agg(min($"ts"), max($"ts"))
+          checkAnswer(
+            result,
+            Row(
+              LocalDateTime.parse("1969-12-31T23:59:59.999999"),
+              LocalDateTime.parse("2024-01-01T00:00:00.123456")))
+          assert(
+            getExecutedPlan(result).exists(_.isInstanceOf[HashAggregateExecBaseTransformer]),
+            result.queryExecution.executedPlan.treeString)
+      }
+    }
+  }
+
+  testGluten("unsupported project falls back") {
+    withSQLConf(
+      SQLConf.ANSI_ENABLED.key -> "false",
+      SQLConf.SESSION_LOCAL_TIMEZONE.key -> "America/Los_Angeles",
+      GlutenConfig.GLUTEN_ANSI_FALLBACK_ENABLED.key -> "false") {
+      withTempPath {
+        path =>
+          Seq("2024-01-01 00:00:00.123456")
+            .toDF("input")
+            .selectExpr("cast(input as timestamp_ntz) as ts")
+            .write
+            .parquet(path.getCanonicalPath)
+
+          val result = spark.read
+            .parquet(path.getCanonicalPath)
+            .selectExpr("to_json(named_struct('ts', ts))")
+          checkAnswer(result, Row("""{"ts":"2024-01-01T00:00:00.123"}"""))
+          assert(
+            !getExecutedPlan(result).exists(_.isInstanceOf[ProjectExecTransformer]),
+            result.queryExecution.executedPlan.treeString)

Review Comment:
   Valid. Updated the test in 6f80ab9cf4 to identify the vanilla ProjectExec whose output attributes match the final to_json result, instead of rejecting any ProjectExecTransformer elsewhere in the plan. The focused Spark 4.1 suite passes.



##########
cpp/velox/substrait/VeloxSubstraitSignature.cc:
##########
@@ -155,6 +158,10 @@ TypePtr VeloxSubstraitSignature::fromSubstraitSignature(const std::string& signa
     return TIMESTAMP();
   }
 
+  if (signature == "tsntz") {

Review Comment:
   No compatibility alias added. These Substrait plans are generated per query and Gluten requires matching JVM/native components; they are not a persisted cross-version format. Also, the old ts_ntz token was never parseable for aggregates because sigToTypes splits on underscores, so accepting it only in fromSubstraitSignature would not restore a valid legacy aggregate encoding.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5534435699

   🔄 Delta Spark UT started by @felipepessoto (~2.5 h). [View run](https://github.com/apache/gluten/actions/runs/33826472162)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r3950606077


##########
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.test.DeltaSQLCommandTest
+
+class GlutenDeltaStatsSuite extends DeltaSQLCommandTest {
+
+  import testImplicits._
+
+  test("collect TIMESTAMP_NTZ statistics natively") {
+    withSQLConf(DeltaSQLConf.DELTA_COLLECT_STATS.key -> "true") {
+      withTempDir {
+        dir =>
+          val path = dir.getCanonicalPath
+          val data = Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr(
+              "cast(input as timestamp_ntz) as ts",
+              "struct(cast(input as timestamp_ntz) as ts) as nested")
+
+          data.coalesce(1).write.format("delta").save(path)
+
+          val actual = spark.read.format("delta").load(path)
+          assert(actual.collect().toSet == data.collect().toSet)
+
+          val addFiles = DeltaLog.forTable(spark, path).update().allFiles.collect()
+          assert(addFiles.length == 1)
+          val stats = addFiles.head.stats
+          assert(stats != null)
+          val statsValues = Seq(stats)
+            .toDF("stats")
+            .selectExpr(
+              "get_json_object(stats, '$.minValues.ts')",
+              "get_json_object(stats, '$.minValues.nested.ts')",
+              "get_json_object(stats, '$.maxValues.ts')",
+              "get_json_object(stats, '$.maxValues.nested.ts')"
+            )
+            .head()
+          assert(
+            statsValues == Row(
+              "1969-12-31T23:59:59.999",
+              "1969-12-31T23:59:59.999",
+              "2024-01-01T00:00:00.123",
+              "2024-01-01T00:00:00.123"),
+            stats)

Review Comment:
   Could you please assert that the plan contains `HashAggregateExecTransformer` to ensure aggregate has been offloaded into native?



##########
cpp/velox/substrait/VeloxSubstraitSignature.cc:
##########
@@ -24,6 +24,9 @@ std::string VeloxSubstraitSignature::toSubstraitSignature(const TypePtr& type) {
   if (type->isDate()) {
     return "date";
   }
+  if (type->equivalent(*TIMESTAMP_UTC())) {
+    return "tsntz";

Review Comment:
   nit: use `ts_ntz` to be aligned with Spark.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r3937146783


##########
backends-velox/src-delta33/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.test.DeltaSQLCommandTest
+
+class GlutenDeltaStatsSuite extends DeltaSQLCommandTest {
+
+  import testImplicits._
+
+  test("collect TIMESTAMP_NTZ statistics natively") {
+    withSQLConf(DeltaSQLConf.DELTA_COLLECT_STATS.key -> "true") {
+      withTempDir {
+        dir =>
+          val path = dir.getCanonicalPath
+          val data = Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr(
+              "cast(input as timestamp_ntz) as ts",
+              "struct(cast(input as timestamp_ntz) as ts) as nested")
+
+          data.coalesce(1).write.format("delta").save(path)
+
+          val actual = spark.read.format("delta").load(path)
+          assert(actual.collect().toSet == data.collect().toSet)
+
+          val addFiles = DeltaLog.forTable(spark, path).update().allFiles.collect()
+          assert(addFiles.length == 1)
+          val stats = addFiles.head.stats
+          assert(stats != null)
+          assert(stats.contains("\"minValues\""), stats)
+          assert(stats.contains("\"maxValues\""), stats)
+          assert(stats.contains("1969-12-31") && stats.contains("23:59:59.999"), stats)
+          assert(stats.contains("2024-01-01") && stats.contains("00:00:00.123"), stats)

Review Comment:
   Valid. Replaced the substring checks with exact JSON-path assertions for minValues.ts, minValues.nested.ts, maxValues.ts, and maxValues.nested.ts in bcfffcd5cd. The Delta 3.3 test passes.



##########
cpp/velox/tests/VeloxSubstraitRoundTripTest.cc:
##########
@@ -205,6 +206,38 @@ TEST_F(VeloxSubstraitRoundTripTest, countAll) {
   assertPlanConversion(plan, "SELECT count(*) as num_price FROM tmp WHERE c6 < 24 GROUP BY c0, c1");
 }
 
+TEST_F(VeloxSubstraitRoundTripTest, minMaxTimestampUtc) {
+  functions::aggregate::sparksql::registerAggregateFunctions("spark_");

Review Comment:
   Valid. Moved the Spark-prefixed aggregate registration to the test binary main(), so it runs once before the suite instead of mutating the global registry from an individual test. Updated in bcfffcd5cd.



##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenTimestampNtzAggregateSuite.scala:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql
+
+import org.apache.gluten.config.GlutenConfig
+import org.apache.gluten.execution.{HashAggregateExecBaseTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.functions.{max, min}
+import org.apache.spark.sql.internal.SQLConf
+
+import java.time.LocalDateTime
+
+class GlutenTimestampNtzAggregateSuite extends GlutenSQLTestsTrait {
+
+  import testImplicits._
+
+  testGluten("min and max") {
+    withSQLConf(
+      SQLConf.ANSI_ENABLED.key -> "false",
+      GlutenConfig.GLUTEN_ANSI_FALLBACK_ENABLED.key -> "false") {
+      withTempPath {
+        path =>
+          Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr("cast(input as timestamp_ntz) as ts")
+            .write
+            .parquet(path.getCanonicalPath)
+
+          val result = spark.read.parquet(path.getCanonicalPath).agg(min($"ts"), max($"ts"))
+          checkAnswer(
+            result,
+            Row(
+              LocalDateTime.parse("1969-12-31T23:59:59.999999"),
+              LocalDateTime.parse("2024-01-01T00:00:00.123456")))
+          assert(
+            getExecutedPlan(result).exists(_.isInstanceOf[HashAggregateExecBaseTransformer]),
+            result.queryExecution.executedPlan.treeString)
+      }
+    }
+  }
+
+  testGluten("unsupported project falls back") {
+    withSQLConf(
+      SQLConf.ANSI_ENABLED.key -> "false",
+      SQLConf.SESSION_LOCAL_TIMEZONE.key -> "America/Los_Angeles",
+      GlutenConfig.GLUTEN_ANSI_FALLBACK_ENABLED.key -> "false") {
+      withTempPath {
+        path =>
+          Seq("2024-01-01 00:00:00.123456")
+            .toDF("input")
+            .selectExpr("cast(input as timestamp_ntz) as ts")
+            .write
+            .parquet(path.getCanonicalPath)
+
+          val result = spark.read
+            .parquet(path.getCanonicalPath)
+            .selectExpr("to_json(named_struct('ts', ts))")
+          checkAnswer(result, Row("""{"ts":"2024-01-01T00:00:00.123"}"""))

Review Comment:
   Keeping the explicit expected string intentionally. This suite is Spark 4.1-specific and pins the session timezone, and the exact value is the correctness oracle for the bug: native execution previously shifted the NTZ value by the session offset. Deriving the expectation through the same expression path would weaken that regression check.



##########
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.test.DeltaSQLCommandTest
+
+class GlutenDeltaStatsSuite extends DeltaSQLCommandTest {
+
+  import testImplicits._
+
+  test("collect TIMESTAMP_NTZ statistics natively") {
+    withSQLConf(DeltaSQLConf.DELTA_COLLECT_STATS.key -> "true") {
+      withTempDir {
+        dir =>
+          val path = dir.getCanonicalPath
+          val data = Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr(
+              "cast(input as timestamp_ntz) as ts",
+              "struct(cast(input as timestamp_ntz) as ts) as nested")
+
+          data.coalesce(1).write.format("delta").save(path)
+
+          val actual = spark.read.format("delta").load(path)
+          assert(actual.collect().toSet == data.collect().toSet)
+
+          val addFiles = DeltaLog.forTable(spark, path).update().allFiles.collect()
+          assert(addFiles.length == 1)
+          val stats = addFiles.head.stats
+          assert(stats != null)
+          assert(stats.contains("\"minValues\""), stats)
+          assert(stats.contains("\"maxValues\""), stats)
+          assert(stats.contains("1969-12-31") && stats.contains("23:59:59.999"), stats)
+          assert(stats.contains("2024-01-01") && stats.contains("00:00:00.123"), stats)

Review Comment:
   Valid. Replaced the substring checks with exact JSON-path assertions for minValues.ts, minValues.nested.ts, maxValues.ts, and maxValues.nested.ts in bcfffcd5cd. The Delta 4.x test passes.



##########
cpp/velox/substrait/VeloxToSubstraitType.cc:
##########
@@ -31,6 +31,13 @@ const ::substrait::Type& VeloxToSubstraitTypeConvertor::toSubstraitType(
     substraitType->set_allocated_date(substraitDate);
     return *substraitType;
   }
+  if (type->equivalent(*velox::TIMESTAMP_UTC())) {
+    auto substraitTimestampNtz = google::protobuf::Arena::CreateMessage<::substrait::Type_PrecisionTimestamp>(&arena);
+    substraitTimestampNtz->set_precision(6);
+    substraitTimestampNtz->set_nullability(::substrait::Type_Nullability_NULLABILITY_NULLABLE);
+    substraitType->set_allocated_precision_timestamp(substraitTimestampNtz);

Review Comment:
   Agreed. Renamed the variable to substraitPrecisionTimestamp in bcfffcd5cd to match the emitted PrecisionTimestamp type.



##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/ConverterUtils.scala:
##########
@@ -414,7 +414,8 @@ object ConverterUtils extends Logging {
       case DoubleType => "fp64"
       case DateType => "date"
       case TimestampType => "ts"
-      case other if other.typeName == "timestamp_ntz" => "ts_ntz"
+      // Underscores delimit arguments in native function signatures.
+      case TimestampNTZType => "tsntz"

Review Comment:
   No compatibility alias added. Gluten generates these Substrait plans per query and requires matching JVM/native components; they are not a persisted cross-version format. More importantly, ts_ntz was never parseable for aggregates because the signature parser splits on underscores, so there is no previously valid aggregate encoding to preserve. Accepting it only in fromSubstraitSignature would not fix sigToTypes and would give a false compatibility guarantee.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5545415126

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5640330160

   > @felipepessoto Yes, I added the `TIMESTAMP_UTC` type in Velox. It was designed to share the same physical representation as `TimestampType` while having different logical semantics. The motivation was to represent the timezone-adjustment semantics correctly while still allowing us to reuse the existing implementations where possible.
   > 
   > My concern is that, since we haven’t yet verified that aggregate functions work correctly with `TIMESTAMP_UTC`, it may be better not to treat them as supported implicitly. Instead, we could explicitly validate the support in Velox by adding unit tests and updating the coverage documentation, even if no implementation changes turn out to be necessary.
   > 
   > Would you be interested in helping add this support for `min`/`max` in Velox first? Thanks.
   
   @rui-mo https://github.com/facebookincubator/velox/pull/18963. I left it in draft, I'm not sure if the tests are ok:
   
   <img width="571" height="77" alt="image" src="https://github.com/user-attachments/assets/defba74d-ed03-4828-8131-2583426e85a9" />
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5780644355

   Thanks @rui-mo, the Velox PR is merged. @zhztheplayer, could you check the Delta changes please?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r4077448462


##########
cpp/velox/tests/VeloxToSubstraitTypeTest.cc:
##########
@@ -62,4 +62,13 @@ TEST_F(VeloxToSubstraitTypeTest, basic) {
   testTypeConversion(ROW({}, {}));
 }
 
+TEST_F(VeloxToSubstraitTypeTest, timestampUtc) {
+  google::protobuf::Arena arena;
+  const auto& substraitType = typeConvertor_->toSubstraitType(arena, TIMESTAMP_UTC());
+
+  ASSERT_TRUE(substraitType.has_precision_timestamp());
+  ASSERT_EQ(substraitType.precision_timestamp().precision(), 6);
+  ASSERT_TRUE(SubstraitParser::parseType(substraitType)->equivalent(*TIMESTAMP_UTC()));

Review Comment:
   I exercised this locally with both required and nullable inputs. Assertions expecting  REQUIRED  to survive the round trip fail for both  TIMESTAMP_UTC  and ordinary  TIMESTAMP ; the proposed  nullable = false  overload does not exist. I’ve kept the reproducer separate for the potential follow-up issue rather than adding assertions that make the existing loss of requiredness an expectation of this PR. The original precision and logical-type coverage remains unchanged.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r4029957127


##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala:
##########
@@ -273,17 +279,21 @@ object Validators {
           case _ => false
         }
         val isSupportedNtz = plan match {
+          case _: HashAggregateExec | _: ObjectHashAggregateExec | _: SortAggregateExec => true
+          case _: ShuffleExchangeExec => true
           case p: ProjectExec =>
             p.projectList.forall {
               expr =>
                 (!containsNTZ(expr.dataType) &&
                   !expr.references.exists(a => containsNTZ(a.dataType))) ||
+                isDirectNtzProjection(expr) ||
                 expr.exists {
                   case Hour(child, _) => containsNTZ(child.dataType)
                   case Minute(child, _) => containsNTZ(child.dataType)
                   case Second(child, _) => containsNTZ(child.dataType)
                   case TimestampAdd(_, _, child, _) => containsNTZ(child.dataType)
                   case c: Cast if isNTZ(c.dataType) || isNTZ(c.child.dataType) => true
+                  case IsNull(child) => containsNTZ(child.dataType)

Review Comment:
   Added explicit `TIMESTAMP_UTC` coverage for Spark's `isnull` and `isnotnull` in [facebookincubator/velox#18963](https://github.com/facebookincubator/velox/pull/18963), commit [fdead606c](https://github.com/facebookincubator/velox/commit/fdead606cf2166c0e492fc5dc813d69a68f12685).
   
   The shared case matrix covers flat, constant, and dictionary inputs, including dictionary-level nulls and nulls in base vectors. Both predicates are compared with explicit non-null Boolean expectations derived from the declared null pattern, not from each other's output. No production implementation changes were needed.
   



##########
docs/velox-configuration.md:
##########
@@ -29,7 +29,7 @@ nav_order: 16
 | spark.gluten.sql.columnar.backend.velox.cudf.shuffleMaxPrefetchBytes             | 🔄 Dynamic    | 1028MB            | Maximum bytes to prefetch in CPU memory during GPU shuffle read while waiting for GPU available.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
                 |
 | spark.gluten.sql.columnar.backend.velox.directorySizeGuess                       | ⚓ Static      | 32KB              | Deprecated, rename to spark.gluten.sql.columnar.backend.velox.footerEstimatedSize                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
                 |
 | spark.gluten.sql.columnar.backend.velox.driverSideBroadcastHashTableBuild        | 🔄 Dynamic    | false             | Enable driver-side broadcast hash table build. When enabled, the hash table is built and serialized on the driver, then broadcast to executors. When disabled, each executor builds its own hash table from the broadcast data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                 |
-| spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation             | 🔄 Dynamic    | false             | Enable validation fallback for TimestampNTZ type. When true, any plan containing TimestampNTZ will fall back to Spark execution. When false, allows native execution for TimestampNTZ scan.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
                 |
+| spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation             | 🔄 Dynamic    | false             | Enable validation fallback for TimestampNTZ type. When true, plans with TimestampNTZ in their input or output schemas fall back to Spark execution. When false, supported TimestampNTZ scans, aggregates, shuffles and projections are eligible for native execution. Supported projections include direct columns and struct fields, casts, hour/minute/second extraction, timestamp addition and null checks. Other operator, expression and native validation rules still apply.                                                                                                                                                                                                                                                                                                                                                                                                                       
                 |

Review Comment:
   Regenerated `docs/velox-configuration.md` with the same high-level description in [5630a57695](https://github.com/apache/gluten/commit/5630a5769566a60b47b90e2997441a2aa1d0eda5).
   



##########
backends-velox/src/main/scala/org/apache/gluten/config/VeloxConfig.scala:
##########
@@ -981,9 +981,12 @@ object VeloxConfig extends ConfigRegistry {
   val ENABLE_TIMESTAMP_NTZ_VALIDATION =
     buildConf("spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation")
       .doc(
-        "Enable validation fallback for TimestampNTZ type. When true, any plan " +
-          "containing TimestampNTZ will fall back to Spark execution. When false, " +
-          "allows native execution for TimestampNTZ scan.")
+        "Enable validation fallback for TimestampNTZ type. When true, plans with " +
+          "TimestampNTZ in their input or output schemas fall back to Spark execution. " +
+          "When false, supported TimestampNTZ scans, aggregates, shuffles and projections " +
+          "are eligible for native execution. Supported projections include direct columns " +
+          "and struct fields, casts, hour/minute/second extraction, timestamp addition " +
+          "and null checks. Other operator, expression and native validation rules still apply.")

Review Comment:
   Simplified in [5630a57695](https://github.com/apache/gluten/commit/5630a5769566a60b47b90e2997441a2aa1d0eda5). Removed the operator/expression enumeration and kept only the flag's high-level fallback behavior and the caveat that other validation rules still apply.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5534433262

   /delta-test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5534996905

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r3937268049


##########
cpp/velox/substrait/VeloxSubstraitSignature.cc:
##########
@@ -155,6 +158,10 @@ TypePtr VeloxSubstraitSignature::fromSubstraitSignature(const std::string& signa
     return TIMESTAMP();
   }
 
+  if (signature == "tsntz") {

Review Comment:
   The signature token for TIMESTAMP_NTZ has changed (previously `ts_ntz`, now `tsntz`). If any previously serialized Substrait plans/signatures still use `ts_ntz`, `fromSubstraitSignature` will fail to interpret them. Consider accepting both `tsntz` and the legacy `ts_ntz` token during parsing (while continuing to emit only `tsntz`) to preserve backward compatibility.



##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenTimestampNtzAggregateSuite.scala:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql
+
+import org.apache.gluten.config.GlutenConfig
+import org.apache.gluten.execution.{HashAggregateExecBaseTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.functions.{max, min}
+import org.apache.spark.sql.internal.SQLConf
+
+import java.time.LocalDateTime
+
+class GlutenTimestampNtzAggregateSuite extends GlutenSQLTestsTrait {
+
+  import testImplicits._
+
+  testGluten("min and max") {
+    withSQLConf(
+      SQLConf.ANSI_ENABLED.key -> "false",
+      GlutenConfig.GLUTEN_ANSI_FALLBACK_ENABLED.key -> "false") {
+      withTempPath {
+        path =>
+          Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr("cast(input as timestamp_ntz) as ts")
+            .write
+            .parquet(path.getCanonicalPath)
+
+          val result = spark.read.parquet(path.getCanonicalPath).agg(min($"ts"), max($"ts"))
+          checkAnswer(
+            result,
+            Row(
+              LocalDateTime.parse("1969-12-31T23:59:59.999999"),
+              LocalDateTime.parse("2024-01-01T00:00:00.123456")))
+          assert(
+            getExecutedPlan(result).exists(_.isInstanceOf[HashAggregateExecBaseTransformer]),
+            result.queryExecution.executedPlan.treeString)
+      }
+    }
+  }
+
+  testGluten("unsupported project falls back") {
+    withSQLConf(
+      SQLConf.ANSI_ENABLED.key -> "false",
+      SQLConf.SESSION_LOCAL_TIMEZONE.key -> "America/Los_Angeles",
+      GlutenConfig.GLUTEN_ANSI_FALLBACK_ENABLED.key -> "false") {
+      withTempPath {
+        path =>
+          Seq("2024-01-01 00:00:00.123456")
+            .toDF("input")
+            .selectExpr("cast(input as timestamp_ntz) as ts")
+            .write
+            .parquet(path.getCanonicalPath)
+
+          val result = spark.read
+            .parquet(path.getCanonicalPath)
+            .selectExpr("to_json(named_struct('ts', ts))")
+          checkAnswer(result, Row("""{"ts":"2024-01-01T00:00:00.123"}"""))
+          assert(
+            !getExecutedPlan(result).exists(_.isInstanceOf[ProjectExecTransformer]),
+            result.queryExecution.executedPlan.treeString)

Review Comment:
   This assertion is brittle because it fails if *any* `ProjectExecTransformer` appears anywhere in the executed plan (even if unrelated to the `to_json(named_struct(...))` projection being tested). To make the test more stable, assert fallback more locally—e.g., identify the specific `ProjectExec` corresponding to the `to_json` expression and verify that node is not transformed (or verify that the plan contains a non-transformer `ProjectExec` for that projection).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto closed pull request #12967: [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation
URL: https://github.com/apache/gluten/pull/12967


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5534435655

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5535878073

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5840663185

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r4077466100


##########
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.gluten.execution.HashAggregateExecTransformer
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.stats.GlutenDeltaJobStatsTracker
+import org.apache.spark.sql.delta.test.DeltaSQLCommandTest
+import org.apache.spark.sql.execution.SparkPlan
+
+import java.util.concurrent.ConcurrentLinkedQueue
+
+import scala.collection.JavaConverters._
+
+class GlutenDeltaStatsSuite extends DeltaSQLCommandTest {
+
+  import testImplicits._
+
+  test("collect TIMESTAMP_NTZ statistics natively") {
+    withSQLConf(DeltaSQLConf.DELTA_COLLECT_STATS.key -> "true") {
+      withTempDir {
+        dir =>
+          val path = dir.getCanonicalPath
+          val data = Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")

Review Comment:
   A shared helper with version-specific wiring is possible, but I’d prefer to defer that refactoring and keep this PR focused unless maintainers prefer the extraction now. This seems to be aligned with other tests



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r3977731425


##########
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.test.DeltaSQLCommandTest
+
+class GlutenDeltaStatsSuite extends DeltaSQLCommandTest {
+
+  import testImplicits._
+
+  test("collect TIMESTAMP_NTZ statistics natively") {
+    withSQLConf(DeltaSQLConf.DELTA_COLLECT_STATS.key -> "true") {
+      withTempDir {
+        dir =>
+          val path = dir.getCanonicalPath
+          val data = Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr(
+              "cast(input as timestamp_ntz) as ts",
+              "struct(cast(input as timestamp_ntz) as ts) as nested")
+
+          data.coalesce(1).write.format("delta").save(path)
+
+          val actual = spark.read.format("delta").load(path)
+          assert(actual.collect().toSet == data.collect().toSet)
+
+          val addFiles = DeltaLog.forTable(spark, path).update().allFiles.collect()
+          assert(addFiles.length == 1)
+          val stats = addFiles.head.stats
+          assert(stats != null)
+          val statsValues = Seq(stats)
+            .toDF("stats")
+            .selectExpr(
+              "get_json_object(stats, '$.minValues.ts')",
+              "get_json_object(stats, '$.minValues.nested.ts')",
+              "get_json_object(stats, '$.maxValues.ts')",
+              "get_json_object(stats, '$.maxValues.nested.ts')"
+            )
+            .head()
+          assert(
+            statsValues == Row(
+              "1969-12-31T23:59:59.999",
+              "1969-12-31T23:59:59.999",
+              "2024-01-01T00:00:00.123",
+              "2024-01-01T00:00:00.123"),
+            stats)

Review Comment:
   > The test could then assert that the hook was invoked and that the captured plan contains HashAggregateExecTransformer. 
   
   Hi @zhztheplayer, do you have any preference for this? My intention was to make sure the test verifies native execution instead of silently falling back to vanilla Spark.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r3985500813


##########
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.test.DeltaSQLCommandTest
+
+class GlutenDeltaStatsSuite extends DeltaSQLCommandTest {
+
+  import testImplicits._
+
+  test("collect TIMESTAMP_NTZ statistics natively") {
+    withSQLConf(DeltaSQLConf.DELTA_COLLECT_STATS.key -> "true") {
+      withTempDir {
+        dir =>
+          val path = dir.getCanonicalPath
+          val data = Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr(
+              "cast(input as timestamp_ntz) as ts",
+              "struct(cast(input as timestamp_ntz) as ts) as nested")
+
+          data.coalesce(1).write.format("delta").save(path)
+
+          val actual = spark.read.format("delta").load(path)
+          assert(actual.collect().toSet == data.collect().toSet)
+
+          val addFiles = DeltaLog.forTable(spark, path).update().allFiles.collect()
+          assert(addFiles.length == 1)
+          val stats = addFiles.head.stats
+          assert(stats != null)
+          val statsValues = Seq(stats)
+            .toDF("stats")
+            .selectExpr(
+              "get_json_object(stats, '$.minValues.ts')",
+              "get_json_object(stats, '$.minValues.nested.ts')",
+              "get_json_object(stats, '$.maxValues.ts')",
+              "get_json_object(stats, '$.maxValues.nested.ts')"
+            )
+            .head()
+          assert(
+            statsValues == Row(
+              "1969-12-31T23:59:59.999",
+              "1969-12-31T23:59:59.999",
+              "2024-01-01T00:00:00.123",
+              "2024-01-01T00:00:00.123"),
+            stats)

Review Comment:
   @rui-mo @felipepessoto 
   
   > The approach I found would require changing production tracker code in both Delta variants to add a test observation hook.
   
   This approach sounds good to me. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r3977688877


##########
cpp/velox/substrait/VeloxSubstraitSignature.cc:
##########
@@ -24,6 +24,9 @@ std::string VeloxSubstraitSignature::toSubstraitSignature(const TypePtr& type) {
   if (type->isDate()) {
     return "date";
   }
+  if (type->equivalent(*TIMESTAMP_UTC())) {
+    return "tsntz";

Review Comment:
   Ok, this works to me as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r3932789529


##########
cpp/velox/tests/VeloxSubstraitRoundTripTest.cc:
##########
@@ -205,6 +205,33 @@ TEST_F(VeloxSubstraitRoundTripTest, countAll) {
   assertPlanConversion(plan, "SELECT count(*) as num_price FROM tmp WHERE c6 < 24 GROUP BY c0, c1");
 }
 
+TEST_F(VeloxSubstraitRoundTripTest, minMaxTimestampUtc) {
+  const auto minTimestamp = Timestamp(-1, 999'000'000);
+  const auto maxTimestamp = Timestamp(1'704'067'200, 123'000'000);

Review Comment:
   The TIMESTAMP_UTC min/max round-trip test uses nanosecond values that correspond to millisecond precision (999ms / 123ms), but this change set maps TIMESTAMP_UTC to Substrait precision_timestamp with precision=6 (microseconds). Using microsecond-level values here would better exercise the intended precision and avoid missing truncation/rounding regressions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r4077456648


##########
backends-velox/src-delta40/main/scala/org/apache/spark/sql/delta/stats/GlutenDeltaJobStatsTracker.scala:
##########
@@ -86,6 +86,22 @@ private[stats] class GlutenDeltaJobStatsTracker(val delegate: DeltaJobStatistics
 }
 
 object GlutenDeltaJobStatsTracker extends Logging {
+  private val statsPlanObserverLock = new Object
+  @volatile private var statsPlanObserver: Option[(Path, SparkPlan) => Unit] = None
+
+  /** Observes task-local statistics plans in local-mode tests; callbacks run on task threads. */
+  private[delta] def withStatsPlanObserver[T](observer: (Path, SparkPlan) => Unit)(f: => T): T =
+    statsPlanObserverLock.synchronized {
+      require(Utils.isTesting, "Statistics plan observation is only available in tests")
+      require(statsPlanObserver.isEmpty, "A statistics plan observer is already registered")
+      statsPlanObserver = Some(observer)
+      try {
+        f
+      } finally {
+        statsPlanObserver = None
+      }
+    }

Review Comment:
   The current implementation serializes observer registration scopes with a lock held across the write, publishes the callback through  @volatile , and clears it in  finally . The caller filters observations by its unique table path and stores matching plans in a  ConcurrentLinkedQueue , so unrelated writes do not contaminate its assertions. A  DynamicVariable  would not reliably propagate into Spark’s existing task threads. I’d retain this scoped local-mode hook unless we need concurrent independent observers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r4077483996


##########
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.gluten.execution.HashAggregateExecTransformer
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.stats.GlutenDeltaJobStatsTracker
+import org.apache.spark.sql.delta.test.DeltaSQLCommandTest
+import org.apache.spark.sql.execution.SparkPlan
+
+import java.util.concurrent.ConcurrentLinkedQueue
+
+import scala.collection.JavaConverters._

Review Comment:
   Not changing it to keep aligned with 3.3 file



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5702848473

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5534452788

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r4074929874


##########
cpp/velox/tests/VeloxToSubstraitTypeTest.cc:
##########
@@ -62,4 +62,13 @@ TEST_F(VeloxToSubstraitTypeTest, basic) {
   testTypeConversion(ROW({}, {}));
 }
 
+TEST_F(VeloxToSubstraitTypeTest, timestampUtc) {
+  google::protobuf::Arena arena;
+  const auto& substraitType = typeConvertor_->toSubstraitType(arena, TIMESTAMP_UTC());
+
+  ASSERT_TRUE(substraitType.has_precision_timestamp());
+  ASSERT_EQ(substraitType.precision_timestamp().precision(), 6);
+  ASSERT_TRUE(SubstraitParser::parseType(substraitType)->equivalent(*TIMESTAMP_UTC()));

Review Comment:
   This test validates precision and type equivalence but doesn’t assert nullability behavior. Add assertions (and/or an additional test case) to verify that `toSubstraitType(..., nullable = false)` produces `NULLABILITY_REQUIRED` (or the project’s equivalent) for `TIMESTAMP_UTC`, to prevent regressions like hard-coded `NULLABLE`.



##########
cpp/velox/substrait/VeloxToSubstraitType.cc:
##########
@@ -31,6 +31,14 @@ const ::substrait::Type& VeloxToSubstraitTypeConvertor::toSubstraitType(
     substraitType->set_allocated_date(substraitDate);
     return *substraitType;
   }
+  if (type->equivalent(*velox::TIMESTAMP_UTC())) {
+    auto substraitPrecisionTimestamp =
+        google::protobuf::Arena::CreateMessage<::substrait::Type_PrecisionTimestamp>(&arena);
+    substraitPrecisionTimestamp->set_precision(6);
+    substraitPrecisionTimestamp->set_nullability(::substrait::Type_Nullability_NULLABILITY_NULLABLE);
+    substraitType->set_allocated_precision_timestamp(substraitPrecisionTimestamp);
+    return *substraitType;
+  }

Review Comment:
   `toSubstraitType` unconditionally sets `PrecisionTimestamp` nullability to `NULLABLE` for `TIMESTAMP_UTC`, which can misrepresent non-nullable schemas. Propagate the method’s nullability input (consistent with other type branches) so required/not-null columns round-trip correctly.



##########
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.gluten.execution.HashAggregateExecTransformer
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.stats.GlutenDeltaJobStatsTracker
+import org.apache.spark.sql.delta.test.DeltaSQLCommandTest
+import org.apache.spark.sql.execution.SparkPlan
+
+import java.util.concurrent.ConcurrentLinkedQueue
+
+import scala.collection.JavaConverters._

Review Comment:
   `scala.collection.JavaConverters._` is deprecated on Scala versions that provide `scala.jdk.CollectionConverters._` (commonly used in Spark 4.x builds). Consider switching to `scala.jdk.CollectionConverters._` in Spark 4.1+/Delta 4.x test code to avoid deprecation warnings and keep the test sources forward-compatible.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r4074765394


##########
cpp/velox/substrait/VeloxToSubstraitType.cc:
##########
@@ -31,6 +31,14 @@ const ::substrait::Type& VeloxToSubstraitTypeConvertor::toSubstraitType(
     substraitType->set_allocated_date(substraitDate);
     return *substraitType;
   }
+  if (type->equivalent(*velox::TIMESTAMP_UTC())) {
+    auto substraitPrecisionTimestamp =
+        google::protobuf::Arena::CreateMessage<::substrait::Type_PrecisionTimestamp>(&arena);
+    substraitPrecisionTimestamp->set_precision(6);
+    substraitPrecisionTimestamp->set_nullability(::substrait::Type_Nullability_NULLABILITY_NULLABLE);
+    substraitType->set_allocated_precision_timestamp(substraitPrecisionTimestamp);
+    return *substraitType;
+  }

Review Comment:
   Nullability is being hard-coded to `NULLABILITY_NULLABLE` for `TIMESTAMP_UTC`. This can misrepresent NOT NULL schemas in the emitted Substrait plan. Please set nullability consistently with how other types in this method handle the `nullable` flag (or equivalent nullability input) rather than forcing nullable.



##########
backends-velox/src-delta40/main/scala/org/apache/spark/sql/delta/stats/GlutenDeltaJobStatsTracker.scala:
##########
@@ -86,6 +86,22 @@ private[stats] class GlutenDeltaJobStatsTracker(val delegate: DeltaJobStatistics
 }
 
 object GlutenDeltaJobStatsTracker extends Logging {
+  private val statsPlanObserverLock = new Object
+  @volatile private var statsPlanObserver: Option[(Path, SparkPlan) => Unit] = None
+
+  /** Observes task-local statistics plans in local-mode tests; callbacks run on task threads. */
+  private[delta] def withStatsPlanObserver[T](observer: (Path, SparkPlan) => Unit)(f: => T): T =
+    statsPlanObserverLock.synchronized {
+      require(Utils.isTesting, "Statistics plan observation is only available in tests")
+      require(statsPlanObserver.isEmpty, "A statistics plan observer is already registered")
+      statsPlanObserver = Some(observer)
+      try {
+        f
+      } finally {
+        statsPlanObserver = None
+      }
+    }

Review Comment:
   This introduces a JVM-global, mutable observer that can be accessed from multiple task threads. Even though it’s gated behind `Utils.isTesting`, it can still create cross-test interference if tests run concurrently in the same JVM (one suite registering while another triggers observation). A more isolated approach is to store the observer in a per-test dynamic scope (e.g., `scala.util.DynamicVariable`) or propagate an identifier via Spark local properties and keep observers keyed per test, so concurrent tests don’t contend on a single global slot.



##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenTimestampNtzAggregateSuite.scala:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql
+
+import org.apache.gluten.config.GlutenConfig
+import org.apache.gluten.execution.{HashAggregateExecBaseTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.execution.ProjectExec
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+import org.apache.spark.sql.functions.{max, min}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.TimestampNTZType
+
+import java.time.LocalDateTime
+
+class GlutenTimestampNtzAggregateSuite extends GlutenSQLTestsTrait {
+
+  import testImplicits._
+
+  testGluten("min and max") {
+    withSQLConf(
+      SQLConf.ANSI_ENABLED.key -> "false",
+      GlutenConfig.GLUTEN_ANSI_FALLBACK_ENABLED.key -> "false") {
+      withTempPath {
+        path =>
+          Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr("cast(input as timestamp_ntz) as ts")
+            .write
+            .parquet(path.getCanonicalPath)
+
+          val result = spark.read.parquet(path.getCanonicalPath).agg(min($"ts"), max($"ts"))
+          checkAnswer(
+            result,
+            Row(
+              LocalDateTime.parse("1969-12-31T23:59:59.999999"),
+              LocalDateTime.parse("2024-01-01T00:00:00.123456")))
+          assert(
+            getExecutedPlan(result).exists(_.isInstanceOf[HashAggregateExecBaseTransformer]),
+            result.queryExecution.executedPlan.treeString)
+      }
+    }
+  }
+
+  testGluten("min and max grouped by timestamp_ntz") {
+    withSQLConf(
+      SQLConf.ANSI_ENABLED.key -> "false",
+      SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
+      SQLConf.SESSION_LOCAL_TIMEZONE.key -> "America/Los_Angeles",
+      SQLConf.SHUFFLE_PARTITIONS.key -> "2",
+      GlutenConfig.GLUTEN_ANSI_FALLBACK_ENABLED.key -> "false",
+      "spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation" -> "false"
+    ) {
+      withTempPath {
+        path =>
+          val beforeEpoch = LocalDateTime.parse("1969-12-31T23:59:59.999999")
+          val afterEpoch = LocalDateTime.parse("1970-01-01T00:00:00.000001")
+          val firstKey = LocalDateTime.parse("2024-01-01T00:00:00.123456")
+          val secondKey = LocalDateTime.parse("2024-01-01T00:00:00.123457")
+          Seq(
+            (firstKey, beforeEpoch),
+            (firstKey, secondKey),
+            (firstKey, null),
+            (secondKey, afterEpoch),
+            (secondKey, firstKey),
+            (null, beforeEpoch),
+            (null, afterEpoch),
+            (beforeEpoch, null)
+          ).toDF("key", "ts")
+            .write
+            .parquet(path.getCanonicalPath)
+
+          val result = spark.read
+            .parquet(path.getCanonicalPath)
+            .groupBy($"key")
+            .agg(min($"ts"), max($"ts"))
+          checkAnswer(
+            result,
+            Seq(
+              Row(firstKey, beforeEpoch, secondKey),
+              Row(secondKey, afterEpoch, firstKey),
+              Row(null, beforeEpoch, afterEpoch),
+              Row(beforeEpoch, null, null)))
+          val aggregates = getExecutedPlan(result).collect {
+            case aggregate: BaseAggregateExec => aggregate
+          }
+          assert(aggregates.nonEmpty, result.queryExecution.executedPlan.treeString)
+          assert(
+            aggregates.forall {
+              case aggregate: HashAggregateExecBaseTransformer =>
+                aggregate.groupingExpressions.map(_.dataType) == Seq(TimestampNTZType)
+              case _ => false
+            },
+            result.queryExecution.executedPlan.treeString
+          )

Review Comment:
   This assertion is brittle because it requires *every* `BaseAggregateExec` in the executed plan to be a `HashAggregateExecBaseTransformer`. If Spark inserts additional aggregates (e.g., multiple stages) or a non-hash aggregate appears alongside the transformer, the test will fail even if the intended native aggregate is present. Prefer filtering/collecting `HashAggregateExecBaseTransformer` nodes directly and asserting on those (and/or separately asserting that at least one such node exists).



##########
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.gluten.execution.HashAggregateExecTransformer
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.stats.GlutenDeltaJobStatsTracker
+import org.apache.spark.sql.delta.test.DeltaSQLCommandTest
+import org.apache.spark.sql.execution.SparkPlan
+
+import java.util.concurrent.ConcurrentLinkedQueue
+
+import scala.collection.JavaConverters._
+
+class GlutenDeltaStatsSuite extends DeltaSQLCommandTest {
+
+  import testImplicits._
+
+  test("collect TIMESTAMP_NTZ statistics natively") {
+    withSQLConf(DeltaSQLConf.DELTA_COLLECT_STATS.key -> "true") {
+      withTempDir {
+        dir =>
+          val path = dir.getCanonicalPath
+          val data = Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")

Review Comment:
   The Delta 3.3 and Delta 4.0 versions of `GlutenDeltaStatsSuite` appear to be identical in this PR. To reduce duplication and keep future changes consistent, consider extracting the shared test logic into a common trait/helper (e.g., under a shared test source directory) and keeping only minimal version-specific wiring in each module.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5668130243

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r4026631740


##########
backends-velox/src/main/scala/org/apache/gluten/config/VeloxConfig.scala:
##########
@@ -981,9 +981,12 @@ object VeloxConfig extends ConfigRegistry {
   val ENABLE_TIMESTAMP_NTZ_VALIDATION =
     buildConf("spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation")
       .doc(
-        "Enable validation fallback for TimestampNTZ type. When true, any plan " +
-          "containing TimestampNTZ will fall back to Spark execution. When false, " +
-          "allows native execution for TimestampNTZ scan.")
+        "Enable validation fallback for TimestampNTZ type. When true, plans with " +
+          "TimestampNTZ in their input or output schemas fall back to Spark execution. " +
+          "When false, supported TimestampNTZ scans, aggregates, shuffles and projections " +
+          "are eligible for native execution. Supported projections include direct columns " +
+          "and struct fields, casts, hour/minute/second extraction, timestamp addition " +
+          "and null checks. Other operator, expression and native validation rules still apply.")

Review Comment:
   > Supported projections include direct columns and struct fields, casts, hour/minute/second extraction, timestamp addition and null checks.
   
   Support for `timestamp_ntz` is evolving quickly and may change frequently, so could we avoid documenting those implementation details for now? It would be better to keep the configuration documentation at a high level.
   



##########
docs/velox-configuration.md:
##########
@@ -29,7 +29,7 @@ nav_order: 16
 | spark.gluten.sql.columnar.backend.velox.cudf.shuffleMaxPrefetchBytes             | 🔄 Dynamic    | 1028MB            | Maximum bytes to prefetch in CPU memory during GPU shuffle read while waiting for GPU available.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
                 |
 | spark.gluten.sql.columnar.backend.velox.directorySizeGuess                       | ⚓ Static      | 32KB              | Deprecated, rename to spark.gluten.sql.columnar.backend.velox.footerEstimatedSize                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
                 |
 | spark.gluten.sql.columnar.backend.velox.driverSideBroadcastHashTableBuild        | 🔄 Dynamic    | false             | Enable driver-side broadcast hash table build. When enabled, the hash table is built and serialized on the driver, then broadcast to executors. When disabled, each executor builds its own hash table from the broadcast data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                 |
-| spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation             | 🔄 Dynamic    | false             | Enable validation fallback for TimestampNTZ type. When true, any plan containing TimestampNTZ will fall back to Spark execution. When false, allows native execution for TimestampNTZ scan.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
                 |
+| spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation             | 🔄 Dynamic    | false             | Enable validation fallback for TimestampNTZ type. When true, plans with TimestampNTZ in their input or output schemas fall back to Spark execution. When false, supported TimestampNTZ scans, aggregates, shuffles and projections are eligible for native execution. Supported projections include direct columns and struct fields, casts, hour/minute/second extraction, timestamp addition and null checks. Other operator, expression and native validation rules still apply.                                                                                                                                                                                                                                                                                                                                                                                                                       
                 |

Review Comment:
   ditto



##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala:
##########
@@ -273,17 +279,21 @@ object Validators {
           case _ => false
         }
         val isSupportedNtz = plan match {
+          case _: HashAggregateExec | _: ObjectHashAggregateExec | _: SortAggregateExec => true
+          case _: ShuffleExchangeExec => true
           case p: ProjectExec =>
             p.projectList.forall {
               expr =>
                 (!containsNTZ(expr.dataType) &&
                   !expr.references.exists(a => containsNTZ(a.dataType))) ||
+                isDirectNtzProjection(expr) ||
                 expr.exists {
                   case Hour(child, _) => containsNTZ(child.dataType)
                   case Minute(child, _) => containsNTZ(child.dataType)
                   case Second(child, _) => containsNTZ(child.dataType)
                   case TimestampAdd(_, _, child, _) => containsNTZ(child.dataType)
                   case c: Cast if isNTZ(c.dataType) || isNTZ(c.child.dataType) => true
+                  case IsNull(child) => containsNTZ(child.dataType)

Review Comment:
   `isnull` is also unverified for TIMESTAMP_UTC type in Velox.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5694533045

   Addressed both suggestions from [this review](https://github.com/apache/gluten/pull/12967#pullrequestreview-5219697921) in [b97a45a9d2](https://github.com/apache/gluten/commit/b97a45a9d2e43d1aa0936b14db2131ead3a9e784).
   
   Added a grouped min/max regression with a `TIMESTAMP_NTZ` grouping key and a separate NTZ value column. It checks exact per-key results and requires every aggregate stage to be native with an NTZ grouping key. The inputs include keys one microsecond apart, null keys, and an all-null value group.
   
   Updated the `VeloxConfig.ENABLE_TIMESTAMP_NTZ_VALIDATION` description and regenerated `docs/velox-configuration.md`. The documentation now explains that `false` permits supported NTZ scans, aggregates, shuffles and projections, while other operator, expression and native validation rules still apply. The flag and its default are unchanged.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r4029736350


##########
cpp/velox/substrait/VeloxToSubstraitType.cc:
##########
@@ -31,6 +31,14 @@ const ::substrait::Type& VeloxToSubstraitTypeConvertor::toSubstraitType(
     substraitType->set_allocated_date(substraitDate);
     return *substraitType;
   }
+  if (type->equivalent(*velox::TIMESTAMP_UTC())) {
+    auto substraitPrecisionTimestamp =
+        google::protobuf::Arena::CreateMessage<::substrait::Type_PrecisionTimestamp>(&arena);
+    substraitPrecisionTimestamp->set_precision(6);
+    substraitPrecisionTimestamp->set_nullability(::substrait::Type_Nullability_NULLABILITY_NULLABLE);

Review Comment:
   This adds the `PrecisionTimestamp` type mapping, but timestamp literals still go through `VeloxToSubstraitExpr`'s `TypeKind::TIMESTAMP` path, which emits `timestamp_tz`; parsing that literal reconstructs `TIMESTAMP()`, not `TIMESTAMP_UTC()`. A plan containing a TIMESTAMP_NTZ literal can therefore lose its logical type during round-trip (and fail or change semantics in mixed NTZ operations). Pass the logical type into literal conversion and emit the NTZ literal form, with a regression test for a literal/Values plan.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5704727906

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5781822557

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5544948909

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5545941085

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r3960735492


##########
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.test.DeltaSQLCommandTest
+
+class GlutenDeltaStatsSuite extends DeltaSQLCommandTest {
+
+  import testImplicits._
+
+  test("collect TIMESTAMP_NTZ statistics natively") {
+    withSQLConf(DeltaSQLConf.DELTA_COLLECT_STATS.key -> "true") {
+      withTempDir {
+        dir =>
+          val path = dir.getCanonicalPath
+          val data = Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr(
+              "cast(input as timestamp_ntz) as ts",
+              "struct(cast(input as timestamp_ntz) as ts) as nested")
+
+          data.coalesce(1).write.format("delta").save(path)
+
+          val actual = spark.read.format("delta").load(path)
+          assert(actual.collect().toSet == data.collect().toSet)
+
+          val addFiles = DeltaLog.forTable(spark, path).update().allFiles.collect()
+          assert(addFiles.length == 1)
+          val stats = addFiles.head.stats
+          assert(stats != null)
+          val statsValues = Seq(stats)
+            .toDF("stats")
+            .selectExpr(
+              "get_json_object(stats, '$.minValues.ts')",
+              "get_json_object(stats, '$.minValues.nested.ts')",
+              "get_json_object(stats, '$.maxValues.ts')",
+              "get_json_object(stats, '$.maxValues.nested.ts')"
+            )
+            .head()
+          assert(
+            statsValues == Row(
+              "1969-12-31T23:59:59.999",
+              "1969-12-31T23:59:59.999",
+              "2024-01-01T00:00:00.123",
+              "2024-01-01T00:00:00.123"),
+            stats)

Review Comment:
   I agree that an explicit offload assertion would strengthen this regression. The statistics aggregate is constructed locally inside `GlutenDeltaJobStatsTracker` in each write task and submitted directly to `NativePlanEvaluator`, so it is not visible in the outer DataFrame's executed plan ([code](https://github.com/apache/gluten/blob/6f80ab9cf4eac841c23c1ea4af4e2b3fb5f66e94/backends-velox/src-delta40/main/scala/org/apache/spark/sql/delta/stats/GlutenDeltaJobStatsTracker.scala#L151-L203)).
   
   The approach I found would require changing production tracker code in both Delta variants to add a test observation hook. The test could then assert that the hook was invoked and that the captured plan contains `HashAggregateExecTransformer`. Would you be comfortable with that, or do you know a simpler approach, such as an existing way to observe this internal plan from tests without adding a new hook?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r3954120668


##########
cpp/velox/substrait/VeloxSubstraitSignature.cc:
##########
@@ -24,6 +24,9 @@ std::string VeloxSubstraitSignature::toSubstraitSignature(const TypePtr& type) {
   if (type->isDate()) {
     return "date";
   }
+  if (type->equivalent(*TIMESTAMP_UTC())) {
+    return "tsntz";

Review Comment:
   `ts_ntz` would align visually with Spark, but it cannot represent one type in Gluten's current function-signature grammar. The native parser uses `_` as the delimiter between argument types
   ([parser](https://github.com/apache/gluten/blob/6f80ab9cf4eac841c23c1ea4af4e2b3fb5f66e94/cpp/velox/substrait/SubstraitParser.cc#L236-L261)), so `min:ts_ntz` is parsed as two argument types, `ts` and `ntz`, and aggregate validation fails when it tries to resolve `ntz`.
   
   `tsntz` is deliberately an internal, separator-free signature token ([mapping](https://github.com/apache/gluten/blob/6f80ab9cf4eac841c23c1ea4af4e2b3fb5f66e94/gluten-substrait/src/main/scala/org/apache/gluten/expression/ConverterUtils.scala#L405-L419)). The Spark-facing type remains `timestamp_ntz`, and the Substrait representation remains `PrecisionTimestamp`. Using `ts_ntz` would require first changing the signature grammar to support escaping or structured type tokens.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r3992030847


##########
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.test.DeltaSQLCommandTest
+
+class GlutenDeltaStatsSuite extends DeltaSQLCommandTest {
+
+  import testImplicits._
+
+  test("collect TIMESTAMP_NTZ statistics natively") {
+    withSQLConf(DeltaSQLConf.DELTA_COLLECT_STATS.key -> "true") {
+      withTempDir {
+        dir =>
+          val path = dir.getCanonicalPath
+          val data = Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr(
+              "cast(input as timestamp_ntz) as ts",
+              "struct(cast(input as timestamp_ntz) as ts) as nested")
+
+          data.coalesce(1).write.format("delta").save(path)
+
+          val actual = spark.read.format("delta").load(path)
+          assert(actual.collect().toSet == data.collect().toSet)
+
+          val addFiles = DeltaLog.forTable(spark, path).update().allFiles.collect()
+          assert(addFiles.length == 1)
+          val stats = addFiles.head.stats
+          assert(stats != null)
+          val statsValues = Seq(stats)
+            .toDF("stats")
+            .selectExpr(
+              "get_json_object(stats, '$.minValues.ts')",
+              "get_json_object(stats, '$.minValues.nested.ts')",
+              "get_json_object(stats, '$.maxValues.ts')",
+              "get_json_object(stats, '$.maxValues.nested.ts')"
+            )
+            .head()
+          assert(
+            statsValues == Row(
+              "1969-12-31T23:59:59.999",
+              "1969-12-31T23:59:59.999",
+              "2024-01-01T00:00:00.123",
+              "2024-01-01T00:00:00.123"),
+            stats)

Review Comment:
   Thanks @rui-mo and @zhztheplayer. Implemented it. Both Delta 3.3 and 4.x regression suites now capture the statistics tracker's task-local transformed plan, assert that at least one plan was observed for the write, and verify that every captured plan contains `HashAggregateExecTransformer`. This prevents a write that falls back entirely to vanilla Delta from silently passing.
   
   The hook is enabled only in test mode. These local-mode tests capture plans in a thread-safe queue, and the observer is scoped around the write with cleanup in `finally`. Normal production execution does not invoke the callback.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r3936638360


##########
cpp/velox/tests/VeloxSubstraitRoundTripTest.cc:
##########
@@ -205,6 +205,33 @@ TEST_F(VeloxSubstraitRoundTripTest, countAll) {
   assertPlanConversion(plan, "SELECT count(*) as num_price FROM tmp WHERE c6 < 24 GROUP BY c0, c1");
 }
 
+TEST_F(VeloxSubstraitRoundTripTest, minMaxTimestampUtc) {
+  const auto minTimestamp = Timestamp(-1, 999'000'000);
+  const auto maxTimestamp = Timestamp(1'704'067'200, 123'000'000);

Review Comment:
   Good catch. Updated the round-trip test to use the Spark-prefixed min/max implementation with microsecond-level values, and added an explicit assertion that the emitted Substrait PrecisionTimestamp has precision 6. The focused native tests pass in e6e42bd55f.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto closed pull request #12967: [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation
URL: https://github.com/apache/gluten/pull/12967


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r4077434434


##########
cpp/velox/substrait/VeloxToSubstraitType.cc:
##########
@@ -31,6 +31,14 @@ const ::substrait::Type& VeloxToSubstraitTypeConvertor::toSubstraitType(
     substraitType->set_allocated_date(substraitDate);
     return *substraitType;
   }
+  if (type->equivalent(*velox::TIMESTAMP_UTC())) {
+    auto substraitPrecisionTimestamp =
+        google::protobuf::Arena::CreateMessage<::substrait::Type_PrecisionTimestamp>(&arena);
+    substraitPrecisionTimestamp->set_precision(6);
+    substraitPrecisionTimestamp->set_nullability(::substrait::Type_Nullability_NULLABILITY_NULLABLE);
+    substraitType->set_allocated_precision_timestamp(substraitPrecisionTimestamp);
+    return *substraitType;
+  }

Review Comment:
   This converter currently accepts only  (arena, type) , with no nullability argument. The parser does not retain nullability in the Velox type, and existing conversion branches—including ordinary  TIMESTAMP —also emit  NULLABLE . Local probes confirmed that required inputs become nullable for both timestamp types, so this behavior is not introduced by the NTZ branch. @rui-mo, should we open a separate issue to track preserving schema nullability end to end, since this is existing converter-wide behavior?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5693569038

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5534959330

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r3936776217


##########
backends-velox/src-delta33/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.test.DeltaSQLCommandTest
+
+class GlutenDeltaStatsSuite extends DeltaSQLCommandTest {
+
+  import testImplicits._
+
+  test("collect TIMESTAMP_NTZ statistics natively") {
+    withSQLConf(DeltaSQLConf.DELTA_COLLECT_STATS.key -> "true") {
+      withTempDir {
+        dir =>
+          val path = dir.getCanonicalPath
+          val data = Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr(
+              "cast(input as timestamp_ntz) as ts",
+              "struct(cast(input as timestamp_ntz) as ts) as nested")
+
+          data.coalesce(1).write.format("delta").save(path)
+
+          val actual = spark.read.format("delta").load(path)
+          assert(actual.collect().toSet == data.collect().toSet)
+
+          val addFiles = DeltaLog.forTable(spark, path).update().allFiles.collect()
+          assert(addFiles.length == 1)
+          val stats = addFiles.head.stats
+          assert(stats != null)
+          assert(stats.contains("\"minValues\""), stats)
+          assert(stats.contains("\"maxValues\""), stats)
+          assert(stats.contains("1969-12-31") && stats.contains("23:59:59.999"), stats)
+          assert(stats.contains("2024-01-01") && stats.contains("00:00:00.123"), stats)

Review Comment:
   Same issue as the Delta 4.0 variant: substring checks can yield false positives and don’t verify the correct JSON paths/columns. Prefer parsing `stats` JSON and asserting exact `minValues`/`maxValues` values for `ts` and `nested.ts`.



##########
cpp/velox/tests/VeloxSubstraitRoundTripTest.cc:
##########
@@ -205,6 +206,38 @@ TEST_F(VeloxSubstraitRoundTripTest, countAll) {
   assertPlanConversion(plan, "SELECT count(*) as num_price FROM tmp WHERE c6 < 24 GROUP BY c0, c1");
 }
 
+TEST_F(VeloxSubstraitRoundTripTest, minMaxTimestampUtc) {
+  functions::aggregate::sparksql::registerAggregateFunctions("spark_");

Review Comment:
   `registerAggregateFunctions` typically mutates a global function registry. Calling it inside a single test can introduce order-dependence or duplicate-registration issues if other tests also register the same prefix (now or in the future). Consider registering once for the whole test suite (e.g., in the fixture’s `SetUpTestSuite()` guarded by `std::once_flag`) to make this deterministic.



##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/ConverterUtils.scala:
##########
@@ -414,7 +414,8 @@ object ConverterUtils extends Logging {
       case DoubleType => "fp64"
       case DateType => "date"
       case TimestampType => "ts"
-      case other if other.typeName == "timestamp_ntz" => "ts_ntz"
+      // Underscores delimit arguments in native function signatures.
+      case TimestampNTZType => "tsntz"

Review Comment:
   Changing the `TIMESTAMP_NTZ` signature from `ts_ntz` to `tsntz` can be a compatibility break for any persisted/cached Substrait plans or cross-version components that still emit/expect `ts_ntz`. Consider adding backward-compatible parsing/aliasing (accept both `ts_ntz` and `tsntz` on the ‘from signature’ path and/or in any signature normalization), and document the deprecation if supported.



##########
cpp/velox/substrait/VeloxToSubstraitType.cc:
##########
@@ -31,6 +31,13 @@ const ::substrait::Type& VeloxToSubstraitTypeConvertor::toSubstraitType(
     substraitType->set_allocated_date(substraitDate);
     return *substraitType;
   }
+  if (type->equivalent(*velox::TIMESTAMP_UTC())) {
+    auto substraitTimestampNtz = google::protobuf::Arena::CreateMessage<::substrait::Type_PrecisionTimestamp>(&arena);
+    substraitTimestampNtz->set_precision(6);
+    substraitTimestampNtz->set_nullability(::substrait::Type_Nullability_NULLABILITY_NULLABLE);
+    substraitType->set_allocated_precision_timestamp(substraitTimestampNtz);

Review Comment:
   `substraitTimestampNtz` is misleading here because the emitted Substrait type is `PrecisionTimestamp` for `TIMESTAMP_UTC`. Renaming this variable to something like `substraitPrecisionTimestamp`/`substraitTimestampUtc` would make the mapping clearer and reduce confusion when debugging type conversions.



##########
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.test.DeltaSQLCommandTest
+
+class GlutenDeltaStatsSuite extends DeltaSQLCommandTest {
+
+  import testImplicits._
+
+  test("collect TIMESTAMP_NTZ statistics natively") {
+    withSQLConf(DeltaSQLConf.DELTA_COLLECT_STATS.key -> "true") {
+      withTempDir {
+        dir =>
+          val path = dir.getCanonicalPath
+          val data = Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr(
+              "cast(input as timestamp_ntz) as ts",
+              "struct(cast(input as timestamp_ntz) as ts) as nested")
+
+          data.coalesce(1).write.format("delta").save(path)
+
+          val actual = spark.read.format("delta").load(path)
+          assert(actual.collect().toSet == data.collect().toSet)
+
+          val addFiles = DeltaLog.forTable(spark, path).update().allFiles.collect()
+          assert(addFiles.length == 1)
+          val stats = addFiles.head.stats
+          assert(stats != null)
+          assert(stats.contains("\"minValues\""), stats)
+          assert(stats.contains("\"maxValues\""), stats)
+          assert(stats.contains("1969-12-31") && stats.contains("23:59:59.999"), stats)
+          assert(stats.contains("2024-01-01") && stats.contains("00:00:00.123"), stats)

Review Comment:
   These assertions are substring-based and can pass even if the stats structure is malformed or if min/max values are present but associated with the wrong column (e.g., `nested.ts` vs `ts`). To make the regression more robust, parse `stats` as JSON and assert the specific `minValues`/`maxValues` entries for both `ts` and `nested.ts` match the expected serialized values.



##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenTimestampNtzAggregateSuite.scala:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql
+
+import org.apache.gluten.config.GlutenConfig
+import org.apache.gluten.execution.{HashAggregateExecBaseTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.functions.{max, min}
+import org.apache.spark.sql.internal.SQLConf
+
+import java.time.LocalDateTime
+
+class GlutenTimestampNtzAggregateSuite extends GlutenSQLTestsTrait {
+
+  import testImplicits._
+
+  testGluten("min and max") {
+    withSQLConf(
+      SQLConf.ANSI_ENABLED.key -> "false",
+      GlutenConfig.GLUTEN_ANSI_FALLBACK_ENABLED.key -> "false") {
+      withTempPath {
+        path =>
+          Seq(
+            "1969-12-31 23:59:59.999999",
+            "2024-01-01 00:00:00.123456"
+          ).toDF("input")
+            .selectExpr("cast(input as timestamp_ntz) as ts")
+            .write
+            .parquet(path.getCanonicalPath)
+
+          val result = spark.read.parquet(path.getCanonicalPath).agg(min($"ts"), max($"ts"))
+          checkAnswer(
+            result,
+            Row(
+              LocalDateTime.parse("1969-12-31T23:59:59.999999"),
+              LocalDateTime.parse("2024-01-01T00:00:00.123456")))
+          assert(
+            getExecutedPlan(result).exists(_.isInstanceOf[HashAggregateExecBaseTransformer]),
+            result.queryExecution.executedPlan.treeString)
+      }
+    }
+  }
+
+  testGluten("unsupported project falls back") {
+    withSQLConf(
+      SQLConf.ANSI_ENABLED.key -> "false",
+      SQLConf.SESSION_LOCAL_TIMEZONE.key -> "America/Los_Angeles",
+      GlutenConfig.GLUTEN_ANSI_FALLBACK_ENABLED.key -> "false") {
+      withTempPath {
+        path =>
+          Seq("2024-01-01 00:00:00.123456")
+            .toDF("input")
+            .selectExpr("cast(input as timestamp_ntz) as ts")
+            .write
+            .parquet(path.getCanonicalPath)
+
+          val result = spark.read
+            .parquet(path.getCanonicalPath)
+            .selectExpr("to_json(named_struct('ts', ts))")
+          checkAnswer(result, Row("""{"ts":"2024-01-01T00:00:00.123"}"""))

Review Comment:
   The expected JSON timestamp string format here is hard-coded and may vary across Spark versions/config (e.g., millisecond vs microsecond rendering). To reduce brittleness, consider deriving the expected JSON using Spark’s own expression evaluation (e.g., build an expected DataFrame using the same `to_json(named_struct(...))` over a literal `timestamp_ntz`) and compare Rows, rather than pinning an exact string representation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5538677060

   @Mariamalmesfer @rui-mo I think you two have worked on other timestamp_ntz PRs. Could you take a look, please?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on code in PR #12967:
URL: https://github.com/apache/gluten/pull/12967#discussion_r4077436672


##########
cpp/velox/substrait/VeloxToSubstraitType.cc:
##########
@@ -31,6 +31,14 @@ const ::substrait::Type& VeloxToSubstraitTypeConvertor::toSubstraitType(
     substraitType->set_allocated_date(substraitDate);
     return *substraitType;
   }
+  if (type->equivalent(*velox::TIMESTAMP_UTC())) {
+    auto substraitPrecisionTimestamp =
+        google::protobuf::Arena::CreateMessage<::substrait::Type_PrecisionTimestamp>(&arena);
+    substraitPrecisionTimestamp->set_precision(6);
+    substraitPrecisionTimestamp->set_nullability(::substrait::Type_Nullability_NULLABILITY_NULLABLE);
+    substraitType->set_allocated_precision_timestamp(substraitPrecisionTimestamp);
+    return *substraitType;
+  }

Review Comment:
   Duplicated of https://github.com/apache/gluten/pull/12967#discussion_r4074929820



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan merged PR #12967:
URL: https://github.com/apache/gluten/pull/12967


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5589139525

   > Thanks @felipepessoto. Could you please clarify whether this implementation is based on the aggregate implementation for the `Timestamp` type in Velox, and whether it can be fully reused for the `TimestampNTZ` type?
   
   Yes, for `min` and `max`, this PR reuses Velox's existing timestamp aggregate implementation; it does not add a separate `TIMESTAMP_NTZ` aggregate kernel ([code](https://github.com/facebookincubator/velox/blob/68c4501e9ae33a2d6041cc3f9ce40e0bec376fc7/velox/functions/lib/aggregates/MinMaxAggregateBase.cpp#L613-L630)).
   
   Velox represents Spark `TIMESTAMP_NTZ` as `TIMESTAMP_UTC`. `TimestampUtcType` derives from `TimestampType`, so it has the same `TypeKind::TIMESTAMP` and native `Timestamp` value representation, while retaining distinct logical type identity and no session-timezone adjustment ([Velox type](https://github.com/facebookincubator/velox/blob/68c4501e9ae33a2d6041cc3f9ce40e0bec376fc7/velox/type/Type.h#L1172-L1201)). The shared min/max factory dispatches `TypeKind::TIMESTAMP` to the existing `SimpleNumericMin/MaxAggregate<Timestamp>` implementation and carries the supplied result type through ([factory](https://github.com/facebookincubator/velox/blob/68c4501e9ae33a2d6041cc3f9ce40e0bec376fc7/velox/functions/lib/aggregates/MinMaxAggregateBase.cpp#L595-L630)). Spark's registration configures that implementation with microsecond precision ([registration](https://github.com/facebookincubator/velox/blob/68c4501e9ae33a2d6041cc3f9ce40e0bec376fc7/velox/functions/sparksql/aggregates/MinMaxAggregate.
 cpp#L23-L78)).
   
   The Gluten changes preserve `TIMESTAMP_NTZ` as `TIMESTAMP_UTC` through the function signature and Substrait output type, then allow the plan to reach native validation. This does not assume every aggregate function supports `TIMESTAMP_NTZ`: each aggregate is still checked by native signature validation, and unsupported functions continue to fall back. The behavior specifically enabled and asserted here is the `min`/`max` path used by Delta statistics.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5547243178

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-12538][VL] Unblock TIMESTAMP_NTZ min/max in Delta statistics [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12967:
URL: https://github.com/apache/gluten/pull/12967#issuecomment-5638557987

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org