You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "liujiayi771 (via GitHub)" <gi...@apache.org> on 2025/11/13 02:51:06 UTC

[PR] [CORE] Use Substrait timestamp_tz for Spark TimestampType to preserve timezone-aware semantics [incubator-gluten]

liujiayi771 opened a new pull request, #11074:
URL: https://github.com/apache/incubator-gluten/pull/11074

   ## What changes are proposed in this pull request?
   
   Spark’s TimestampType is timezone-aware: it internally stores timestamps in UTC (by converting input values to UTC based on the session time zone or just read UTC timestamp from parquet file) and represents an absolute point in time. This semantics aligns with Substrait’s timestamp_tz type, which also denotes a timezone-aware timestamp that can be unambiguously mapped to a moment on the timeline.
   
   To maintain semantic consistency between Spark and Substrait, this PR maps Spark’s TimestampType to Substrait’s timestamp_tz.
   
   https://substrait.io/types/type_classes
   
   This approach is consistent with other projects—for example, Apache Iceberg also maps Spark’s TimestampType to TIMESTAMP WITH TIME ZONE and Spark’s TimestampNTZ to TIMESTAMP WITHOUT TIME ZONE.
   
   Note: For future support of Spark’s TimestampNTZType (timezone-naive timestamps), we should map it to Substrait’s timestamp type instead.
   
   ## How was this patch tested?
   
   The existing tests already cover this change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [CORE] Use Substrait timestamp_tz for Spark TimestampType to preserve timezone-aware semantics [incubator-gluten]

Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on PR #11074:
URL: https://github.com/apache/incubator-gluten/pull/11074#issuecomment-3552115899

   > @rui-mo @zhztheplayer Could you please help review this? @taiyang-li @lgbo-ustc @zzcclp could you confirm whether the changes here have any impact on the ClickHouse backend?
   
   CH backend changes are ok to me. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [CORE] Use Substrait timestamp_tz for Spark TimestampType to preserve timezone-aware semantics [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11074:
URL: https://github.com/apache/incubator-gluten/pull/11074#issuecomment-3525125415

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [CORE] Use Substrait timestamp_tz for Spark TimestampType to preserve timezone-aware semantics [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11074:
URL: https://github.com/apache/incubator-gluten/pull/11074#issuecomment-3525673338

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [CORE] Use Substrait timestamp_tz for Spark TimestampType to preserve timezone-aware semantics [incubator-gluten]

Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #11074:
URL: https://github.com/apache/incubator-gluten/pull/11074#issuecomment-3537115133

   Spark3.5 added TIMESTAMP_NTZ data type. @rui-mo does velox support it? have we enabled the UT for for?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [CORE] Use Substrait timestamp_tz for Spark TimestampType to preserve timezone-aware semantics [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11074:
URL: https://github.com/apache/incubator-gluten/pull/11074#issuecomment-3526597241

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [CORE] Use Substrait timestamp_tz for Spark TimestampType to preserve timezone-aware semantics [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11074:
URL: https://github.com/apache/incubator-gluten/pull/11074#issuecomment-3525692864

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [CORE] Use Substrait timestamp_tz for Spark TimestampType to preserve timezone-aware semantics [incubator-gluten]

Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on PR #11074:
URL: https://github.com/apache/incubator-gluten/pull/11074#issuecomment-3537201074

   > Spark3.5 added TIMESTAMP_NTZ data type. @rui-mo does velox support it? have we enabled the UT for for?
   
   This PR only modified the Substrait mapping, reserving the corresponding mapping type for future support of timestamp_ntz, the UT cannot be emabled yet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [CORE] Use Substrait timestamp_tz for Spark TimestampType to preserve timezone-aware semantics [incubator-gluten]

Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on PR #11074:
URL: https://github.com/apache/incubator-gluten/pull/11074#issuecomment-3530455369

   @rui-mo @zhztheplayer Could you please help review this? @taiyang-li @lgbo-ustc @zzcclp could you confirm whether the changes here have any impact on the ClickHouse backend?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [CORE] Use Substrait timestamp_tz for Spark TimestampType to preserve timezone-aware semantics [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11074:
URL: https://github.com/apache/incubator-gluten/pull/11074#issuecomment-3525162197

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [CORE] Use Substrait timestamp_tz for Spark TimestampType to preserve timezone-aware semantics [incubator-gluten]

Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW merged PR #11074:
URL: https://github.com/apache/incubator-gluten/pull/11074


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [CORE] Use Substrait timestamp_tz for Spark TimestampType to preserve timezone-aware semantics [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11074:
URL: https://github.com/apache/incubator-gluten/pull/11074#issuecomment-3524948232

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [PR] [CORE] Use Substrait timestamp_tz for Spark TimestampType to preserve timezone-aware semantics [incubator-gluten]

Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #11074:
URL: https://github.com/apache/incubator-gluten/pull/11074#issuecomment-3537456245

   We need a through clean up of timestamp and timezone support in Gluten sometime later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org