You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/01/14 19:46:42 UTC
[I] [Feature] Support Spark expression: make_timestamp_ntz [datafusion-comet]
andygrove opened a new issue, #3108:
URL: https://github.com/apache/datafusion-comet/issues/3108
## What is the problem the feature request solves?
> **Note:** This issue was generated with AI assistance. The specification details have been extracted from Spark documentation and may need verification.
Comet does not currently support the Spark `make_timestamp_ntz` function, causing queries using this function to fall back to Spark's JVM execution instead of running natively on DataFusion.
`MakeTimestampNTZ` creates a timestamp without timezone (NTZ) by combining a date with a time value. This expression is a runtime replaceable binary expression that delegates its implementation to the `DateTimeUtils.makeTimestampNTZ` method.
Supporting this expression would allow more Spark workloads to benefit from Comet's native acceleration.
## Describe the potential solution
### Spark Specification
**Syntax:**
```sql
make_timestamp_ntz(date_expr, time_expr)
```
```scala
// DataFrame API
col("date_col").make_timestamp_ntz(col("time_col"))
```
**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| left | DateType | The date component for the timestamp |
| right | AnyTimeType | The time component (can be time or timestamp type) |
**Return Type:** `TimestampNTZType` - A timestamp without timezone information.
**Supported Data Types:**
- **Left operand**: `DateType` only
- **Right operand**: Any time-related type (`AnyTimeType` which includes time and timestamp types)
**Edge Cases:**
- **Null handling**: If either input is null, the result will be null
- **Invalid dates**: Behavior depends on the underlying `DateTimeUtils.makeTimestampNTZ` implementation
- **Time overflow**: Time values that exceed 24-hour boundaries may wrap or cause errors depending on implementation
- **Timezone considerations**: Output explicitly excludes timezone information (NTZ = No Time Zone)
**Examples:**
```sql
-- Create timestamp NTZ from date and time
SELECT make_timestamp_ntz(DATE '2023-12-25', TIME '14:30:00') as christmas_afternoon;
-- Using with table columns
SELECT make_timestamp_ntz(date_col, time_col) as full_timestamp
FROM events_table;
```
```scala
// DataFrame API usage
import org.apache.spark.sql.functions._
df.select(expr("make_timestamp_ntz(event_date, event_time)"))
// With column references
df.withColumn("full_timestamp", expr("make_timestamp_ntz(date_col, time_col)"))
```
### Implementation Approach
See the [Comet guide on adding new expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html) for detailed instructions.
1. **Scala Serde**: Add expression handler in `spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has built-in support first)
## Additional context
**Difficulty:** Medium
**Spark Expression Class:** `org.apache.spark.sql.catalyst.expressions.MakeTimestampNTZ`
**Related:**
- `MakeDate` - Creates date values from year, month, day components
- `MakeTimestamp` - Creates timestamp with timezone
- `DateTimeUtils` - Underlying utility class containing the implementation
- `TimestampNTZType` - The return data type for timezone-naive timestamps
---
*This issue was auto-generated from Spark reference documentation.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Feature] Support Spark expression: make_timestamp_ntz [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #3108: [Feature] Support Spark expression: make_timestamp_ntz
URL: https://github.com/apache/datafusion-comet/issues/3108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Feature] Support Spark expression: make_timestamp_ntz [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #3108:
URL: https://github.com/apache/datafusion-comet/issues/3108#issuecomment-5113038843
`MakeTimestampNTZ` is a `RuntimeReplaceable` expression. The `ReplaceExpressions` rule (Finish Analysis batch) substitutes it with its `replacement` before physical planning, so the class never appears in a physical plan and Comet's expression serde never sees it. Registering a `CometExpressionSerde` for it would be dead code.
Its replacement is a `StaticInvoke` into `DateTimeUtils.makeTimestampNTZ`, so this function is never gated on a serde for `MakeTimestampNTZ`. Native support for `make_timestamp_ntz` would have to come from supporting that replacement, which is a different piece of work than this issue describes.
Closing as not actionable as specified. If the fallback matters in practice, a fresh issue should track the replacement path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org