You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/01/14 19:47:05 UTC
[I] [Feature] Support Spark expression: try_make_timestamp_from_date_time [datafusion-comet]
andygrove opened a new issue, #3118:
URL: https://github.com/apache/datafusion-comet/issues/3118
## What is the problem the feature request solves?
> **Note:** This issue was generated with AI assistance. The specification details have been extracted from Spark documentation and may need verification.
Comet does not currently support the Spark `try_make_timestamp_from_date_time` function, causing queries using this function to fall back to Spark's JVM execution instead of running natively on DataFusion.
`TryMakeTimestampFromDateTime` is a runtime-replaceable Catalyst expression that attempts to construct a timestamp from date and optional time components. Unlike its non-try counterpart, this expression returns null instead of throwing an exception when the input values cannot be converted to a valid timestamp.
Supporting this expression would allow more Spark workloads to benefit from Comet's native acceleration.
## Describe the potential solution
### Spark Specification
**Syntax:**
```sql
try_make_timestamp(date [, time] [, timezone])
```
```scala
// DataFrame API usage would depend on the function registry
col("date_col").expr("try_make_timestamp(date_col)")
```
**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| date | Expression | The date component for the timestamp |
| time | Expression (optional) | The time component to combine with the date |
| timezone | Expression (optional) | The timezone for the resulting timestamp |
| replacement | Expression | Internal replacement expression used during runtime |
**Return Type:** Returns `TimestampType` on successful conversion, or `null` if the conversion fails.
**Supported Data Types:**
- **date**: Date types, string representations of dates
- **time**: Time types, string representations of time
- **timezone**: String representations of valid timezone identifiers
**Edge Cases:**
- **Null date input**: Returns null without attempting conversion
- **Invalid date formats**: Returns null instead of throwing parse exceptions
- **Invalid timezone**: Returns null for unrecognized timezone identifiers
- **Leap second handling**: Behavior depends on the underlying datetime implementation
- **Daylight saving transitions**: May return null for ambiguous times during DST changes
**Examples:**
```sql
-- Basic usage with date only
SELECT try_make_timestamp('2023-12-25') as christmas_timestamp;
-- With date and time components
SELECT try_make_timestamp('2023-12-25', '15:30:00') as afternoon_timestamp;
-- With timezone specification
SELECT try_make_timestamp('2023-12-25', '15:30:00', 'UTC') as utc_timestamp;
-- Handling invalid input (returns null)
SELECT try_make_timestamp('invalid-date') as null_result;
```
```scala
// DataFrame API usage (conceptual)
import org.apache.spark.sql.functions._
df.select(expr("try_make_timestamp(date_col)").as("timestamp_col"))
df.select(expr("try_make_timestamp(date_col, time_col, 'PST')").as("pst_timestamp"))
```
### Implementation Approach
See the [Comet guide on adding new expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html) for detailed instructions.
1. **Scala Serde**: Add expression handler in `spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has built-in support first)
## Additional context
**Difficulty:** Medium
**Spark Expression Class:** `org.apache.spark.sql.catalyst.expressions.TryMakeTimestampFromDateTime`
**Related:**
- `MakeTimestampFromDateTime` - The non-try version that throws exceptions on invalid input
- `MakeTimestamp` - Creates timestamps from individual year/month/day/hour/minute/second components
- `ToTimestamp` - Converts string representations to timestamps with format patterns
---
*This issue was auto-generated from Spark reference documentation.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Feature] Support Spark expression: try_make_timestamp_from_date_time [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #3118: [Feature] Support Spark expression: try_make_timestamp_from_date_time
URL: https://github.com/apache/datafusion-comet/issues/3118
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Feature] Support Spark expression: try_make_timestamp_from_date_time [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #3118:
URL: https://github.com/apache/datafusion-comet/issues/3118#issuecomment-5113039322
`TryMakeTimestampFromDateTime` is a `RuntimeReplaceable` expression. The `ReplaceExpressions` rule (Finish Analysis batch) substitutes it with its `replacement` before physical planning, so the class never appears in a physical plan and Comet's expression serde never sees it. Registering a `CometExpressionSerde` for it would be dead code.
Its replacement is `MakeTimestampFromDateTime`, which is itself `RuntimeReplaceable` and resolves to a `StaticInvoke` into `DateTimeUtils`, so this function is never gated on a serde for `TryMakeTimestampFromDateTime`. Native support for `try_make_timestamp_from_date_time` would have to come from supporting that replacement, which is a different piece of work than this issue describes.
Closing as not actionable as specified. If the fallback matters in practice, a fresh issue should track the replacement path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org