You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/01/14 19:46:06 UTC

[I] [Feature] Support Spark expression: divide_dt_interval [datafusion-comet]

andygrove opened a new issue, #3096:
URL: https://github.com/apache/datafusion-comet/issues/3096

   ## What is the problem the feature request solves?
   
   > **Note:** This issue was generated with AI assistance. The specification details have been extracted from Spark documentation and may need verification.
   
   Comet does not currently support the Spark `divide_dt_interval` function, causing queries using this function to fall back to Spark's JVM execution instead of running natively on DataFusion.
   
   The `DivideDTInterval` expression divides a day-time interval by a numeric value, returning a new day-time interval. This operation performs division with `HALF_UP` rounding mode and includes comprehensive overflow and divide-by-zero checks to ensure safe arithmetic operations.
   
   Supporting this expression would allow more Spark workloads to benefit from Comet's native acceleration.
   
   ## Describe the potential solution
   
   ### Spark Specification
   
   **Syntax:**
   ```sql
   day_time_interval / numeric_value
   ```
   
   **Arguments:**
   | Argument | Type | Description |
   |----------|------|-------------|
   | `interval` | `DayTimeIntervalType` | The day-time interval to be divided |
   | `num` | `NumericType` | The numeric divisor (integral, decimal, or fractional types) |
   
   **Return Type:** `DayTimeIntervalType()` - Returns a day-time interval representing the result of the division operation.
   
   **Supported Data Types:**
   - **Left operand (interval)**: `DayTimeIntervalType` only
   - **Right operand (num)**: All numeric types including:
     - `IntegralType` (Byte, Short, Int, Long)
     - `DecimalType` 
     - `FractionalType` (Float, Double)
   
   **Edge Cases:**
   - **Null handling**: Expression is null-intolerant (`nullIntolerant = true`) - returns null if either operand is null
   - **Divide by zero**: Throws `QueryExecutionErrors` when divisor is zero
   - **Overflow behavior**: 
     - Checks for `Long.MinValue / -1` overflow condition
     - Throws `QueryExecutionErrors.overflowInIntegralDivideError` on overflow
     - Uses `longValueExact()` for decimal operations to detect overflow
   - **Precision**: All division operations use `HALF_UP` rounding mode for consistent behavior
   
   **Examples:**
   ```sql
   -- Divide a day-time interval by an integer
   SELECT INTERVAL '2 12:30:45.123' DAY TO SECOND / 2;
   -- Result: INTERVAL '1 06:15:22.561500' DAY TO SECOND
   
   -- Divide by decimal value
   SELECT INTERVAL '5 00:00:00' DAY TO SECOND / 2.5;
   -- Result: INTERVAL '2 00:00:00' DAY TO SECOND
   ```
   
   ```scala
   // Example DataFrame API usage
   import org.apache.spark.sql.functions._
   
   // Divide interval column by numeric literal
   df.select(col("day_time_interval") / lit(3))
   
   // Divide interval by another numeric column
   df.select(col("day_time_interval") / col("divisor"))
   ```
   
   ### Implementation Approach
   
   See the [Comet guide on adding new expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html) for detailed instructions.
   
   1. **Scala Serde**: Add expression handler in `spark/src/main/scala/org/apache/comet/serde/`
   2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
   3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if needed
   4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has built-in support first)
   
   
   ## Additional context
   
   **Difficulty:** Medium
   **Spark Expression Class:** `org.apache.spark.sql.catalyst.expressions.DivideDTInterval`
   
   **Related:**
   - `DivideYMInterval` - Division operation for year-month intervals
   - `MultiplyDTInterval` - Multiplication of day-time intervals
   - `IntervalDivide` - Base trait for interval division operations
   - `BinaryExpression` - Parent class for binary operations
   
   ---
   *This issue was auto-generated from Spark reference documentation.*
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] [Feature] Support Spark expression: divide_dt_interval [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao closed issue #3096: [Feature] Support Spark expression: divide_dt_interval
URL: https://github.com/apache/datafusion-comet/issues/3096


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] [Feature] Support Spark expression: divide_dt_interval [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on issue #3096:
URL: https://github.com/apache/datafusion-comet/issues/3096#issuecomment-4951258290

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org