You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "the-sakthi (via GitHub)" <gi...@apache.org> on 2025/03/20 11:56:12 UTC
[PR] [WIP][SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
the-sakthi opened a new pull request, #50336:
URL: https://github.com/apache/spark/pull/50336
### What changes were proposed in this pull request?
This PR adds support for a new function current_time() which returns the current time at the start of query evaluation.
```bash
scala> spark.sql("SELECT current_time(0);").show()
+--------------+
|current_time()|
+--------------+
| 04:48:47|
+--------------+
scala> spark.sql("SELECT current_time();").show()
+---------------+
| current_time()|
+---------------+
|04:48:51.609168|
+---------------+
scala> spark.sql("SELECT current_time(3);").show()
+--------------+
|current_time()|
+--------------+
| 04:48:54.846|
+--------------+
scala> spark.sql("SELECT current_time(6);").show()
+---------------+
| current_time()|
+---------------+
|04:48:59.222348|
+---------------+
scala> spark.sql("SELECT current_time(7);").show()
org.apache.spark.sql.catalyst.ExtendedAnalysisException: [DATATYPE_MISMATCH.TYPE_CHECK_FAILURE_WITH_HINT] Cannot resolve "current_time()" due to data type mismatch: Invalid precision 7. Must be between 0 and 6.. SQLSTATE: 42K09; line 1 pos 7;
'Project [unresolvedalias(current_time(7))]
+- OneRowRelation
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:52)
...
```
### Why are the changes needed?
Adds a built-in current_time([n]) function returning just the time portion (in a TIME(n) type). This aligns Spark with other SQL systems offering a native time function, improves convenience for time-only queries, and complements existing functions like current_date and current_timestamp.
### Does this PR introduce _any_ user-facing change?
Yes, adds a new function. Users can now get the current time using this function.
### How was this patch tested?
Manual testing.
### Pending
- Support for current_time [i.e. without any braces]
- Implementation of codegen (currently extending CodegenFallback)
- Addition of UTs.
- Use inbuilt functions for `computeMicrosAtPrecision`
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #50336:
URL: https://github.com/apache/spark/pull/50336#discussion_r2021600007
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:
##########
@@ -133,6 +134,32 @@ trait SparkDateTimeUtils {
}
}
+ /**
+ * Gets the number of microseconds since midnight using the session time zone.
+ */
+ def instantToMicrosOfDay(instant: Instant): Long = {
+ val zoneId = getZoneId(SqlApiConf.get.sessionLocalTimeZone)
+ val localDateTime = LocalDateTime.ofInstant(instant, zoneId)
+ localDateTime.toLocalTime.toNanoOfDay / 1000
Review Comment:
```suggestion
localDateTime.toLocalTime.getLong(MICRO_OF_DAY)
```
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala:
##########
@@ -290,3 +291,63 @@ object HourExpressionBuilder extends ExpressionBuilder {
}
}
}
+
+case class CurrentTime(precision: Int = 6) extends CurrentTimestampLike with ExpectsInputTypes {
+ // The function returns a TIME(n).
+ override def dataType: DataType = TimeType(precision)
+
+ override def prettyName: String = "current_time"
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(IntegerType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (precision < TimeType.MIN_PRECISION || precision > TimeType.MICROS_PRECISION) {
+ TypeCheckFailure(s"Invalid precision $precision. Must be between 0 and 6.")
+ } else {
+ TypeCheckSuccess
+ }
+ }
+}
+
+/**
+ * Returns the current time at the start of query evaluation.
+ * There is no code generation since this expression should get constant folded by the optimizer.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_() - Returns the current time at the start of query evaluation. All calls of current_time within the same query return the same value.
+
+ _FUNC_ - Returns the current time at the start of query evaluation.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_();
+ 15:49:11.914120
+ > SELECT _FUNC_;
+ 15:49:11.914120
+ > SELECT _FUNC_(0);
+ 15:49:11
+ > SELECT _FUNC_(3);
+ 15:49:11.914
+ """,
+ group = "datetime_funcs",
+ since = "4.1.0")
+// scalastyle:on line.size.limit
+object CurrentTimeExpressionBuilder extends ExpressionBuilder {
+ override def build(funcName: String, expressions: Seq[Expression]): Expression = {
+ expressions match {
+ case Nil => CurrentTime(TimeType.MICROS_PRECISION)
+
+ case Seq(Literal(precisionValue, IntegerType)) =>
+ CurrentTime(precisionValue.asInstanceOf[Int])
+
+ case Seq(_) =>
+ val child = expressions.head
+ throw QueryCompilationErrors.nonFoldableInputError("precision", child, IntegerType)
Review Comment:
A foldable expression could be non-Literal, for example `1+1`
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:
##########
@@ -133,6 +134,32 @@ trait SparkDateTimeUtils {
}
}
+ /**
+ * Gets the number of microseconds since midnight using the session time zone.
+ */
+ def instantToMicrosOfDay(instant: Instant): Long = {
+ val zoneId = getZoneId(SqlApiConf.get.sessionLocalTimeZone)
Review Comment:
Most of the functions in the file are pure, and don't depend on global variables. Let's follow the convention, and pass the time zone to the function. It is available at the caller side, right?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala:
##########
@@ -2148,6 +2148,21 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat
))
}
+ // scalastyle:off line.size.limit
+ def nonFoldableInputError(argumentName: String, expression: Expression, requiredType: DataType): Throwable = {
Review Comment:
Just put parameters on separate lines, see https://github.com/databricks/scala-style-guide?tab=readme-ov-file#spacing-and-indentation
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala:
##########
@@ -290,3 +291,63 @@ object HourExpressionBuilder extends ExpressionBuilder {
}
}
}
+
+case class CurrentTime(precision: Int = 6) extends CurrentTimestampLike with ExpectsInputTypes {
+ // The function returns a TIME(n).
+ override def dataType: DataType = TimeType(precision)
+
+ override def prettyName: String = "current_time"
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(IntegerType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (precision < TimeType.MIN_PRECISION || precision > TimeType.MICROS_PRECISION) {
+ TypeCheckFailure(s"Invalid precision $precision. Must be between 0 and 6.")
+ } else {
+ TypeCheckSuccess
+ }
+ }
+}
+
+/**
+ * Returns the current time at the start of query evaluation.
+ * There is no code generation since this expression should get constant folded by the optimizer.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_() - Returns the current time at the start of query evaluation. All calls of current_time within the same query return the same value.
+
+ _FUNC_ - Returns the current time at the start of query evaluation.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_();
+ 15:49:11.914120
+ > SELECT _FUNC_;
+ 15:49:11.914120
+ > SELECT _FUNC_(0);
+ 15:49:11
+ > SELECT _FUNC_(3);
+ 15:49:11.914
+ """,
+ group = "datetime_funcs",
+ since = "4.1.0")
+// scalastyle:on line.size.limit
+object CurrentTimeExpressionBuilder extends ExpressionBuilder {
Review Comment:
You can just add constructors with different parameters to `CurrentTime` instead of using `ExpressionBuilder`.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala:
##########
@@ -129,6 +131,11 @@ object ComputeCurrentTime extends Rule[LogicalPlan] {
Literal.create(
DateTimeUtils.microsToDays(currentTimestampMicros, cd.zoneId), DateType)
})
+ case currentTimeType : CurrentTime =>
+ // scalastyle:off line.size.limit
Review Comment:
Just put split the line below, and remove this.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala:
##########
@@ -129,6 +131,11 @@ object ComputeCurrentTime extends Rule[LogicalPlan] {
Literal.create(
DateTimeUtils.microsToDays(currentTimestampMicros, cd.zoneId), DateType)
})
+ case currentTimeType : CurrentTime =>
+ // scalastyle:off line.size.limit
+ val truncatedTime = truncateTimeMicrosToPrecision(currentTimeOfDayMicros, currentTimeType.precision)
Review Comment:
```suggestion
val truncatedTime = truncateTimeMicrosToPrecision(currentTimeOfDayMicros,
currentTimeType.precision)```
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:
##########
@@ -133,6 +134,32 @@ trait SparkDateTimeUtils {
}
}
+ /**
+ * Gets the number of microseconds since midnight using the session time zone.
+ */
+ def instantToMicrosOfDay(instant: Instant): Long = {
+ val zoneId = getZoneId(SqlApiConf.get.sessionLocalTimeZone)
+ val localDateTime = LocalDateTime.ofInstant(instant, zoneId)
+ localDateTime.toLocalTime.toNanoOfDay / 1000
+ }
+
+ /**
+ * Truncates a time value (in microseconds) to the specified fractional precision `p`.
+ *
+ * For example, if `p = 3`, we keep millisecond resolution and discard any digits beyond
+ * the thousand-microsecond place. So a value like `123456` microseconds (12:34:56.123456)
+ * becomes `123000` microseconds (12:34:56.123).
+ *
+ * @param micros The original time in microseconds.
+ * @param p The fractional second precision (range 0 to 6).
Review Comment:
Could you add an assert for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #50336:
URL: https://github.com/apache/spark/pull/50336#discussion_r2034896002
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala:
##########
@@ -290,3 +294,100 @@ object HourExpressionBuilder extends ExpressionBuilder {
}
}
}
+
+@ExpressionDescription(
+ usage = """
+ _FUNC_([precision]) - Returns the current time at the start of query evaluation.
Review Comment:
It would be nice if you describe the `precision` parameter since this is visible to users. Could you `arguments` like
https://github.com/apache/spark/blob/f4ce0ab8cee043d0b1a8d935fb13ce617d2de873/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala#L136-L141
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "the-sakthi (via GitHub)" <gi...@apache.org>.
the-sakthi commented on PR #50336:
URL: https://github.com/apache/spark/pull/50336#issuecomment-2796364972
Thank you very much @MaxGekk !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "the-sakthi (via GitHub)" <gi...@apache.org>.
the-sakthi commented on PR #50336:
URL: https://github.com/apache/spark/pull/50336#issuecomment-2766027730
Thanks for the feedback @MaxGekk , let me know how this one looks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "the-sakthi (via GitHub)" <gi...@apache.org>.
the-sakthi commented on PR #50336:
URL: https://github.com/apache/spark/pull/50336#issuecomment-2770961375
Thanks for the feedback @MaxGekk , let me know how this revision one looks! Hopefully this is better aligned to the review comments!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "the-sakthi (via GitHub)" <gi...@apache.org>.
the-sakthi commented on code in PR #50336:
URL: https://github.com/apache/spark/pull/50336#discussion_r2036843551
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala:
##########
@@ -349,3 +353,109 @@ object SecondExpressionBuilder extends ExpressionBuilder {
}
}
+/**
+ * Returns the current time at the start of query evaluation.
+ * There is no code generation since this expression should get constant folded by the optimizer.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_([precision]) - Returns the current time at the start of query evaluation.
+ All calls of current_time within the same query return the same value.
+
+ _FUNC_ - Returns the current time at the start of query evaluation.
+ """,
+ arguments = """
+ Arguments:
+ * precision - An optional integer literal in the range [${TimeType.MIN_PRECISION}..${TimeType.MICROS_PRECISION}], indicating how many
Review Comment:
Yeah you are right. I don't think it works. Even with string interpolation it errors out. Let me change to hardcoded values for simplicity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "the-sakthi (via GitHub)" <gi...@apache.org>.
the-sakthi commented on PR #50336:
URL: https://github.com/apache/spark/pull/50336#issuecomment-2775006830
Thanks for the response, Max, even while traveling. Will await your review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "the-sakthi (via GitHub)" <gi...@apache.org>.
the-sakthi commented on PR #50336:
URL: https://github.com/apache/spark/pull/50336#issuecomment-2787460679
Sure thanks @MaxGekk ! Will wait.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #50336:
URL: https://github.com/apache/spark/pull/50336#issuecomment-2789577275
@the-sakthi Please, resolve conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #50336:
URL: https://github.com/apache/spark/pull/50336#issuecomment-2774891187
@the-sakthi Thank you for the ping. I am traveling right now. I will look at the PR slightly latter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "the-sakthi (via GitHub)" <gi...@apache.org>.
the-sakthi commented on code in PR #50336:
URL: https://github.com/apache/spark/pull/50336#discussion_r2023835403
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala:
##########
@@ -290,3 +291,63 @@ object HourExpressionBuilder extends ExpressionBuilder {
}
}
}
+
+case class CurrentTime(precision: Int = 6) extends CurrentTimestampLike with ExpectsInputTypes {
+ // The function returns a TIME(n).
+ override def dataType: DataType = TimeType(precision)
+
+ override def prettyName: String = "current_time"
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(IntegerType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (precision < TimeType.MIN_PRECISION || precision > TimeType.MICROS_PRECISION) {
+ TypeCheckFailure(s"Invalid precision $precision. Must be between 0 and 6.")
+ } else {
+ TypeCheckSuccess
+ }
+ }
+}
+
+/**
+ * Returns the current time at the start of query evaluation.
+ * There is no code generation since this expression should get constant folded by the optimizer.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_() - Returns the current time at the start of query evaluation. All calls of current_time within the same query return the same value.
+
+ _FUNC_ - Returns the current time at the start of query evaluation.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_();
+ 15:49:11.914120
+ > SELECT _FUNC_;
+ 15:49:11.914120
+ > SELECT _FUNC_(0);
+ 15:49:11
+ > SELECT _FUNC_(3);
+ 15:49:11.914
+ """,
+ group = "datetime_funcs",
+ since = "4.1.0")
+// scalastyle:on line.size.limit
+object CurrentTimeExpressionBuilder extends ExpressionBuilder {
+ override def build(funcName: String, expressions: Seq[Expression]): Expression = {
+ expressions match {
+ case Nil => CurrentTime(TimeType.MICROS_PRECISION)
+
+ case Seq(Literal(precisionValue, IntegerType)) =>
+ CurrentTime(precisionValue.asInstanceOf[Int])
+
+ case Seq(_) =>
+ val child = expressions.head
+ throw QueryCompilationErrors.nonFoldableInputError("precision", child, IntegerType)
Review Comment:
Ah totally! Have catered to this in the new revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #50336:
URL: https://github.com/apache/spark/pull/50336#issuecomment-2796312063
+1, LGTM. Merging to master.
Thank you, @the-sakthi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "the-sakthi (via GitHub)" <gi...@apache.org>.
the-sakthi commented on code in PR #50336:
URL: https://github.com/apache/spark/pull/50336#discussion_r2023835597
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala:
##########
@@ -290,3 +291,63 @@ object HourExpressionBuilder extends ExpressionBuilder {
}
}
}
+
+case class CurrentTime(precision: Int = 6) extends CurrentTimestampLike with ExpectsInputTypes {
+ // The function returns a TIME(n).
+ override def dataType: DataType = TimeType(precision)
+
+ override def prettyName: String = "current_time"
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(IntegerType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (precision < TimeType.MIN_PRECISION || precision > TimeType.MICROS_PRECISION) {
+ TypeCheckFailure(s"Invalid precision $precision. Must be between 0 and 6.")
+ } else {
+ TypeCheckSuccess
+ }
+ }
+}
+
+/**
+ * Returns the current time at the start of query evaluation.
+ * There is no code generation since this expression should get constant folded by the optimizer.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_() - Returns the current time at the start of query evaluation. All calls of current_time within the same query return the same value.
+
+ _FUNC_ - Returns the current time at the start of query evaluation.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_();
+ 15:49:11.914120
+ > SELECT _FUNC_;
+ 15:49:11.914120
+ > SELECT _FUNC_(0);
+ 15:49:11
+ > SELECT _FUNC_(3);
+ 15:49:11.914
+ """,
+ group = "datetime_funcs",
+ since = "4.1.0")
+// scalastyle:on line.size.limit
+object CurrentTimeExpressionBuilder extends ExpressionBuilder {
Review Comment:
Makes sense! Rev 2 accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "the-sakthi (via GitHub)" <gi...@apache.org>.
the-sakthi commented on PR #50336:
URL: https://github.com/apache/spark/pull/50336#issuecomment-2744671757
Ahh sure! Thanks for the early review, Max. Will refer `CurrentTimestampLike` to better cater to the requirement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk closed pull request #50336: [SPARK-51423][SQL] Add the current_time() function for TIME datatype
URL: https://github.com/apache/spark/pull/50336
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "the-sakthi (via GitHub)" <gi...@apache.org>.
the-sakthi commented on code in PR #50336:
URL: https://github.com/apache/spark/pull/50336#discussion_r2022326295
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala:
##########
@@ -2148,6 +2148,21 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat
))
}
+ // scalastyle:off line.size.limit
+ def nonFoldableInputError(argumentName: String, expression: Expression, requiredType: DataType): Throwable = {
Review Comment:
Very helpful resource. Thanks, Max.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "the-sakthi (via GitHub)" <gi...@apache.org>.
the-sakthi commented on code in PR #50336:
URL: https://github.com/apache/spark/pull/50336#discussion_r2022286788
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:
##########
@@ -133,6 +134,32 @@ trait SparkDateTimeUtils {
}
}
+ /**
+ * Gets the number of microseconds since midnight using the session time zone.
+ */
+ def instantToMicrosOfDay(instant: Instant): Long = {
+ val zoneId = getZoneId(SqlApiConf.get.sessionLocalTimeZone)
Review Comment:
Ah sure. Makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "the-sakthi (via GitHub)" <gi...@apache.org>.
the-sakthi commented on PR #50336:
URL: https://github.com/apache/spark/pull/50336#issuecomment-2791011469
@MaxGekk The latest push has the updated user description and rebased with latest main branch. Let me know!
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #50336:
URL: https://github.com/apache/spark/pull/50336#discussion_r2036803770
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala:
##########
@@ -349,3 +353,109 @@ object SecondExpressionBuilder extends ExpressionBuilder {
}
}
+/**
+ * Returns the current time at the start of query evaluation.
+ * There is no code generation since this expression should get constant folded by the optimizer.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_([precision]) - Returns the current time at the start of query evaluation.
+ All calls of current_time within the same query return the same value.
+
+ _FUNC_ - Returns the current time at the start of query evaluation.
+ """,
+ arguments = """
+ Arguments:
+ * precision - An optional integer literal in the range [${TimeType.MIN_PRECISION}..${TimeType.MICROS_PRECISION}], indicating how many
Review Comment:
I guess the constants ${TimeType.MIN_PRECISION} won't embedded in the string. It should start from s" at least. Could you generate the doc, and double check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "the-sakthi (via GitHub)" <gi...@apache.org>.
the-sakthi commented on code in PR #50336:
URL: https://github.com/apache/spark/pull/50336#discussion_r2035714407
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala:
##########
@@ -290,3 +294,100 @@ object HourExpressionBuilder extends ExpressionBuilder {
}
}
}
+
+@ExpressionDescription(
+ usage = """
+ _FUNC_([precision]) - Returns the current time at the start of query evaluation.
Review Comment:
Definitely!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51423][SQL] Add the current_time() function for TIME datatype [spark]
Posted by "the-sakthi (via GitHub)" <gi...@apache.org>.
the-sakthi commented on code in PR #50336:
URL: https://github.com/apache/spark/pull/50336#discussion_r2036861770
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala:
##########
@@ -349,3 +353,109 @@ object SecondExpressionBuilder extends ExpressionBuilder {
}
}
+/**
+ * Returns the current time at the start of query evaluation.
+ * There is no code generation since this expression should get constant folded by the optimizer.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_([precision]) - Returns the current time at the start of query evaluation.
+ All calls of current_time within the same query return the same value.
+
+ _FUNC_ - Returns the current time at the start of query evaluation.
+ """,
+ arguments = """
+ Arguments:
+ * precision - An optional integer literal in the range [${TimeType.MIN_PRECISION}..${TimeType.MICROS_PRECISION}], indicating how many
Review Comment:
Changed to hardcoded versions in the new revision.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala:
##########
@@ -349,3 +353,109 @@ object SecondExpressionBuilder extends ExpressionBuilder {
}
}
+/**
+ * Returns the current time at the start of query evaluation.
+ * There is no code generation since this expression should get constant folded by the optimizer.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_([precision]) - Returns the current time at the start of query evaluation.
+ All calls of current_time within the same query return the same value.
+
+ _FUNC_ - Returns the current time at the start of query evaluation.
+ """,
+ arguments = """
+ Arguments:
+ * precision - An optional integer literal in the range [${TimeType.MIN_PRECISION}..${TimeType.MICROS_PRECISION}], indicating how many
Review Comment:
Changed to hardcoded versions in the new revision, @MaxGekk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org