You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/10/23 23:51:13 UTC
[GitHub] [spark] HeartSaVioR opened a new pull request, #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls
HeartSaVioR opened a new pull request, #38361:
URL: https://github.com/apache/spark/pull/38361
### What changes were proposed in this pull request?
This PR proposes to loosen the requirement of window_time rule to allow multiple distinct window_time calls. After this change, users can call the window_time function with different windows in the same logical node (select, where, etc.).
Given that we allow multiple calls of window_time in projection, we no longer be able to use the reserved column name "window_time". This PR picked up the SQL representation of the WindowTime, to distinguish each distinct function call.
### Why are the changes needed?
The rule for window time followed the existing rules of time window / session window which only allows a single function call in a same projection (strictly saying, it considers the call of function as once if the function is called with same parameters).
For time window/session window rules , the restriction makes sense since allowing this would produce cartesian product of rows (although Spark can handle it). But given that window_time only produces one value, the restriction no longer makes sense.
### Does this PR introduce _any_ user-facing change?
Yes since it changes the resulting column name from window_time function call, but the function is not released yet.
### How was this patch tested?
New test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on code in PR #38361:
URL: https://github.com/apache/spark/pull/38361#discussion_r1002857134
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala:
##########
@@ -292,53 +292,59 @@ object ResolveWindowTime extends Rule[LogicalPlan] {
val windowTimeExpressions =
p.expressions.flatMap(_.collect { case w: WindowTime => w }).toSet
- if (windowTimeExpressions.size == 1 &&
- windowTimeExpressions.head.windowColumn.resolved &&
- windowTimeExpressions.head.checkInputDataTypes().isSuccess) {
+ val allWindowTimeExprsResolved = windowTimeExpressions.forall { w =>
+ w.windowColumn.resolved && w.checkInputDataTypes().isSuccess
+ }
- val windowTime = windowTimeExpressions.head
+ if (windowTimeExpressions.nonEmpty && allWindowTimeExprsResolved) {
+ val windowTimeToAttrAndNewColumn = windowTimeExpressions.map { windowTime =>
+ val metadata = windowTime.windowColumn match {
+ case a: Attribute => a.metadata
+ case _ => Metadata.empty
+ }
- val metadata = windowTime.windowColumn match {
- case a: Attribute => a.metadata
- case _ => Metadata.empty
- }
+ if (!metadata.contains(TimeWindow.marker) &&
+ !metadata.contains(SessionWindow.marker)) {
+ // FIXME: error framework?
+ throw new AnalysisException(
+ "The input is not a correct window column: $windowTime", plan = Some(p))
+ }
- if (!metadata.contains(TimeWindow.marker) &&
- !metadata.contains(SessionWindow.marker)) {
- // FIXME: error framework?
- throw new AnalysisException(
- "The input is not a correct window column: $windowTime", plan = Some(p))
- }
+ val newMetadata = new MetadataBuilder()
+ .withMetadata(metadata)
+ .remove(TimeWindow.marker)
+ .remove(SessionWindow.marker)
+ .build()
- val newMetadata = new MetadataBuilder()
- .withMetadata(metadata)
- .remove(TimeWindow.marker)
- .remove(SessionWindow.marker)
- .build()
+ val colName = windowTime.sql
Review Comment:
In practice, end users will apply another time window function against the resulting column of window time, hence the final resulting column will be another "window".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #38361:
URL: https://github.com/apache/spark/pull/38361#issuecomment-1288496655
cc. @viirya I'd be lucky if I can get your traction to review this, who are familiar with both areas. Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #38361:
URL: https://github.com/apache/spark/pull/38361#issuecomment-1294361918
Thanks @cloud-fan ! Given this PR stayed for 4 days and no feedback so far, I'm merging this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #38361:
URL: https://github.com/apache/spark/pull/38361#issuecomment-1288235975
cc. @cloud-fan @alex-balikov
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #38361:
URL: https://github.com/apache/spark/pull/38361#discussion_r1002847203
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala:
##########
@@ -292,53 +292,59 @@ object ResolveWindowTime extends Rule[LogicalPlan] {
val windowTimeExpressions =
p.expressions.flatMap(_.collect { case w: WindowTime => w }).toSet
- if (windowTimeExpressions.size == 1 &&
- windowTimeExpressions.head.windowColumn.resolved &&
- windowTimeExpressions.head.checkInputDataTypes().isSuccess) {
+ val allWindowTimeExprsResolved = windowTimeExpressions.forall { w =>
+ w.windowColumn.resolved && w.checkInputDataTypes().isSuccess
+ }
- val windowTime = windowTimeExpressions.head
+ if (windowTimeExpressions.nonEmpty && allWindowTimeExprsResolved) {
+ val windowTimeToAttrAndNewColumn = windowTimeExpressions.map { windowTime =>
+ val metadata = windowTime.windowColumn match {
+ case a: Attribute => a.metadata
+ case _ => Metadata.empty
+ }
- val metadata = windowTime.windowColumn match {
- case a: Attribute => a.metadata
- case _ => Metadata.empty
- }
+ if (!metadata.contains(TimeWindow.marker) &&
+ !metadata.contains(SessionWindow.marker)) {
+ // FIXME: error framework?
+ throw new AnalysisException(
+ "The input is not a correct window column: $windowTime", plan = Some(p))
+ }
- if (!metadata.contains(TimeWindow.marker) &&
- !metadata.contains(SessionWindow.marker)) {
- // FIXME: error framework?
- throw new AnalysisException(
- "The input is not a correct window column: $windowTime", plan = Some(p))
- }
+ val newMetadata = new MetadataBuilder()
+ .withMetadata(metadata)
+ .remove(TimeWindow.marker)
+ .remove(SessionWindow.marker)
+ .build()
- val newMetadata = new MetadataBuilder()
- .withMetadata(metadata)
- .remove(TimeWindow.marker)
- .remove(SessionWindow.marker)
- .build()
+ val colName = windowTime.sql
Review Comment:
will this be materialized in the checkpoint or state store? The SQL string for an expression is unstable, as it depends on resolved expression, and resolution may change overtime (e.g. type coercion may add cast differently).
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala:
##########
@@ -292,53 +292,59 @@ object ResolveWindowTime extends Rule[LogicalPlan] {
val windowTimeExpressions =
p.expressions.flatMap(_.collect { case w: WindowTime => w }).toSet
- if (windowTimeExpressions.size == 1 &&
- windowTimeExpressions.head.windowColumn.resolved &&
- windowTimeExpressions.head.checkInputDataTypes().isSuccess) {
+ val allWindowTimeExprsResolved = windowTimeExpressions.forall { w =>
+ w.windowColumn.resolved && w.checkInputDataTypes().isSuccess
+ }
- val windowTime = windowTimeExpressions.head
+ if (windowTimeExpressions.nonEmpty && allWindowTimeExprsResolved) {
+ val windowTimeToAttrAndNewColumn = windowTimeExpressions.map { windowTime =>
+ val metadata = windowTime.windowColumn match {
+ case a: Attribute => a.metadata
+ case _ => Metadata.empty
+ }
- val metadata = windowTime.windowColumn match {
- case a: Attribute => a.metadata
- case _ => Metadata.empty
- }
+ if (!metadata.contains(TimeWindow.marker) &&
+ !metadata.contains(SessionWindow.marker)) {
+ // FIXME: error framework?
+ throw new AnalysisException(
+ "The input is not a correct window column: $windowTime", plan = Some(p))
+ }
- if (!metadata.contains(TimeWindow.marker) &&
- !metadata.contains(SessionWindow.marker)) {
- // FIXME: error framework?
- throw new AnalysisException(
- "The input is not a correct window column: $windowTime", plan = Some(p))
- }
+ val newMetadata = new MetadataBuilder()
+ .withMetadata(metadata)
+ .remove(TimeWindow.marker)
+ .remove(SessionWindow.marker)
+ .build()
- val newMetadata = new MetadataBuilder()
- .withMetadata(metadata)
- .remove(TimeWindow.marker)
- .remove(SessionWindow.marker)
- .build()
+ val colName = windowTime.sql
Review Comment:
will this be materialized in the checkpoint or state store? The SQL string for an expression is unstable, as it depends on resolved expression, and resolution may change over time (e.g. type coercion may add cast differently).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on code in PR #38361:
URL: https://github.com/apache/spark/pull/38361#discussion_r1002794765
##########
sql/core/src/test/scala/org/apache/spark/sql/DataFrameTimeWindowingSuite.scala:
##########
@@ -599,23 +599,37 @@ class DataFrameTimeWindowingSuite extends QueryTest with SharedSparkSession {
("2016-03-27 19:38:18"), ("2016-03-27 19:39:25")
).toDF("time")
- val e = intercept[AnalysisException] {
- df
- .withColumn("time2", expr("time - INTERVAL 5 minutes"))
- .select(
- window($"time", "10 seconds").as("window1"),
- window($"time2", "10 seconds").as("window2")
- )
- .select(
- $"window1.end".cast("string"),
- window_time($"window1").cast("string"),
- $"window2.end".cast("string"),
- window_time($"window2").cast("string")
- )
- }
- assert(e.getMessage.contains(
- "Multiple time/session window expressions would result in a cartesian product of rows, " +
- "therefore they are currently not supported"))
+ val df2 = df
+ .withColumn("time2", expr("time - INTERVAL 15 minutes"))
+ .select(window($"time", "10 seconds").as("window1"), $"time2")
Review Comment:
We have to do select twice since time window rule does not allow multiple time window function call to co-exist in same projection. time_window/session_window function is effectively a TVF.
##########
sql/core/src/test/scala/org/apache/spark/sql/DataFrameTimeWindowingSuite.scala:
##########
@@ -599,23 +599,37 @@ class DataFrameTimeWindowingSuite extends QueryTest with SharedSparkSession {
("2016-03-27 19:38:18"), ("2016-03-27 19:39:25")
).toDF("time")
- val e = intercept[AnalysisException] {
- df
- .withColumn("time2", expr("time - INTERVAL 5 minutes"))
- .select(
- window($"time", "10 seconds").as("window1"),
- window($"time2", "10 seconds").as("window2")
- )
- .select(
- $"window1.end".cast("string"),
- window_time($"window1").cast("string"),
- $"window2.end".cast("string"),
- window_time($"window2").cast("string")
- )
- }
- assert(e.getMessage.contains(
- "Multiple time/session window expressions would result in a cartesian product of rows, " +
- "therefore they are currently not supported"))
+ val df2 = df
+ .withColumn("time2", expr("time - INTERVAL 15 minutes"))
+ .select(window($"time", "10 seconds").as("window1"), $"time2")
+ .select($"window1", window($"time2", "10 seconds").as("window2"))
+
+ checkAnswer(
+ df2.select(
+ $"window1.end".cast("string"),
+ window_time($"window1").cast("string"),
+ $"window2.end".cast("string"),
+ window_time($"window2").cast("string")),
+ Seq(
+ Row("2016-03-27 19:38:20", "2016-03-27 19:38:19.999999",
+ "2016-03-27 19:23:20", "2016-03-27 19:23:19.999999"),
+ Row("2016-03-27 19:39:30", "2016-03-27 19:39:29.999999",
+ "2016-03-27 19:24:30", "2016-03-27 19:24:29.999999"))
+ )
+
+ // check column names
+ val df3 = df2
+ .select(
+ window_time($"window1").cast("string"),
+ window_time($"window2").cast("string"),
+ window_time($"window2").as("wt2_aliased").cast("string")
Review Comment:
This checks the functionality that "same" call of window_time function won't bring conflict, and can be tagged with different column name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on code in PR #38361:
URL: https://github.com/apache/spark/pull/38361#discussion_r1002851334
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala:
##########
@@ -292,53 +292,59 @@ object ResolveWindowTime extends Rule[LogicalPlan] {
val windowTimeExpressions =
p.expressions.flatMap(_.collect { case w: WindowTime => w }).toSet
- if (windowTimeExpressions.size == 1 &&
- windowTimeExpressions.head.windowColumn.resolved &&
- windowTimeExpressions.head.checkInputDataTypes().isSuccess) {
+ val allWindowTimeExprsResolved = windowTimeExpressions.forall { w =>
+ w.windowColumn.resolved && w.checkInputDataTypes().isSuccess
+ }
- val windowTime = windowTimeExpressions.head
+ if (windowTimeExpressions.nonEmpty && allWindowTimeExprsResolved) {
+ val windowTimeToAttrAndNewColumn = windowTimeExpressions.map { windowTime =>
+ val metadata = windowTime.windowColumn match {
+ case a: Attribute => a.metadata
+ case _ => Metadata.empty
+ }
- val metadata = windowTime.windowColumn match {
- case a: Attribute => a.metadata
- case _ => Metadata.empty
- }
+ if (!metadata.contains(TimeWindow.marker) &&
+ !metadata.contains(SessionWindow.marker)) {
+ // FIXME: error framework?
+ throw new AnalysisException(
+ "The input is not a correct window column: $windowTime", plan = Some(p))
+ }
- if (!metadata.contains(TimeWindow.marker) &&
- !metadata.contains(SessionWindow.marker)) {
- // FIXME: error framework?
- throw new AnalysisException(
- "The input is not a correct window column: $windowTime", plan = Some(p))
- }
+ val newMetadata = new MetadataBuilder()
+ .withMetadata(metadata)
+ .remove(TimeWindow.marker)
+ .remove(SessionWindow.marker)
+ .build()
- val newMetadata = new MetadataBuilder()
- .withMetadata(metadata)
- .remove(TimeWindow.marker)
- .remove(SessionWindow.marker)
- .build()
+ val colName = windowTime.sql
Review Comment:
The schema checker of state store allows changing name on the column. It majority checks type and nullability.
The ideal column name would be exactly the same as what users call as it is, but I can't find the way to do (since there are multiple ways to call the SQL function, and here we do not have a context on SQL). Seems like this is a best effort, if this way is already a thing to define the resulting column name for other SQL functions as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on code in PR #38361:
URL: https://github.com/apache/spark/pull/38361#discussion_r1002851334
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala:
##########
@@ -292,53 +292,59 @@ object ResolveWindowTime extends Rule[LogicalPlan] {
val windowTimeExpressions =
p.expressions.flatMap(_.collect { case w: WindowTime => w }).toSet
- if (windowTimeExpressions.size == 1 &&
- windowTimeExpressions.head.windowColumn.resolved &&
- windowTimeExpressions.head.checkInputDataTypes().isSuccess) {
+ val allWindowTimeExprsResolved = windowTimeExpressions.forall { w =>
+ w.windowColumn.resolved && w.checkInputDataTypes().isSuccess
+ }
- val windowTime = windowTimeExpressions.head
+ if (windowTimeExpressions.nonEmpty && allWindowTimeExprsResolved) {
+ val windowTimeToAttrAndNewColumn = windowTimeExpressions.map { windowTime =>
+ val metadata = windowTime.windowColumn match {
+ case a: Attribute => a.metadata
+ case _ => Metadata.empty
+ }
- val metadata = windowTime.windowColumn match {
- case a: Attribute => a.metadata
- case _ => Metadata.empty
- }
+ if (!metadata.contains(TimeWindow.marker) &&
+ !metadata.contains(SessionWindow.marker)) {
+ // FIXME: error framework?
+ throw new AnalysisException(
+ "The input is not a correct window column: $windowTime", plan = Some(p))
+ }
- if (!metadata.contains(TimeWindow.marker) &&
- !metadata.contains(SessionWindow.marker)) {
- // FIXME: error framework?
- throw new AnalysisException(
- "The input is not a correct window column: $windowTime", plan = Some(p))
- }
+ val newMetadata = new MetadataBuilder()
+ .withMetadata(metadata)
+ .remove(TimeWindow.marker)
+ .remove(SessionWindow.marker)
+ .build()
- val newMetadata = new MetadataBuilder()
- .withMetadata(metadata)
- .remove(TimeWindow.marker)
- .remove(SessionWindow.marker)
- .build()
+ val colName = windowTime.sql
Review Comment:
The schema checker of state store allows changing name on the column. It majority checks type and nullability.
The ideal column name would be exactly the same as what users call as it is, but I can't find the way to do, and I'm not sure it is available for all usages since there are multiple ways to call the SQL function. Seems like this is a best effort, if this way is already a thing to define the resulting column name for other SQL functions as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #38361:
URL: https://github.com/apache/spark/pull/38361#discussion_r1002855960
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala:
##########
@@ -292,53 +292,59 @@ object ResolveWindowTime extends Rule[LogicalPlan] {
val windowTimeExpressions =
p.expressions.flatMap(_.collect { case w: WindowTime => w }).toSet
- if (windowTimeExpressions.size == 1 &&
- windowTimeExpressions.head.windowColumn.resolved &&
- windowTimeExpressions.head.checkInputDataTypes().isSuccess) {
+ val allWindowTimeExprsResolved = windowTimeExpressions.forall { w =>
+ w.windowColumn.resolved && w.checkInputDataTypes().isSuccess
+ }
- val windowTime = windowTimeExpressions.head
+ if (windowTimeExpressions.nonEmpty && allWindowTimeExprsResolved) {
+ val windowTimeToAttrAndNewColumn = windowTimeExpressions.map { windowTime =>
+ val metadata = windowTime.windowColumn match {
+ case a: Attribute => a.metadata
+ case _ => Metadata.empty
+ }
- val metadata = windowTime.windowColumn match {
- case a: Attribute => a.metadata
- case _ => Metadata.empty
- }
+ if (!metadata.contains(TimeWindow.marker) &&
+ !metadata.contains(SessionWindow.marker)) {
+ // FIXME: error framework?
+ throw new AnalysisException(
+ "The input is not a correct window column: $windowTime", plan = Some(p))
+ }
- if (!metadata.contains(TimeWindow.marker) &&
- !metadata.contains(SessionWindow.marker)) {
- // FIXME: error framework?
- throw new AnalysisException(
- "The input is not a correct window column: $windowTime", plan = Some(p))
- }
+ val newMetadata = new MetadataBuilder()
+ .withMetadata(metadata)
+ .remove(TimeWindow.marker)
+ .remove(SessionWindow.marker)
+ .build()
- val newMetadata = new MetadataBuilder()
- .withMetadata(metadata)
- .remove(TimeWindow.marker)
- .remove(SessionWindow.marker)
- .build()
+ val colName = windowTime.sql
Review Comment:
If we allow name change, this is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls
URL: https://github.com/apache/spark/pull/38361
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org