Re: [PR] [SPARK-46906][SS] Add a check for stateful operator change for streaming [spark]

["jingz-db (via GitHub)" <gi...@apache.org>]

jingz-db commented on code in PR #44927:
URL: https://github.com/apache/spark/pull/44927#discussion_r1488336417


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -184,6 +185,41 @@ class IncrementalExecution(
     }
   }
 
+  /**

Review Comment:
   Thanks for the review Jungtaek! I also like the idea of adding a map.
   > So the check can be also done after executing physical planning rules, maybe at the end of state.apply()
   
   I tried to add the check after `WriteStatefulOperatorMetadataRule` but this will miss detecting for adding an operator after restart (because the additional operator is already written to metadata). So I keep the check before `WriteStatefulOperatorMetadataRule` and will omit the check if metadata is empty.
   It is also worth noting that if we do not perform the check before writing to metadata and fail the query, untruthful info will be written to state metadata.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org