You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/05/02 18:55:22 UTC

[GitHub] [spark] dongjoon-hyun opened a new pull request #24516: [SPARK-27624][CORE] Fix CalenderInterval to show an empty interval correctly

dongjoon-hyun opened a new pull request #24516: [SPARK-27624][CORE] Fix CalenderInterval to show an empty interval correctly
URL: https://github.com/apache/spark/pull/24516
 
 
   ## What changes were proposed in this pull request?
   
   If the interval is `0`, it doesn't show the value `0` at all. For example, this happens in the explain plans and Spark Web UI on `EventTimeWatermark` diagram.
   
   **BEFORE**
   ```scala
   scala> spark.readStream.schema("ts timestamp").parquet("/tmp/t").withWatermark("ts", "1 microsecond").explain
   == Physical Plan ==
   EventTimeWatermark ts#0: timestamp, interval 1 microseconds
   +- StreamingRelation FileSource[/tmp/t], [ts#0]
   
   scala> spark.readStream.schema("ts timestamp").parquet("/tmp/t").withWatermark("ts", "0 microsecond").explain
   == Physical Plan ==
   EventTimeWatermark ts#3: timestamp, interval
   +- StreamingRelation FileSource[/tmp/t], [ts#3]
   ```
   
   **AFTER**
   ```scala
   scala> spark.readStream.schema("ts timestamp").parquet("/tmp/t").withWatermark("ts", "1 microsecond").explain
   == Physical Plan ==
   EventTimeWatermark ts#0: timestamp, interval 1 microseconds
   +- StreamingRelation FileSource[/tmp/t], [ts#0]
   
   scala> spark.readStream.schema("ts timestamp").parquet("/tmp/t").withWatermark("ts", "0 microsecond").explain
   == Physical Plan ==
   EventTimeWatermark ts#3: timestamp, interval 0 microseconds
   +- StreamingRelation FileSource[/tmp/t], [ts#3]
   ```
   
   ## How was this patch tested?
   
   Pass the Jenkins with the updated test case.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org