You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean R. Owen (Jira)" <ji...@apache.org> on 2021/09/26 00:53:00 UTC
[jira] [Resolved] (SPARK-35862) Watermark timestamp only can be
format in UTC timeZone, unfriendly to users in other time zones
[ https://issues.apache.org/jira/browse/SPARK-35862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean R. Owen resolved SPARK-35862.
----------------------------------
Resolution: Won't Fix
> Watermark timestamp only can be format in UTC timeZone, unfriendly to users in other time zones
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-35862
> URL: https://issues.apache.org/jira/browse/SPARK-35862
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.1.2
> Reporter: Yazhi Wang
> Priority: Minor
>
> Timestamp is formatted in `ProgressReporter` by `formatTimestamp` for watermark and eventTime stats. the timestampFormat is hardcoded in UTC time zone.
> `
> private val timestampFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'") // ISO8601
> timestampFormat.setTimeZone(DateTimeUtils.getTimeZone("UTC"))
> `
> When users set the different timezone by java options `-Duser.timezone` , they may be confused by the information mixed with different timezone.
> eg
> `
> {color:#FF0000}*2021-06-23 16:12:07*{color} [stream execution thread for [id = 92f4f363-df85-48e9-aef9-5ea6f2b70316, runId = 5733ef8e-11d1-46c4-95cc-219bde6e7a20]] INFO [MicroBatchExecution:54]: Streaming query made progress: {
> "id" : "92f4f363-df85-48e9-aef9-5ea6f2b70316",
> "runId" : "5733ef8e-11d1-46c4-95cc-219bde6e7a20",
> "name" : null,
> "timestamp" : "2021-06-23T08:11:56.790Z",
> "batchId" : 91740,
> "numInputRows" : 2577,
> "inputRowsPerSecond" : 155.33453887884266,
> "processedRowsPerSecond" : 242.29033471229786,
> "durationMs" :
> { "addBatch" : 8671, "getBatch" : 3, "getOffset" : 1139, "queryPlanning" : 79, "triggerExecution" : 10636, "walCommit" : 162 }
> ,
> "eventTime" :
> {color:#FF0000}*{ "avg" : "2021-06-23T08:11:46.307Z", "max" : "2021-06-23T08:11:55.000Z", "min" : "2021-06-23T08:11:37.000Z", "watermark" : "2021-06-23T07:41:39.000Z" }*{color}
> ,
> `
> maybe we need to unified the timezone for time format
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org