You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yang Jie (Jira)" <ji...@apache.org> on 2022/12/23 03:48:00 UTC
[jira] [Commented] (SPARK-41677) Protobuf serializer for StreamingQueryProgressWrapper
[ https://issues.apache.org/jira/browse/SPARK-41677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17651476#comment-17651476 ]
Yang Jie commented on SPARK-41677:
----------------------------------
{code:java}
private[sql] class StreamingQueryProgressWrapper(val progress: StreamingQueryProgress) {
@JsonIgnore @KVIndex
private val uniqueId: String = getUniqueId(progress.runId, progress.batchId, progress.timestamp)
@JsonIgnore @KVIndex("runId")
private def runIdIndex: String = progress.runId.toString
}
class StreamingQueryProgress private[sql](
val id: UUID,
val runId: UUID,
val name: String,
val timestamp: String,
val batchId: Long,
val batchDuration: Long,
val durationMs: ju.Map[String, JLong],
val eventTime: ju.Map[String, String],
val stateOperators: Array[StateOperatorProgress],
val sources: Array[SourceProgress],
val sink: SinkProgress,
@JsonDeserialize(contentAs = classOf[GenericRowWithSchema])
val observedMetrics: ju.Map[String, Row]) extends Serializable { {code}
This one is complicated, new to solve how to serialize Row with protobuf
> Protobuf serializer for StreamingQueryProgressWrapper
> -----------------------------------------------------
>
> Key: SPARK-41677
> URL: https://issues.apache.org/jira/browse/SPARK-41677
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Yang Jie
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org