You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Gabor Somogyi (JIRA)" <ji...@apache.org> on 2019/07/03 13:49:00 UTC
[jira] [Comment Edited] (SPARK-27975) ConsoleSink should display
alias and options for streaming progress
[ https://issues.apache.org/jira/browse/SPARK-27975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16877836#comment-16877836 ]
Gabor Somogyi edited comment on SPARK-27975 at 7/3/19 1:48 PM:
---------------------------------------------------------------
After I've moved console sink from v1 to v2 the progress looks like this:
{code:java}
{
"id" : "b0a8ffaa-900b-4a8b-8769-c2f43b9cf99b",
"runId" : "6b4d01ed-7d0f-44de-8997-0b31bf2a106e",
"name" : null,
"timestamp" : "2019-07-03T12:55:17.104Z",
"batchId" : 0,
"numInputRows" : 1,
"processedRowsPerSecond" : 0.5115089514066496,
"durationMs" : {
"addBatch" : 1486,
"getBatch" : 3,
"latestOffset" : 0,
"queryPlanning" : 335,
"triggerExecution" : 1955,
"walCommit" : 69
},
"stateOperators" : [ ],
"sources" : [ {
"description" : "MemoryStream[value#1]",
"startOffset" : null,
"endOffset" : 0,
"numInputRows" : 1,
"processedRowsPerSecond" : 0.5115089514066496
} ],
"sink" : {
"description" : "org.apache.spark.sql.execution.streaming.ConsoleTable@373e6cb2",
"numOutputRows" : 1
}
}
{code}
I think TextSocketV2 is different from ConsoleTable. In TextSocketV2 the mentioned parameters are keys for the instance which is not the same for ConsoleTable. Additionally in DSv2, table and writer concept is split up and requires extra references to get information from writer.
I agree at the moment it's not possible to see the parameters of ConsoleWrite but this case I suggest to add a log entry (let's consider a sink where maybe 10+ different params are there for example Kafka).
was (Author: gsomogyi):
After I've moved console sink from v1 to v2 the progress looks like this:
{code:java}
{
"id" : "b0a8ffaa-900b-4a8b-8769-c2f43b9cf99b",
"runId" : "6b4d01ed-7d0f-44de-8997-0b31bf2a106e",
"name" : null,
"timestamp" : "2019-07-03T12:55:17.104Z",
"batchId" : 0,
"numInputRows" : 1,
"processedRowsPerSecond" : 0.5115089514066496,
"durationMs" : {
"addBatch" : 1486,
"getBatch" : 3,
"latestOffset" : 0,
"queryPlanning" : 335,
"triggerExecution" : 1955,
"walCommit" : 69
},
"stateOperators" : [ ],
"sources" : [ {
"description" : "MemoryStream[value#1]",
"startOffset" : null,
"endOffset" : 0,
"numInputRows" : 1,
"processedRowsPerSecond" : 0.5115089514066496
} ],
"sink" : {
"description" : "org.apache.spark.sql.execution.streaming.ConsoleTable@373e6cb2",
"numOutputRows" : 1
}
}
{code}
I think TextSocketV2 is different from ConsoleTable. In TextSocketV2 the mentioned parameters are keys for the instance which is not the same for ConsoleTable.
I agree at the moment it's not possible to see the parameters of ConsoleWrite but this case I suggest to add a log entry (let's consider a sink where maybe 10+ different params are there for example Kafka).
> ConsoleSink should display alias and options for streaming progress
> -------------------------------------------------------------------
>
> Key: SPARK-27975
> URL: https://issues.apache.org/jira/browse/SPARK-27975
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.0.0
> Reporter: Jacek Laskowski
> Priority: Minor
>
> {{console}} sink shows itself in progress with this internal instance representation as follows:
> {code:json}
> "sink" : {
> "description" : "org.apache.spark.sql.execution.streaming.ConsoleSinkProvider@12fa674a"
> }
> {code}
> That is not very user-friendly and would be much better for debugging if it included the alias and options as {{socket}} does:
> {code}
> "sources" : [ {
> "description" : "TextSocketV2[host: localhost, port: 8888]",
> ...
> } ],
> {code}
> The entire sample progress looks as follows:
> {code}
> 19/06/07 11:47:18 INFO MicroBatchExecution: Streaming query made progress: {
> "id" : "26bedc9f-076f-4b15-8e17-f09609aaecac",
> "runId" : "8c365e74-7ac9-4fad-bf1b-397eb086661e",
> "name" : "socket-console",
> "timestamp" : "2019-06-07T09:47:18.969Z",
> "batchId" : 2,
> "numInputRows" : 0,
> "inputRowsPerSecond" : 0.0,
> "durationMs" : {
> "getEndOffset" : 0,
> "setOffsetRange" : 0,
> "triggerExecution" : 0
> },
> "stateOperators" : [ ],
> "sources" : [ {
> "description" : "TextSocketV2[host: localhost, port: 8888]",
> "startOffset" : 0,
> "endOffset" : 0,
> "numInputRows" : 0,
> "inputRowsPerSecond" : 0.0
> } ],
> "sink" : {
> "description" : "org.apache.spark.sql.execution.streaming.ConsoleSinkProvider@12fa674a"
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org