You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by M Singh <ma...@yahoo.com.INVALID> on 2018/03/28 17:10:38 UTC

Apache Spark - Structured Streaming StreamExecution Stats Description

Hi:
I am using spark structured streaming 2.2.1 and am using flatMapGroupWithState and a groupBy count operators.
 In the StreamExecution logs I see two enteries for stateOperators
"stateOperators" : [ {
    "numRowsTotal" : 1617339,
    "numRowsUpdated" : 9647
  }, {
    "numRowsTotal" : 1326355,
    "numRowsUpdated" : 1398672
  } ],
My questions are:1. Is there way to figure out which stats is for flatMapGroupWithState and which one for groupBy count ?  In my case, I can guess based on my data but want to be definitive about it.2. For the second stats - how can the numRowsTotal (1326355) be less than numRowsUpdated (1398672) ?
If there in documentation I can use to understand the debug output, please let me know.

Thanks