[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total estimated size of states ...

[arunmahadevan <gi...@git.apache.org>]

Github user arunmahadevan commented on the issue:

    https://github.com/apache/spark/pull/21469
  
    @HeartSaVioR , may be then this should be reported in the "memoryUsedBytes" in the StateOperatorProgress (value reported in StreamingQueryProgress) because currently the usage reported does not reflect the memory used for the cache.
    
    Question: in the screenshot "Estimated size of states cache in provider total" is 3.3 MB whereas the "memory used by state total" is 20.6 KB with "total number of state rows" = 2. This 150x difference is expected with just 2 rows in the state? Were there 100 versions of the map in the sample output you posted?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org