You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@samza.apache.org by "Lakshmi Manasa Gaduputi (Jira)" <ji...@apache.org> on 2022/06/27 20:19:00 UTC

[jira] [Updated] (SAMZA-2752) DiskSpace monitor reports 0 disk usage even for a stateful job with lots of disk usage

|  ![](cid:jira-generated-image-avatar-8cc8f668-1a65-42d3-8d79-324ea1abb84d) |
[Lakshmi Manasa
Gaduputi](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=lakshmi-
manasa) **updated** an issue  
---|---  
|  
---  
|  [Samza](https://issues.apache.org/jira/browse/SAMZA) / [![Bug](cid:jira-
generated-image-
avatar-f799b94b-9691-45a8-988e-23c35505aaa5)](https://issues.apache.org/jira/browse/SAMZA-2752)
[SAMZA-2752](https://issues.apache.org/jira/browse/SAMZA-2752)  
---  
[DiskSpace monitor reports 0 disk usage even for a stateful job with lots of
disk usage](https://issues.apache.org/jira/browse/SAMZA-2752)  
| Change By: |  [Lakshmi Manasa
Gaduputi](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=lakshmi-
manasa)  
---|---  
|  Stateful jobs after PR (#1491): report 0 disk usage for their disk spaced
used by their rocksdb kv stores.  
  
Reason:  
# [PR|https://github.com/apache/samza/pull/1491] changed the init of rocksdb
task stores from constructor of ContainerStorageManager to
ContainerStorageManager.restoreStores() (which is invoked at run and not in
the constructor). see these code changes of the PR for file
ContainerStorageManager  
#  However, SamzaContainer creates the disk space monitor during [setup (here)
|https://github.com/apache/samza/blob/master/samza-
core/src/main/scala/org/apache/samza/container/SamzaContainer.scala#L661]
(before run() which invokes restoreStores()) and SamzaContainer uses the store
paths returned by ContainerStorageManager after its constructor is done.  
# Hence, the task stores (not side input) are not part of the disk space
monitoring.  
# logs show "2022-06-22 20:46:41.897 [main] SamzaContainer$ [INFO] Initialized
disk space monitor watch paths to: [] // basically disk space monitor is not
looking at anything" after this PR but should be showing a non-empty set of
paths  
  
  
  
  
---  
|  |  [ ![Add Comment](cid:jira-generated-image-static-comment-
icon-50bd7332-9c72-4dcc-81e8-a0679fa00382)
](https://issues.apache.org/jira/browse/SAMZA-2752#add-comment "Add Comment")
|  [Add Comment](https://issues.apache.org/jira/browse/SAMZA-2752#add-comment
"Add Comment")  
---|---  
  
|  This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9) |  |
![Atlassian logo](https://issues.apache.org/jira/images/mail/atlassian-email-
logo.png)  
---