You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@storm.apache.org by "Aaron Gresch (Jira)" <ji...@apache.org> on 2020/12/16 17:18:00 UTC

[jira] [Created] (STORM-3724) Use blobstore dir modtime to avoid update lookups by HDFSBlobstore

Aaron Gresch created STORM-3724:
-----------------------------------

             Summary: Use blobstore dir modtime to avoid update lookups by HDFSBlobstore
                 Key: STORM-3724
                 URL: https://issues.apache.org/jira/browse/STORM-3724
             Project: Apache Storm
          Issue Type: Improvement
            Reporter: Aaron Gresch
            Assignee: Aaron Gresch


We have multiple storm clusters with 100's of supervisors polling for blob updates.  This causes high load on our Hadoop namenodes that are also used by multiple other clusters.

 

An improvement would be for the AsyncLocalizer to check the remote blobstore last mod time once and then skip checking each individual blob if it was already checked for the same mod time.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)