[jira] [Commented] (STORM-636) UI/Monitor is slow for topologies with a large number of components

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287775#comment-14287775 ] 

ASF GitHub Bot commented on STORM-636:
--------------------------------------

GitHub user d2r opened a pull request:

    https://github.com/apache/storm/pull/392

    [STORM-636] Faster, optional retrieval of last component error

    We want to speed up interactions with the topology that call getTopologyInfo, including the Topology Page in the UI.
    
    Before change:
    - Errors written to /errors/topo-id/comp-name/eN, where N is a sequence number.
    - getTopologyInfo grabs all /errors/topo-id/*/* errors from ZK
    
    After this change:
    - Errors written to /errors/topo-id/comp-name/eN, where N is a sequence number. (unchanged)
    - Errors also written to /errors/topo-id/comp-name-last-error (this means there is an extra ZK write per error)
    - New Nimbus thrift interface method `getTopologyInfoWithOpts`
      - Only option right now is how many errors to retrieve from ZK: 0, 1, or all.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/d2r/storm storm-636-ui-errors

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/storm/pull/392.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #392
    
----
commit 1cfa190f2efb06f8798984b43dec801e5ff20ad5
Author: Derek Dagit <de...@yahoo-inc.com>
Date:   2015-01-22T16:46:03Z

    Faster, optional retrieval of last component error

----


> UI/Monitor is slow for topologies with a large number of components
> -------------------------------------------------------------------
>
>                 Key: STORM-636
>                 URL: https://issues.apache.org/jira/browse/STORM-636
>             Project: Apache Storm
>          Issue Type: Bug
>    Affects Versions: 0.10.0
>            Reporter: Derek Dagit
>            Assignee: Derek Dagit
>            Priority: Minor
>
> The getTopologyInfo method in nimbus fetches from ZK all errors reported by all components.  This becomes too slow for topologies with a larger numbers of components  (bolts/spouts).
> In one example, the UI consistently took over 5 minutes to load the topology page for a topology with nearly 500 components while ZK was under load.
> Errors are currently stored in ZooKeeper under individual znodes per component.  This means that each call to getTopologyInfo needs to list children of each znode and then download the error znodes it finds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)