[jira] [Comment Edited] (STORM-2736) o.a.s.b.BlobStoreUtils [ERROR] Could not update the blob with key

["Pydi Prasanna Sai Kamanuru (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16732533#comment-16732533 ] 

Pydi Prasanna Sai Kamanuru edited comment on STORM-2736 at 1/3/19 12:07 AM:
----------------------------------------------------------------------------

Hey [~revans2] [~hmcc] ,

I just encountered this issue in our prod environment ( Running a older version ). We only run one nimbus node (not HA), similiary zk issues lead to the situation where there are no children to the node 
 * /storm.104/blobstore/key

Shouldn't nimbus automatically create the children with hostname:port-sequencenumber automatically if it is up and running ? In my case, nimbus did not crash but the ephemeral nodes( the children in the case) vanished  when the connection was reset between nimbus and zookeeper. I don't see any code path that creates the children if they are missing in the /blobstore zk path.

 


was (Author: pkamanuru):
Hey [~revans2] [~hmcc] ,

I just encountered this issue in our prod environment ( Running a older version ). We only run one nimbus node (not HA), similiary zk issues lead to the situation where there are no children to the node 
 * /storm.104/blobstore/key

Shouldn't nimbus automatically create the children with hostname:port-sequencenumber automatically if it is up and running ?

 

> o.a.s.b.BlobStoreUtils [ERROR] Could not update the blob with key
> -----------------------------------------------------------------
>
>                 Key: STORM-2736
>                 URL: https://issues.apache.org/jira/browse/STORM-2736
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: storm-core
>    Affects Versions: 1.1.1
>            Reporter: Heather McCartney
>            Assignee: Heather McCartney
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 2.0.0, 1.2.0, 1.1.2
>
>          Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Sometimes, after our topologies have been running for a while, Zookeeper does not respond within an appropriate time and we see
> {code}
> 2017-08-16 10:18:38.859 o.a.s.zookeeper [INFO] ip-10-181-20-70.ec2.internal lost leadership.
> 2017-08-16 10:21:31.144 o.a.s.zookeeper [INFO] ip-10-181-20-70.ec2.internal gained leadership, checking if it has all the topology code locally.
> 2017-08-16 10:21:46.201 o.a.s.zookeeper [INFO] Accepting leadership, all active topology found localy.
> {code}
> That's fine, and we probably need to allocate more resources. But after a new leader is chosen, we then see:
> {code}
> o.a.s.b.BlobStoreUtils [ERROR] Could not update the blob with key<key>
> {code}
> over and over.
> I can't figure out yet how to cause the conditions that lead to Zookeeper becoming unresponsive, but it is possible to reproduce the {{BlobStoreUtils}} error by restarting Zookeeper.
> The problem, I think, is that the loop [here|https://github.com/apache/storm/blob/v1.1.1/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java#L175] never executes because the {{nimbusInfos}} list is empty. If I add a check similar to [this|https://github.com/apache/storm/blob/v1.1.1/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java#L244] for a node which exists but has no children, the error goes away.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)