You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Heather McCartney (JIRA)" <ji...@apache.org> on 2017/09/11 12:14:00 UTC
[jira] [Updated] (STORM-2736) o.a.s.b.BlobStoreUtils [ERROR] Could
not update the blob with key
[ https://issues.apache.org/jira/browse/STORM-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Heather McCartney updated STORM-2736:
-------------------------------------
Description:
Sometimes, after our topologies have been running for a while, Zookeeper does not respond within an appropriate time and we see
{code}
2017-08-16 10:18:38.859 o.a.s.zookeeper [INFO] ip-10-181-20-70.ec2.internal lost leadership.
2017-08-16 10:21:31.144 o.a.s.zookeeper [INFO] ip-10-181-20-70.ec2.internal gained leadership, checking if it has all the topology code locally.
2017-08-16 10:21:46.201 o.a.s.zookeeper [INFO] Accepting leadership, all active topology found localy.
{code}
That's fine, and we probably need to allocate more resources. But after a new leader is chosen, we then see:
{code}
o.a.s.b.BlobStoreUtils [ERROR] Could not update the blob with key<key>
{code}
over and over.
I can't figure out yet how to cause the conditions that lead to Zookeeper becoming unresponsive, but it is possible to reproduce the BlobStoreUtils error by restarting Zookeeper.
The problem, I think, is that the loop [here|https://github.com/apache/storm/blob/v1.1.1/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java#L175] never executes because the {{NimbusInfos}} list is empty. If I add a check similar to [this|https://github.com/apache/storm/blob/v1.1.1/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java#L244] for a node which exists but has no children, the error goes away.
was:
Sometimes, after our topologies have been running for a while, Zookeeper does not respond within an appropriate time and we see
{{
2017-08-16 10:18:38.859 o.a.s.zookeeper [INFO] ip-10-181-20-70.ec2.internal lost leadership.
2017-08-16 10:21:31.144 o.a.s.zookeeper [INFO] ip-10-181-20-70.ec2.internal gained leadership, checking if it has all the topology code locally.
2017-08-16 10:21:46.201 o.a.s.zookeeper [INFO] Accepting leadership, all active topology found localy.
}}
That's fine, and we probably need to allocate more resources in the long run. But after a new leader is chosen, we then see:
{{
o.a.s.b.BlobStoreUtils [ERROR] Could not update the blob with key<key>
}}
over and over.
I can't figure out yet how to cause the conditions that lead to Zookeeper becoming unresponsive, but it is possible to reproduce the BlobStoreUtils error by restarting Zookeeper.
The problem, I think, is that the loop [here](https://github.com/apache/storm/blob/v1.1.1/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java#L175) never executes because the {NimbusInfos} list is empty. If I add a check similar to [this](https://github.com/apache/storm/blob/v1.1.1/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java#L244) for a node which exists but has no children, the error goes away.
> o.a.s.b.BlobStoreUtils [ERROR] Could not update the blob with key
> -----------------------------------------------------------------
>
> Key: STORM-2736
> URL: https://issues.apache.org/jira/browse/STORM-2736
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
> Affects Versions: 1.1.1
> Reporter: Heather McCartney
> Assignee: Heather McCartney
> Priority: Minor
>
> Sometimes, after our topologies have been running for a while, Zookeeper does not respond within an appropriate time and we see
> {code}
> 2017-08-16 10:18:38.859 o.a.s.zookeeper [INFO] ip-10-181-20-70.ec2.internal lost leadership.
> 2017-08-16 10:21:31.144 o.a.s.zookeeper [INFO] ip-10-181-20-70.ec2.internal gained leadership, checking if it has all the topology code locally.
> 2017-08-16 10:21:46.201 o.a.s.zookeeper [INFO] Accepting leadership, all active topology found localy.
> {code}
> That's fine, and we probably need to allocate more resources. But after a new leader is chosen, we then see:
> {code}
> o.a.s.b.BlobStoreUtils [ERROR] Could not update the blob with key<key>
> {code}
> over and over.
> I can't figure out yet how to cause the conditions that lead to Zookeeper becoming unresponsive, but it is possible to reproduce the BlobStoreUtils error by restarting Zookeeper.
> The problem, I think, is that the loop [here|https://github.com/apache/storm/blob/v1.1.1/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java#L175] never executes because the {{NimbusInfos}} list is empty. If I add a check similar to [this|https://github.com/apache/storm/blob/v1.1.1/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java#L244] for a node which exists but has no children, the error goes away.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)