You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Antoine Tran (JIRA)" <ji...@apache.org> on 2017/03/01 15:55:45 UTC
[jira] [Commented] (STORM-1469) Unable to deploy large topologies
on apache storm
[ https://issues.apache.org/jira/browse/STORM-1469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890425#comment-15890425 ]
Antoine Tran commented on STORM-1469:
-------------------------------------
In 1.0.1, using PlainSaslTransportPlugin will break supervisors with an ACL error. I suspect the uberjar couldn't be put in the blobstore.
The STORM-1661 fix this by putting permissive Acl in blobstore.
> Unable to deploy large topologies on apache storm
> -------------------------------------------------
>
> Key: STORM-1469
> URL: https://issues.apache.org/jira/browse/STORM-1469
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
> Affects Versions: 1.0.0, 2.0.0
> Reporter: Rudra Sharma
> Assignee: Kishor Patil
> Fix For: 1.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When deploying to a nimbus a topology which is larger in size >17MB, we get an exception. In storm 0.9.3 this could be mitigated by using the following config on the storm.yaml to increse the buffer size to handle the topology size. i.e. 50MB would be
> nimbus.thrift.max_buffer_size: 50000000
> This configuration does not resolve the issue in the master branch of storm and we cannot deploy topologies which are large in size.
> Here is the log on the client side when attempting to deploy to the nimbus node:
> java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException
> at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:251) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:272) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:155) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at com.trustwave.siem.storm.topology.deployer.TopologyDeployer.deploy(TopologyDeployer.java:149) [siem-ng-storm-deployer-cloud.jar:]
> at com.trustwave.siem.storm.topology.deployer.TopologyDeployer.main(TopologyDeployer.java:87) [siem-ng-storm-deployer-cloud.jar:]
> Caused by: org.apache.thrift7.transport.TTransportException
> at org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:77) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:238) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:222) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:237) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> ... 4 more
> Here is the log on the server side (nimbus.log):
> 2016-01-13 10:48:07.206 o.a.s.d.nimbus [INFO] Cleaning inbox ... deleted: stormjar-c8666220-fa19-426b-a7e4-c62dfb57f1f0.jar
> 2016-01-13 10:55:09.823 o.a.s.d.nimbus [INFO] Uploading file from client to /var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar
> 2016-01-13 10:55:11.910 o.a.s.d.nimbus [INFO] Finished uploading file from client: /var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar
> 2016-01-13 10:55:12.084 o.a.t.s.AbstractNonblockingServer$FrameBuffer [WARN] Exception while invoking!
> org.apache.thrift7.transport.TTransportException: Frame size (17435758) larger than max length (16384000)!
> at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
> at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86)
> at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:27)
> at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
> at org.apache.thrift7.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
> at org.apache.thrift7.server.Invocation.run(Invocation.java:18)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)