You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "EL YOUSSFI Nordine (Jira)" <ji...@apache.org> on 2019/10/28 15:03:00 UTC
[jira] [Comment Edited] (STORM-1469) Unable to deploy large
topologies on apache storm
[ https://issues.apache.org/jira/browse/STORM-1469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961129#comment-16961129 ]
EL YOUSSFI Nordine edited comment on STORM-1469 at 10/28/19 3:02 PM:
---------------------------------------------------------------------
This issue related to the size of the serialized topology has been fixed on Storm release 2.0.0. The parameter nimbus.thrift.max_buffer_size is now take into account and allows to submit topologies to the nimbus according to the maximum size configured.
was (Author: nordineelyoussfi):
This issue related to the size of the serialized topology has been fixed on Storm release 2.0.0. The parameter nimbus.thrift.max_buffer_size is now take into account and allows to submit topologies to the nimbus according to the size configured.
> Unable to deploy large topologies on apache storm
> -------------------------------------------------
>
> Key: STORM-1469
> URL: https://issues.apache.org/jira/browse/STORM-1469
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
> Affects Versions: 1.0.0, 2.0.0
> Reporter: Rudra Sharma
> Assignee: Kishor Patil
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When deploying to a nimbus a topology which is larger in size >17MB, we get an exception. In storm 0.9.3 this could be mitigated by using the following config on the storm.yaml to increse the buffer size to handle the topology size. i.e. 50MB would be
> nimbus.thrift.max_buffer_size: 50000000
> This configuration does not resolve the issue in the master branch of storm and we cannot deploy topologies which are large in size.
> Here is the log on the client side when attempting to deploy to the nimbus node:
> java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException
> at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:251) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:272) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:155) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at com.trustwave.siem.storm.topology.deployer.TopologyDeployer.deploy(TopologyDeployer.java:149) [siem-ng-storm-deployer-cloud.jar:]
> at com.trustwave.siem.storm.topology.deployer.TopologyDeployer.main(TopologyDeployer.java:87) [siem-ng-storm-deployer-cloud.jar:]
> Caused by: org.apache.thrift7.transport.TTransportException
> at org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:77) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:238) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:222) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:237) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
> ... 4 more
> Here is the log on the server side (nimbus.log):
> 2016-01-13 10:48:07.206 o.a.s.d.nimbus [INFO] Cleaning inbox ... deleted: stormjar-c8666220-fa19-426b-a7e4-c62dfb57f1f0.jar
> 2016-01-13 10:55:09.823 o.a.s.d.nimbus [INFO] Uploading file from client to /var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar
> 2016-01-13 10:55:11.910 o.a.s.d.nimbus [INFO] Finished uploading file from client: /var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar
> 2016-01-13 10:55:12.084 o.a.t.s.AbstractNonblockingServer$FrameBuffer [WARN] Exception while invoking!
> org.apache.thrift7.transport.TTransportException: Frame size (17435758) larger than max length (16384000)!
> at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
> at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86)
> at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:27)
> at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
> at org.apache.thrift7.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
> at org.apache.thrift7.server.Invocation.run(Invocation.java:18)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)