You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by pradeep s <sr...@gmail.com> on 2016/12/10 03:26:16 UTC
Spout failures too high
Hi,
We are running a 5 node cluster(2x large ec2) with below config. Topology
is consuming message from SQS and writing to RDS and S3. Even if there are
no bolt failures , we are seeing many spout failures.
Can you please help in checking the config.Also i am setting tasks as
parallelism count * 2. Is this fine?
TOPOLOGY_NAME=MDP_STORM_PRD
MARIA_BOLT_PARALLELISM=50
S3_BOLT_PARALLELISM=50
SQS_DELETE_BOLT_PARALLELISM=100
SPOUT_PARALLELISM=50
NUMBER_OF_WORKERS=5
NUMBER_OF_ACKERS=5
SPOUT_MAX_PENDING=5000
MESSAGE_TIMEOUT_SECONDS=240
Topology Code
=============
Config config = new Config();
config.setNumWorkers(numWorkers);
config.setDebug(false);
config.setNumAckers(numAckers);
config.setMaxSpoutPending(maxSpoutPending);
config.setMessageTimeoutSecs(messageTimeoutSecs);
topologyBuilder.setSpout(spoutId, new
SQSMessageReaderSpout(sqsUtils.getSQSUrl(dataQueue), properties),
spoutParallelism).*setNumTasks(spoutParallelism * TWO);*
topologyBuilder.setBolt(mariaBoltId, new MariaDbBolt(properties),
mariaBoltParallelism)
.*setNumTasks(mariaBoltParallelism * TWO)*.fieldsGrouping(spoutId,
new Fields(MESSAGE_ID));
topologyBuilder.setBolt(s3BoltId, new S3WriteBolt(properties,
s3Properties), s3BoltParallelism)
.*setNumTasks(s3BoltParallelism * TWO).*
shuffleGrouping(mariaBoltId);
topologyBuilder
.setBolt(sqsDeleteBoltId, new
SQSMessageDeleteBolt(sqsUtils.getSQSUrl(dataQueue)), sqsBoltParallelism)
.*setNumTasks(sqsBoltParallelism * TWO)*.shuffleGrouping(s3BoltId);
StormSubmitter.submitTopology(topologyName, config,
topologyBuilder.createTopology());
Regards
Pradeep S
Re: Spout failures too high
Posted by Kevin Peek <kp...@salesforce.com>.
The most common reason for spout failures but no bolt failures is tuple
timeouts. Try increasing MESSAGE_TIMEOUT_SECONDS. If you are seeing very
low execute latencies for your bolts but very high complete latency, you
can also try decreasing SPOUT_MAX_PENDING.
Hopefully this helps.
On Fri, Dec 9, 2016 at 10:26 PM, pradeep s <sr...@gmail.com>
wrote:
> Hi,
> We are running a 5 node cluster(2x large ec2) with below config. Topology
> is consuming message from SQS and writing to RDS and S3. Even if there are
> no bolt failures , we are seeing many spout failures.
> Can you please help in checking the config.Also i am setting tasks as
> parallelism count * 2. Is this fine?
>
> TOPOLOGY_NAME=MDP_STORM_PRD
> MARIA_BOLT_PARALLELISM=50
> S3_BOLT_PARALLELISM=50
> SQS_DELETE_BOLT_PARALLELISM=100
> SPOUT_PARALLELISM=50
> NUMBER_OF_WORKERS=5
> NUMBER_OF_ACKERS=5
> SPOUT_MAX_PENDING=5000
> MESSAGE_TIMEOUT_SECONDS=240
>
>
> Topology Code
> =============
>
> Config config = new Config();
> config.setNumWorkers(numWorkers);
> config.setDebug(false);
> config.setNumAckers(numAckers);
> config.setMaxSpoutPending(maxSpoutPending);
> config.setMessageTimeoutSecs(messageTimeoutSecs);
>
>
> topologyBuilder.setSpout(spoutId, new SQSMessageReaderSpout(sqsUtils.getSQSUrl(dataQueue),
> properties),
> spoutParallelism).*setNumTasks(spoutParallelism * TWO);*
> topologyBuilder.setBolt(mariaBoltId, new MariaDbBolt(properties),
> mariaBoltParallelism)
> .*setNumTasks(mariaBoltParallelism * TWO)*.fieldsGrouping(spoutId,
> new Fields(MESSAGE_ID));
>
> topologyBuilder.setBolt(s3BoltId, new S3WriteBolt(properties,
> s3Properties), s3BoltParallelism)
> .*setNumTasks(s3BoltParallelism * TWO).*shuffleGrouping(
> mariaBoltId);
>
> topologyBuilder
> .setBolt(sqsDeleteBoltId, new SQSMessageDeleteBolt(sqsUtils.getSQSUrl(dataQueue)),
> sqsBoltParallelism)
> .*setNumTasks(sqsBoltParallelism * TWO)*.shuffleGrouping(s3BoltId)
> ;
>
> StormSubmitter.submitTopology(topologyName, config, topologyBuilder.
> createTopology());
>
>
> Regards
>
> Pradeep S
>
Re: Spout failures too high
Posted by Li Wang <wa...@gmail.com>.
Hi Pradeep,
Your config looks fine to me. Could you please check the log and see what causes the failure?
Regards,
Li
Sent from my iPhone
> On 10 Dec 2016, at 11:26 AM, pradeep s <sr...@gmail.com> wrote:
>
> Hi,
> We are running a 5 node cluster(2x large ec2) with below config. Topology
> is consuming message from SQS and writing to RDS and S3. Even if there are
> no bolt failures , we are seeing many spout failures.
> Can you please help in checking the config.Also i am setting tasks as
> parallelism count * 2. Is this fine?
>
> TOPOLOGY_NAME=MDP_STORM_PRD
> MARIA_BOLT_PARALLELISM=50
> S3_BOLT_PARALLELISM=50
> SQS_DELETE_BOLT_PARALLELISM=100
> SPOUT_PARALLELISM=50
> NUMBER_OF_WORKERS=5
> NUMBER_OF_ACKERS=5
> SPOUT_MAX_PENDING=5000
> MESSAGE_TIMEOUT_SECONDS=240
>
>
> Topology Code
> =============
>
> Config config = new Config();
> config.setNumWorkers(numWorkers);
> config.setDebug(false);
> config.setNumAckers(numAckers);
> config.setMaxSpoutPending(maxSpoutPending);
> config.setMessageTimeoutSecs(messageTimeoutSecs);
>
>
> topologyBuilder.setSpout(spoutId, new
> SQSMessageReaderSpout(sqsUtils.getSQSUrl(dataQueue), properties),
> spoutParallelism).*setNumTasks(spoutParallelism * TWO);*
> topologyBuilder.setBolt(mariaBoltId, new MariaDbBolt(properties),
> mariaBoltParallelism)
> .*setNumTasks(mariaBoltParallelism * TWO)*.fieldsGrouping(spoutId,
> new Fields(MESSAGE_ID));
>
> topologyBuilder.setBolt(s3BoltId, new S3WriteBolt(properties,
> s3Properties), s3BoltParallelism)
> .*setNumTasks(s3BoltParallelism * TWO).*
> shuffleGrouping(mariaBoltId);
>
> topologyBuilder
> .setBolt(sqsDeleteBoltId, new
> SQSMessageDeleteBolt(sqsUtils.getSQSUrl(dataQueue)), sqsBoltParallelism)
> .*setNumTasks(sqsBoltParallelism * TWO)*.shuffleGrouping(s3BoltId);
>
> StormSubmitter.submitTopology(topologyName, config,
> topologyBuilder.createTopology());
>
>
> Regards
>
> Pradeep S