You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Michael Schonfeld <mi...@modernmast.com> on 2015/03/27 16:17:15 UTC

ReturnResultsReducer bottleneck in Trident

Hey guys,

We're experiencing a severe bottleneck issue with our trident topology.
Basically, we have a DRPC function that queries an Elasticsearch db. Said
query may take a few seconds to respond, and because of
ReturnResultsReducer, even with an increased parallelismHint, all future
DRPC queries' responses must wait until the slow Elasticsearch query is
done.

We've tried staging this part of the trident topology both as a State
Query, and a .each Base Function. In both cases, everything funnels down to
a single bolt - ReturnResultsReducer.

```
topology.newDRPCStream("search", localDRPC)
  .stateQuery(reportDb, new Fields("args"), new ReportQuery(), new
Fields("response"))
  .parallelismHint(5);

```

```
topology.newDRPCStream("search", localDRPC)
  .each(new Fields("args"), new DRPCSearch(), new Fields("response"))
  .parallelismHint(5);
```

Any ideas how we can get around this bottleneck?

Thanks,
- Michael