You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2019/04/03 16:40:02 UTC
[jira] [Commented] (NIFI-6180) DruidTranquilityController: expose
`firehoseGracePeriod` for higher index task throughput
[ https://issues.apache.org/jira/browse/NIFI-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808898#comment-16808898 ]
ASF subversion and git services commented on NIFI-6180:
-------------------------------------------------------
Commit 3a63de2ae2debea95244bc5c424d8669d5833cd9 in nifi's branch refs/heads/master from Endre Zoltan Kovacs
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3a63de2 ]
NIFI-6180: exposing firehose grace period to DruidTranquilityController
this allows for configuring `druidBeam.firehoseGracePeriod`
(https://github.com/druid-io/tranquility/blob/master/docs/configuration.md#properties)
NIFI-6180: Corrected typo in DruidTranquilityController
Signed-off-by: Matthew Burgess <ma...@apache.org>
This closes #3403
> DruidTranquilityController: expose `firehoseGracePeriod` for higher index task throughput
> -----------------------------------------------------------------------------------------
>
> Key: NIFI-6180
> URL: https://issues.apache.org/jira/browse/NIFI-6180
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Endre Kovacs
> Assignee: Endre Kovacs
> Priority: Minor
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> During integration testing NiFi with Druid, I noticed a constant, overhead in the realtime index tasks: eg.: even if i set the
> {code:java}
> druid-cs-window-period=PT1M{code}
> the task duration was still at 426726ms (~7 minutes)
> 5 of that 7 minutes were due to firehose's grace period.
> My suggestion is to expose `druidBeam.firehoseGracePeriod` ([https://github.com/druid-io/tranquility/blob/master/docs/configuration.md#properties]) to the controller service: DruidTranquilityController, and build it into
> [https://github.com/apache/nifi/blob/3696b5bfcf0bd9e12ee4e9472f3413a93c9c0fcd/nifi-nar-bundles/nifi-druid-bundle/nifi-druid-controller-service/src/main/java/org/apache/nifi/controller/druid/DruidTranquilityController.java#L452-L457] the beam config.
> Currently this constant takes the default value of 5 minutes, causing a constant overhead on each of the indexing tasks initiated by NiFi, no matter how few/small the workload window.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)