[jira] [Commented] (ATLAS-3305) Unable to scale atlas kafka consumers

["Adam Rempter (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ATLAS-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985964#comment-16985964 ] 

Adam Rempter commented on ATLAS-3305:
-------------------------------------

Updated patch with latest master.  In our case such change really allowed atlas to swallow growing changes delta. 

Please see grafana screenshot for details. !multiple_consumers_perf.png!

> Unable to scale atlas kafka consumers
> -------------------------------------
>
>                 Key: ATLAS-3305
>                 URL: https://issues.apache.org/jira/browse/ATLAS-3305
>             Project: Atlas
>          Issue Type: Bug
>          Components:  atlas-core, atlas-intg
>    Affects Versions: 1.1.0, 2.0.0
>            Reporter: Adam Rempter
>            Priority: Major
>              Labels: performance
>         Attachments: ATLAS-3305_multiple_kafka_consumers.patch, multiple_consumers_perf.png
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> We wanted to scale kafka consumers for atlas, as we are getting many lineage messages and processing them just with one consumer is not enough. 
>  
> There is parameter atlas.notification.hook.numthreads to scale consumers in  
> NotificationHookConsumer.
> But the method:
>  
> notificationInterface.createConsumers(NotificationType.HOOK, numThreads)
>  
> is always returning one element list, which effectively always starts one consumer
> List<NotificationConsumer<T>> consumers = Collections.singletonList(kafkaConsumer);
>  
> Log incorrectly says that nuber of consumers has been created:
> LOG.info("<== KafkaNotification.createConsumers(notificationType={}, numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers, autoCommitEnabled)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)