[jira] [Commented] (ATLAS-3305) Unable to scale atlas kafka consumers

["Bolke de Bruin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ATLAS-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903328#comment-16903328 ] 

Bolke de Bruin commented on ATLAS-3305:
---------------------------------------

Why not use partitions with hashed ordering by qualified name? Or is this what you meant [~arempter]?

Can we have consensus here [~sarath.kum4r@gmail.com] cause this scalability issue is stopping deployment at the moment for us

> Unable to scale atlas kafka consumers
> -------------------------------------
>
>                 Key: ATLAS-3305
>                 URL: https://issues.apache.org/jira/browse/ATLAS-3305
>             Project: Atlas
>          Issue Type: Bug
>          Components:  atlas-core, atlas-intg
>    Affects Versions: 1.1.0, 2.0.0
>            Reporter: Adam Rempter
>            Priority: Major
>              Labels: performance
>
> We wanted to scale kafka consumers for atlas, as we are getting many lineage messages and processing them just with one consumer is not enough. 
>  
> There is parameter atlas.notification.hook.numthreads to scale consumers in  
> NotificationHookConsumer.
> But the method:
>  
> notificationInterface.createConsumers(NotificationType.HOOK, numThreads)
>  
> is always returning one element list, which effectively always starts one consumer
> List<NotificationConsumer<T>> consumers = Collections.singletonList(kafkaConsumer);
>  
> Log incorrectly says that nuber of consumers has been created:
> LOG.info("<== KafkaNotification.createConsumers(notificationType={}, numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers, autoCommitEnabled)



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)