[jira] [Created] (ATLAS-3305) Unable to scale atlas kafka consumers

["Adam Rempter (JIRA)" <ji...@apache.org>]

Adam Rempter created ATLAS-3305:
-----------------------------------

             Summary: Unable to scale atlas kafka consumers
                 Key: ATLAS-3305
                 URL: https://issues.apache.org/jira/browse/ATLAS-3305
             Project: Atlas
          Issue Type: Bug
          Components:  atlas-core, atlas-intg
    Affects Versions: 2.0.0, 1.1.0
            Reporter: Adam Rempter


We wanted to scale kafka consumers for atlas, as we are getting many lineage messages and processing them just with one consumer is not enough. 

 

There is parameter atlas.notification.hook.numthreads to scale consumers in  

NotificationHookConsumer.

But the method:

 

notificationInterface.createConsumers(NotificationType.HOOK, numThreads)

 

is always returning one element list, which effectively always starts one consumer

List<NotificationConsumer<T>> consumers = Collections.singletonList(kafkaConsumer);

 

Log incorrectly says that nuber of consumers has been created:

LOG.info("<== KafkaNotification.createConsumers(notificationType={}, numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers, autoCommitEnabled)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)