You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Ernani Pereira de Mattos Junior (JIRA)" <ji...@apache.org> on 2019/04/24 21:01:00 UTC

[jira] [Updated] (ATLAS-3169) Create priority Queue for Atlas messages coming from 2 different sources;

     [ https://issues.apache.org/jira/browse/ATLAS-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ernani Pereira de Mattos Junior updated ATLAS-3169:
---------------------------------------------------
    Description: 
One of the sources is creating a long lag, and hence the data from other sources are not getting processed until the lag is cleared. Is there a way to make a queue take priority?
 ==

The customer would like to have Kafka use some type of priority Queue so it would avoid the FIFO nature of Kafka's topics. This is regarding the ATLAS_HOOK in which the customer want other resources besides Hive h
 ook to have a faster processing.

Basically the idea is to have some resource to publish messages in different levels of priority.

1) high_priority_queue
 2) low_priority_queue

Atlas would have to consumer these messages accordingly to the Queue priority.

Business Case:

Topics or prioritizing messages for the ATLAS_HOOK topic.

HIVECLUSTER_ATLAS – publish --> KAFKA-ATLAS_HOOK – subscribed --> ATLAS-1.1
 OTHER_Services– publish --> KAFKA-ATLAS_HOOK – subscribed --> ATLAS-1.1

note:  Hive is the bigger publisher, They have a LAG of about a week on the amount of messages produced by Hive. Currently the LAG produce by Hive impacts the other services publishers;

Some References:
 [https://stackoverflow.com/a/30686523]
 [https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics] - KAFKA-6690
 "Single-Event Processing" - book:Kafka: The Definitive Guide: Real-Time Data And Stream Processing At Scale

  was:
One of the sources is creating a long lag, and hence the data from other sources are not getting processed until the lag is cleared. Is there a way to make a queue take priority?
==

The customer would like to have Kafka use some type of priority Queue so it would avoid the FIFO nature of Kafka's topics. This is regarding the ATLAS_HOOK in which the customer want other resources besides Hive h
ook to have a faster processing.

Basically the idea is to have some resource to publish messages in different levels of priority.

1) high_priority_queue
2) low_priority_queue

Atlas would have to consumer these messages accordingly to the Queue priority.

Business Case:

Topics or prioritizing messages for the ATLAS_HOOK topic.

<MELD>_HIVECLUSTER_ATLAS – publish --> KAFKA-ATLAS_HOOK – subscribed --> ATLAS-1.1
OTHERCLUSTER_ATLAS-1.1 – publish --> KAFKA-ATLAS_HOOK – subscribed --> ATLAS-1.1

note: MELD Hive is the bigger publisher, They have a LAG of about a week on the amount of messages produced by Hive. Currently the LAG produce by Hive impacts the other services publishers;

Some References:
[https://stackoverflow.com/a/30686523]
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics] - KAFKA-6690
"Single-Event Processing" - book:Kafka: The Definitive Guide: Real-Time Data And Stream Processing At Scale


> Create priority Queue for Atlas messages coming from 2 different sources; 
> --------------------------------------------------------------------------
>
>                 Key: ATLAS-3169
>                 URL: https://issues.apache.org/jira/browse/ATLAS-3169
>             Project: Atlas
>          Issue Type: Improvement
>          Components: atlas-intg
>    Affects Versions: 1.1.0
>            Reporter: Ernani Pereira de Mattos Junior
>            Priority: Critical
>
> One of the sources is creating a long lag, and hence the data from other sources are not getting processed until the lag is cleared. Is there a way to make a queue take priority?
>  ==
> The customer would like to have Kafka use some type of priority Queue so it would avoid the FIFO nature of Kafka's topics. This is regarding the ATLAS_HOOK in which the customer want other resources besides Hive h
>  ook to have a faster processing.
> Basically the idea is to have some resource to publish messages in different levels of priority.
> 1) high_priority_queue
>  2) low_priority_queue
> Atlas would have to consumer these messages accordingly to the Queue priority.
> Business Case:
> Topics or prioritizing messages for the ATLAS_HOOK topic.
> HIVECLUSTER_ATLAS – publish --> KAFKA-ATLAS_HOOK – subscribed --> ATLAS-1.1
>  OTHER_Services– publish --> KAFKA-ATLAS_HOOK – subscribed --> ATLAS-1.1
> note:  Hive is the bigger publisher, They have a LAG of about a week on the amount of messages produced by Hive. Currently the LAG produce by Hive impacts the other services publishers;
> Some References:
>  [https://stackoverflow.com/a/30686523]
>  [https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics] - KAFKA-6690
>  "Single-Event Processing" - book:Kafka: The Definitive Guide: Real-Time Data And Stream Processing At Scale



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)