You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/10/22 21:09:00 UTC
[jira] [Commented] (ATLAS-3427) Atlas hooks enhancements for better
fault-tolerance i.e. Kafka unavailability
[ https://issues.apache.org/jira/browse/ATLAS-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17219347#comment-17219347 ]
ASF subversion and git services commented on ATLAS-3427:
--------------------------------------------------------
Commit de87bc5022627d82cb8e6048b6728e7028a4af25 in atlas's branch refs/heads/master from Ashutosh Mestry
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=de87bc5 ]
ATLAS-3427: Atlas Hook Enhancements for improved resiliancy.
> Atlas hooks enhancements for better fault-tolerance i.e. Kafka unavailability
> -----------------------------------------------------------------------------
>
> Key: ATLAS-3427
> URL: https://issues.apache.org/jira/browse/ATLAS-3427
> Project: Atlas
> Issue Type: Improvement
> Reporter: Nixon Rodrigues
> Assignee: Ashutosh Mestry
> Priority: Major
>
> *Background*
> Existing implementation for sending messages from the hooks relies on Kafka. Which means that if Kafka is not available for any reasons, the messages do not make it to Atlas. They are simply added to the _failed_._log_.
> *Solution*
> Introduce a mechanism which preserves messages if destination for the messages is not reachable. This new mechanism will also have logic to retry these messages.
> *Requirements*
> The mechanism should be:
> * Transparent: Existing hooks should not have to rework their implementation.
> * Configurable: This should be introduced optionally. If hook chooses not to use this, it should fallback to existing implementation. The behavior should be configurable via properties.
> * High-performance: Using this mechanism should not introduce additional overhead. Ideally, there should not be additional serialization introduced.
> * Reliable: The new mechanism should relay these messages reliably. Existing message formats should be supported viz. plain, compressed, compress & multi-part.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)