Review Request 71631: ATLAS-3427 - Atlas hooks enhancements for better fault-tolerance i.e. Kafka unavailability

[Nixon Rodrigues <ni...@freestoneinfotech.com>]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71631/
-----------------------------------------------------------

Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.


Bugs: ATLAS-3427
    https://issues.apache.org/jira/browse/ATLAS-3427


Repository: atlas


Description
-------

This patch includes changes to add file spooling mechanism to make atlas Hooks fault tolerant when Kafka destination is down.


Configuration required.

atlas.hook.spool.filecache.is.enabled = true [By Default false]

atlas.hook.spool.filecache.filespool.dir =  "/tmp"  [Directory where spool file  are stored, require proper write permissions for hook application]
atlas.hook.spool.filecache.filespool.archive.dir =  "/tmp/archive" [Directory where spool file  are archieved after syncing]
atlas.hook.spool.filecache.filespool.index.filename =  "indexFile" [Name of index file which has info of spool file and its status]
atlas.hook.spool.filecache.filespool.file.prefix = "atlas-hook"  [Prefix of index file]
atlas.hook.spool.filecache.filespool.file.rollover.sec  = 60 [ Roll over time for closing the spool file in sec]
atlas.hook.spool.filecache.filespool.destination.retry.ms [Retry time for Syncing the message from spool to kafka]  (Default 30 seconds)
atlas.hook.spool.filecache.filespool.archive.max.files = [Max files stored in archive directory]
atlas.hook.spool.filecache.filespool.app.type [application type such as hive, hbase etc]


Diffs
-----

  notification/src/main/java/org/apache/atlas/hook/AtlasHook.java cc6546be8 
  notification/src/main/java/org/apache/atlas/hook/FailedMessagesLogger.java b319e81b8 
  notification/src/main/java/org/apache/atlas/notification/NotificationException.java 2dd9c9fa0 
  notification/src/main/java/org/apache/atlas/spool/AtlasFileCacheProvider.java PRE-CREATION 
  notification/src/main/java/org/apache/atlas/spool/AtlasFileSpool.java PRE-CREATION 
  notification/src/main/java/org/apache/atlas/spool/AtlasKafkaMessageHandler.java PRE-CREATION 
  notification/src/main/java/org/apache/atlas/spool/MessageHandler.java PRE-CREATION 
  notification/src/main/java/org/apache/atlas/spool/MessageIndexRecord.java PRE-CREATION 
  notification/src/main/java/org/apache/atlas/spool/SpoolFileStatus.java PRE-CREATION 
  notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 1ae7c278c 


Diff: https://reviews.apache.org/r/71631/diff/1/


Testing
-------

Tested spooling on hive with atlas hive hook enabled in simple and kerberos env.

Existing testcase working fine.


Thanks,

Nixon Rodrigues