You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "Otto Fowler (JIRA)" <ji...@apache.org> on 2018/01/29 13:25:00 UTC
[jira] [Updated] (METRON-534) Metron Should have service to monitor
HDFS for changes to files
[ https://issues.apache.org/jira/browse/METRON-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Otto Fowler updated METRON-534:
-------------------------------
Description:
With the INotify library it is possible to receive notifications from HDFS on file changes.
A service that utilized this and was configured with "rules" to process notifications on files of interest ( configurations, rules etc ) in the system would be able to send notifications to Zookeeper
This may allow for things that are better stored in HDFS to remain there, and not force the system to put things in Zookeeper just for change notifications.
Issues:
* Inotify requires hdfs super user, what are the implications on a secure cluster etc?
* performance / scalability and load
* Can it be deployed with yarn?
* Is Zookeeper actually the right place for all the configurations
---------------------------------------------------------------------
!inotify-ascii.png!
was:
With the INotify library it is possible to receive notifications from HDFS on file changes.
A service that utilized this and was configured with "rules" to process notifications on files of interest ( configurations, rules etc ) in the system would be able to send notifications to Zookeeper
This may allow for things that are better stored in HDFS to remain there, and not force the system to put things in Zookeeper just for change notifications.
Issues:
* Inotify requires hdfs super user, what are the implications on a secure cluster etc?
* performance / scalability and load
* Can it be deployed with yarn?
* Is Zookeeper actually the right place for all the configurations
> Metron Should have service to monitor HDFS for changes to files
> ---------------------------------------------------------------
>
> Key: METRON-534
> URL: https://issues.apache.org/jira/browse/METRON-534
> Project: Metron
> Issue Type: New Feature
> Reporter: Otto Fowler
> Priority: Minor
> Attachments: inotify-ascii.png
>
>
> With the INotify library it is possible to receive notifications from HDFS on file changes.
> A service that utilized this and was configured with "rules" to process notifications on files of interest ( configurations, rules etc ) in the system would be able to send notifications to Zookeeper
> This may allow for things that are better stored in HDFS to remain there, and not force the system to put things in Zookeeper just for change notifications.
> Issues:
> * Inotify requires hdfs super user, what are the implications on a secure cluster etc?
> * performance / scalability and load
> * Can it be deployed with yarn?
> * Is Zookeeper actually the right place for all the configurations
> ---------------------------------------------------------------------
>
> !inotify-ascii.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)