You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@eagle.apache.org by Edward Zhang <yo...@apache.org> on 2015/12/07 06:12:14 UTC
[Discuss] support complex policy template gracefully
I want to start some discussion on how to support complex policy template
gracefully.
Today if we want to support a policy like "alert when a user deletes some
sensitivity file", then user has to compose very complex policy because in
Hdfs file deletion will spawn multiple granular hdfs audit events. It is
hard for user to define such a simple policy in a straightforward way.
I want to propose to solve the problem with the following approach
EAGLE-68 <https://issues.apache.org/jira/browse/EAGLE-68>, EAGLE-14
<https://issues.apache.org/jira/browse/EAGLE-14>
First in stream processing phase, Eagle will reassemble user level command
from granular audit event which is defined by EAGLE-14
<https://issues.apache.org/jira/browse/EAGLE-14>
Second, in UI we provide a general feature for user to import a predefined
policy template and those policy templates can be hosted in eagle source
code externalPolices for example. this is defined in EAGLE-68
<https://issues.apache.org/jira/browse/EAGLE-68>
With this approach, we don't need customize HDFS policy UI and I hope we
can always avoid customizing a UI for a specified data source.
Please suggest.
Thanks
Edward Zhang
Re: [Discuss] support complex policy template gracefully
Posted by "Chen, Hao" <Ha...@ebay.com>.
We could refer to logstash extensible patterns: https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
I think logstash is well designed as a general-purpose pipeline for stream processing though single process only now :-)
Thanks,
Hao
On 12/7/15, 2:29 PM, "Zhang, Edward (GDI Hadoop)" <yo...@ebay.com> wrote:
>I have not figured out what the policy template looks like, but like you
>said, that should include variable. and this template should be populated
>into UI.
>
>Eagle-68 was previously proposed by Hemanth by customizing HDFS policy UI
>to simplify complex policy onboard, but I think we can do better.
>
>Edward
>
>On 12/6/15, 22:15, "Liangfei.Su" <su...@gmail.com> wrote:
>
>>I would second this template way to keep the user from the error-prone
>>command assembling define.
>>What kind of json schema as you mentioned in EAGLE-68? Is the simple
>>policy
>>DSL definition enough here (with template variable)?
>>
>>Thanks,
>>Ralph
>>
>>On Mon, Dec 7, 2015 at 1:12 PM, Edward Zhang <yo...@apache.org>
>>wrote:
>>
>>> I want to start some discussion on how to support complex policy
>>>template
>>> gracefully.
>>>
>>> Today if we want to support a policy like "alert when a user deletes
>>>some
>>> sensitivity file", then user has to compose very complex policy because
>>>in
>>> Hdfs file deletion will spawn multiple granular hdfs audit events. It is
>>> hard for user to define such a simple policy in a straightforward way.
>>>
>>> I want to propose to solve the problem with the following approach
>>> EAGLE-68 <https://issues.apache.org/jira/browse/EAGLE-68>, EAGLE-14
>>> <https://issues.apache.org/jira/browse/EAGLE-14>
>>>
>>> First in stream processing phase, Eagle will reassemble user level
>>>command
>>> from granular audit event which is defined by EAGLE-14
>>> <https://issues.apache.org/jira/browse/EAGLE-14>
>>> Second, in UI we provide a general feature for user to import a
>>>predefined
>>> policy template and those policy templates can be hosted in eagle source
>>> code externalPolices for example. this is defined in EAGLE-68
>>> <https://issues.apache.org/jira/browse/EAGLE-68>
>>>
>>> With this approach, we don't need customize HDFS policy UI and I hope we
>>> can always avoid customizing a UI for a specified data source.
>>>
>>> Please suggest.
>>>
>>> Thanks
>>> Edward Zhang
>>>
>
Re: [Discuss] support complex policy template gracefully
Posted by "Chen, Hao" <Ha...@ebay.com>.
We could refer to logstash extensible patterns: https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
I think logstash is well designed as a general-purpose pipeline for stream processing though single process only now :-)
Thanks,
Hao
On 12/7/15, 2:29 PM, "Zhang, Edward (GDI Hadoop)" <yo...@ebay.com> wrote:
>I have not figured out what the policy template looks like, but like you
>said, that should include variable. and this template should be populated
>into UI.
>
>Eagle-68 was previously proposed by Hemanth by customizing HDFS policy UI
>to simplify complex policy onboard, but I think we can do better.
>
>Edward
>
>On 12/6/15, 22:15, "Liangfei.Su" <su...@gmail.com> wrote:
>
>>I would second this template way to keep the user from the error-prone
>>command assembling define.
>>What kind of json schema as you mentioned in EAGLE-68? Is the simple
>>policy
>>DSL definition enough here (with template variable)?
>>
>>Thanks,
>>Ralph
>>
>>On Mon, Dec 7, 2015 at 1:12 PM, Edward Zhang <yo...@apache.org>
>>wrote:
>>
>>> I want to start some discussion on how to support complex policy
>>>template
>>> gracefully.
>>>
>>> Today if we want to support a policy like "alert when a user deletes
>>>some
>>> sensitivity file", then user has to compose very complex policy because
>>>in
>>> Hdfs file deletion will spawn multiple granular hdfs audit events. It is
>>> hard for user to define such a simple policy in a straightforward way.
>>>
>>> I want to propose to solve the problem with the following approach
>>> EAGLE-68 <https://issues.apache.org/jira/browse/EAGLE-68>, EAGLE-14
>>> <https://issues.apache.org/jira/browse/EAGLE-14>
>>>
>>> First in stream processing phase, Eagle will reassemble user level
>>>command
>>> from granular audit event which is defined by EAGLE-14
>>> <https://issues.apache.org/jira/browse/EAGLE-14>
>>> Second, in UI we provide a general feature for user to import a
>>>predefined
>>> policy template and those policy templates can be hosted in eagle source
>>> code externalPolices for example. this is defined in EAGLE-68
>>> <https://issues.apache.org/jira/browse/EAGLE-68>
>>>
>>> With this approach, we don't need customize HDFS policy UI and I hope we
>>> can always avoid customizing a UI for a specified data source.
>>>
>>> Please suggest.
>>>
>>> Thanks
>>> Edward Zhang
>>>
>
Re: [Discuss] support complex policy template gracefully
Posted by "Zhang, Edward (GDI Hadoop)" <yo...@ebay.com>.
I have not figured out what the policy template looks like, but like you
said, that should include variable. and this template should be populated
into UI.
Eagle-68 was previously proposed by Hemanth by customizing HDFS policy UI
to simplify complex policy onboard, but I think we can do better.
Edward
On 12/6/15, 22:15, "Liangfei.Su" <su...@gmail.com> wrote:
>I would second this template way to keep the user from the error-prone
>command assembling define.
>What kind of json schema as you mentioned in EAGLE-68? Is the simple
>policy
>DSL definition enough here (with template variable)?
>
>Thanks,
>Ralph
>
>On Mon, Dec 7, 2015 at 1:12 PM, Edward Zhang <yo...@apache.org>
>wrote:
>
>> I want to start some discussion on how to support complex policy
>>template
>> gracefully.
>>
>> Today if we want to support a policy like "alert when a user deletes
>>some
>> sensitivity file", then user has to compose very complex policy because
>>in
>> Hdfs file deletion will spawn multiple granular hdfs audit events. It is
>> hard for user to define such a simple policy in a straightforward way.
>>
>> I want to propose to solve the problem with the following approach
>> EAGLE-68 <https://issues.apache.org/jira/browse/EAGLE-68>, EAGLE-14
>> <https://issues.apache.org/jira/browse/EAGLE-14>
>>
>> First in stream processing phase, Eagle will reassemble user level
>>command
>> from granular audit event which is defined by EAGLE-14
>> <https://issues.apache.org/jira/browse/EAGLE-14>
>> Second, in UI we provide a general feature for user to import a
>>predefined
>> policy template and those policy templates can be hosted in eagle source
>> code externalPolices for example. this is defined in EAGLE-68
>> <https://issues.apache.org/jira/browse/EAGLE-68>
>>
>> With this approach, we don't need customize HDFS policy UI and I hope we
>> can always avoid customizing a UI for a specified data source.
>>
>> Please suggest.
>>
>> Thanks
>> Edward Zhang
>>
Re: [Discuss] support complex policy template gracefully
Posted by "Zhang, Edward (GDI Hadoop)" <yo...@ebay.com>.
I have not figured out what the policy template looks like, but like you
said, that should include variable. and this template should be populated
into UI.
Eagle-68 was previously proposed by Hemanth by customizing HDFS policy UI
to simplify complex policy onboard, but I think we can do better.
Edward
On 12/6/15, 22:15, "Liangfei.Su" <su...@gmail.com> wrote:
>I would second this template way to keep the user from the error-prone
>command assembling define.
>What kind of json schema as you mentioned in EAGLE-68? Is the simple
>policy
>DSL definition enough here (with template variable)?
>
>Thanks,
>Ralph
>
>On Mon, Dec 7, 2015 at 1:12 PM, Edward Zhang <yo...@apache.org>
>wrote:
>
>> I want to start some discussion on how to support complex policy
>>template
>> gracefully.
>>
>> Today if we want to support a policy like "alert when a user deletes
>>some
>> sensitivity file", then user has to compose very complex policy because
>>in
>> Hdfs file deletion will spawn multiple granular hdfs audit events. It is
>> hard for user to define such a simple policy in a straightforward way.
>>
>> I want to propose to solve the problem with the following approach
>> EAGLE-68 <https://issues.apache.org/jira/browse/EAGLE-68>, EAGLE-14
>> <https://issues.apache.org/jira/browse/EAGLE-14>
>>
>> First in stream processing phase, Eagle will reassemble user level
>>command
>> from granular audit event which is defined by EAGLE-14
>> <https://issues.apache.org/jira/browse/EAGLE-14>
>> Second, in UI we provide a general feature for user to import a
>>predefined
>> policy template and those policy templates can be hosted in eagle source
>> code externalPolices for example. this is defined in EAGLE-68
>> <https://issues.apache.org/jira/browse/EAGLE-68>
>>
>> With this approach, we don't need customize HDFS policy UI and I hope we
>> can always avoid customizing a UI for a specified data source.
>>
>> Please suggest.
>>
>> Thanks
>> Edward Zhang
>>
Re: [Discuss] support complex policy template gracefully
Posted by "Liangfei.Su" <su...@gmail.com>.
I would second this template way to keep the user from the error-prone
command assembling define.
What kind of json schema as you mentioned in EAGLE-68? Is the simple policy
DSL definition enough here (with template variable)?
Thanks,
Ralph
On Mon, Dec 7, 2015 at 1:12 PM, Edward Zhang <yo...@apache.org>
wrote:
> I want to start some discussion on how to support complex policy template
> gracefully.
>
> Today if we want to support a policy like "alert when a user deletes some
> sensitivity file", then user has to compose very complex policy because in
> Hdfs file deletion will spawn multiple granular hdfs audit events. It is
> hard for user to define such a simple policy in a straightforward way.
>
> I want to propose to solve the problem with the following approach
> EAGLE-68 <https://issues.apache.org/jira/browse/EAGLE-68>, EAGLE-14
> <https://issues.apache.org/jira/browse/EAGLE-14>
>
> First in stream processing phase, Eagle will reassemble user level command
> from granular audit event which is defined by EAGLE-14
> <https://issues.apache.org/jira/browse/EAGLE-14>
> Second, in UI we provide a general feature for user to import a predefined
> policy template and those policy templates can be hosted in eagle source
> code externalPolices for example. this is defined in EAGLE-68
> <https://issues.apache.org/jira/browse/EAGLE-68>
>
> With this approach, we don't need customize HDFS policy UI and I hope we
> can always avoid customizing a UI for a specified data source.
>
> Please suggest.
>
> Thanks
> Edward Zhang
>