You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Viktor Somogyi-Vass <vi...@gmail.com> on 2020/08/28 09:35:29 UTC
Re: [DISCUSS] KIP-567: Kafka Cluster Audit
Hi folks,
I have a use-case and a non-trivial implementation with Apache Atlas for
this KIP and since this kip seems to be dormant for a while now, I'd take
it over and drive it to completion if you don't mind.
The current state of the PoC can be found on my fork at
https://github.com/viktorsomogyi/kafka/tree/atlas-audit-impl.
Viktor
On Wed, Jan 29, 2020 at 9:32 AM Игорь Мартемьянов <le...@gmail.com>
wrote:
> Hello, Nikolai.
>
>
>
> > Can you, please, make it more specific?
>
> > Why does a business want to have this information?
>
> It is very demanded for security department to know who/when/where create
> or edit ACL settings. The same situation about topics.
>
>
>
> > What are the use-cases for it?
>
> This KIP are able to help catching some unauthorized changes of cluster
> resources or save history these changes.
>
>
>
> > Who will be analyzing these events and how?
>
> Events could be analyzed by some security department or developers who
> builds applications that can change cluster resources.
>
>
>
> > Why it’s not convenient to implement it with third-party tools?
>
> It is imposible to my mind getting detailed information about these events
> through third-party tools, because we don`t have the ability to catch these
> events outside Kafka core.
>
>
>
> > 1. `audit` name sounds too general for me. How about `onEvent`?
>
> Thanks, good point. I`ll change the method name.
>
>
>
> > 2. Should we introduce a special marker interface for audit events?
>
> > `AuditableRequest`, for example?
>
> No, we shouldn`t. We are going to create these events if we process special
> request in KafkaApis class (ApiKeys.CREATE_TOPICS, ApiKeys.DELETE_TOPICS,
> ApiKeys.CREATE_ACLS, ApiKeys.DELETE_ACLS, ApiKeys.DESCRIBE_ACLS,
> ApiKeys.ALTER_CONFIGS)
>
>
>
> >> public interface AuditEvent<T extends AbstractRequest> {
>
> >> String guid();
>
>
>
> > Where this `guid` comes from?
>
> Guid generated automatically when the audit event is created.
>
>
>
> > Will it be the same on each node that receives an auditable event?
>
> Event is creating while request that changes cluster resource is processed
> and will destroyed after the prosessing will finished on each node.
>
>
>
> > Do we have `guid` for any extensions of `AbstractRequest`?
>
> No, we don`t.
>
>
>
> > If this field is `guid` why do we format this as a String on the API
> level?
>
> Thanks for the point.
>
> I have changed event interface recently.
>
>
>
> > Can you, please, add a full list of proposed new configuration properties
> and
>
> > examples for each to clarify your intentions?
>
> Yes, I have added some examples of new configuration properties.
>
> ср, 29 янв. 2020 г., 10:23 Владимир Беруненко <vv...@gmail.com>:
>
> > Hi Nikolai!
> >
> > >Can you, please, make it more specific?
> > Why does a business want to have this information?
> > >What are the use-cases for it?
> > >Who will be analyzing these events and how?
> > >Why it’s not convenient to implement it with third-party tools?
> >
> > This is required by the guys from information security to detect
> potential
> > threats of violation the rules for providing access to the data layer.
> > Analysis in the audit system is usually based on identifying
> > uncharacteristic integrations. The key feature of the audit, is that
> > cluster administrators do not have access to modification of audit
> events.
> > Third-party tools are great for data analysis, but its not good idea to
> use
> > it for audit events collection.
> >
> > >It’s not clear for me where and when AuditEvents will be sent?
> > Who will be the receiver of events?
> >
> > In my opinion, sending an audit event should be initiated when the broker
> > receives a request that matches the audit parameters. Each organization
> has
> > its own receiver system, so a common interface is required that the
> > organization’s development team can implement to integrate with their
> audit
> > system.
> >
> > Best wishes, Vladimir
> >
> > Hello, Igor.
> > >
> > > Thanks for the KIP.
> > > I have a couple of comments for it:
> > >
> > > > Motivation
> > > > It is highly demanded in most businesses to have the ability of
> > > obtaining audit information in case someone changes cluster
> configuration
> > > (like creation/deletion/modify/description of any topic or ACLs).
> > >
> > > Can you, please, make it more specific?
> > > Why does a business want to have this information?
> > > What are the use-cases for it?
> > > Who will be analyzing these events and how?
> > > Why it’s not convenient to implement it with third-party tools?
> > >
> > > It’s not clear for me where and when AuditEvents will be sent?
> > > Who will be the receiver of events?
> > >
> > > > void audit<T extends AbstractRequest>(AuditEvent<T> event);
> > >
> > > 1. `audit` name sounds too general for me. How about `onEvent`?
> > > 2. Should we introduce a special marker interface for audit events?
> > > `AuditableRequest`, for example?
> > >
> > > > public interface AuditEvent<T extends AbstractRequest> {
> > > > String guid();
> > >
> > > Where this `guid` comes from?
> > > Will it be the same on each node that receives an auditable event?
> > > Do we have `guid` for any extensions of `AbstractRequest`?
> > > If this field is `guid` why do we format this as a String on the API
> > level?
> > >
> > > > One or more of AuditExtension's implementations can be configured via
> > > the configuration audit.extension.classes
> > >
> > > Can you, please, add a full list of proposed new configuration
> properties
> > > and examples for each to clarify your intentions?
> > >
> > >
> > > > 24 янв. 2020 г., в 18:12, Alexander Dunayevsky <
> > a.a.dunayevsky@gmail.com>
> > > написал(а):
> > > >
> > > > Hello Igor,
> > > >
> > > > Thanks for your KIP 🙌🏽
> > > > It would be great to adopt this functionality and getting the best of
> > > > tracking cluster activity.
> > > >
> > > > +1 vote from me
> > > >
> > > > Cheers,
> > > > Alex Dunayevsky
> > > >
> > > >
> > > > On Fri, 24 Jan 2020, 15:35 Игорь Мартемьянов, <le...@gmail.com>
> > > wrote:
> > > >
> > > >> Motivation:
> > > >>
> > > >>
> > > >> *It is highly demanded in most businesses to have the ability of
> > > obtaining
> > > >> audit information in case someone changes cluster configuration
> (like
> > > >> creation/deletion/modify/description of any topic or ACLs).We may
> add
> > > this
> > > >> ability. Since audit requirements are so broad, it's impractical to
> > > support
> > > >> all of them.Hence we have to provide ability for users to plug
> > resources
> > > >> helping to achieve required capabilities.*
> > > >>
> > > >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-567%3A+Kafka+Cluster+Audit
> > > >>
> > > >>
> > > >> пт, 24 янв. 2020 г., 17:29 Игорь Мартемьянов <le...@gmail.com>:
> > > >>
> > > >>> Hello there.
> > > >>> Please review this KIP.
> > > >>> Thanks.
> > > >>>
> > > >>
> > >
> > >
> >
>
Re: [DISCUSS] KIP-567: Kafka Cluster Audit
Posted by Viktor Somogyi-Vass <vi...@gmail.com>.
Hi Nikolay,
I actually have a somewhat different approach that is somewhat similar to
the Authorizer interface.
I have updated the KIP to reflect that.
I'm happy to collaborate during the implementation. I have a code change
not yet published but I can publish it and we can see what's the best way
to collaborate :).
Viktor
On Mon, Sep 7, 2020 at 12:20 PM Nikolay Izhikov <ni...@apache.org> wrote:
> Hello, Viktor.
>
> Do you want to implement the exact approach as it described in the current
> KIP?
> Or you have another proposal on how it has to be implemented?
>
> I abandoned this KIP due to lack of interest from community.
> Guess we can collaborate during implementation.
>
> > 7 сент. 2020 г., в 13:13, Viktor Somogyi-Vass <vi...@gmail.com>
> написал(а):
> >
> > Hi folks,
> >
> > It's been a few days since I last pinged and nobody replied so I assume
> > that this KIP is abandoned and I can take this over (but please let me
> know
> > if it's not). I will keep the current version of the KIP and move it to a
> > sub-page if it's ever needed.
> >
> > Thanks,
> > Viktor
> >
> > On Fri, Aug 28, 2020 at 11:35 AM Viktor Somogyi-Vass <
> > viktorsomogyi@gmail.com> wrote:
> >
> >> Hi folks,
> >>
> >> I have a use-case and a non-trivial implementation with Apache Atlas for
> >> this KIP and since this kip seems to be dormant for a while now, I'd
> take
> >> it over and drive it to completion if you don't mind.
> >> The current state of the PoC can be found on my fork at
> >> https://github.com/viktorsomogyi/kafka/tree/atlas-audit-impl.
> >>
> >> Viktor
> >>
> >> On Wed, Jan 29, 2020 at 9:32 AM Игорь Мартемьянов <le...@gmail.com>
> >> wrote:
> >>
> >>> Hello, Nikolai.
> >>>
> >>>
> >>>
> >>>> Can you, please, make it more specific?
> >>>
> >>>> Why does a business want to have this information?
> >>>
> >>> It is very demanded for security department to know who/when/where
> create
> >>> or edit ACL settings. The same situation about topics.
> >>>
> >>>
> >>>
> >>>> What are the use-cases for it?
> >>>
> >>> This KIP are able to help catching some unauthorized changes of cluster
> >>> resources or save history these changes.
> >>>
> >>>
> >>>
> >>>> Who will be analyzing these events and how?
> >>>
> >>> Events could be analyzed by some security department or developers who
> >>> builds applications that can change cluster resources.
> >>>
> >>>
> >>>
> >>>> Why it’s not convenient to implement it with third-party tools?
> >>>
> >>> It is imposible to my mind getting detailed information about these
> events
> >>> through third-party tools, because we don`t have the ability to catch
> >>> these
> >>> events outside Kafka core.
> >>>
> >>>
> >>>
> >>>> 1. `audit` name sounds too general for me. How about `onEvent`?
> >>>
> >>> Thanks, good point. I`ll change the method name.
> >>>
> >>>
> >>>
> >>>> 2. Should we introduce a special marker interface for audit events?
> >>>
> >>>> `AuditableRequest`, for example?
> >>>
> >>> No, we shouldn`t. We are going to create these events if we process
> >>> special
> >>> request in KafkaApis class (ApiKeys.CREATE_TOPICS,
> ApiKeys.DELETE_TOPICS,
> >>> ApiKeys.CREATE_ACLS, ApiKeys.DELETE_ACLS, ApiKeys.DESCRIBE_ACLS,
> >>> ApiKeys.ALTER_CONFIGS)
> >>>
> >>>
> >>>
> >>>>> public interface AuditEvent<T extends AbstractRequest> {
> >>>
> >>>>> String guid();
> >>>
> >>>
> >>>
> >>>> Where this `guid` comes from?
> >>>
> >>> Guid generated automatically when the audit event is created.
> >>>
> >>>
> >>>
> >>>> Will it be the same on each node that receives an auditable event?
> >>>
> >>> Event is creating while request that changes cluster resource is
> processed
> >>> and will destroyed after the prosessing will finished on each node.
> >>>
> >>>
> >>>
> >>>> Do we have `guid` for any extensions of `AbstractRequest`?
> >>>
> >>> No, we don`t.
> >>>
> >>>
> >>>
> >>>> If this field is `guid` why do we format this as a String on the API
> >>> level?
> >>>
> >>> Thanks for the point.
> >>>
> >>> I have changed event interface recently.
> >>>
> >>>
> >>>
> >>>> Can you, please, add a full list of proposed new configuration
> >>> properties
> >>> and
> >>>
> >>>> examples for each to clarify your intentions?
> >>>
> >>> Yes, I have added some examples of new configuration properties.
> >>>
> >>> ср, 29 янв. 2020 г., 10:23 Владимир Беруненко <vv...@gmail.com>:
> >>>
> >>>> Hi Nikolai!
> >>>>
> >>>>> Can you, please, make it more specific?
> >>>> Why does a business want to have this information?
> >>>>> What are the use-cases for it?
> >>>>> Who will be analyzing these events and how?
> >>>>> Why it’s not convenient to implement it with third-party tools?
> >>>>
> >>>> This is required by the guys from information security to detect
> >>> potential
> >>>> threats of violation the rules for providing access to the data layer.
> >>>> Analysis in the audit system is usually based on identifying
> >>>> uncharacteristic integrations. The key feature of the audit, is that
> >>>> cluster administrators do not have access to modification of audit
> >>> events.
> >>>> Third-party tools are great for data analysis, but its not good idea
> to
> >>> use
> >>>> it for audit events collection.
> >>>>
> >>>>> It’s not clear for me where and when AuditEvents will be sent?
> >>>> Who will be the receiver of events?
> >>>>
> >>>> In my opinion, sending an audit event should be initiated when the
> >>> broker
> >>>> receives a request that matches the audit parameters. Each
> organization
> >>> has
> >>>> its own receiver system, so a common interface is required that the
> >>>> organization’s development team can implement to integrate with their
> >>> audit
> >>>> system.
> >>>>
> >>>> Best wishes, Vladimir
> >>>>
> >>>> Hello, Igor.
> >>>>>
> >>>>> Thanks for the KIP.
> >>>>> I have a couple of comments for it:
> >>>>>
> >>>>>> Motivation
> >>>>>> It is highly demanded in most businesses to have the ability of
> >>>>> obtaining audit information in case someone changes cluster
> >>> configuration
> >>>>> (like creation/deletion/modify/description of any topic or ACLs).
> >>>>>
> >>>>> Can you, please, make it more specific?
> >>>>> Why does a business want to have this information?
> >>>>> What are the use-cases for it?
> >>>>> Who will be analyzing these events and how?
> >>>>> Why it’s not convenient to implement it with third-party tools?
> >>>>>
> >>>>> It’s not clear for me where and when AuditEvents will be sent?
> >>>>> Who will be the receiver of events?
> >>>>>
> >>>>>> void audit<T extends AbstractRequest>(AuditEvent<T> event);
> >>>>>
> >>>>> 1. `audit` name sounds too general for me. How about `onEvent`?
> >>>>> 2. Should we introduce a special marker interface for audit events?
> >>>>> `AuditableRequest`, for example?
> >>>>>
> >>>>>> public interface AuditEvent<T extends AbstractRequest> {
> >>>>>> String guid();
> >>>>>
> >>>>> Where this `guid` comes from?
> >>>>> Will it be the same on each node that receives an auditable event?
> >>>>> Do we have `guid` for any extensions of `AbstractRequest`?
> >>>>> If this field is `guid` why do we format this as a String on the API
> >>>> level?
> >>>>>
> >>>>>> One or more of AuditExtension's implementations can be configured
> >>> via
> >>>>> the configuration audit.extension.classes
> >>>>>
> >>>>> Can you, please, add a full list of proposed new configuration
> >>> properties
> >>>>> and examples for each to clarify your intentions?
> >>>>>
> >>>>>
> >>>>>> 24 янв. 2020 г., в 18:12, Alexander Dunayevsky <
> >>>> a.a.dunayevsky@gmail.com>
> >>>>> написал(а):
> >>>>>>
> >>>>>> Hello Igor,
> >>>>>>
> >>>>>> Thanks for your KIP 🙌🏽
> >>>>>> It would be great to adopt this functionality and getting the best
> >>> of
> >>>>>> tracking cluster activity.
> >>>>>>
> >>>>>> +1 vote from me
> >>>>>>
> >>>>>> Cheers,
> >>>>>> Alex Dunayevsky
> >>>>>>
> >>>>>>
> >>>>>> On Fri, 24 Jan 2020, 15:35 Игорь Мартемьянов, <le...@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>>> Motivation:
> >>>>>>>
> >>>>>>>
> >>>>>>> *It is highly demanded in most businesses to have the ability of
> >>>>> obtaining
> >>>>>>> audit information in case someone changes cluster configuration
> >>> (like
> >>>>>>> creation/deletion/modify/description of any topic or ACLs).We may
> >>> add
> >>>>> this
> >>>>>>> ability. Since audit requirements are so broad, it's impractical to
> >>>>> support
> >>>>>>> all of them.Hence we have to provide ability for users to plug
> >>>> resources
> >>>>>>> helping to achieve required capabilities.*
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-567%3A+Kafka+Cluster+Audit
> >>>>>>>
> >>>>>>>
> >>>>>>> пт, 24 янв. 2020 г., 17:29 Игорь Мартемьянов <ledostuff@gmail.com
> >>>> :
> >>>>>>>
> >>>>>>>> Hello there.
> >>>>>>>> Please review this KIP.
> >>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: [DISCUSS] KIP-567: Kafka Cluster Audit
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Viktor.
Do you want to implement the exact approach as it described in the current KIP?
Or you have another proposal on how it has to be implemented?
I abandoned this KIP due to lack of interest from community.
Guess we can collaborate during implementation.
> 7 сент. 2020 г., в 13:13, Viktor Somogyi-Vass <vi...@gmail.com> написал(а):
>
> Hi folks,
>
> It's been a few days since I last pinged and nobody replied so I assume
> that this KIP is abandoned and I can take this over (but please let me know
> if it's not). I will keep the current version of the KIP and move it to a
> sub-page if it's ever needed.
>
> Thanks,
> Viktor
>
> On Fri, Aug 28, 2020 at 11:35 AM Viktor Somogyi-Vass <
> viktorsomogyi@gmail.com> wrote:
>
>> Hi folks,
>>
>> I have a use-case and a non-trivial implementation with Apache Atlas for
>> this KIP and since this kip seems to be dormant for a while now, I'd take
>> it over and drive it to completion if you don't mind.
>> The current state of the PoC can be found on my fork at
>> https://github.com/viktorsomogyi/kafka/tree/atlas-audit-impl.
>>
>> Viktor
>>
>> On Wed, Jan 29, 2020 at 9:32 AM Игорь Мартемьянов <le...@gmail.com>
>> wrote:
>>
>>> Hello, Nikolai.
>>>
>>>
>>>
>>>> Can you, please, make it more specific?
>>>
>>>> Why does a business want to have this information?
>>>
>>> It is very demanded for security department to know who/when/where create
>>> or edit ACL settings. The same situation about topics.
>>>
>>>
>>>
>>>> What are the use-cases for it?
>>>
>>> This KIP are able to help catching some unauthorized changes of cluster
>>> resources or save history these changes.
>>>
>>>
>>>
>>>> Who will be analyzing these events and how?
>>>
>>> Events could be analyzed by some security department or developers who
>>> builds applications that can change cluster resources.
>>>
>>>
>>>
>>>> Why it’s not convenient to implement it with third-party tools?
>>>
>>> It is imposible to my mind getting detailed information about these events
>>> through third-party tools, because we don`t have the ability to catch
>>> these
>>> events outside Kafka core.
>>>
>>>
>>>
>>>> 1. `audit` name sounds too general for me. How about `onEvent`?
>>>
>>> Thanks, good point. I`ll change the method name.
>>>
>>>
>>>
>>>> 2. Should we introduce a special marker interface for audit events?
>>>
>>>> `AuditableRequest`, for example?
>>>
>>> No, we shouldn`t. We are going to create these events if we process
>>> special
>>> request in KafkaApis class (ApiKeys.CREATE_TOPICS, ApiKeys.DELETE_TOPICS,
>>> ApiKeys.CREATE_ACLS, ApiKeys.DELETE_ACLS, ApiKeys.DESCRIBE_ACLS,
>>> ApiKeys.ALTER_CONFIGS)
>>>
>>>
>>>
>>>>> public interface AuditEvent<T extends AbstractRequest> {
>>>
>>>>> String guid();
>>>
>>>
>>>
>>>> Where this `guid` comes from?
>>>
>>> Guid generated automatically when the audit event is created.
>>>
>>>
>>>
>>>> Will it be the same on each node that receives an auditable event?
>>>
>>> Event is creating while request that changes cluster resource is processed
>>> and will destroyed after the prosessing will finished on each node.
>>>
>>>
>>>
>>>> Do we have `guid` for any extensions of `AbstractRequest`?
>>>
>>> No, we don`t.
>>>
>>>
>>>
>>>> If this field is `guid` why do we format this as a String on the API
>>> level?
>>>
>>> Thanks for the point.
>>>
>>> I have changed event interface recently.
>>>
>>>
>>>
>>>> Can you, please, add a full list of proposed new configuration
>>> properties
>>> and
>>>
>>>> examples for each to clarify your intentions?
>>>
>>> Yes, I have added some examples of new configuration properties.
>>>
>>> ср, 29 янв. 2020 г., 10:23 Владимир Беруненко <vv...@gmail.com>:
>>>
>>>> Hi Nikolai!
>>>>
>>>>> Can you, please, make it more specific?
>>>> Why does a business want to have this information?
>>>>> What are the use-cases for it?
>>>>> Who will be analyzing these events and how?
>>>>> Why it’s not convenient to implement it with third-party tools?
>>>>
>>>> This is required by the guys from information security to detect
>>> potential
>>>> threats of violation the rules for providing access to the data layer.
>>>> Analysis in the audit system is usually based on identifying
>>>> uncharacteristic integrations. The key feature of the audit, is that
>>>> cluster administrators do not have access to modification of audit
>>> events.
>>>> Third-party tools are great for data analysis, but its not good idea to
>>> use
>>>> it for audit events collection.
>>>>
>>>>> It’s not clear for me where and when AuditEvents will be sent?
>>>> Who will be the receiver of events?
>>>>
>>>> In my opinion, sending an audit event should be initiated when the
>>> broker
>>>> receives a request that matches the audit parameters. Each organization
>>> has
>>>> its own receiver system, so a common interface is required that the
>>>> organization’s development team can implement to integrate with their
>>> audit
>>>> system.
>>>>
>>>> Best wishes, Vladimir
>>>>
>>>> Hello, Igor.
>>>>>
>>>>> Thanks for the KIP.
>>>>> I have a couple of comments for it:
>>>>>
>>>>>> Motivation
>>>>>> It is highly demanded in most businesses to have the ability of
>>>>> obtaining audit information in case someone changes cluster
>>> configuration
>>>>> (like creation/deletion/modify/description of any topic or ACLs).
>>>>>
>>>>> Can you, please, make it more specific?
>>>>> Why does a business want to have this information?
>>>>> What are the use-cases for it?
>>>>> Who will be analyzing these events and how?
>>>>> Why it’s not convenient to implement it with third-party tools?
>>>>>
>>>>> It’s not clear for me where and when AuditEvents will be sent?
>>>>> Who will be the receiver of events?
>>>>>
>>>>>> void audit<T extends AbstractRequest>(AuditEvent<T> event);
>>>>>
>>>>> 1. `audit` name sounds too general for me. How about `onEvent`?
>>>>> 2. Should we introduce a special marker interface for audit events?
>>>>> `AuditableRequest`, for example?
>>>>>
>>>>>> public interface AuditEvent<T extends AbstractRequest> {
>>>>>> String guid();
>>>>>
>>>>> Where this `guid` comes from?
>>>>> Will it be the same on each node that receives an auditable event?
>>>>> Do we have `guid` for any extensions of `AbstractRequest`?
>>>>> If this field is `guid` why do we format this as a String on the API
>>>> level?
>>>>>
>>>>>> One or more of AuditExtension's implementations can be configured
>>> via
>>>>> the configuration audit.extension.classes
>>>>>
>>>>> Can you, please, add a full list of proposed new configuration
>>> properties
>>>>> and examples for each to clarify your intentions?
>>>>>
>>>>>
>>>>>> 24 янв. 2020 г., в 18:12, Alexander Dunayevsky <
>>>> a.a.dunayevsky@gmail.com>
>>>>> написал(а):
>>>>>>
>>>>>> Hello Igor,
>>>>>>
>>>>>> Thanks for your KIP 🙌🏽
>>>>>> It would be great to adopt this functionality and getting the best
>>> of
>>>>>> tracking cluster activity.
>>>>>>
>>>>>> +1 vote from me
>>>>>>
>>>>>> Cheers,
>>>>>> Alex Dunayevsky
>>>>>>
>>>>>>
>>>>>> On Fri, 24 Jan 2020, 15:35 Игорь Мартемьянов, <le...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>>> Motivation:
>>>>>>>
>>>>>>>
>>>>>>> *It is highly demanded in most businesses to have the ability of
>>>>> obtaining
>>>>>>> audit information in case someone changes cluster configuration
>>> (like
>>>>>>> creation/deletion/modify/description of any topic or ACLs).We may
>>> add
>>>>> this
>>>>>>> ability. Since audit requirements are so broad, it's impractical to
>>>>> support
>>>>>>> all of them.Hence we have to provide ability for users to plug
>>>> resources
>>>>>>> helping to achieve required capabilities.*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-567%3A+Kafka+Cluster+Audit
>>>>>>>
>>>>>>>
>>>>>>> пт, 24 янв. 2020 г., 17:29 Игорь Мартемьянов <ledostuff@gmail.com
>>>> :
>>>>>>>
>>>>>>>> Hello there.
>>>>>>>> Please review this KIP.
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
Re: [DISCUSS] KIP-567: Kafka Cluster Audit
Posted by Viktor Somogyi-Vass <vi...@gmail.com>.
Hi folks,
It's been a few days since I last pinged and nobody replied so I assume
that this KIP is abandoned and I can take this over (but please let me know
if it's not). I will keep the current version of the KIP and move it to a
sub-page if it's ever needed.
Thanks,
Viktor
On Fri, Aug 28, 2020 at 11:35 AM Viktor Somogyi-Vass <
viktorsomogyi@gmail.com> wrote:
> Hi folks,
>
> I have a use-case and a non-trivial implementation with Apache Atlas for
> this KIP and since this kip seems to be dormant for a while now, I'd take
> it over and drive it to completion if you don't mind.
> The current state of the PoC can be found on my fork at
> https://github.com/viktorsomogyi/kafka/tree/atlas-audit-impl.
>
> Viktor
>
> On Wed, Jan 29, 2020 at 9:32 AM Игорь Мартемьянов <le...@gmail.com>
> wrote:
>
>> Hello, Nikolai.
>>
>>
>>
>> > Can you, please, make it more specific?
>>
>> > Why does a business want to have this information?
>>
>> It is very demanded for security department to know who/when/where create
>> or edit ACL settings. The same situation about topics.
>>
>>
>>
>> > What are the use-cases for it?
>>
>> This KIP are able to help catching some unauthorized changes of cluster
>> resources or save history these changes.
>>
>>
>>
>> > Who will be analyzing these events and how?
>>
>> Events could be analyzed by some security department or developers who
>> builds applications that can change cluster resources.
>>
>>
>>
>> > Why it’s not convenient to implement it with third-party tools?
>>
>> It is imposible to my mind getting detailed information about these events
>> through third-party tools, because we don`t have the ability to catch
>> these
>> events outside Kafka core.
>>
>>
>>
>> > 1. `audit` name sounds too general for me. How about `onEvent`?
>>
>> Thanks, good point. I`ll change the method name.
>>
>>
>>
>> > 2. Should we introduce a special marker interface for audit events?
>>
>> > `AuditableRequest`, for example?
>>
>> No, we shouldn`t. We are going to create these events if we process
>> special
>> request in KafkaApis class (ApiKeys.CREATE_TOPICS, ApiKeys.DELETE_TOPICS,
>> ApiKeys.CREATE_ACLS, ApiKeys.DELETE_ACLS, ApiKeys.DESCRIBE_ACLS,
>> ApiKeys.ALTER_CONFIGS)
>>
>>
>>
>> >> public interface AuditEvent<T extends AbstractRequest> {
>>
>> >> String guid();
>>
>>
>>
>> > Where this `guid` comes from?
>>
>> Guid generated automatically when the audit event is created.
>>
>>
>>
>> > Will it be the same on each node that receives an auditable event?
>>
>> Event is creating while request that changes cluster resource is processed
>> and will destroyed after the prosessing will finished on each node.
>>
>>
>>
>> > Do we have `guid` for any extensions of `AbstractRequest`?
>>
>> No, we don`t.
>>
>>
>>
>> > If this field is `guid` why do we format this as a String on the API
>> level?
>>
>> Thanks for the point.
>>
>> I have changed event interface recently.
>>
>>
>>
>> > Can you, please, add a full list of proposed new configuration
>> properties
>> and
>>
>> > examples for each to clarify your intentions?
>>
>> Yes, I have added some examples of new configuration properties.
>>
>> ср, 29 янв. 2020 г., 10:23 Владимир Беруненко <vv...@gmail.com>:
>>
>> > Hi Nikolai!
>> >
>> > >Can you, please, make it more specific?
>> > Why does a business want to have this information?
>> > >What are the use-cases for it?
>> > >Who will be analyzing these events and how?
>> > >Why it’s not convenient to implement it with third-party tools?
>> >
>> > This is required by the guys from information security to detect
>> potential
>> > threats of violation the rules for providing access to the data layer.
>> > Analysis in the audit system is usually based on identifying
>> > uncharacteristic integrations. The key feature of the audit, is that
>> > cluster administrators do not have access to modification of audit
>> events.
>> > Third-party tools are great for data analysis, but its not good idea to
>> use
>> > it for audit events collection.
>> >
>> > >It’s not clear for me where and when AuditEvents will be sent?
>> > Who will be the receiver of events?
>> >
>> > In my opinion, sending an audit event should be initiated when the
>> broker
>> > receives a request that matches the audit parameters. Each organization
>> has
>> > its own receiver system, so a common interface is required that the
>> > organization’s development team can implement to integrate with their
>> audit
>> > system.
>> >
>> > Best wishes, Vladimir
>> >
>> > Hello, Igor.
>> > >
>> > > Thanks for the KIP.
>> > > I have a couple of comments for it:
>> > >
>> > > > Motivation
>> > > > It is highly demanded in most businesses to have the ability of
>> > > obtaining audit information in case someone changes cluster
>> configuration
>> > > (like creation/deletion/modify/description of any topic or ACLs).
>> > >
>> > > Can you, please, make it more specific?
>> > > Why does a business want to have this information?
>> > > What are the use-cases for it?
>> > > Who will be analyzing these events and how?
>> > > Why it’s not convenient to implement it with third-party tools?
>> > >
>> > > It’s not clear for me where and when AuditEvents will be sent?
>> > > Who will be the receiver of events?
>> > >
>> > > > void audit<T extends AbstractRequest>(AuditEvent<T> event);
>> > >
>> > > 1. `audit` name sounds too general for me. How about `onEvent`?
>> > > 2. Should we introduce a special marker interface for audit events?
>> > > `AuditableRequest`, for example?
>> > >
>> > > > public interface AuditEvent<T extends AbstractRequest> {
>> > > > String guid();
>> > >
>> > > Where this `guid` comes from?
>> > > Will it be the same on each node that receives an auditable event?
>> > > Do we have `guid` for any extensions of `AbstractRequest`?
>> > > If this field is `guid` why do we format this as a String on the API
>> > level?
>> > >
>> > > > One or more of AuditExtension's implementations can be configured
>> via
>> > > the configuration audit.extension.classes
>> > >
>> > > Can you, please, add a full list of proposed new configuration
>> properties
>> > > and examples for each to clarify your intentions?
>> > >
>> > >
>> > > > 24 янв. 2020 г., в 18:12, Alexander Dunayevsky <
>> > a.a.dunayevsky@gmail.com>
>> > > написал(а):
>> > > >
>> > > > Hello Igor,
>> > > >
>> > > > Thanks for your KIP 🙌🏽
>> > > > It would be great to adopt this functionality and getting the best
>> of
>> > > > tracking cluster activity.
>> > > >
>> > > > +1 vote from me
>> > > >
>> > > > Cheers,
>> > > > Alex Dunayevsky
>> > > >
>> > > >
>> > > > On Fri, 24 Jan 2020, 15:35 Игорь Мартемьянов, <le...@gmail.com>
>> > > wrote:
>> > > >
>> > > >> Motivation:
>> > > >>
>> > > >>
>> > > >> *It is highly demanded in most businesses to have the ability of
>> > > obtaining
>> > > >> audit information in case someone changes cluster configuration
>> (like
>> > > >> creation/deletion/modify/description of any topic or ACLs).We may
>> add
>> > > this
>> > > >> ability. Since audit requirements are so broad, it's impractical to
>> > > support
>> > > >> all of them.Hence we have to provide ability for users to plug
>> > resources
>> > > >> helping to achieve required capabilities.*
>> > > >>
>> > > >>
>> > > >>
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-567%3A+Kafka+Cluster+Audit
>> > > >>
>> > > >>
>> > > >> пт, 24 янв. 2020 г., 17:29 Игорь Мартемьянов <ledostuff@gmail.com
>> >:
>> > > >>
>> > > >>> Hello there.
>> > > >>> Please review this KIP.
>> > > >>> Thanks.
>> > > >>>
>> > > >>
>> > >
>> > >
>> >
>>
>