You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ozone.apache.org by Janus Chow <yi...@gmail.com> on 2022/04/04 02:58:50 UTC
Disscuss of enabling READ audit log by default
Hi Ozone dev,
When checking the audit logs from Ozone components, we found that by
default Ozone only logs WRITE operations. In order to enable the audit log
for READ operations, we need to change the configurations in
audit-log4j2.properties.
That brings some confusion for users when comparing it to some other
storage systems, like HDFS, in which audit logs are enabled for both READ
and WRITE by default.
We have a Jira ticket(https://issues.apache.org/jira/browse/HDDS-6532) and
PR (https://github.com/apache/ozone/pull/3255) about adding audit logs for
READ operations by default.
Could you help to check and comment if there are any specific concerns
not to enable READ audit logs?
Yiyang
Thank you very much.
Re: Disscuss of enabling READ audit log by default
Posted by Hui Fei <fe...@gmail.com>.
Thanks for starting this discussion.
I also prefer to enable READ audit.
In my experience data security is very important.
Janus Chow <yi...@gmail.com> 于2022年4月4日周一 10:59写道：
> Hi Ozone dev,
>
> When checking the audit logs from Ozone components, we found that by
> default Ozone only logs WRITE operations. In order to enable the audit log
> for READ operations, we need to change the configurations in
> audit-log4j2.properties.
> That brings some confusion for users when comparing it to some other
> storage systems, like HDFS, in which audit logs are enabled for both READ
> and WRITE by default.
>
> We have a Jira ticket(https://issues.apache.org/jira/browse/HDDS-6532) and
> PR (https://github.com/apache/ozone/pull/3255) about adding audit logs for
> READ operations by default.
> Could you help to check and comment if there are any specific concerns
> not to enable READ audit logs?
>
> Yiyang
> Thank you very much.
>
Re: Disscuss of enabling READ audit log by default
Posted by Janus Chow <yi...@gmail.com>.
Hello,
The feature to exclude certain operations for audit log is ready: https://github.com/apache/ozone/pull/3289 <https://github.com/apache/ozone/pull/3289>
Please help to review.
We also noticed it might be inconvenient for users to update the configuration, since it needs to restart the service to reload the configurations.
So we also build a dynamic refresh feature to refresh the configuration, will raise the ticket after the above PR finishes.
Also hope to get reviewed.
Thanks
Symious
> On 7 Apr 2022, at 2:33 PM, Janus Chow <yi...@gmail.com> wrote:
>
> Hello,
>
> @Arp @feihui, Thanks for the reply.
>
> Have created a ticket for the feature to exclude operations on demand. (https://issues.apache.org/jira/browse/HDDS-6562 <https://issues.apache.org/jira/browse/HDDS-6562>)
>
> Thanks
> Yiyang
>
> Arpit Agarwal <aa...@cloudera.com.invalid> 于2022年4月6日周三 21:36写道：
> Hi Yiyang,
>
> +1 to enable if we have a way to exclude on demand.
>
> Thanks,
> Arpit
>
>
> > On Apr 3, 2022, at 9:37 PM, Janus Chow <yiyang0203@gmail.com <ma...@gmail.com>> wrote:
> >
> > Thanks for the reply. @Arp
> >
> > From the commit in
> > https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml#L3190 <https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml#L3190>,
> > I think by default the configuration is empty.
> >
> > In a releted ticket, https://issues.apache.org/jira/browse/HDFS-9828 <https://issues.apache.org/jira/browse/HDFS-9828>, It's
> > also recommended not to disable READ audit logs.
> >
> > Currently, we only enabled audit for READ in our UAT cluster, the
> > request/second is not very high, in PROD cluster, it should be quite higher.
> >
> > IMHO, the read audit log is quite useful, the problem is that we don't have
> > a similar way like HDFS to exclude some read operations. If we have a
> > similar exclude mechanism, is it ok to enable READ operation audit log by
> > default?
> >
> >
> > Thanks
> > Yiyang
> >
> > Arpit Agarwal <aa...@cloudera.com.invalid> 于2022年4月4日周一 12:01写道：
> >
> >> Hi Janus,
> >>
> >> Performance will be the main concern. In busy HDFS clusters admins are
> >> likely to use dfs.namenode.audit.log.debug.cmdlist.
> >>
> >> Have you enabled read audit logging in your Ozone cluster? What is the
> >> number of requests/second?
> >>
> >> Thanks,
> >> Arpit
> >>
> >>
> >>> On Apr 3, 2022, at 7:58 PM, Janus Chow <yiyang0203@gmail.com <ma...@gmail.com>> wrote:
> >>>
> >>> Hi Ozone dev,
> >>>
> >>> When checking the audit logs from Ozone components, we found that by
> >>> default Ozone only logs WRITE operations. In order to enable the audit
> >> log
> >>> for READ operations, we need to change the configurations in
> >>> audit-log4j2.properties.
> >>> That brings some confusion for users when comparing it to some other
> >>> storage systems, like HDFS, in which audit logs are enabled for both READ
> >>> and WRITE by default.
> >>>
> >>> We have a Jira ticket(https://issues.apache.org/jira/browse/HDDS-6532 <https://issues.apache.org/jira/browse/HDDS-6532>)
> >> and
> >>> PR (https://github.com/apache/ozone/pull/3255 <https://github.com/apache/ozone/pull/3255>) about adding audit logs
> >> for
> >>> READ operations by default.
> >>> Could you help to check and comment if there are any specific concerns
> >>> not to enable READ audit logs?
> >>>
> >>> Yiyang
> >>> Thank you very much.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@ozone.apache.org <ma...@ozone.apache.org>
> >> For additional commands, e-mail: dev-help@ozone.apache.org <ma...@ozone.apache.org>
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ozone.apache.org <ma...@ozone.apache.org>
> For additional commands, e-mail: dev-help@ozone.apache.org <ma...@ozone.apache.org>
>
Re: Disscuss of enabling READ audit log by default
Posted by Janus Chow <yi...@gmail.com>.
Hello,
@Arp @feihui, Thanks for the reply.
Have created a ticket for the feature to exclude operations on demand. (
https://issues.apache.org/jira/browse/HDDS-6562)
Thanks
Yiyang
Arpit Agarwal <aa...@cloudera.com.invalid> 于2022年4月6日周三 21:36写道：
> Hi Yiyang,
>
> +1 to enable if we have a way to exclude on demand.
>
> Thanks,
> Arpit
>
>
> > On Apr 3, 2022, at 9:37 PM, Janus Chow <yi...@gmail.com> wrote:
> >
> > Thanks for the reply. @Arp
> >
> > From the commit in
> >
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml#L3190
> ,
> > I think by default the configuration is empty.
> >
> > In a releted ticket, https://issues.apache.org/jira/browse/HDFS-9828,
> It's
> > also recommended not to disable READ audit logs.
> >
> > Currently, we only enabled audit for READ in our UAT cluster, the
> > request/second is not very high, in PROD cluster, it should be quite
> higher.
> >
> > IMHO, the read audit log is quite useful, the problem is that we don't
> have
> > a similar way like HDFS to exclude some read operations. If we have a
> > similar exclude mechanism, is it ok to enable READ operation audit log by
> > default?
> >
> >
> > Thanks
> > Yiyang
> >
> > Arpit Agarwal <aa...@cloudera.com.invalid> 于2022年4月4日周一 12:01写道：
> >
> >> Hi Janus,
> >>
> >> Performance will be the main concern. In busy HDFS clusters admins are
> >> likely to use dfs.namenode.audit.log.debug.cmdlist.
> >>
> >> Have you enabled read audit logging in your Ozone cluster? What is the
> >> number of requests/second?
> >>
> >> Thanks,
> >> Arpit
> >>
> >>
> >>> On Apr 3, 2022, at 7:58 PM, Janus Chow <yi...@gmail.com> wrote:
> >>>
> >>> Hi Ozone dev,
> >>>
> >>> When checking the audit logs from Ozone components, we found that by
> >>> default Ozone only logs WRITE operations. In order to enable the audit
> >> log
> >>> for READ operations, we need to change the configurations in
> >>> audit-log4j2.properties.
> >>> That brings some confusion for users when comparing it to some other
> >>> storage systems, like HDFS, in which audit logs are enabled for both
> READ
> >>> and WRITE by default.
> >>>
> >>> We have a Jira ticket(https://issues.apache.org/jira/browse/HDDS-6532)
> >> and
> >>> PR (https://github.com/apache/ozone/pull/3255) about adding audit logs
> >> for
> >>> READ operations by default.
> >>> Could you help to check and comment if there are any specific concerns
> >>> not to enable READ audit logs?
> >>>
> >>> Yiyang
> >>> Thank you very much.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@ozone.apache.org
> >> For additional commands, e-mail: dev-help@ozone.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ozone.apache.org
> For additional commands, e-mail: dev-help@ozone.apache.org
>
>
Re: Disscuss of enabling READ audit log by default
Posted by Arpit Agarwal <aa...@cloudera.com.INVALID>.
Hi Yiyang,
+1 to enable if we have a way to exclude on demand.
Thanks,
Arpit
> On Apr 3, 2022, at 9:37 PM, Janus Chow <yi...@gmail.com> wrote:
>
> Thanks for the reply. @Arp
>
> From the commit in
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml#L3190,
> I think by default the configuration is empty.
>
> In a releted ticket, https://issues.apache.org/jira/browse/HDFS-9828, It's
> also recommended not to disable READ audit logs.
>
> Currently, we only enabled audit for READ in our UAT cluster, the
> request/second is not very high, in PROD cluster, it should be quite higher.
>
> IMHO, the read audit log is quite useful, the problem is that we don't have
> a similar way like HDFS to exclude some read operations. If we have a
> similar exclude mechanism, is it ok to enable READ operation audit log by
> default?
>
>
> Thanks
> Yiyang
>
> Arpit Agarwal <aa...@cloudera.com.invalid> 于2022年4月4日周一 12:01写道：
>
>> Hi Janus,
>>
>> Performance will be the main concern. In busy HDFS clusters admins are
>> likely to use dfs.namenode.audit.log.debug.cmdlist.
>>
>> Have you enabled read audit logging in your Ozone cluster? What is the
>> number of requests/second?
>>
>> Thanks,
>> Arpit
>>
>>
>>> On Apr 3, 2022, at 7:58 PM, Janus Chow <yi...@gmail.com> wrote:
>>>
>>> Hi Ozone dev,
>>>
>>> When checking the audit logs from Ozone components, we found that by
>>> default Ozone only logs WRITE operations. In order to enable the audit
>> log
>>> for READ operations, we need to change the configurations in
>>> audit-log4j2.properties.
>>> That brings some confusion for users when comparing it to some other
>>> storage systems, like HDFS, in which audit logs are enabled for both READ
>>> and WRITE by default.
>>>
>>> We have a Jira ticket(https://issues.apache.org/jira/browse/HDDS-6532)
>> and
>>> PR (https://github.com/apache/ozone/pull/3255) about adding audit logs
>> for
>>> READ operations by default.
>>> Could you help to check and comment if there are any specific concerns
>>> not to enable READ audit logs?
>>>
>>> Yiyang
>>> Thank you very much.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@ozone.apache.org
>> For additional commands, e-mail: dev-help@ozone.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ozone.apache.org
For additional commands, e-mail: dev-help@ozone.apache.org
Re: Disscuss of enabling READ audit log by default
Posted by Janus Chow <yi...@gmail.com>.
Thanks for the reply. @Arp
From the commit in
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml#L3190,
I think by default the configuration is empty.
In a releted ticket, https://issues.apache.org/jira/browse/HDFS-9828, It's
also recommended not to disable READ audit logs.
Currently, we only enabled audit for READ in our UAT cluster, the
request/second is not very high, in PROD cluster, it should be quite higher.
IMHO, the read audit log is quite useful, the problem is that we don't have
a similar way like HDFS to exclude some read operations. If we have a
similar exclude mechanism, is it ok to enable READ operation audit log by
default?
Thanks
Yiyang
Arpit Agarwal <aa...@cloudera.com.invalid> 于2022年4月4日周一 12:01写道：
> Hi Janus,
>
> Performance will be the main concern. In busy HDFS clusters admins are
> likely to use dfs.namenode.audit.log.debug.cmdlist.
>
> Have you enabled read audit logging in your Ozone cluster? What is the
> number of requests/second?
>
> Thanks,
> Arpit
>
>
> > On Apr 3, 2022, at 7:58 PM, Janus Chow <yi...@gmail.com> wrote:
> >
> > Hi Ozone dev,
> >
> > When checking the audit logs from Ozone components, we found that by
> > default Ozone only logs WRITE operations. In order to enable the audit
> log
> > for READ operations, we need to change the configurations in
> > audit-log4j2.properties.
> > That brings some confusion for users when comparing it to some other
> > storage systems, like HDFS, in which audit logs are enabled for both READ
> > and WRITE by default.
> >
> > We have a Jira ticket(https://issues.apache.org/jira/browse/HDDS-6532)
> and
> > PR (https://github.com/apache/ozone/pull/3255) about adding audit logs
> for
> > READ operations by default.
> > Could you help to check and comment if there are any specific concerns
> > not to enable READ audit logs?
> >
> > Yiyang
> > Thank you very much.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ozone.apache.org
> For additional commands, e-mail: dev-help@ozone.apache.org
>
>
Re: Disscuss of enabling READ audit log by default
Posted by Arpit Agarwal <aa...@cloudera.com.INVALID>.
Hi Janus,
Performance will be the main concern. In busy HDFS clusters admins are likely to use dfs.namenode.audit.log.debug.cmdlist.
Have you enabled read audit logging in your Ozone cluster? What is the number of requests/second?
Thanks,
Arpit
> On Apr 3, 2022, at 7:58 PM, Janus Chow <yi...@gmail.com> wrote:
>
> Hi Ozone dev,
>
> When checking the audit logs from Ozone components, we found that by
> default Ozone only logs WRITE operations. In order to enable the audit log
> for READ operations, we need to change the configurations in
> audit-log4j2.properties.
> That brings some confusion for users when comparing it to some other
> storage systems, like HDFS, in which audit logs are enabled for both READ
> and WRITE by default.
>
> We have a Jira ticket(https://issues.apache.org/jira/browse/HDDS-6532) and
> PR (https://github.com/apache/ozone/pull/3255) about adding audit logs for
> READ operations by default.
> Could you help to check and comment if there are any specific concerns
> not to enable READ audit logs?
>
> Yiyang
> Thank you very much.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ozone.apache.org
For additional commands, e-mail: dev-help@ozone.apache.org