You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Duong (Jira)" <ji...@apache.org> on 2022/06/16 19:41:00 UTC
[jira] [Comment Edited] (HDDS-6559) S3G client logging review
[ https://issues.apache.org/jira/browse/HDDS-6559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17555269#comment-17555269 ]
Duong edited comment on HDDS-6559 at 6/16/22 7:40 PM:
------------------------------------------------------
That makes sense, [~kerneltime]. It's a good practice to not log customer data/events to application log for security and privacy purposes.
I got thought the S3G code and identified some points.
1. Customer events are sometimes logged in application log, e.g. [here|#L250-L250].] We need to review and clean up those.
2. OMException (which indicates user errors?) is sometimes logged in application log as well they're sent to audit log already. Those logs can just be removed as they're also in audit logs already.
3. OMExceptions are always pushed to framework level (Jersey/Netty). However, there's no appropriate exception handler plugged to the framework and those OMExceptions are consider as unknown exception by default. This result in wrong API responses (always 500 Internal Server Error while I believe most OMExceptions should be 4xx) and logs like.
```
{{2022-06-16 18:14:29,340 [qtp1912821769-23] WARN server.HttpChannel: handleException /bucket_2 INVALID_BUCKET_NAME org.apache.hadoop.ozone.om.exceptions.OMException: Bucket or Volume name has an unsupported character : }}_{{{}2022-06-16 18:14:29,341 [qtp1912821769-23] WARN server.HttpChannelState: unhandled due to prior sendError{}}}{{{}javax.servlet.ServletException: javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: INVALID_BUCKET_NAME org.apache.hadoop.ozone.om.exceptions.OMException: Bucket or Volume name has an unsupported character :{}}}_ {{{}at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162){}}}{{{}at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127){}}}{{{}at org.eclipse.jetty.server.Server.handle(Server.java:516){}}}{{{}at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388){}}}{{{}at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633){}}}{{{}at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380){}}}{{{}at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277){}}}{{{}at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311){}}}{{{}at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105){}}}{{{}at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104){}}}
```
We should implement proper exception handler to 1) map Exceptions to correct API responses and 2) log correctly.
I believe fixing those points would results a clean status for the Application logs (and also the API).
was (Author: JIRAUSER290990):
That makes sense, [~kerneltime]. It's a good practice to not log customer data/events to application log for security and privacy purposes.
I got thought the S3G code and identified some points.
# Customer events are sometimes logged in application log, e.g. [here|[https://github.com/duongnguyen0/ozone/blob/7c64595ee01f05a77675a3c5abf156d63378d727/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java#L250-L250].] We need to review and clean up those.
# OMException (which indicates user errors?) is sometimes logged in application log as well they're sent to audit log already. Those logs can just be removed as they're also in audit logs already.
#
OMExceptions are always pushed to framework level (Jersey/Netty). However, there's no appropriate exception handler plugged to the framework and those OMExceptions are consider as unknown exception by default. This result in wrong API responses (always 500 Internal Server Error while I believe most OMExceptions should be 4xx) and logs like.
{{{}2022-06-16 18:14:29,340 [qtp1912821769-23] WARN server.HttpChannel: handleException /bucket_2 INVALID_BUCKET_NAME org.apache.hadoop.ozone.om.exceptions.OMException: Bucket or Volume name has an unsupported character : _{}}}{{{}2022-06-16 18:14:29,341 [qtp1912821769-23] WARN server.HttpChannelState: unhandled due to prior sendError{}}}{{{}javax.servlet.ServletException: javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: INVALID_BUCKET_NAME org.apache.hadoop.ozone.om.exceptions.OMException: Bucket or Volume name has an unsupported character : _{}}}{{{}at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162){}}}{{{}at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127){}}}{{{}at org.eclipse.jetty.server.Server.handle(Server.java:516){}}}{{{}at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388){}}}{{{}at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633){}}}{{{}at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380){}}}{{{}at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277){}}}{{{}at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311){}}}{{{}at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105){}}}{{{}at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104){}}}
We should implement proper exception handler to 1) map Exceptions to correct API responses and 2) log correctly.
I believe fixing those points would results a clean status for the Application logs (and also the API).
> S3G client logging review
> -------------------------
>
> Key: HDDS-6559
> URL: https://issues.apache.org/jira/browse/HDDS-6559
> Project: Apache Ozone
> Issue Type: Improvement
> Components: S3
> Reporter: Ritesh H Shukla
> Assignee: Duong
> Priority: Major
>
> Currently a broad range of errors get logged to S3G's log which include errors that can be induced by a client. Ideally these should not be logged (Example invalid header), there can be an access log which only captures HTTP requests and responses but the S3G log should be for the internal state of S3G.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org