You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2025/03/07 17:40:00 UTC
[jira] [Commented] (HADOOP-19455) S3A: support metrics publishers
[ https://issues.apache.org/jira/browse/HADOOP-19455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17933383#comment-17933383 ]
Steve Loughran commented on HADOOP-19455:
-----------------------------------------
It'd be nicest to wire up the logging publisher so it'd automatically to the pretty logs at trace level -so could be turned on when needed.
However, that'd require some reflection to invoke a constructor which, in the shaded bundle.jar, needs a reference to the shaded slf4j enum.
Instead, it'll be set to wire up to log at INFO, but only if DefaultS3ClientFactory is set to log at trace
{code}
log4j.logger.org.apache.hadoop.fs.s3a.DefaultS3ClientFactory=TRACE
{code}
{code}
2025-03-07 17:34:26,086 [s3a-transfer-stevel-london-bounded-pool2-t1] DEBUG impl.S3AStoreImpl (DurationInfo.java:<init>(80)) - Starting: deleting test/testXAttrFile
2025-03-07 17:34:26,154 [s3a-transfer-stevel-london-bounded-pool2-t1] INFO metrics.LoggingMetricPublisher (LoggerAdapter.java:info(165)) - Metrics published: MetricCollection(name=ApiCall, metrics=[MetricRecord(metric=MarshallingDuration, value=PT0.000092041S), MetricRecord(metric=RetryCount, value=0), MetricRecord(metric=ApiCallSuccessful, value=true), MetricRecord(metric=OperationName, value=DeleteObject), MetricRecord(metric=EndpointResolveDuration, value=PT0.000132792S), MetricRecord(metric=ApiCallDuration, value=PT0.064890875S), MetricRecord(metric=CredentialsFetchDuration, value=PT0.000017458S), MetricRecord(metric=ServiceEndpoint, value=https://stevel-london.s3.eu-west-2.amazonaws.com), MetricRecord(metric=ServiceId, value=S3)], children=[MetricCollection(name=ApiCallAttempt, metrics=[MetricRecord(metric=TimeToFirstByte, value=PT0.06260225S), MetricRecord(metric=SigningDuration, value=PT0.000293083S), MetricRecord(metric=ReadThroughput, value=0.0), MetricRecord(metric=ServiceCallDuration, value=PT0.06260225S), MetricRecord(metric=AwsExtendedRequestId, value=zvfoe8Vr8JFB42aDPawtJ3VfqV/xm0zyJ2jt9GG6027aVErpWdQ3UYdLEG8TVIAldxUovsyVJdw=), MetricRecord(metric=HttpStatusCode, value=204), MetricRecord(metric=BackoffDelayDuration, value=PT0S), MetricRecord(metric=TimeToLastByte, value=PT0.064313667S), MetricRecord(metric=AwsRequestId, value=RKZD44SE5DW91K1G)], children=[MetricCollection(name=HttpClient, metrics=[MetricRecord(metric=AvailableConcurrency, value=1), MetricRecord(metric=LeasedConcurrency, value=0), MetricRecord(metric=ConcurrencyAcquireDuration, value=PT0S), MetricRecord(metric=PendingConcurrencyAcquires, value=0), MetricRecord(metric=MaxConcurrency, value=512), MetricRecord(metric=HttpClientName, value=Apache)], children=[])])])
{code}
> S3A: support metrics publishers
> -------------------------------
>
> Key: HADOOP-19455
> URL: https://issues.apache.org/jira/browse/HADOOP-19455
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
>
> Add options to enable logging and Cloudwatch metrics
> always turn on logging metrics but log at trace, so never evaluated or visible unless requested.
> Cloudwatch would be interesting but complicated, because lots of config. Also it should be a singleton: first configuration wins.
> Whoever wants cloudwatch gets to implement that themselves with docs and tests (how?)
> https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/metrics/LoggingMetricPublisher.html
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org