[jira] [Created] (HADOOP-17847) S3AInstrumentation Closing output stream statistics while data is still marked as pending upload in OutputStreamStatistics

["Li Rong (Jira)" <ji...@apache.org>]

Li Rong created HADOOP-17847:
--------------------------------

             Summary: S3AInstrumentation Closing output stream statistics while data is still marked as pending upload in OutputStreamStatistics
                 Key: HADOOP-17847
                 URL: https://issues.apache.org/jira/browse/HADOOP-17847
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs/s3
    Affects Versions: 3.2.1
         Environment: hadoop: 3.2.1

spark: 3.0.2

k8s server version: 1.18

aws.java.sdk.bundle.version:1.11.1033
            Reporter: Li Rong
         Attachments: logs.txt

When using hadoop s3a file upload for spark event Logs, the logs were queued up and not uploaded before the process is shut down:
{code:java}
// 21/08/13 12:22:39 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
21/08/13 12:22:39 WARN S3AInstrumentation: Closing output stream statistics while data is still marked as pending upload in OutputStreamStatistics{blocksSubmitted=1, blocksInQueue=1, blocksActive=0, blockUploadsCompleted=0, blockUploadsFailed=0, bytesPendingUpload=106716, bytesUploaded=0, blocksAllocated=1, blocksReleased=1, blocksActivelyAllocated=0, exceptionsInMultipartFinalize=0, transferDuration=0 ms, queueDuration=0 ms, averageQueueTime=0 ms, totalUploadDuration=0 ms, effectiveBandwidth=0.0 bytes/s}{code}
details see logs attached



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org