You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2016/08/31 15:34:20 UTC
[jira] [Commented] (HADOOP-13569) S3AFastOutputStream to take
ProgressListener in file create()
[ https://issues.apache.org/jira/browse/HADOOP-13569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15452550#comment-15452550 ]
Steve Loughran commented on HADOOP-13569:
-----------------------------------------
What the output would look like
{code}
2016-08-31 16:22:15,353 [JUnit] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:test_010_CreateHugeFile(142)) - [100%] Written 256.00 MB out of 256.00 MB; PUT = 209715200 bytes in 2 operations
2016-08-31 16:22:15,353 [JUnit] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:test_010_CreateHugeFile(154)) - Closing file and completing write operation
2016-08-31 16:22:15,372 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event TRANSFER_PART_STARTED_EVENT, bytes: 0
2016-08-31 16:22:15,372 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event CLIENT_REQUEST_STARTED_EVENT, bytes: 0
2016-08-31 16:22:15,373 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event HTTP_REQUEST_STARTED_EVENT, bytes: 0
2016-08-31 16:23:40,083 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event HTTP_REQUEST_COMPLETED_EVENT, bytes: 0
2016-08-31 16:23:40,083 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event HTTP_RESPONSE_STARTED_EVENT, bytes: 0
2016-08-31 16:23:40,084 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event HTTP_RESPONSE_COMPLETED_EVENT, bytes: 0
2016-08-31 16:23:40,084 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event CLIENT_REQUEST_SUCCESS_EVENT, bytes: 0
2016-08-31 16:23:40,084 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event TRANSFER_PART_COMPLETED_EVENT, bytes: 0
2016-08-31 16:24:42,754 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event HTTP_REQUEST_COMPLETED_EVENT, bytes: 0
2016-08-31 16:24:42,754 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event HTTP_RESPONSE_STARTED_EVENT, bytes: 0
2016-08-31 16:24:42,755 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event HTTP_RESPONSE_COMPLETED_EVENT, bytes: 0
2016-08-31 16:24:42,755 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event CLIENT_REQUEST_SUCCESS_EVENT, bytes: 0
2016-08-31 16:24:42,755 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event TRANSFER_PART_COMPLETED_EVENT, bytes: 0
2016-08-31 16:25:08,954 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event HTTP_REQUEST_COMPLETED_EVENT, bytes: 0
2016-08-31 16:25:08,954 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event HTTP_RESPONSE_STARTED_EVENT, bytes: 0
2016-08-31 16:25:08,954 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event HTTP_RESPONSE_COMPLETED_EVENT, bytes: 0
2016-08-31 16:25:08,955 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event CLIENT_REQUEST_SUCCESS_EVENT, bytes: 0
2016-08-31 16:25:08,955 [java-sdk-progress-listener-callback-thread] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:progressChanged(199)) - Event TRANSFER_PART_COMPLETED_EVENT, bytes: 0
2016-08-31 16:25:09,327 [JUnit] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1365)) - Duration of Time to close() output stream: 173,972,783,472 nS
2016-08-31 16:25:09,327 [JUnit] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1365)) - Duration of Time to write 256 MB in blocks of 65536: 174,530,704,251 nS
2016-08-31 16:25:09,328 [JUnit] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:logFSState(232)) - File System state after operation:
S3AFileSystem{ inputPolicy=normal, partSize=104857600, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=5242880, statistics {0 bytes read, 268435456 bytes written, 4 read ops, 0 large read ops, 4 write ops}, metrics {{Context=S3AFileSystem} { {files_created=1} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=4} {op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=0} {op_mkdirs=0} {op_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=2} {object_continue_list_requests=0} {object_metadata_requests=6} {object_multipart_aborted=0} {object_put_bytes=327155712} {object_put_requests=4} {stream_read_fully_operations=0} {stream_bytes_skipped_on_seek=0} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=0} {streamOpened=0} {stream_read_operations_incomplete=0} {stream_bytes_discarded_in_abort=0} {stream_close_operations=0} {stream_read_operations=0} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0} {streamClosed=0} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }}
2016-08-31 16:25:09,328 [JUnit] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:test_010_CreateHugeFile(164)) - Time per MB to write = 681,760,563 nS
2016-08-31 16:25:09,328 [JUnit] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:test_010_CreateHugeFile(165)) - Effective Bandwidth: 1.5380414417737729 MB/s
2016-08-31 16:25:09,328 [JUnit] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:test_010_CreateHugeFile(168)) - PUT 327155712 bytes in 4 operations; 8.5761906966528E13 MB/operation
2016-08-31 16:25:09,328 [JUnit] INFO scale.STestS3AHugeFileCreate (STestS3AHugeFileCreate.java:test_010_CreateHugeFile(171)) - Time per PUT 43,632,676,062 nS
{code}
> S3AFastOutputStream to take ProgressListener in file create()
> -------------------------------------------------------------
>
> Key: HADOOP-13569
> URL: https://issues.apache.org/jira/browse/HADOOP-13569
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
>
> For scale testing I'd like more meaningful progress than the Hadoop {{Progressable}} offers.
> Proposed: having {{S3AFastOutputStream}} check to see if the progressable passed in is also an instance of {{com.amazonaws.event.ProgressListener}} —and if so, wire it up directly.
> This allows tests to directly track state of upload, log it and perhaps even assert on it
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org