[jira] [Commented] (HADOOP-18611) AbstractSTestS3AHugeFiles put request assert is using wrong iostatistic

["Steve Loughran (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-18611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683401#comment-17683401 ] 

Steve Loughran commented on HADOOP-18611:
-----------------------------------------

stack
{code}

[ERROR] Failures: 
[ERROR]   ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.test_010_CreateHugeFile:247 [Put request count from filesystem stats counters=((action_file_opened.failures=0) (committer_materialize_file.failures=0) (store_exists_probe.failures=0) (committer_bytes_committed=0) (object_bulk_delete_request=0) (stream_read_vectored_incoming_ranges=0) (files_delete_rejected=0) (committer_load_single_pending_file.failures=0) (stream_write_total_time=64654) (op_is_directory=0) (fake_directories_created=0) (stream_write_block_uploads=16) (object_put_request.failures=0) (stream_write_block_uploads_aborted=0) (multipart_upload_abort_under_path_invoked=0) (object_put_bytes=67108864) (committer_commit_job=0) (op_abort.failures=0) (op_rename=0) (stream_write_bytes=67108864) (stream_read_bytes_backwards_on_seek=0) (delegation_tokens_issued=0) (stream_read_version_mismatches=0) (op_mkdirs=1) (action_http_get_request=0) (committer_commits.failures=0) (op_get_delegation_token=0) (object_delete_request=1) (committer_bytes_uploaded=0) (object_metadata_request=1) (multipart_upload_started=0) (files_deleted=1) (op_get_file_status=0) (action_http_get_request.failures=0) (committer_tasks_failed=0) (object_list_request.failures=0) (committer_stage_file_upload=0) (committer_commits_created=0) (op_is_directory.failures=0) (fake_directories_deleted=0) (stream_read_bytes=0) (stream_read_operations=0) (op_xattr_get_map.failures=0) (op_access=0) (op_mkdirs.failures=0) (committer_stage_file_upload.failures=0) (directories_deleted=0) (object_continue_list_request=0) (op_copy_from_local_file.failures=0) (stream_read_bytes_discarded_in_close=0) (store_exists_probe=0) (object_select_requests=0) (stream_read_remote_stream_drain=0) (action_file_opened=0) (stream_read_opened=0) (multipart_upload_part_put_bytes=0) (audit_request_execution=16) (committer_commits_completed=0) (op_list_status=0) (op_xattr_get_named_map=0) (stream_read_seek_forward_operations=0) (stream_read_unbuffered=0) (op_get_delegation_token.failures=0) (stream_read_vectored_operations=0) (stream_aborted=0) (action_executor_acquired=8) (files_copied_bytes=0) (committer_magic_files_created=0) (op_glob_status=0) (op_exists.failures=0) (stream_read_exceptions=0) (op_delete.failures=0) (committer_jobs_completed=0) (files_copied=0) (delegation_tokens_issued.failures=0) (committer_magic_marker_put=0) (op_xattr_get_map=0) (stream_read_seek_bytes_discarded=0) (object_put_request_completed=8) (stream_read_operations_incomplete=0) (multipart_upload_list.failures=0) (op_list_located_status=0) (stream_read_fully_operations=0) (audit_span_creation=4) (object_multipart_initiated.failures=0) (stream_read_closed=0) (stream_read_close_operations=0) (op_get_content_summary=0) (op_list_status.failures=0) (object_delete_objects=1) (op_xattr_get_named_map.failures=0) (object_list_request=3) (committer_tasks_completed=0) (action_http_head_request=1) (store_io_throttled=0) (op_is_file.failures=0) (committer_load_single_pending_file=0) (op_xattr_list=0) (multipart_instantiated=0) (stream_write_exceptions=0) (object_multipart_initiated=2) (store_io_request=16) (op_list_files.failures=0) (store_io_retry=0) (committer_magic_marker_put.failures=0) (op_copy_from_local_file=0) (op_get_file_status.failures=0) (multipart_upload_list=0) (stream_write_exceptions_completing_upload=0) (stream_write_total_data=134217728) (stream_write_queue_duration.failures=0) (stream_read_remote_stream_aborted=0) (op_get_file_checksum.failures=0) (op_get_content_summary.failures=0) (op_exists=0) (object_multipart_aborted=0) (op_hsync=0) (op_xattr_list.failures=0) (committer_commits_reverted=0) (action_executor_acquired.failures=0) (op_access.failures=0) (object_copy_requests=0) (multipart_upload_completed=1) (audit_failure=0) (stream_read_total_bytes=0) (object_continue_list_request.failures=0) (stream_read_remote_stream_aborted.failures=0) (op_createfile.failures=0) (op_rename.failures=0) (committer_jobs_failed=0) (stream_read_vectored_combined_ranges=0) (op_create.failures=0) (op_get_file_checksum=0) (object_bulk_delete_request.failures=0) (op_open=0) (object_delete_request.failures=0) (multipart_upload_part_put=0) (stream_read_remote_stream_drain.failures=0) (committer_commits_aborted=0) (stream_write_queue_duration=10) (committer_commit_job.failures=0) (stream_read_vectored_read_bytes_discarded=0) (op_abort=0) (object_put_request=0) (op_xattr_get_named=0) (op_create_non_recursive=0) (ignored_errors=0) (directories_created=0) (op_xattr_get_named.failures=0) (op_create=1) (op_delete=1) (stream_read_seek_backward_operations=0) (op_glob_status.failures=0) (action_http_head_request.failures=0) (op_openfile=0) (multipart_upload_aborted=0) (stream_read_bytes_discarded_in_abort=0) (op_createfile=0) (op_is_file=0) (stream_read_seek_policy_changed=0) (stream_write_block_uploads_committed=0) (object_multipart_aborted.failures=0) (stream_read_seek_operations=0) (files_created=1) (op_hflush=0) (audit_access_check_failure=0) (op_list_files=0) (committer_materialize_file=0) (stream_read_seek_bytes_skipped=0));
gauges=((stream_read_active_prefetch_operations=0) (stream_read_blocks_in_cache=0) (stream_write_block_uploads_active=0) (stream_write_block_uploads_data_pending=0) (object_put_request_active=0) (client_side_encryption_enabled=0) (stream_read_active_memory_in_use=0) (stream_write_block_uploads_pending=0) (object_put_bytes_pending=0));
minimums=((object_multipart_aborted.min=-1) (committer_stage_file_upload.min=-1) (op_mkdirs.failures.min=-1) (op_list_status.min=-1) (op_get_file_status.failures.min=-1) (op_xattr_get_named_map.min=-1) (action_executor_acquired.min=0) (object_multipart_initiated.min=62) (committer_load_single_pending_file.min=-1) (object_delete_request.failures.min=-1) (op_is_directory.min=-1) (op_xattr_list.min=-1) (op_delete.failures.min=-1) (op_get_content_summary.failures.min=-1) (object_multipart_aborted.failures.min=-1) (committer_materialize_file.failures.min=-1) (op_access.min=-1) (delegation_tokens_issued.failures.min=-1) (stream_read_remote_stream_drain.failures.min=-1) (op_get_file_checksum.min=-1) (object_list_request.min=24) (op_mkdirs.min=590) (committer_stage_file_upload.failures.min=-1) (store_exists_probe.failures.min=-1) (op_exists.min=-1) (op_rename.failures.min=-1) (op_copy_from_local_file.failures.min=-1) (stream_write_queue_duration.failures.min=-1) (op_list_files.failures.min=-1) (action_executor_acquired.failures.min=-1) (object_bulk_delete_request.min=-1) (op_get_file_status.min=-1) (op_glob_status.failures.min=-1) (op_copy_from_local_file.min=-1) (op_exists.failures.min=-1) (op_abort.min=-1) (multipart_upload_list.failures.min=-1) (op_is_directory.failures.min=-1) (op_list_files.min=-1) (object_multipart_initiated.failures.min=-1) (stream_read_remote_stream_aborted.min=-1) (object_list_request.failures.min=-1) (committer_load_single_pending_file.failures.min=-1) (committer_commit_job.min=-1) (op_list_status.failures.min=-1) (object_put_request.failures.min=-1) (op_rename.min=-1) (op_is_file.min=-1) (committer_magic_marker_put.min=-1) (op_delete.min=32) (committer_commit_job.failures.min=-1) (op_get_delegation_token.min=-1) (op_createfile.failures.min=-1) (op_xattr_list.failures.min=-1) (op_create.min=29) (op_xattr_get_named_map.failures.min=-1) (object_continue_list_request.failures.min=-1) (action_file_opened.failures.min=-1) (store_exists_probe.min=-1) (op_is_file.failures.min=-1) (op_xattr_get_map.min=-1) (op_get_content_summary.min=-1) (object_put_request.min=-1) (delegation_tokens_issued.min=-1) (action_http_head_request.min=37) (stream_write_queue_duration.min=-1) (op_glob_status.min=-1) (op_abort.failures.min=-1) (action_http_get_request.failures.min=-1) (op_get_delegation_token.failures.min=-1) (stream_read_remote_stream_drain.min=-1) (op_get_file_checksum.failures.min=-1) (action_http_get_request.min=-1) (op_create.failures.min=-1) (multipart_upload_list.min=-1) (op_createfile.min=-1) (action_http_head_request.failures.min=-1) (op_xattr_get_named.min=-1) (object_delete_request.min=29) (stream_read_remote_stream_aborted.failures.min=-1) (op_access.failures.min=-1) (object_continue_list_request.min=-1) (op_xattr_get_named.failures.min=-1) (object_bulk_delete_request.failures.min=-1) (op_xattr_get_map.failures.min=-1) (committer_materialize_file.min=-1) (action_file_opened.min=-1) (committer_magic_marker_put.failures.min=-1));
maximums=((object_put_request.failures.max=-1) (op_exists.failures.max=-1) (op_glob_status.failures.max=-1) (committer_magic_marker_put.failures.max=-1) (action_file_opened.max=-1) (op_xattr_get_named_map.max=-1) (op_list_status.failures.max=-1) (op_xattr_list.max=-1) (op_xattr_get_map.max=-1) (op_get_delegation_token.failures.max=-1) (op_is_file.max=-1) (op_createfile.failures.max=-1) (op_rename.max=-1) (op_abort.failures.max=-1) (committer_commit_job.failures.max=-1) (stream_read_remote_stream_aborted.failures.max=-1) (op_glob_status.max=-1) (op_get_file_checksum.failures.max=-1) (object_continue_list_request.failures.max=-1) (op_list_status.max=-1) (op_list_files.max=-1) (stream_write_queue_duration.max=-1) (store_exists_probe.max=-1) (action_executor_acquired.failures.max=-1) (op_is_directory.max=-1) (op_delete.max=32) (op_get_file_checksum.max=-1) (object_multipart_aborted.max=-1) (op_xattr_get_named_map.failures.max=-1) (committer_materialize_file.failures.max=-1) (delegation_tokens_issued.failures.max=-1) (op_list_files.failures.max=-1) (multipart_upload_list.max=-1) (op_xattr_get_map.failures.max=-1) (committer_load_single_pending_file.max=-1) (op_xattr_list.failures.max=-1) (op_xattr_get_named.max=-1) (op_get_file_status.max=-1) (committer_stage_file_upload.max=-1) (action_http_get_request.max=-1) (op_get_delegation_token.max=-1) (op_copy_from_local_file.max=-1) (delegation_tokens_issued.max=-1) (object_list_request.max=587) (object_list_request.failures.max=-1) (op_copy_from_local_file.failures.max=-1) (op_get_content_summary.failures.max=-1) (multipart_upload_list.failures.max=-1) (object_multipart_initiated.max=62) (op_mkdirs.max=590) (action_file_opened.failures.max=-1) (committer_commit_job.max=-1) (object_multipart_initiated.failures.max=-1) (committer_materialize_file.max=-1) (op_create.failures.max=-1) (op_access.max=-1) (committer_magic_marker_put.max=-1) (store_exists_probe.failures.max=-1) (op_access.failures.max=-1) (stream_read_remote_stream_drain.max=-1) (object_bulk_delete_request.failures.max=-1) (action_http_get_request.failures.max=-1) (op_mkdirs.failures.max=-1) (op_delete.failures.max=-1) (object_continue_list_request.max=-1) (op_abort.max=-1) (committer_load_single_pending_file.failures.max=-1) (action_executor_acquired.max=7) (op_createfile.max=-1) (object_delete_request.failures.max=-1) (op_is_directory.failures.max=-1) (op_rename.failures.max=-1) (op_get_content_summary.max=-1) (action_http_head_request.failures.max=-1) (object_multipart_aborted.failures.max=-1) (object_bulk_delete_request.max=-1) (stream_read_remote_stream_aborted.max=-1) (object_delete_request.max=29) (stream_read_remote_stream_drain.failures.max=-1) (op_xattr_get_named.failures.max=-1) (committer_stage_file_upload.failures.max=-1) (stream_write_queue_duration.failures.max=-1) (op_get_file_status.failures.max=-1) (op_exists.max=-1) (object_put_request.max=-1) (action_http_head_request.max=37) (op_is_file.failures.max=-1) (op_create.max=29));
means=((op_xattr_get_named_map.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_http_head_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_get_file_status.mean=(samples=0, sum=0, mean=0.0000)) (delegation_tokens_issued.mean=(samples=0, sum=0, mean=0.0000)) (op_get_content_summary.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.mean=(samples=0, sum=0, mean=0.0000)) (stream_read_remote_stream_drain.mean=(samples=0, sum=0, mean=0.0000)) (object_bulk_delete_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (stream_read_remote_stream_aborted.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_is_file.mean=(samples=0, sum=0, mean=0.0000)) (action_http_get_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_exists.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_is_directory.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_get_file_checksum.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_list.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_createfile.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_get_delegation_token.mean=(samples=0, sum=0, mean=0.0000)) (action_http_get_request.mean=(samples=0, sum=0, mean=0.0000)) (committer_load_single_pending_file.mean=(samples=0, sum=0, mean=0.0000)) (op_access.mean=(samples=0, sum=0, mean=0.0000)) (committer_materialize_file.mean=(samples=0, sum=0, mean=0.0000)) (op_delete.failures.mean=(samples=0, sum=0, mean=0.0000)) (committer_stage_file_upload.mean=(samples=0, sum=0, mean=0.0000)) (op_mkdirs.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_initiated.failures.mean=(samples=0, sum=0, mean=0.0000)) (committer_magic_marker_put.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.mean=(samples=0, sum=0, mean=0.0000)) (op_copy_from_local_file.mean=(samples=0, sum=0, mean=0.0000)) (committer_commit_job.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_put_request.mean=(samples=0, sum=0, mean=0.0000)) (action_http_head_request.mean=(samples=1, sum=37, mean=37.0000)) (store_exists_probe.mean=(samples=0, sum=0, mean=0.0000)) (object_bulk_delete_request.mean=(samples=0, sum=0, mean=0.0000)) (op_list_status.mean=(samples=0, sum=0, mean=0.0000)) (object_put_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_list.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.mean=(samples=16, sum=10, mean=0.6250)) (committer_stage_file_upload.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_get_delegation_token.failures.mean=(samples=0, sum=0, mean=0.0000)) (delegation_tokens_issued.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_list_files.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_map.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_list_status.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_map.mean=(samples=0, sum=0, mean=0.0000)) (committer_commit_job.mean=(samples=0, sum=0, mean=0.0000)) (object_list_request.mean=(samples=3, sum=637, mean=212.3333)) (object_continue_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (stream_write_queue_duration.mean=(samples=0, sum=0, mean=0.0000)) (op_delete.mean=(samples=1, sum=32, mean=32.0000)) (op_glob_status.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_get_file_status.failures.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_list.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_list_files.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_rename.mean=(samples=0, sum=0, mean=0.0000)) (committer_materialize_file.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_delete_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (store_exists_probe.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_create.mean=(samples=1, sum=29, mean=29.0000)) (op_create.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_named_map.mean=(samples=0, sum=0, mean=0.0000)) (stream_write_queue_duration.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_initiated.mean=(samples=2, sum=122, mean=61.0000)) (op_glob_status.mean=(samples=0, sum=0, mean=0.0000)) (object_continue_list_request.mean=(samples=0, sum=0, mean=0.0000)) (op_get_content_summary.mean=(samples=0, sum=0, mean=0.0000)) (object_delete_request.mean=(samples=1, sum=29, mean=29.0000)) (op_xattr_list.failures.mean=(samples=0, sum=0, mean=0.0000)) (stream_read_remote_stream_drain.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_rename.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_copy_from_local_file.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_mkdirs.mean=(samples=1, sum=590, mean=590.0000)) (object_multipart_aborted.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_named.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_access.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_get_file_checksum.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_named.mean=(samples=0, sum=0, mean=0.0000)) (committer_magic_marker_put.mean=(samples=0, sum=0, mean=0.0000)) (op_is_file.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_file_opened.mean=(samples=0, sum=0, mean=0.0000)) (stream_read_remote_stream_aborted.mean=(samples=0, sum=0, mean=0.0000)) (action_file_opened.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_exists.mean=(samples=0, sum=0, mean=0.0000)) (committer_load_single_pending_file.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_createfile.mean=(samples=0, sum=0, mean=0.0000)) (op_is_directory.failures.mean=(samples=0, sum=0, mean=0.0000)));
] 
Expecting:
 <0L>
to be greater than:
 <0L> 

{code}
   

> AbstractSTestS3AHugeFiles put request assert is using wrong iostatistic
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-18611
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18611
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3, test
>    Affects Versions: 3.3.5
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>
> The huge file test case {{AbstractSTestS3AHugeFiles.test_010_CreateHugeFile()}} asserts that the put file count > 0. But this stat measures the count of HTTP PUT requests sent to S3; if a multipart PUT was used then the count is zero. As a result, this test case can fail on bigger files than the default.
> Fix. assert on OBJECT_PUT_REQUESTS_COMPLETED; this will be 1 for a simple PUT, and #of multipart blocks uploaded on a larger file. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org