You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2016/12/07 16:27:58 UTC
[jira] [Comment Edited] (HADOOP-13871)
ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks
performance on branch-2.8 awful
[ https://issues.apache.org/jira/browse/HADOOP-13871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729168#comment-15729168 ]
Steve Loughran edited comment on HADOOP-13871 at 12/7/16 4:27 PM:
------------------------------------------------------------------
rerun on 2.8, logs this time show a SocketTimeout -> restart and new https link, with the next attempt having the same numbers as the new SDK
{code}
12-07 16:09:18,492 [Thread-1] INFO contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:setup(184)) - Test filesystem = s3a://hwdev-steve-ireland-new implemented by S3AFileSystem{uri=s3a://hwdev-steve-ireland-new, workingDir=s3a://hwdev-steve-ireland-new/user/stevel, inputPolicy=normal, partSize=8000000, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647, blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@2c5b7418, executor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=30, available=30, waiting=0}, activeCount=0}, statistics {0 bytes read, 0 bytes written, 0 read ops, 0 large read ops, 0 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=be5a5251-b83f-441b-b43c-774586f7bbde-hwdev-steve-ireland-new} {fsURI=s3a://hwdev-steve-ireland-new/} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {fake_directories_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=0} {op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=0} {op_mkdirs=0} {op_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=0} {object_continue_list_requests=0} {object_metadata_requests=0} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {object_put_requests_completed=0} {stream_write_failures=0} {stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {stream_write_block_uploads_aborted=0} {stream_write_total_time=0} {stream_write_total_data=0} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=0} {stream_write_block_uploads_data_pending=0} {stream_read_fully_operations=0} {stream_opened=0} {stream_bytes_skipped_on_seek=0} {stream_closed=0} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=0} {stream_read_operations_incomplete=0} {stream_bytes_discarded_in_abort=0} {stream_close_operations=0} {stream_read_operations=0} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }}
2016-12-07 16:09:20,294 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO s3a.AbstractS3ATestBase (AbstractS3ATestBase.java:describe(87)) -
testTimeToOpenAndReadWholeFileBlocks: Open the test file s3a://landsat-pds/scene_list.gz and read it in blocks of size 1048576
2016-12-07 16:10:13,635 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 0 in 121 reads: 53,171,733,017 nS
2016-12-07 16:11:11,834 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 58,198,581,698 nS
2016-12-07 16:12:07,894 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 2 in 121 reads: 56,060,315,378 nS
2016-12-07 16:13:09,745 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 3 in 122 reads: 61,851,755,314 nS
2016-12-07 16:14:00,233 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 4 in 122 reads: 50,487,446,260 nS
2016-12-07 16:15:05,391 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 5 in 121 reads: 65,158,241,571 nS
2016-12-07 16:16:03,961 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 6 in 121 reads: 58,570,460,343 nS
2016-12-07 16:17:05,090 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 7 in 121 reads: 61,129,325,214 nS
2016-12-07 16:17:58,719 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 8 in 123 reads: 53,628,801,572 nS
2016-12-07 16:19:36,115 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 9 in 121 reads: 97,396,659,275 nS
2016-12-07 16:24:23,031 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO s3a.S3AFileSystem (S3AInputStream.java:onReadFailure(334)) - Got exception while trying to read from stream s3a://landsat-pds/scene_list.gz trying to recover: java.net.SocketTimeoutException: Read timed out
2016-12-07 16:24:24,569 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 10 in 120 reads: 288,457,023,727 nS
2016-12-07 16:24:25,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 11 in 121 reads: 892,929,231 nS
2016-12-07 16:24:25,965 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 12 in 123 reads: 501,503,328 nS
2016-12-07 16:24:26,308 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 13 in 121 reads: 341,840,365 nS
2016-12-07 16:24:26,556 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 14 in 121 reads: 247,359,257 nS
2016-12-07 16:24:26,794 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 15 in 86 reads: 238,286,855 nS
2016-12-07 16:24:27,001 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 16 in 69 reads: 206,087,428 nS
2016-12-07 16:24:27,190 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 17 in 69 reads: 188,955,262 nS
2016-12-07 16:24:27,385 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 18 in 69 reads: 194,649,152 nS
2016-12-07 16:24:27,563 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 19 in 69 reads: 177,754,120 nS
2016-12-07 16:24:27,738 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 20 in 69 reads: 173,799,688 nS
2016-12-07 16:24:27,913 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 21 in 69 reads: 175,191,414 nS
2016-12-07 16:24:28,087 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 22 in 69 reads: 173,933,288 nS
2016-12-07 16:24:28,264 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 23 in 69 reads: 175,654,086 nS
2016-12-07 16:24:28,437 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 24 in 69 reads: 173,505,198 nS
2016-12-07 16:24:28,612 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 25 in 69 reads: 173,746,842 nS
2016-12-07 16:24:28,785 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 26 in 69 reads: 173,089,830 nS
2016-12-07 16:24:28,958 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 27 in 69 reads: 172,631,269 nS
2016-12-07 16:24:29,133 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 28 in 69 reads: 174,324,505 nS
2016-12-07 16:24:29,306 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 29 in 69 reads: 172,987,396 nS
2016-12-07 16:24:29,482 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 30 in 69 reads: 175,804,772 nS
2016-12-07 16:24:29,658 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 31 in 69 reads: 174,854,033 nS
2016-12-07 16:24:29,835 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 32 in 69 reads: 177,342,092 nS
2016-12-07 16:24:30,010 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 33 in 69 reads: 174,078,949 nS
2016-12-07 16:24:30,184 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 34 in 69 reads: 173,946,606 nS
2016-12-07 16:24:30,358 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 35 in 69 reads: 173,489,121 nS
2016-12-07 16:24:30,533 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 36 in 69 reads: 174,367,264 nS
2016-12-07 16:24:30,533 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Time to read 38820130 bytes in 37 blocks: 910,083,350,959 nS
2016-12-07 16:24:30,535 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:bandwidth(963)) - Bandwidth = 0.040656 MB/S
2016-12-07 16:24:30,535 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO scale.ITestS3AInputStreamPerformance (ITestS3AInputStreamPerformance.java:logStreamStatistics(324)) - Stream Statistics
StreamStatistics{OpenOperations=2, CloseOperations=1, Closed=0, Aborted=1, SeekOperations=0, ReadExceptions=1, ForwardSeekOperations=0, BackwardSeekOperations=0, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=0, BytesRead=38797312, BytesRead excluding skipped=38797312, ReadOperations=3355, ReadFullyOperations=0, ReadsIncomplete=3318, BytesReadInClose=0, BytesDiscardedInAbort=27401974}
2016-12-07 16:24:30,535 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:describe(255)) - cleanup
2016-12-07 16:24:30,864 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:describe(255)) - closing file system
{code}
was (Author: stevel@apache.org):
rerun on 2.8, logs this time show a SocketTimeout -> restart and new https link, with the next attempt having the same numbers as the new SDK
{code}
testTimeToOpenAndReadWholeFileBlocks: Open the test file s3a://landsat-pds/scene_list.gz and read it in blocks of size 1048576
2016-12-07 16:10:13,635 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 0 in 121 reads: 53,171,733,017 nS
2016-12-07 16:11:11,834 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 58,198,581,698 nS
2016-12-07 16:12:07,894 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 2 in 121 reads: 56,060,315,378 nS
2016-12-07 16:13:09,745 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 3 in 122 reads: 61,851,755,314 nS
2016-12-07 16:14:00,233 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 4 in 122 reads: 50,487,446,260 nS
2016-12-07 16:15:05,391 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 5 in 121 reads: 65,158,241,571 nS
2016-12-07 16:16:03,961 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 6 in 121 reads: 58,570,460,343 nS
2016-12-07 16:17:05,090 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 7 in 121 reads: 61,129,325,214 nS
2016-12-07 16:17:58,719 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 8 in 123 reads: 53,628,801,572 nS
2016-12-07 16:19:36,115 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 9 in 121 reads: 97,396,659,275 nS
2016-12-07 16:24:23,031 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO s3a.S3AFileSystem (S3AInputStream.java:onReadFailure(334)) - Got exception while trying to read from stream s3a://landsat-pds/scene_list.gz trying to recover: java.net.SocketTimeoutException: Read timed out
2016-12-07 16:24:24,569 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 10 in 120 reads: 288,457,023,727 nS
2016-12-07 16:24:25,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 11 in 121 reads: 892,929,231 nS
2016-12-07 16:24:25,965 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 12 in 123 reads: 501,503,328 nS
2016-12-07 16:24:26,308 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 13 in 121 reads: 341,840,365 nS
2016-12-07 16:24:26,556 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 14 in 121 reads: 247,359,257 nS
2016-12-07 16:24:26,794 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 15 in 86 reads: 238,286,855 nS
2016-12-07 16:24:27,001 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 16 in 69 reads: 206,087,428 nS
2016-12-07 16:24:27,190 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 17 in 69 reads: 188,955,262 nS
2016-12-07 16:24:27,385 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 18 in 69 reads: 194,649,152 nS
2016-12-07 16:24:27,563 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 19 in 69 reads: 177,754,120 nS
2016-12-07 16:24:27,738 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 20 in 69 reads: 173,799,688 nS
2016-12-07 16:24:27,913 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 21 in 69 reads: 175,191,414 nS
2016-12-07 16:24:28,087 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 22 in 69 reads: 173,933,288 nS
2016-12-07 16:24:28,264 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 23 in 69 reads: 175,654,086 nS
2016-12-07 16:24:28,437 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 24 in 69 reads: 173,505,198 nS
2016-12-07 16:24:28,612 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 25 in 69 reads: 173,746,842 nS
2016-12-07 16:24:28,785 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 26 in 69 reads: 173,089,830 nS
2016-12-07 16:24:28,958 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 27 in 69 reads: 172,631,269 nS
2016-12-07 16:24:29,133 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 28 in 69 reads: 174,324,505 nS
2016-12-07 16:24:29,306 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 29 in 69 reads: 172,987,396 nS
2016-12-07 16:24:29,482 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 30 in 69 reads: 175,804,772 nS
2016-12-07 16:24:29,658 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 31 in 69 reads: 174,854,033 nS
{code}
> ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance on branch-2.8 awful
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-13871
> URL: https://issues.apache.org/jira/browse/HADOOP-13871
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.0
> Environment: landsat bucket from the UK
> Reporter: Steve Loughran
> Assignee: Steve Loughran
>
> The ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks takes 15s on branch-2, but is now taking minutes on branch-2.8.
> This is a regression, and it's surfacing on some internal branches too. Even where the code hasn't changed. It does not happen on branch-2, which has a later SDK.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org