You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Ahmed Hussein (Jira)" <ji...@apache.org> on 2020/07/17 16:11:00 UTC
[jira] [Commented] (HADOOP-17100) Replace Guava Supplier with
Java8+ Supplier in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17160029#comment-17160029 ]
Ahmed Hussein commented on HADOOP-17100:
----------------------------------------
[^HADOOP-17100.006.patch] submission results looks okay. I reviewed the failing unit tests.
The patch did not break any test and the failures either exist on trunk, or caused by existing broken tests.
|TestDistributedShell|Test is broken on trunk. see YARN-10334|Broken|
|TestRMHATimelineCollectors|Test is broken on trunk. (YARN-10337)|Broken|
|TestHDFSContractMultipartUploader|Broken on Trunk. HDFS-15471.|Broken|
|TestFixKerberosTicketOrder|intermittently fails on trunk |Flaky|
|TestGetFileChecksum|for HADOOP-17101, Filed HDFS-1546. an old jira exist HDFS-4723|Flaky|
|TestGroupsCaching|intermittently fails due to other UT failures|Flaky|
|TestNameNodeRetryCacheMetrics|filed a jira HDFS-1548|Flaky|
|TestRaceWhenRelogin|intermittently fails due to other UT failures|Flaky|
|TestRollingUpgrade|intermittently fails due to other UT failures |Flaky|
|TestBalancerWithHANameNodes|intermittently fails due to other UT failures |Flaky|
|TestDelegationTokenRenewer|intermittently fails due to other UT failures|Flaky|
cc [~weichiu], can you please take a look at the patch and merge into 3.2.2, 3.3.1, 3.4.0, 3.1.5?
The patch is very straightforward, changing import line in the files.
> Replace Guava Supplier with Java8+ Supplier in Hadoop
> -----------------------------------------------------
>
> Key: HADOOP-17100
> URL: https://issues.apache.org/jira/browse/HADOOP-17100
> Project: Hadoop Common
> Issue Type: Sub-task
> Reporter: Ahmed Hussein
> Assignee: Ahmed Hussein
> Priority: Major
> Attachments: HADOOP-17100.001.patch, HADOOP-17100.002.patch, HADOOP-17100.003.patch, HADOOP-17100.006.patch
>
>
> Replacing Usage of {{guava.Supplier<>}} are in Unit tests {{GenereicTestUtils.waitFor()}} in Hadoop project.
> * To make things more convenient for reviewers, I decided:
> ** Not to replace Object instantiation by lambda expressions because this will increase the patch size significantly and require code adjustments to pass the checkstyle scripts.
> ** Not to refactor the imports because this will make reading the patch more difficult.
> * Merge should be done to the following branches: 3.2.2, 3.3.1, 3.4.0, trunk
> The task is straightforward because {{java.util.Supplier}} has the same API as {{guava.Supplier<>}} and the vast majority of usage comes from Test-units.
> Therefore, we need only to do the following a "one-line" change in all 147 files.
> {code:bash}
>
> -import com.google.common.base.Supplier;
> +import java.util.function.Supplier;
> {code}
> The code change needs to be applied to the following list of files:
> {code:java}
>
> Targets
> Occurrences of 'com.google.common.base.Supplier' in project with mask '*.java'
> Found Occurrences (146 usages found)
> org.apache.hadoop.conf (1 usage found)
> TestReconfiguration.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> org.apache.hadoop.crypto.key.kms.server (1 usage found)
> TestKMS.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> org.apache.hadoop.fs (2 usages found)
> FCStatisticsBaseTest.java (1 usage found)
> 40 import com.google.common.base.Supplier;
> TestEnhancedByteBufferAccess.java (1 usage found)
> 75 import com.google.common.base.Supplier;
> org.apache.hadoop.fs.viewfs (1 usage found)
> TestViewFileSystemWithTruncate.java (1 usage found)
> 23 import com.google.common.base.Supplier;
> org.apache.hadoop.ha (1 usage found)
> TestZKFailoverController.java (1 usage found)
> 25 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs (20 usages found)
> DFSTestUtil.java (1 usage found)
> 79 import com.google.common.base.Supplier;
> MiniDFSCluster.java (1 usage found)
> 78 import com.google.common.base.Supplier;
> TestBalancerBandwidth.java (1 usage found)
> 29 import com.google.common.base.Supplier;
> TestClientProtocolForPipelineRecovery.java (1 usage found)
> 30 import com.google.common.base.Supplier;
> TestDatanodeRegistration.java (1 usage found)
> 44 import com.google.common.base.Supplier;
> TestDataTransferKeepalive.java (1 usage found)
> 47 import com.google.common.base.Supplier;
> TestDeadNodeDetection.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestDecommission.java (1 usage found)
> 41 import com.google.common.base.Supplier;
> TestDFSShell.java (1 usage found)
> 37 import com.google.common.base.Supplier;
> TestEncryptedTransfer.java (1 usage found)
> 35 import com.google.common.base.Supplier;
> TestEncryptionZonesWithKMS.java (1 usage found)
> 22 import com.google.common.base.Supplier;
> TestFileCorruption.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> TestLeaseRecovery2.java (1 usage found)
> 32 import com.google.common.base.Supplier;
> TestLeaseRecoveryStriped.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> TestMaintenanceState.java (1 usage found)
> 63 import com.google.common.base.Supplier;
> TestPread.java (1 usage found)
> 61 import com.google.common.base.Supplier;
> TestQuota.java (1 usage found)
> 39 import com.google.common.base.Supplier;
> TestReplaceDatanodeOnFailure.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestReplication.java (1 usage found)
> 27 import com.google.common.base.Supplier;
> TestSafeMode.java (1 usage found)
> 62 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.client.impl (2 usages found)
> TestBlockReaderLocalMetrics.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestLeaseRenewer.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.qjournal (1 usage found)
> MiniJournalCluster.java (1 usage found)
> 31 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.qjournal.client (1 usage found)
> TestIPCLoggerChannel.java (1 usage found)
> 43 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.qjournal.server (1 usage found)
> TestJournalNodeSync.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.blockmanagement (7 usages found)
> TestBlockManagerSafeMode.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestBlockReportRateLimiting.java (1 usage found)
> 25 import com.google.common.base.Supplier;
> TestNameNodePrunesMissingStorages.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> TestPendingInvalidateBlock.java (1 usage found)
> 43 import com.google.common.base.Supplier;
> TestPendingReconstruction.java (1 usage found)
> 34 import com.google.common.base.Supplier;
> TestRBWBlockInvalidation.java (1 usage found)
> 49 import com.google.common.base.Supplier;
> TestSlowDiskTracker.java (1 usage found)
> 48 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.datanode (13 usages found)
> DataNodeTestUtils.java (1 usage found)
> 40 import com.google.common.base.Supplier;
> TestBlockRecovery.java (1 usage found)
> 120 import com.google.common.base.Supplier;
> TestBlockScanner.java (1 usage found)
> 43 import com.google.common.base.Supplier;
> TestBPOfferService.java (1 usage found)
> 92 import com.google.common.base.Supplier;
> TestCorruptMetadataFile.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestDataNodeLifeline.java (1 usage found)
> 74 import com.google.common.base.Supplier;
> TestDataNodeMetrics.java (1 usage found)
> 37 import com.google.common.base.Supplier;
> TestDataNodeMetricsLogger.java (1 usage found)
> 57 import com.google.common.base.Supplier;
> TestDataNodeMultipleRegistrations.java (1 usage found)
> 33 import com.google.common.base.Supplier;
> TestDataNodeMXBean.java (1 usage found)
> 31 import com.google.common.base.Supplier;
> TestDatanodeProtocolRetryPolicy.java (1 usage found)
> 32 import com.google.common.base.Supplier;
> TestDataNodeVolumeFailure.java (1 usage found)
> 94 import com.google.common.base.Supplier;
> TestDiskError.java (1 usage found)
> 31 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.datanode.checker (1 usage found)
> TestThrottledAsyncChecker.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl (7 usages found)
> TestCacheByPmemMappableBlockLoader.java (1 usage found)
> 65 import com.google.common.base.Supplier;
> TestFsDatasetCache.java (1 usage found)
> 94 import com.google.common.base.Supplier;
> TestFsDatasetImpl.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestFsVolumeList.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestLazyPersistLockedMemory.java (1 usage found)
> 22 import com.google.common.base.Supplier;
> TestPmemCacheRecovery.java (1 usage found)
> 65 import com.google.common.base.Supplier;
> TestSpaceReservation.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.datanode.metrics (1 usage found)
> TestDataNodeOutlierDetectionViaMetrics.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs (1 usage found)
> TestDataNodeUGIProvider.java (1 usage found)
> 54 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.diskbalancer (2 usages found)
> TestDiskBalancer.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestDiskBalancerWithMockMover.java (1 usage found)
> 23 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.federation (1 usage found)
> FederationTestUtils.java (1 usage found)
> 95 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.federation.router (4 usages found)
> TestRouterAdminCLI.java (1 usage found)
> 67 import com.google.common.base.Supplier;
> TestRouterQuota.java (1 usage found)
> 77 import com.google.common.base.Supplier;
> TestRouterRpc.java (1 usage found)
> 131 import com.google.common.base.Supplier;
> TestRouterRPCClientRetries.java (1 usage found)
> 57 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.mover (1 usage found)
> TestMover.java (1 usage found)
> 98 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.namenode (17 usages found)
> TestAddStripedBlockInFBR.java (1 usage found)
> 43 import com.google.common.base.Supplier;
> TestBackupNode.java (1 usage found)
> 59 import com.google.common.base.Supplier;
> TestCacheDirectives.java (1 usage found)
> 99 import com.google.common.base.Supplier;
> TestCheckpoint.java (1 usage found)
> 98 import com.google.common.base.Supplier;
> TestDeadDatanode.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestEditLogAutoroll.java (1 usage found)
> 49 import com.google.common.base.Supplier;
> TestEditLogRace.java (1 usage found)
> 41 import com.google.common.base.Supplier;
> TestFsck.java (1 usage found)
> 62 import com.google.common.base.Supplier;
> TestFSNamesystemLock.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> TestMetaSave.java (1 usage found)
> 33 import com.google.common.base.Supplier;
> TestNameNodeMetadataConsistency.java (1 usage found)
> 33 import com.google.common.base.Supplier;
> TestNameNodeMetricsLogger.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> TestNameNodeMXBean.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> TestNameNodeStatusMXBean.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestPersistentStoragePolicySatisfier.java (1 usage found)
> 40 import com.google.common.base.Supplier;
> TestReencryption.java (1 usage found)
> 34 import com.google.common.base.Supplier;
> TestUpgradeDomainBlockPlacementPolicy.java (1 usage found)
> 52 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.namenode.ha (11 usages found)
> HATestUtil.java (1 usage found)
> 64 import com.google.common.base.Supplier;
> TestBootstrapStandby.java (1 usage found)
> 30 import com.google.common.base.Supplier;
> TestDNFencing.java (1 usage found)
> 32 import com.google.common.base.Supplier;
> TestDNFencingWithReplication.java (1 usage found)
> 39 import com.google.common.base.Supplier;
> TestEditLogTailer.java (1 usage found)
> 65 import com.google.common.base.Supplier;
> TestHASafeMode.java (1 usage found)
> 75 import com.google.common.base.Supplier;
> TestPendingCorruptDnMessages.java (1 usage found)
> 44 import com.google.common.base.Supplier;
> TestPipelinesFailover.java (1 usage found)
> 65 import com.google.common.base.Supplier;
> TestStandbyCheckpoints.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> TestStandbyInProgressTail.java (1 usage found)
> 52 import com.google.common.base.Supplier;
> TestStandbyIsHot.java (1 usage found)
> 46 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.namenode.snapshot (1 usage found)
> TestRandomOpsWithSnapshots.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.namenode.sps (1 usage found)
> TestStoragePolicySatisfierWithStripedFile.java (1 usage found)
> 55 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.server.sps (1 usage found)
> TestExternalStoragePolicySatisfier.java (1 usage found)
> 101 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.shortcircuit (1 usage found)
> TestShortCircuitCache.java (1 usage found)
> 93 import com.google.common.base.Supplier;
> org.apache.hadoop.hdfs.tools (2 usages found)
> TestDFSAdmin.java (1 usage found)
> 28 import com.google.common.base.Supplier;
> TestDFSZKFailoverController.java (1 usage found)
> 58 import com.google.common.base.Supplier;
> org.apache.hadoop.http (1 usage found)
> TestSSLHttpServerConfigs.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> org.apache.hadoop.ipc (1 usage found)
> TestIPC.java (1 usage found)
> 101 import com.google.common.base.Supplier;
> org.apache.hadoop.mapred (2 usages found)
> TestTaskAttemptListenerImpl.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> UtilsForTests.java (1 usage found)
> 64 import com.google.common.base.Supplier;
> org.apache.hadoop.mapreduce.v2.app (4 usages found)
> TestFetchFailure.java (1 usage found)
> 29 import com.google.common.base.Supplier;
> TestMRApp.java (1 usage found)
> 31 import com.google.common.base.Supplier;
> TestRecovery.java (1 usage found)
> 31 import com.google.common.base.Supplier;
> TestTaskHeartbeatHandler.java (1 usage found)
> 28 import com.google.common.base.Supplier;
> org.apache.hadoop.mapreduce.v2.app.rm (1 usage found)
> TestRMContainerAllocator.java (1 usage found)
> 156 import com.google.common.base.Supplier;
> org.apache.hadoop.mapreduce.v2.hs (1 usage found)
> TestJHSDelegationTokenSecretManager.java (1 usage found)
> 30 import com.google.common.base.Supplier;
> org.apache.hadoop.metrics2.impl (1 usage found)
> TestMetricsSystemImpl.java (1 usage found)
> 42 import com.google.common.base.Supplier;
> org.apache.hadoop.metrics2.lib (1 usage found)
> TestMutableRollingAverages.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> org.apache.hadoop.security (1 usage found)
> TestGroupsCaching.java (1 usage found)
> 37 import com.google.common.base.Supplier;
> org.apache.hadoop.security.ssl (1 usage found)
> TestReloadingX509TrustManager.java (1 usage found)
> 24 import com.google.common.base.Supplier;
> org.apache.hadoop.security.token.delegation (1 usage found)
> TestZKDelegationTokenSecretManager.java (1 usage found)
> 27 import com.google.common.base.Supplier;
> org.apache.hadoop.test (2 usages found)
> GenericTestUtils.java (1 usage found)
> 64 import com.google.common.base.Supplier;
> TestGenericTestUtils.java (1 usage found)
> 26 import com.google.common.base.Supplier;
> org.apache.hadoop.tracing (1 usage found)
> SetSpanReceiver.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> org.apache.hadoop.util (1 usage found)
> TestShell.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.applications.distributedshell (1 usage found)
> TestDistributedShell.java (1 usage found)
> 43 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.client (1 usage found)
> TestRMFailover.java (1 usage found)
> 64 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.client.api.impl (1 usage found)
> TestYarnClientWithReservation.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.server.nodemanager.containermanager (1 usage found)
> TestContainerManager.java (1 usage found)
> 51 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher (1 usage found)
> TestContainerLaunch.java (1 usage found)
> 57 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer (1 usage found)
> TestContainerLocalizer.java (1 usage found)
> 97 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation (1 usage found)
> TestLogAggregationService.java (1 usage found)
> 150 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor (1 usage found)
> TestContainersMonitor.java (1 usage found)
> 40 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.server.nodemanager.logaggregation.tracker (1 usage found)
> TestNMLogAggregationStatusTracker.java (1 usage found)
> 24 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.server.resourcemanager (6 usages found)
> TestApplicationMasterLauncher.java (1 usage found)
> 95 import com.google.common.base.Supplier;
> TestLeaderElectorService.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> TestRM.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> TestRMHA.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> TestRMRestart.java (1 usage found)
> 137 import com.google.common.base.Supplier;
> TestWorkPreservingRMRestart.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.server.resourcemanager.recovery (1 usage found)
> TestZKRMStateStore.java (1 usage found)
> 75 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity (1 usage found)
> TestCapacityScheduler.java (1 usage found)
> 192 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair (1 usage found)
> TestContinuousScheduling.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.server.resourcemanager.security (2 usages found)
> TestDelegationTokenRenewer.java (1 usage found)
> 117 import com.google.common.base.Supplier;
> TestRMDelegationTokens.java (1 usage found)
> 29 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.server.router.webapp (1 usage found)
> TestRouterWebServicesREST.java (1 usage found)
> 135 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.server.webproxy.amfilter (1 usage found)
> TestAmFilter.java (1 usage found)
> 53 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.service (1 usage found)
> MockServiceAM.java (1 usage found)
> 21 import com.google.common.base.Supplier;
> org.apache.hadoop.yarn.sls.nodemanager (1 usage found)
> TestNMSimulator.java (1 usage found)
> 20 import com.google.common.base.Supplier;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org