You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "László Bodor (Jira)" <ji...@apache.org> on 2020/05/09 06:50:00 UTC
[jira] [Commented] (TEZ-4160) [Umbrella] Speed up unit tests
[ https://issues.apache.org/jira/browse/TEZ-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17103136#comment-17103136 ]
László Bodor commented on TEZ-4160:
-----------------------------------
run unit tests locally, results in [^mvntest.log]
{code}
grep "Tests run" mvntest.log | grep "org.apache" | awk '{split($0,a,":"); print a[6]}' | sort -n -r
{code}
so here is what should be in focus:
{code}
237.902 s - in org.apache.tez.test.TestSecureShuffle
211.856 s - in org.apache.tez.test.TestTezJobs
205.894 s - in org.apache.tez.test.TestAMRecovery
201.936 s - in org.apache.tez.test.TestFaultTolerance
143.215 s - in org.apache.tez.mapreduce.TestMRRJobsDAGApi
126.698 s - in org.apache.tez.auxservices.TestShuffleHandlerJobs
126.051 s - in org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
112.679 s - in org.apache.tez.tests.TestExternalTezServicesErrors
111.228 s - in org.apache.tez.analyzer.TestAnalyzer
99.141 s - in org.apache.tez.history.TestHistoryParser
85.936 s - in org.apache.tez.mapreduce.TestMRRJobs
72.701 s - in org.apache.tez.test.TestDAGRecovery
72.549 s - in org.apache.tez.dag.history.ats.acls.TestATSHistoryWithACLs
63.041 s - in org.apache.tez.tests.TestExternalTezServices
59.068 s - in org.apache.tez.runtime.library.common.sort.impl.TestPipelinedSorter
58.179 s - in org.apache.tez.dag.history.ats.acls.TestATSHistoryV15
54.081 s - in org.apache.tez.test.TestDAGRecovery2
46.971 s - in org.apache.tez.test.TestPipelinedShuffle
45.966 s - in org.apache.tez.test.TestLocalMode
44.721 s - in org.apache.tez.dag.history.logging.ats.TestATSHistoryWithMiniCluster
44.116 s - in org.apache.tez.test.TestExceptionPropagation
42.946 s - in org.apache.tez.client.TestTezClient
41.207 s - in org.apache.tez.dag.history.recovery.TestRecoveryService
18.657 s - in org.apache.tez.dag.history.logging.ats.TestATSHistoryLoggingService
14.601 s - in org.apache.tez.runtime.library.common.shuffle.impl.TestShuffleManager
12.898 s - in org.apache.tez.dag.api.client.rpc.TestDAGClientAMProtocolBlockingPBServerImpl
12.088 s - in org.apache.tez.dag.app.dag.impl.TestDAGImpl
11.628 s - in org.apache.tez.http.TestHttpConnection
11.202 s - in org.apache.tez.tests.TestExtServicesWithLocalMode
11.132 s - in org.apache.tez.runtime.library.common.sort.impl.TestTezMerger
8.518 s - in org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestShuffleScheduler
8.335 s - in org.apache.tez.dag.app.TestSpeculation
7.967 s - in org.apache.tez.dag.app.TestRecoveryParser
6.987 s - in org.apache.tez.dag.app.TestMockDAGAppMaster
6.56 s - in org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestMergeManager
6.273 s - in org.apache.tez.dag.app.rm.TestContainerReuse
6.218 s - in org.apache.tez.dag.app.rm.TestTaskScheduler
6.182 s - in org.apache.tez.common.TestTezCommonUtils
6.16 s - in org.apache.tez.dag.api.client.rpc.TestDAGClient
6.006 s - in org.apache.tez.mapreduce.processor.map.TestMapProcessor
5.826 s - in org.apache.tez.dag.app.dag.impl.TestCommit
5.149 s - in org.apache.tez.mapreduce.hadoop.TestMRInputHelpers
4.986 s - in org.apache.tez.test.TestMiniTezCluster
4.638 s - in org.apache.tez.runtime.library.common.TestValuesIterator
4.206 s - in org.apache.tez.test.TestTaskErrorsUsingLocalMode
3.945 s - in org.apache.tez.dag.app.dag.impl.TestVertexImpl
3.659 s - in org.apache.tez.dag.app.TestPreemption
3.387 s - in org.apache.tez.runtime.library.common.sort.impl.dflt.TestDefaultSorter
3.238 s - in org.apache.tez.dag.library.vertexmanager.TestFairShuffleVertexManager
3.164 s - in org.apache.tez.client.TestTezClientUtils
3.061 s - in org.apache.tez.runtime.library.output.TestOnFileSortedOutput
2.77 s - in org.apache.tez.dag.app.dag.impl.TestDAGRecovery
2.365 s - in org.apache.tez.runtime.library.common.sort.impl.TestIFile
2.362 s - in org.apache.tez.dag.app.TestDAGAppMaster
1.934 s - in org.apache.hadoop.mapred.split.TestGroupedSplits
1.817 s - in org.apache.tez.auxservices.TestShuffleHandler
1.625 s - in org.apache.tez.mapreduce.output.TestMultiMROutput
1.487 s - in org.apache.tez.mapreduce.processor.reduce.TestReduceProcessor
1.415 s - in org.apache.tez.dag.history.logging.ats.TestATSV15HistoryLoggingService
1.39 s - in org.apache.tez.mapreduce.common.TestMRInputAMSplitGenerator
1.349 s - in org.apache.tez.dag.history.logging.proto.TestProtoHistoryLoggingService
1.347 s - in org.apache.tez.dag.app.rm.TestTaskSchedulerManager
1.277 s - in org.apache.tez.mapreduce.input.TestMultiMRInput
1.269 s - in org.apache.tez.dag.app.dag.impl.TestTaskAttempt
1.243 s - in org.apache.tez.auxservices.TestIndexCache
1.242 s - in org.apache.tez.dag.history.logging.proto.TestDagManifestFileScanner
1.226 s - in org.apache.tez.dag.app.launcher.TestTezLocalCacheManager
1.223 s - in org.apache.tez.runtime.library.output.TestOnFileUnorderedKVOutput
1.191 s - in org.apache.tez.dag.app.rm.TestLocalTaskSchedulerService
1.159 s - in org.apache.tez.common.TestTezUtils
1.104 s - in org.apache.tez.mapreduce.output.TestMROutput
1.103 s - in org.apache.tez.runtime.TestInputReadyTracker
1.095 s - in org.apache.tez.mapreduce.output.TestMROutputLegacy
1.087 s - in org.apache.tez.dag.app.dag.impl.TestTaskImpl
1.053 s - in org.apache.tez.dag.app.TestTaskCommunicatorManager1
1.042 s - in org.apache.tez.runtime.task.TestTaskExecution2
0.99 s - in org.apache.tez.common.TestTezSharedExecutor
0.984 s - in org.apache.tez.runtime.library.common.shuffle.impl.TestShuffleInputEventHandlerImpl
0.979 s - in org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestShuffle
0.97 s - in org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestShuffleInputEventHandlerOrderedGrouped
0.969 s - in org.apache.tez.dag.app.rm.TestDagAwareYarnTaskScheduler
0.923 s - in org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestFetcher
0.91 s - in org.apache.tez.common.security.TestTokenCache
0.908 s - in org.apache.tez.runtime.library.common.shuffle.TestShuffleUtils
0.908 s - in org.apache.tez.dag.app.web.TestAMWebController
0.902 s - in org.apache.tez.runtime.library.output.TestOrderedPartitionedKVOutput2
0.883 s - in org.apache.tez.dag.library.vertexmanager.TestShuffleVertexManagerBase
0.857 s - in org.apache.tez.dag.api.client.TestDAGClientServer
0.846 s - in org.apache.tez.dag.app.dag.impl.TestVertexImpl2
0.841 s - in org.apache.tez.mapreduce.input.TestMRInput
0.818 s - in org.apache.tez.runtime.library.common.readers.TestUnorderedKVReader
0.801 s - in org.apache.tez.dag.api.client.TestATSHttpClient
0.797 s - in org.apache.tez.mapreduce.combine.TestMRCombiner
0.791 s - in org.apache.tez.dag.app.rm.container.TestAMContainer
0.787 s - in org.apache.tez.runtime.library.output.TestUnorderedKVOutput2
0.783 s - in org.apache.tez.runtime.task.TestTaskReporter
0.762 s - in org.apache.tez.dag.app.dag.impl.TestVertexManager
0.744 s - in org.apache.tez.runtime.library.common.shuffle.TestFetcher
0.74 s - in org.apache.tez.dag.app.rm.container.TestAMContainerMap
0.712 s - in org.apache.tez.mapreduce.output.TestMROutputConfigBuilder
0.71 s - in org.apache.tez.dag.app.TestTaskCommunicatorManager2
0.703 s - in org.apache.tez.runtime.common.resources.TestWeightedScalingMemoryDistributor
0.682 s - in org.apache.tez.dag.app.dag.impl.TestRootInputVertexManager
0.681 s - in org.apache.tez.dag.app.rm.node.TestAMNodeTracker
0.673 s - in org.apache.tez.dag.history.TestHistoryEventHandler
0.668 s - in org.apache.tez.mapreduce.lib.TestKVReadersWithMR
0.658 s - in org.apache.tez.dag.api.client.TestDAGClientHandler
0.646 s - in org.apache.tez.runtime.library.conf.TestOrderedPartitionedKVEdgeConfig
0.629 s - in org.apache.tez.runtime.library.output.TestUnorderedPartitionedKVOutput2
0.629 s - in org.apache.tez.runtime.TestLogicalIOProcessorRuntimeTask
0.621 s - in org.apache.tez.dag.history.logging.proto.TestHistoryEventProtoConverter
0.614 s - in org.apache.tez.runtime.library.cartesianproduct.TestFairCartesianProductVertexManager
0.614 s - in org.apache.tez.dag.api.client.TestTimelineReaderFactory
0.608 s - in org.apache.tez.dag.api.TestDAGVerify
0.604 s - in org.apache.tez.dag.library.vertexmanager.TestShuffleVertexManager
0.6 s - in org.apache.tez.dag.app.dag.impl.TestEdge
0.59 s - in org.apache.tez.dag.app.TestTaskCommunicatorManager
0.59 s - in org.apache.tez.common.TestReflectionUtils
0.589 s - in org.apache.tez.runtime.library.conf.TestUnorderedPartitionedKVEdgeConfig
0.589 s - in org.apache.tez.runtime.library.common.shuffle.impl.TestSimpleFetchedInputAllocator
0.587 s - in org.apache.tez.dag.app.rm.TestLocalTaskScheduler
0.583 s - in org.apache.tez.dag.history.events.TestHistoryEventsProtoConversion
0.578 s - in org.apache.tez.runtime.library.input.TestOrderedGroupedKVInput
0.576 s - in org.apache.tez.runtime.library.conf.TestUnorderedKVEdgeConfig
0.564 s - in org.apache.tez.runtime.common.resources.TestMemoryDistributor
0.553 s - in org.apache.tez.dag.app.dag.TestRootInputInitializerManager
0.552 s - in org.apache.tez.dag.api.TestDAG
0.549 s - in org.apache.tez.runtime.api.impl.TestProcessorContext
0.549 s - in org.apache.tez.dag.app.rm.TestTezAMRMClient
0.549 s - in org.apache.tez.dag.app.dag.impl.TestDAGSchedulerNaturalOrderControlled
0.549 s - in org.apache.tez.dag.api.TestDAGPlan
0.535 s - in org.apache.tez.dag.history.utils.TestDAGUtils
0.534 s - in org.apache.tez.dag.app.launcher.TestContainerLauncherManager
0.531 s - in org.apache.tez.dag.library.vertexmanager.TestVertexManagerWithConcurrentInput
0.526 s - in org.apache.tez.runtime.task.TestTezTaskRunner2
0.515 s - in org.apache.tez.dag.history.logging.ats.TestTimelineCachePluginImpl
0.514 s - in org.apache.tez.runtime.library.cartesianproduct.TestCartesianProductVertexManager
0.499 s - in org.apache.tez.dag.history.logging.ats.TestHistoryEventTimelineConversion
0.486 s - in org.apache.tez.common.security.TestACLManager
0.479 s - in org.apache.tez.mapreduce.hadoop.TestMRHelpers
0.469 s - in org.apache.tez.mapreduce.common.TestMRInputSplitDistributor
0.463 s - in org.apache.tez.runtime.library.conf.TestOrderedGroupedMergedKVInputConfig
0.46 s - in org.apache.tez.dag.app.dag.TestStateChangeNotifier
0.448 s - in org.apache.tez.runtime.library.conf.TestOrderedPartitionedKVOutputConfig
0.445 s - in org.apache.tez.runtime.library.conf.TestUnorderedPartitionedKVOutputConfig
0.428 s - in org.apache.tez.dag.app.TestTaskCommunicatorContextImpl
0.426 s - in org.apache.tez.dag.app.dag.impl.TestDAGScheduler
0.425 s - in org.apache.tez.dag.app.launcher.TestDeletionTracker
0.424 s - in org.apache.tez.dag.app.dag.app.TestTezTaskCommunicatorManager
0.417 s - in org.apache.tez.dag.api.TestEntityDescriptor
0.413 s - in org.apache.tez.dag.app.rm.TestTaskSchedulerWrapper
0.405 s - in org.apache.tez.runtime.library.conf.TestUnorderedKVInputConfig
0.405 s - in org.apache.tez.runtime.library.cartesianproduct.TestCartesianProductVertexManagerPartitioned
0.394 s - in org.apache.tez.runtime.library.conf.TestUnorderedKVOutputConfig
0.393 s - in org.apache.tez.runtime.library.cartesianproduct.TestCartesianProductConfig
0.388 s - in org.apache.tez.mapreduce.hadoop.TestConfigTranslationMRToTez
0.382 s - in org.apache.tez.mapreduce.hadoop.TestDeprecatedKeys
0.379 s - in org.apache.tez.dag.api.TestTezConfiguration
0.357 s - in org.apache.tez.util.TestTezMxBeanResourceCalculator
0.329 s - in org.apache.tez.common.TestTezYARNUtils
0.321 s - in org.apache.tez.dag.utils.TestTaskSpecificLaunchCmdOption
0.32 s - in org.apache.tez.runtime.api.impl.TestTezEvent
0.308 s - in org.apache.tez.dag.history.logging.impl.TestHistoryEventJsonConversion
0.301 s - in org.apache.tez.dag.app.TestTaskCommunicatorWrapper
0.298 s - in org.apache.tez.dag.library.vertexmanager.TestInputReadyVertexManager
0.294 s - in org.apache.tez.dag.api.TestDagTypeConverters
0.282 s - in org.apache.tez.common.TestEnvironmentUpdateUtils
0.276 s - in org.apache.tez.runtime.library.input.TestSortedGroupedMergedInput
0.27 s - in org.apache.tez.dag.app.launcher.TestContainerLauncherWrapper
0.267 s - in org.apache.tez.dag.app.dag.impl.TestImmediateStartVertexManager
0.263 s - in org.apache.tez.runtime.task.TestContainerExecution
0.259 s - in org.apache.tez.runtime.api.impl.TestTaskSpec
0.257 s - in org.apache.tez.dag.app.dag.impl.TestVertexStats
0.244 s - in org.apache.tez.dag.records.TestTezIds
0.228 s - in org.apache.tez.runtime.library.api.TestTezRuntimeConfiguration
0.205 s - in org.apache.tez.runtime.library.cartesianproduct.TestCartesianProductEdgeManagerPartitioned
0.196 s - in org.apache.tez.runtime.library.cartesianproduct.TestFairCartesianProductEdgeManager
0.191 s - in org.apache.tez.runtime.library.cartesianproduct.TestCartesianProductEdgeManager
0.188 s - in org.apache.tez.common.TestAsyncDispatcherConcurrent
0.178 s - in org.apache.tez.common.security.TestDAGAccessControls
0.173 s - in org.apache.tez.common.TestAsyncDispatcher
0.154 s - in org.apache.tez.runtime.library.common.comparator.TestProxyComparator
0.152 s - in org.apache.tez.hadoop.shim.TestHadoopShim28Provider
0.152 s - in org.apache.tez.common.security.TestACLConfigurationParser
0.137 s - in org.apache.tez.hadoop.shim.TestHadoopShimsLoader
0.126 s - in org.apache.tez.dag.utils.TestSimple2LevelVersionComparator
0.124 s - in org.apache.tez.common.TestVersionInfo
0.123 s - in org.apache.tez.dag.api.TestHistoryLogLevel
0.122 s - in org.apache.tez.common.TestJavaOptsChecker
0.091 s - in org.apache.tez.dag.app.TestMemoryWithEvents
0.07 s - in org.apache.tez.dag.api.TestTaskLocationHint
0.067 s - in org.apache.tez.common.TestRPCUtil
0.066 s - in org.apache.tez.util.TestNumberFormat
0.062 s - in org.apache.tez.runtime.api.event.TestCompositeDataMovementEvent
0.061 s - in org.apache.tez.runtime.library.cartesianproduct.TestCartesianProductCombination
0.061 s - in org.apache.tez.dag.app.dag.speculation.legacy.TestDataStatistics
0.06 s - in org.apache.tez.dag.api.client.TestVertexStatusBuilder
0.059 s - in org.apache.tez.dag.history.TestHistoryEventType
0.058 s - in org.apache.tez.runtime.common.objectregistry.TestObjectRegistry
0.055 s - in org.apache.tez.util.TestStopWatch
0.052 s - in org.apache.tez.runtime.library.cartesianproduct.TestGrouper
0.047 s - in org.apache.tez.runtime.library.common.TestInputIdentifiers
0.044 s - in org.apache.tez.hadoop.shim.TestHadoopShim28
{code}
> [Umbrella] Speed up unit tests
> ------------------------------
>
> Key: TEZ-4160
> URL: https://issues.apache.org/jira/browse/TEZ-4160
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Jonathan Turner Eagles
> Priority: Major
> Attachments: mvntest.log
>
>
> Umbrella ticket to document and speed up unit tests in tez. This is an umbrella ticket and shouldn't be assigned to anyone. Add subtasks to track individual test classes, packages, or individual tests so they can be worked on in small pieces.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)