You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/21 11:25:58 UTC
[GitHub] [spark] wangyum commented on pull request #31901: [SPARK-34802][SQL] Move simplify expression rules before operator push down
wangyum commented on pull request #31901:
URL: https://github.com/apache/spark/pull/31901#issuecomment-803558990
Before this pr:
```
19:23:58.074 WARN org.apache.spark.sql.TPCDSQuerySuite:
=== Metrics of Analyzer/Optimizer Rules ===
Total number of runs: 407441
Total time: 41.217085463 seconds
Rule Effective Time / Total Time Effective Runs / Total Runs
org.apache.spark.sql.catalyst.optimizer.Optimizer$OptimizeSubqueries 3459364804 / 7273617141 49 / 2778
org.apache.spark.sql.catalyst.optimizer.ColumnPruning 554550102 / 2852279204 338 / 6402
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions 830781928 / 2459212474 49 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveSubquery 1917946262 / 2067630600 51 / 2141
org.apache.spark.sql.catalyst.analysis.DecimalPrecision 1456077159 / 1681591081 361 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences 741392690 / 902859224 815 / 2141
org.apache.spark.sql.catalyst.optimizer.PruneFilters 21731791 / 866239104 5 / 5013
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$ImplicitTypeCasts 330274264 / 782021776 78 / 2141
org.apache.spark.sql.catalyst.optimizer.InferFiltersFromConstraints 644004888 / 776597471 284 / 1389
org.apache.spark.sql.catalyst.optimizer.ReorderJoin 647382677 / 729812324 177 / 3624
org.apache.spark.sql.catalyst.analysis.UpdateAttributeNullability 10876695 / 693912132 12 / 3448
org.apache.spark.sql.catalyst.optimizer.BooleanSimplification 14626317 / 658650063 16 / 3624
org.apache.spark.sql.execution.datasources.PruneFileSourcePartitions 567434042 / 610800777 209 / 1389
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveMissingReferences 11282754 / 603081344 10 / 2141
org.apache.spark.sql.catalyst.optimizer.PushDownPredicates 447678284 / 580794636 1002 / 6635
org.apache.spark.sql.catalyst.optimizer.RemoveRedundantAliases 21103967 / 522693682 29 / 3624
org.apache.spark.sql.catalyst.optimizer.OptimizeCsvJsonExprs 0 / 452119884 0 / 3624
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$FunctionArgumentConversion 236070931 / 447828356 22 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$WidenSetOperationTypes 291743240 / 426849131 24 / 2141
org.apache.spark.sql.execution.datasources.SchemaPruning 0 / 392620647 0 / 1389
org.apache.spark.sql.catalyst.optimizer.OptimizeUpdateFields 0 / 371380231 0 / 4359
org.apache.spark.sql.catalyst.optimizer.NullPropagation 25185090 / 364626484 60 / 3624
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$CaseWhenCoercion 148082198 / 355006700 33 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ExtractGenerator 0 / 345835417 0 / 2141
org.apache.spark.sql.catalyst.optimizer.FoldablePropagation 2839171 / 321384305 6 / 3624
org.apache.spark.sql.catalyst.optimizer.RewriteCorrelatedScalarSubquery 5107341 / 309518387 8 / 3624
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions 136245772 / 299997876 426 / 2141
org.apache.spark.sql.catalyst.optimizer.SimplifyBinaryComparison 255511 / 276737636 1 / 3624
org.apache.spark.sql.catalyst.analysis.ResolveSessionCatalog 155606665 / 275166367 24 / 2141
org.apache.spark.sql.catalyst.optimizer.SimplifyConditionals 0 / 257901787 0 / 3624
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$DateTimeOperations 0 / 257026064 0 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$IntegralDivision 0 / 252694314 0 / 2141
org.apache.spark.sql.catalyst.optimizer.PushFoldableIntoBranches 6912323 / 251903640 19 / 3624
org.apache.spark.sql.catalyst.optimizer.SimplifyExtractValueOps 0 / 243932140 0 / 3624
org.apache.spark.sql.catalyst.optimizer.ConstantFolding 69263491 / 239972524 221 / 3624
org.apache.spark.sql.catalyst.analysis.TypeCoercion$PromoteStrings 8767486 / 239312714 11 / 2141
org.apache.spark.sql.catalyst.optimizer.UnwrapCastInBinaryComparison 4464067 / 232861362 8 / 3624
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$Division 18825655 / 230242583 10 / 2141
org.apache.spark.sql.catalyst.optimizer.ReorderAssociativeOperator 0 / 227408914 0 / 3624
org.apache.spark.sql.catalyst.analysis.ResolveTimeZone 108526256 / 227205992 537 / 2141
org.apache.spark.sql.catalyst.optimizer.SimplifyCaseConversionExpressions 0 / 225276712 0 / 3624
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$InConversion 3227516 / 224466926 4 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveWindowFrame 12058194 / 219537686 37 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercion$BooleanEquality 0 / 215863072 0 / 2141
org.apache.spark.sql.catalyst.optimizer.RemoveDispensableExpressions 0 / 209234062 0 / 3624
org.apache.spark.sql.execution.datasources.FindDataSourceTable 177261972 / 208675610 296 / 2141
org.apache.spark.sql.catalyst.analysis.CTESubstitution 44178100 / 207656236 58 / 735
org.apache.spark.sql.catalyst.analysis.ResolveCreateNamedStruct 0 / 205571016 0 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$IfCoercion 0 / 202086288 0 / 2141
org.apache.spark.sql.catalyst.optimizer.OptimizeIn 9926506 / 200067451 27 / 3624
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$StringLiteralCoercion 0 / 200004878 0 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$EltCoercion 0 / 192989682 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveBinaryArithmetic 5938308 / 192018980 25 / 2141
org.apache.spark.sql.catalyst.optimizer.SimplifyCasts 19440256 / 189351500 83 / 3624
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$ConcatCoercion 0 / 185365923 0 / 2141
org.apache.spark.sql.catalyst.optimizer.LikeSimplification 674471 / 177286705 1 / 3624
org.apache.spark.sql.catalyst.optimizer.ReplaceNullWithFalseInPredicate 2842551 / 177044278 17 / 3624
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations 118491086 / 176494105 296 / 2151
org.apache.spark.sql.catalyst.analysis.PullOutNondeterministic 0 / 168290835 0 / 665
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveWindowOrder 9828199 / 167493133 28 / 2141
org.apache.spark.sql.catalyst.analysis.TimeWindowing 0 / 162946057 0 / 2141
org.apache.spark.sql.catalyst.optimizer.OptimizeWindowFunctions 0 / 162401843 0 / 3624
org.apache.spark.sql.catalyst.analysis.Analyzer$AddMetadataColumns 0 / 160907757 0 / 2151
org.apache.spark.sql.catalyst.optimizer.RemoveNoopOperators 20361364 / 159451417 214 / 7791
org.apache.spark.sql.execution.aggregate.ResolveEncodersInScalaAgg 0 / 150540658 0 / 2141
org.apache.spark.sql.catalyst.analysis.ResolveHigherOrderFunctions 0 / 150173456 0 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$WindowFrameCoercion 0 / 137450079 0 / 2141
org.apache.spark.sql.execution.dynamicpruning.PartitionPruning 122398390 / 131163171 222 / 1389
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$StackCoercion 0 / 128164923 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ExtractWindowExpressions 37023937 / 127981109 38 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$MapZipWithCoercion 0 / 127855021 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGroupingAnalytics 23205519 / 118920458 12 / 2141
org.apache.spark.sql.catalyst.analysis.ApplyCharTypePadding 64824954 / 112460168 106 / 665
org.apache.spark.sql.catalyst.optimizer.ConstantPropagation 0 / 106118066 0 / 3624
org.apache.spark.sql.catalyst.optimizer.DecimalAggregates 56241264 / 103958587 124 / 1513
org.apache.spark.sql.catalyst.analysis.ResolveLambdaVariables 0 / 98114927 0 / 2141
org.apache.spark.sql.catalyst.optimizer.PushDownLeftSemiAntiJoin 4284759 / 93655171 32 / 3624
org.apache.spark.sql.catalyst.optimizer.RemoveNoopUnion 0 / 90742352 0 / 1389
org.apache.spark.sql.execution.python.ExtractPythonUDFs 0 / 90481091 0 / 1389
org.apache.spark.sql.catalyst.optimizer.CollapseProject 32147700 / 88138920 191 / 5013
org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown 0 / 85994865 0 / 1389
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRandomSeed 0 / 83825129 0 / 2141
org.apache.spark.sql.catalyst.analysis.CleanupAliases 45394242 / 80992428 514 / 1179
org.apache.spark.sql.catalyst.optimizer.RemoveRedundantAggregates 0 / 76924970 0 / 3624
org.apache.spark.sql.execution.dynamicpruning.CleanupDynamicPruningFilters 0 / 73585707 0 / 1389
org.apache.spark.sql.catalyst.optimizer.PullupCorrelatedPredicates 18971095 / 72745695 16 / 1389
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveTables 0 / 72132943 0 / 2151
org.apache.spark.sql.catalyst.optimizer.CollapseRepartition 0 / 71574775 0 / 3624
org.apache.spark.sql.catalyst.optimizer.EliminateOuterJoin 9824553 / 70741859 15 / 3624
org.apache.spark.sql.catalyst.optimizer.ComputeCurrentTime 0 / 70233534 0 / 1389
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAliases 9516490 / 66119504 53 / 2141
org.apache.spark.sql.catalyst.optimizer.SimplifyConditionalsInPredicate 1950867 / 65731553 19 / 3624
org.apache.spark.sql.catalyst.optimizer.OptimizeRepartition 0 / 64678672 0 / 3624
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveDeserializer 0 / 64325053 0 / 2141
org.apache.spark.sql.catalyst.optimizer.CombineUnions 7361644 / 63872643 48 / 5013
org.apache.spark.sql.catalyst.optimizer.GetCurrentDatabaseAndCatalog 0 / 60852185 0 / 1389
org.apache.spark.sql.catalyst.optimizer.LimitPushDownThroughWindow 0 / 59078747 0 / 3624
org.apache.spark.sql.catalyst.optimizer.EliminateSerialization 0 / 58972531 0 / 3624
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveNewInstance 0 / 57356096 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$GlobalAggregates 3158587 / 56607389 82 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveUpCast 0 / 55416152 0 / 2141
org.apache.spark.sql.catalyst.optimizer.TransposeWindow 0 / 53243101 0 / 3624
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveOrdinalInOrderByAndGroupBy 2689837 / 51448574 8 / 2141
org.apache.spark.sql.catalyst.optimizer.ReplaceExpressions 241958 / 49203258 2 / 1389
org.apache.spark.sql.catalyst.optimizer.ReplaceUpdateFieldsExpression 0 / 49170134 0 / 1389
org.apache.spark.sql.catalyst.optimizer.EliminateLimits 428978 / 48305281 11 / 3624
org.apache.spark.sql.execution.datasources.ResolveSQLOnFile 0 / 48206191 0 / 2141
org.apache.spark.sql.catalyst.analysis.ResolveUnion 0 / 47625269 0 / 2141
org.apache.spark.sql.catalyst.optimizer.CollapseWindow 0 / 46484925 0 / 3624
org.apache.spark.sql.execution.datasources.PreprocessTableCreation 7488318 / 46444682 7 / 665
org.apache.spark.sql.catalyst.optimizer.PushProjectionThroughUnion 1969261 / 45163860 4 / 3624
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggAliasInGroupBy 0 / 44659099 0 / 2141
org.apache.spark.sql.catalyst.analysis.ResolveInlineTables 0 / 43737249 0 / 2141
org.apache.spark.sql.catalyst.optimizer.RewritePredicateSubquery 24748250 / 43218989 26 / 1389
org.apache.spark.sql.catalyst.optimizer.ReassignLambdaVariableID 0 / 42470875 0 / 1389
org.apache.spark.sql.catalyst.optimizer.EliminateResolvedHint 0 / 42470534 0 / 1389
org.apache.spark.sql.catalyst.optimizer.PushLeftSemiLeftAntiThroughJoin 1391712 / 41855536 3 / 3624
org.apache.spark.sql.catalyst.analysis.ResolveCatalogs 0 / 41744035 0 / 2151
org.apache.spark.sql.catalyst.optimizer.EliminateMapObjects 0 / 40267282 0 / 1389
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveNamespace 0 / 40103109 0 / 2151
org.apache.spark.sql.catalyst.optimizer.LimitPushDown 0 / 39885579 0 / 3624
org.apache.spark.sql.catalyst.optimizer.CombineFilters 0 / 39179703 0 / 3624
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveSubqueryColumnAliases 0 / 37673126 0 / 2141
org.apache.spark.sql.catalyst.optimizer.NormalizeFloatingNumbers 0 / 37289065 0 / 1389
org.apache.spark.sql.catalyst.optimizer.PropagateEmptyRelation 4064628 / 36452001 5 / 2783
org.apache.spark.sql.catalyst.analysis.Analyzer$HandleNullInputsForUDF 0 / 36314386 0 / 665
org.apache.spark.sql.catalyst.optimizer.RewriteNonCorrelatedExists 0 / 35675344 0 / 1389
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate 0 / 34874853 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions 0 / 33665275 0 / 675
org.apache.spark.sql.catalyst.analysis.ResolveTableValuedFunctions 0 / 33609320 0 / 2151
org.apache.spark.sql.catalyst.optimizer.EliminateSorts 557492 / 33057122 2 / 1389
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolvePivot 0 / 32398981 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveNaturalAndUsingJoin 0 / 32114392 0 / 2141
org.apache.spark.sql.execution.datasources.FallBackFileSourceV2 0 / 31687366 0 / 2141
org.apache.spark.sql.catalyst.optimizer.ReplaceDeduplicateWithAggregate 0 / 31540710 0 / 1431
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveInsertInto 0 / 31303611 0 / 2151
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveOutputRelation 0 / 31269729 0 / 2141
org.apache.spark.sql.catalyst.analysis.ResolvePartitionSpec 0 / 31159186 0 / 2151
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveEncodersInUDF 0 / 30312585 0 / 665
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveUserSpecifiedColumns 0 / 29266793 0 / 2151
org.apache.spark.sql.catalyst.optimizer.PushExtraPredicateThroughJoin 4074877 / 29060408 7 / 1393
org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases 22386100 / 27379356 296 / 1389
org.apache.spark.sql.catalyst.optimizer.PushPredicateThroughNonJoin 0 / 19727080 0 / 1389
org.apache.spark.sql.execution.analysis.DetectAmbiguousSelfJoin 0 / 17321108 0 / 665
org.apache.spark.sql.catalyst.optimizer.RewriteDistinctAggregates 0 / 17130396 0 / 1389
org.apache.spark.sql.catalyst.optimizer.ExtractPythonUDFFromJoinCondition 0 / 16212421 0 / 1389
org.apache.spark.sql.execution.python.ExtractPythonUDFFromAggregate 0 / 16097371 0 / 1389
org.apache.spark.sql.catalyst.analysis.SubstituteUnresolvedOrdinals 2291646 / 15083864 8 / 735
org.apache.spark.sql.catalyst.analysis.EliminateView 0 / 14423140 0 / 1389
org.apache.spark.sql.execution.python.ExtractGroupingPythonUDFFromAggregate 0 / 13924484 0 / 1389
org.apache.spark.sql.catalyst.optimizer.ReplaceExceptWithFilter 0 / 13882442 0 / 1431
org.apache.spark.sql.catalyst.optimizer.RemoveRepetitionFromGroupExpressions 1114414 / 13842538 2 / 1391
org.apache.spark.sql.catalyst.optimizer.RewriteExceptAll 0 / 12267008 0 / 1431
org.apache.spark.sql.execution.datasources.v2.V2Writes 0 / 11154273 0 / 1389
org.apache.spark.sql.catalyst.optimizer.InferFiltersFromGenerate 0 / 11076409 0 / 1389
org.apache.spark.sql.catalyst.optimizer.ObjectSerializerPruning 0 / 10973496 0 / 1389
org.apache.spark.sql.catalyst.optimizer.RemoveLiteralFromGroupExpressions 0 / 10820735 0 / 1391
org.apache.spark.sql.catalyst.analysis.Analyzer$WindowsSubstitution 0 / 10494983 0 / 735
org.apache.spark.sql.catalyst.optimizer.CombineTypedFilters 0 / 10316684 0 / 1389
org.apache.spark.sql.catalyst.optimizer.ReplaceIntersectWithSemiJoin 1784460 / 10130166 24 / 1431
org.apache.spark.sql.catalyst.optimizer.ReplaceExceptWithAntiJoin 973501 / 9767367 1 / 1431
org.apache.spark.sql.catalyst.optimizer.ReplaceDistinctWithAggregate 2202293 / 9597890 42 / 1431
org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero 0 / 9000817 0 / 1389
org.apache.spark.sql.catalyst.optimizer.RewriteIntersectAll 0 / 8834253 0 / 1431
org.apache.spark.sql.catalyst.analysis.UpdateOuterReferences 0 / 8764186 0 / 665
org.apache.spark.sql.execution.datasources.DataSourceAnalysis 2456760 / 8498247 24 / 665
org.apache.spark.sql.catalyst.analysis.EliminateUnions 0 / 8494086 0 / 735
org.apache.spark.sql.catalyst.analysis.ResolveHints$ResolveJoinStrategyHints 0 / 7425893 0 / 675
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAlterTableChanges 0 / 6966019 0 / 665
org.apache.spark.sql.catalyst.analysis.ResolveHints$ResolveCoalesceHints 0 / 6911635 0 / 675
org.apache.spark.sql.catalyst.analysis.ResolveCommandsWithIfExists 0 / 6744681 0 / 665
org.apache.spark.sql.execution.datasources.PreprocessTableInsertion 0 / 6354045 0 / 665
org.apache.spark.sql.catalyst.analysis.ResolveHints$RemoveAllHints 0 / 6145726 0 / 665
org.apache.spark.sql.catalyst.optimizer.EliminateAggregateFilter 0 / 6011771 0 / 3624
org.apache.spark.sql.catalyst.optimizer.CheckCartesianProducts 0 / 5949828 0 / 2778
org.apache.spark.sql.catalyst.optimizer.CombineConcats 0 / 4467882 0 / 3624
org.apache.spark.sql.catalyst.optimizer.CostBasedJoinReorder 0 / 3704450 0 / 1389
org.apache.spark.sql.execution.OptimizeMetadataOnlyQuery 0 / 3488431 0 / 1389
org.apache.spark.sql.catalyst.optimizer.EliminateDistinct 0 / 3068115 0 / 1389
org.apache.spark.sql.catalyst.analysis.ResolveHints$DisableHints 0 / 1531442 0 / 675
```
After this pr:
```
19:20:49.183 WARN org.apache.spark.sql.TPCDSQuerySuite:
=== Metrics of Analyzer/Optimizer Rules ===
Total number of runs: 407395
Total time: 34.801544775 seconds
Rule Effective Time / Total Time Effective Runs / Total Runs
org.apache.spark.sql.catalyst.optimizer.Optimizer$OptimizeSubqueries 3187947814 / 6644582977 49 / 2778
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions 791813409 / 2288352046 49 / 2141
org.apache.spark.sql.catalyst.optimizer.ColumnPruning 485946590 / 2276408285 335 / 6401
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveSubquery 1895942724 / 2006154058 51 / 2141
org.apache.spark.sql.catalyst.analysis.DecimalPrecision 1210679550 / 1391366516 361 / 2141
org.apache.spark.sql.catalyst.optimizer.InferFiltersFromConstraints 721109407 / 835950522 284 / 1389
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences 572499783 / 725995054 815 / 2141
org.apache.spark.sql.catalyst.optimizer.PruneFilters 19135828 / 696976637 5 / 5012
org.apache.spark.sql.catalyst.analysis.UpdateAttributeNullability 10856939 / 647795278 12 / 3448
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$ImplicitTypeCasts 301313199 / 631767828 78 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveMissingReferences 10171735 / 571812382 10 / 2141
org.apache.spark.sql.catalyst.optimizer.BooleanSimplification 12116466 / 567023853 16 / 3623
org.apache.spark.sql.catalyst.optimizer.RemoveRedundantAliases 14709526 / 470582394 29 / 3623
org.apache.spark.sql.execution.datasources.PruneFileSourcePartitions 428210354 / 470267955 209 / 1389
org.apache.spark.sql.catalyst.optimizer.PushDownPredicates 368708753 / 459852514 999 / 6634
org.apache.spark.sql.catalyst.optimizer.ReorderJoin 335167605 / 398352520 176 / 3623
org.apache.spark.sql.catalyst.optimizer.OptimizeCsvJsonExprs 0 / 376867104 0 / 3623
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$WidenSetOperationTypes 246240254 / 369841628 24 / 2141
org.apache.spark.sql.execution.datasources.SchemaPruning 0 / 355418010 0 / 1389
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$FunctionArgumentConversion 148160582 / 321067114 22 / 2141
org.apache.spark.sql.catalyst.optimizer.FoldablePropagation 3512250 / 298636116 6 / 3623
org.apache.spark.sql.catalyst.analysis.Analyzer$ExtractGenerator 0 / 265088799 0 / 2141
org.apache.spark.sql.catalyst.analysis.ResolveSessionCatalog 157355983 / 259126590 24 / 2141
org.apache.spark.sql.catalyst.optimizer.NullPropagation 16283311 / 251786857 44 / 3623
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions 108190512 / 242785071 426 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$CaseWhenCoercion 131938944 / 241967046 33 / 2141
org.apache.spark.sql.catalyst.optimizer.SimplifyBinaryComparison 272907 / 238604794 1 / 3623
org.apache.spark.sql.catalyst.optimizer.SimplifyConditionals 0 / 212754945 0 / 3623
org.apache.spark.sql.catalyst.optimizer.PushFoldableIntoBranches 6538075 / 212124151 20 / 3623
org.apache.spark.sql.catalyst.optimizer.UnwrapCastInBinaryComparison 4048477 / 210167101 8 / 3623
org.apache.spark.sql.catalyst.optimizer.ConstantFolding 62704250 / 207895760 219 / 3623
org.apache.spark.sql.catalyst.analysis.CTESubstitution 42936893 / 204813186 58 / 735
org.apache.spark.sql.catalyst.analysis.TypeCoercion$PromoteStrings 8090984 / 200608865 11 / 2141
org.apache.spark.sql.catalyst.optimizer.SimplifyCaseConversionExpressions 0 / 193802033 0 / 3623
org.apache.spark.sql.catalyst.analysis.ResolveTimeZone 98996067 / 189887683 537 / 2141
org.apache.spark.sql.execution.datasources.FindDataSourceTable 163652549 / 189723965 296 / 2141
org.apache.spark.sql.catalyst.optimizer.RewriteCorrelatedScalarSubquery 5072527 / 187662600 8 / 3623
org.apache.spark.sql.catalyst.optimizer.ReorderAssociativeOperator 0 / 184300830 0 / 3623
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$Division 19447010 / 183294771 10 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$InConversion 3278833 / 180420638 4 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercion$BooleanEquality 0 / 179031146 0 / 2141
org.apache.spark.sql.catalyst.optimizer.SimplifyExtractValueOps 0 / 178917033 0 / 3623
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$StringLiteralCoercion 0 / 170818541 0 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$DateTimeOperations 0 / 169515865 0 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$IfCoercion 0 / 169027389 0 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$IntegralDivision 0 / 167919905 0 / 2141
org.apache.spark.sql.catalyst.optimizer.OptimizeUpdateFields 0 / 166598755 0 / 4358
org.apache.spark.sql.catalyst.optimizer.OptimizeIn 7078167 / 166071136 27 / 3623
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveBinaryArithmetic 5831247 / 163639681 25 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations 108927809 / 159568405 296 / 2151
org.apache.spark.sql.catalyst.optimizer.SimplifyCasts 16133997 / 158711617 83 / 3623
org.apache.spark.sql.catalyst.optimizer.RemoveDispensableExpressions 0 / 157812395 0 / 3623
org.apache.spark.sql.catalyst.optimizer.LikeSimplification 731219 / 153753185 1 / 3623
org.apache.spark.sql.catalyst.optimizer.ReplaceNullWithFalseInPredicate 1989842 / 149298864 17 / 3623
org.apache.spark.sql.catalyst.analysis.ResolveCreateNamedStruct 0 / 140379120 0 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$ConcatCoercion 0 / 138616802 0 / 2141
org.apache.spark.sql.catalyst.optimizer.OptimizeWindowFunctions 0 / 132845471 0 / 3623
org.apache.spark.sql.catalyst.analysis.TimeWindowing 0 / 131163499 0 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$EltCoercion 0 / 130479486 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveWindowOrder 6574334 / 121824438 28 / 2141
org.apache.spark.sql.catalyst.analysis.ResolveHigherOrderFunctions 0 / 114267659 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveWindowFrame 5553055 / 110975706 37 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$MapZipWithCoercion 0 / 110007546 0 / 2141
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$WindowFrameCoercion 0 / 109886578 0 / 2141
org.apache.spark.sql.catalyst.optimizer.RemoveNoopOperators 16885284 / 109470038 213 / 7790
org.apache.spark.sql.execution.dynamicpruning.PartitionPruning 101132556 / 109418855 222 / 1389
org.apache.spark.sql.catalyst.analysis.ApplyCharTypePadding 64700613 / 109056775 106 / 665
org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$StackCoercion 0 / 107711558 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ExtractWindowExpressions 26581041 / 107586045 38 / 2141
org.apache.spark.sql.execution.aggregate.ResolveEncodersInScalaAgg 0 / 107451058 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$AddMetadataColumns 0 / 105330817 0 / 2151
org.apache.spark.sql.catalyst.optimizer.ConstantPropagation 0 / 85703067 0 / 3623
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGroupingAnalytics 21926898 / 84729481 12 / 2141
org.apache.spark.sql.execution.python.ExtractPythonUDFs 0 / 84225080 0 / 1389
org.apache.spark.sql.catalyst.optimizer.RemoveNoopUnion 0 / 84138791 0 / 1389
org.apache.spark.sql.catalyst.optimizer.PushDownLeftSemiAntiJoin 3828841 / 81042619 32 / 3623
org.apache.spark.sql.catalyst.optimizer.CollapseProject 29006524 / 80522092 188 / 5012
org.apache.spark.sql.catalyst.analysis.ResolveLambdaVariables 0 / 79404306 0 / 2141
org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown 0 / 76819282 0 / 1389
org.apache.spark.sql.catalyst.analysis.CleanupAliases 41832073 / 73999977 514 / 1179
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRandomSeed 0 / 72832470 0 / 2141
org.apache.spark.sql.catalyst.optimizer.DecimalAggregates 27441059 / 70954995 124 / 1513
org.apache.spark.sql.execution.dynamicpruning.CleanupDynamicPruningFilters 0 / 66211569 0 / 1389
org.apache.spark.sql.catalyst.optimizer.CollapseRepartition 0 / 62738683 0 / 3623
org.apache.spark.sql.catalyst.optimizer.ReplaceUpdateFieldsExpression 0 / 62148688 0 / 1389
org.apache.spark.sql.catalyst.optimizer.EliminateOuterJoin 8447285 / 60438670 15 / 3623
org.apache.spark.sql.catalyst.analysis.PullOutNondeterministic 0 / 59813330 0 / 665
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveTables 0 / 58117890 0 / 2151
org.apache.spark.sql.catalyst.optimizer.SimplifyConditionalsInPredicate 1508788 / 57342124 20 / 3623
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAliases 10736150 / 57202130 53 / 2141
org.apache.spark.sql.catalyst.optimizer.ComputeCurrentTime 0 / 56992025 0 / 1389
org.apache.spark.sql.catalyst.optimizer.CombineUnions 6366672 / 56483290 48 / 5012
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveDeserializer 0 / 53273714 0 / 2141
org.apache.spark.sql.catalyst.optimizer.GetCurrentDatabaseAndCatalog 0 / 52510777 0 / 1389
org.apache.spark.sql.catalyst.optimizer.RemoveRedundantAggregates 0 / 48833758 0 / 3623
org.apache.spark.sql.catalyst.analysis.Analyzer$GlobalAggregates 3017199 / 48774564 82 / 2141
org.apache.spark.sql.catalyst.optimizer.EliminateLimits 505068 / 48749551 11 / 3623
org.apache.spark.sql.catalyst.optimizer.EliminateSerialization 0 / 48564056 0 / 3623
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveNewInstance 0 / 48417148 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveUpCast 0 / 48092004 0 / 2141
org.apache.spark.sql.catalyst.optimizer.RewritePredicateSubquery 17909018 / 46921409 26 / 1389
org.apache.spark.sql.catalyst.optimizer.NormalizeFloatingNumbers 0 / 45951107 0 / 1389
org.apache.spark.sql.catalyst.optimizer.PullupCorrelatedPredicates 17548952 / 42800713 16 / 1389
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveOrdinalInOrderByAndGroupBy 1319508 / 42612744 8 / 2141
org.apache.spark.sql.catalyst.optimizer.TransposeWindow 0 / 41810245 0 / 3623
org.apache.spark.sql.catalyst.optimizer.ReplaceExpressions 246644 / 41676496 2 / 1389
org.apache.spark.sql.catalyst.optimizer.CollapseWindow 0 / 40713724 0 / 3623
org.apache.spark.sql.catalyst.optimizer.ReassignLambdaVariableID 0 / 40113987 0 / 1389
org.apache.spark.sql.catalyst.analysis.ResolveCatalogs 0 / 38439435 0 / 2151
org.apache.spark.sql.catalyst.analysis.ResolveUnion 0 / 37929673 0 / 2141
org.apache.spark.sql.catalyst.optimizer.RewriteNonCorrelatedExists 0 / 37915363 0 / 1389
org.apache.spark.sql.catalyst.optimizer.EliminateMapObjects 0 / 37676082 0 / 1389
org.apache.spark.sql.catalyst.optimizer.PushLeftSemiLeftAntiThroughJoin 1441014 / 37218033 3 / 3623
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggAliasInGroupBy 0 / 37119917 0 / 2141
org.apache.spark.sql.catalyst.optimizer.PushProjectionThroughUnion 1989323 / 36987655 4 / 3623
org.apache.spark.sql.catalyst.optimizer.EliminateResolvedHint 0 / 35106688 0 / 1389
org.apache.spark.sql.catalyst.optimizer.LimitPushDownThroughWindow 0 / 35025111 0 / 3623
org.apache.spark.sql.catalyst.optimizer.LimitPushDown 0 / 35003526 0 / 3623
org.apache.spark.sql.catalyst.optimizer.CombineFilters 0 / 35002506 0 / 3623
org.apache.spark.sql.execution.datasources.PreprocessTableCreation 6997955 / 34806015 7 / 665
org.apache.spark.sql.catalyst.optimizer.OptimizeRepartition 0 / 34460212 0 / 3623
org.apache.spark.sql.execution.datasources.FallBackFileSourceV2 0 / 33817781 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveNamespace 0 / 33816969 0 / 2151
org.apache.spark.sql.catalyst.analysis.Analyzer$HandleNullInputsForUDF 0 / 33606387 0 / 665
org.apache.spark.sql.catalyst.optimizer.PropagateEmptyRelation 3507699 / 32850408 5 / 2783
org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions 0 / 30858879 0 / 675
org.apache.spark.sql.execution.datasources.ResolveSQLOnFile 0 / 30577197 0 / 2141
org.apache.spark.sql.catalyst.optimizer.EliminateSorts 705474 / 30334385 2 / 1389
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolvePivot 0 / 29561352 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate 0 / 28867184 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveOutputRelation 0 / 28505610 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveInsertInto 0 / 28160498 0 / 2151
org.apache.spark.sql.catalyst.optimizer.ReplaceDeduplicateWithAggregate 0 / 27949918 0 / 1431
org.apache.spark.sql.catalyst.analysis.ResolveTableValuedFunctions 0 / 27795588 0 / 2151
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveEncodersInUDF 0 / 27686323 0 / 665
org.apache.spark.sql.catalyst.analysis.ResolveInlineTables 0 / 27576064 0 / 2141
org.apache.spark.sql.catalyst.analysis.ResolvePartitionSpec 0 / 27425328 0 / 2151
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveSubqueryColumnAliases 0 / 27046064 0 / 2141
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveNaturalAndUsingJoin 0 / 26922025 0 / 2141
org.apache.spark.sql.catalyst.optimizer.PushExtraPredicateThroughJoin 4260815 / 25801044 7 / 1393
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveUserSpecifiedColumns 0 / 25738491 0 / 2151
org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases 16536843 / 21034764 296 / 1389
org.apache.spark.sql.catalyst.optimizer.PushPredicateThroughNonJoin 0 / 18230332 0 / 1389
org.apache.spark.sql.catalyst.optimizer.RewriteDistinctAggregates 0 / 15619359 0 / 1389
org.apache.spark.sql.execution.analysis.DetectAmbiguousSelfJoin 0 / 15424755 0 / 665
org.apache.spark.sql.catalyst.analysis.SubstituteUnresolvedOrdinals 2152410 / 14785799 8 / 735
org.apache.spark.sql.catalyst.optimizer.ExtractPythonUDFFromJoinCondition 0 / 14691822 0 / 1389
org.apache.spark.sql.execution.python.ExtractPythonUDFFromAggregate 0 / 14488308 0 / 1389
org.apache.spark.sql.execution.python.ExtractGroupingPythonUDFFromAggregate 0 / 12947972 0 / 1389
org.apache.spark.sql.catalyst.analysis.EliminateView 0 / 12601664 0 / 1389
org.apache.spark.sql.catalyst.optimizer.RemoveRepetitionFromGroupExpressions 923862 / 12317031 2 / 1391
org.apache.spark.sql.catalyst.optimizer.ReplaceExceptWithFilter 0 / 11704687 0 / 1431
org.apache.spark.sql.catalyst.optimizer.RewriteExceptAll 0 / 11161156 0 / 1431
org.apache.spark.sql.catalyst.optimizer.InferFiltersFromGenerate 0 / 10349077 0 / 1389
org.apache.spark.sql.catalyst.optimizer.ObjectSerializerPruning 0 / 10090026 0 / 1389
org.apache.spark.sql.execution.datasources.v2.V2Writes 0 / 10062388 0 / 1389
org.apache.spark.sql.catalyst.optimizer.CombineTypedFilters 0 / 9887692 0 / 1389
org.apache.spark.sql.catalyst.analysis.Analyzer$WindowsSubstitution 0 / 9816823 0 / 735
org.apache.spark.sql.catalyst.optimizer.RemoveLiteralFromGroupExpressions 0 / 9542760 0 / 1391
org.apache.spark.sql.catalyst.optimizer.ReplaceIntersectWithSemiJoin 1859861 / 9281119 24 / 1431
org.apache.spark.sql.catalyst.optimizer.ReplaceDistinctWithAggregate 1986456 / 8370500 42 / 1431
org.apache.spark.sql.catalyst.optimizer.RewriteIntersectAll 0 / 8125096 0 / 1431
org.apache.spark.sql.catalyst.analysis.EliminateUnions 0 / 8104101 0 / 735
org.apache.spark.sql.catalyst.optimizer.ReplaceExceptWithAntiJoin 507389 / 8092301 1 / 1431
org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero 0 / 7931249 0 / 1389
org.apache.spark.sql.execution.datasources.DataSourceAnalysis 2462611 / 7913135 24 / 665
org.apache.spark.sql.catalyst.analysis.UpdateOuterReferences 0 / 7500488 0 / 665
org.apache.spark.sql.catalyst.analysis.ResolveHints$ResolveJoinStrategyHints 0 / 7303284 0 / 675
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAlterTableChanges 0 / 6274915 0 / 665
org.apache.spark.sql.catalyst.analysis.ResolveCommandsWithIfExists 0 / 6211839 0 / 665
org.apache.spark.sql.catalyst.analysis.ResolveHints$ResolveCoalesceHints 0 / 6150465 0 / 675
org.apache.spark.sql.execution.datasources.PreprocessTableInsertion 0 / 6003880 0 / 665
org.apache.spark.sql.catalyst.analysis.ResolveHints$RemoveAllHints 0 / 5882558 0 / 665
org.apache.spark.sql.catalyst.optimizer.CheckCartesianProducts 0 / 5336536 0 / 2778
org.apache.spark.sql.catalyst.optimizer.EliminateAggregateFilter 0 / 5205627 0 / 3623
org.apache.spark.sql.catalyst.optimizer.CombineConcats 0 / 3983214 0 / 3623
org.apache.spark.sql.execution.OptimizeMetadataOnlyQuery 0 / 3364063 0 / 1389
org.apache.spark.sql.catalyst.optimizer.CostBasedJoinReorder 0 / 3272873 0 / 1389
org.apache.spark.sql.catalyst.optimizer.EliminateDistinct 0 / 2649763 0 / 1389
org.apache.spark.sql.catalyst.analysis.ResolveHints$DisableHints 0 / 1525243 0 / 675
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org